Ep 55: Head of Amazon AGI Lab David Luan on DeepSeek’s Significance, What’s Next for Agents & Lessons from OpenAI

David is an OG in AI who has been at the forefront of many of the major breakthroughs of the past decade. His resume: VP of Engineering at OpenAI, a key contributor to Google Brain, co-founder of Adept, and now leading Amazon’s SF AGI Lab. In this episode we focused on how far test-time compute gets us, the real implications of DeepSeek, what agents milestones he’s looking for and more.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  David Luan is the head of the AGISF lab at Amazon. He was previously the co-founder and CEO of Adept, a company that raised over $400 million to build AI agents. And he was the VP of Engineering at OpenAI during a lot of their critical breakthroughs. I'm Jacob Effron, and today on Unsupervised Learning, David and I hit a bunch of different, interesting topics, including his reaction to DeepSeek and his predictions for the future progress of models. We talked about the state of agents today, and what's required to make them reliable and when they'll be ubiquitous. And he shared some really interesting stories from the early days of OpenAI and what made the culture there so special. This was a really fun one, as David and I have been friends for a long time. I think folks will really enjoy it. Without further ado, here's David.

• 0:37Jacob Effron?

  David, thanks so much for coming on the podcast.

• 0:39Speaker 01

  Yeah, thanks for having me. This is gonna be a lot of fun because we've known each other for, what, like more than 10 years now?

• 0:45Jacob Effron?

  You know, I remember when you originally joined OpenAI, and I was like, "That seems interesting, but I wonder, like, you know, a cool career move." And then, obviously, you were, as always, prescient, long before everyone else.

• 0:55Speaker 01

  I got really lucky, you know, just that I always was into robotics, and the biggest constraint for robotics was how smart the underlying algorithms were. So I started working in AI, and it's just been so cool to see that this stuff is working in our lifetimes.

• 1:08Jacob Effron?

  Well, there's a bunch of things I want to hit with you today. I thought I'd just start first topical. You know, obviously, there was this huge reaction to DeepSeek over the last, you know, few weeks. You know, the Nvidia stock crash, people were saying it was bad for OpenAI and Anthropic. I feel like now there's been kind of a coming back to, you know, less freaking out. But I'm curious, like, what people got right about the implications of this and maybe what they got wrong in the broader discourse.

• 1:33Speaker 01

  Yeah, so I still remember the morning that everybody started waking up to DeepSeek news. I woke up, I looked at my phone, I had like five missed calls. I was like, "What is going on?" And the last time something like that happened was when SVB collapsed, because all my investors were calling me to get our funds out from like SVB and First Republic and all that stuff. And so I was like, "Something really bad must be happening." And I checked the news, and it's like, "Stocks are crashing because DeepSeek R1 is out and all that stuff." I instantly was like, "Wow, people have really missed the memo on what actually happened here." DeepSeek, I think, was incredibly good. Work from I have so many thoughts on like team culture and composition and all that other stuff we can get to later.

• 2:14Speaker 01

  But it was it was just really incredible work, but it's part of this broader like story arc of we figure first out how to make new ML systems smarter. And then we figure out how to make them more efficient. And so this was like the the talk to the tick of oh one really. And what everybody got wrong was just because you could make more intelligence happen at a lesser price doesn't mean that you stop consuming more intelligence. If anything, you consume even more of it. So once I think the market woke up to that, now we're back to sanity.

• 2:44Jacob Effron?

  Given that obviously some of the at least base model there seems to have been trained on outputs of of OpenAI. And, you know, I guess you can get the the base deep peak model to say that it's chat GPT in in various ways. Do you think going forward, you know, given what's happened at distillation, that OpenAI Anthropic maybe stop releasing some of these models more publicly?

• 3:00Speaker 01

  You know, I think what's going to happen is that is that people want to build the smartest models possible. But that sometimes it's not always inference efficient. So I think what we're going to start seeing more and more of is whether people talk explicitly about this or not. People are going to train these humongous teacher models on as much compute as they can get their hands on. And then they're going to try to figure out internally in their own labs, how to render it down to something that runs really fast and is like efficient for customers. Like the biggest thing I'm seeing right now is that is that, you know, every I kind of think about AI use cases and concentric circles of complexity. Right? So like maybe at the very, very like inner circle of complexity will be something like just having a good old chat conversation with a base LLM.

• 3:43Speaker 01

  Right? Like we were able to do that back in GPT two pretty competently. And every incremental circle of intelligence, maybe it's like being able to do mental math or coding or like later on agents or later on like drug discovery or whatever. Require smarter smarter models, but every previous ring of intelligence becomes almost so cheap as to be commoditized.

• 4:00Jacob Effron?

  Kind of goes into obviously, you know, I feel like there's been this massive test time compete wave. It seems like a really exciting path forward for coding, for math, for these, you know, easily verifiable domains. How far does this kind of paradigm get us?

• 4:11Speaker 01

  So there's actually an interesting paper trail and podcast trail of me talking about the recipe for how to build AGI. That's like years old at this point.

• 4:20Jacob Effron?

  Let's contribute to that trail.

• 4:22Speaker 01

  Yeah, so now we get to, this is like, this is a proof that we talked about this conversation at this point in time. But like, back in, even back in 2020, right? We always said, you know, at the time, we were starting to see GPT-2 had come out, GPT-3, I think, was cooking or maybe done by then. We were starting to think about four. We were living in this world where people were not sure whether or not all you needed was next token prediction to solve all of AGI. And my view and a couple of views of people around me was actually like, the answer would be no. And the reason why the answer was no is because an LLM trained to do next token prediction, by definition, is penalized for discovering new knowledge because new knowledge was not part of the training set.

• 5:05Speaker 01

  And as a result, what we needed to do was we needed to go look at what are the other ML paradigms we know can actually discover new knowledge? And we know RL can do, RL and search can do that, right? Yes, like, this is such a long trail, but even AlphaGo, right? Was maybe the first, like, time it went to public consciousness that we could discover new knowledge using RL. And the question always was when we were going to combine LLMs with RL to get systems that had all of the knowledge of what humanity already knew and the ability to go build upon it. Like, the reason why the initial DeepMind path of just doing RL alone didn't work was because it was initialized randomly. Those models that were playing, you know, Atari, incredible result or whatever, like, the amount of time it would take something that knew nothing about the world to then rediscover human language

• 5:50Speaker 01

  and rediscover how to coordinate and like, learn details about how to file your taxes would take forever if done purely in an RL setting. And so now I think that philosophy has really been borne out by seeing how successful these like, models that combine both these paradigms are.

• 6:05Jacob Effron?

  Yeah. And do you think, I mean, like, for domains that aren't as easily verifiable, let's take, you know, healthcare or law, like, do these, you know, does following this kind of test time compute paradigm get us to models that can do that? Or like, are we going to get exceptionally good at coding and math, but still not be able to like, you know, tell a joke or something?

• 6:20Speaker 01

  This is a great topic of debate. I have a very strong view.

• 6:23Jacob Effron?

  What's the strong view?

• 6:24Speaker 01

  The answer is that these models are better at generalizing than you think. Everybody's like, ah, you know, like, I played with O1 and it seems like it's a little better for math, but to wait for it to think and it kind of like, maybe it's a little bit worse at chat or whatever, I think those are just blips on the way to glory for how these things are built. Today, like, we already have signs that getting better through problems where you have an explicit ability to test whether or not the model's correctly solved it, which is what we've seen from DeepSeek.

• 6:56Jacob Effron?

  Yeah.

• 6:57Speaker 01

  Does lead to transfer on some slightly fuzzier problems that seem in a similar domain. And I think the field is like working hard, like my team, others, they're just working so hard to figure out how to learn like human preferences around these much more complicated tasks and then just do RL to go satisfy those.

• 7:15Jacob Effron?

  Right. And you obviously had to be able to build like a model to essentially verify like the, you know, hey, that output is like good law or that output is like, you know, a good healthcare diagnosis. Like, obviously a much harder problem than like, you know, verifying a math proof or, you know, did that code run?

• 7:30Speaker 01

  The fundamental thing that I view we are arbitraging is this, is this gap between how good these models are, the same set of neural, the same set of neural network weights, how good they are at determining whether it did a good job compared to generating the right answer.

• 7:45Jacob Effron?

  Yeah.

• 7:45Speaker 01

  We've always seen that these models are better at determining whether they've done a good job than they are generating the answer. And to some extent, what we're doing with all this RL stuff is exploiting that to force it to try over and over and over again to satisfy its own sense of whether or not it did a good job.

• 7:58Jacob Effron?

  Talk to me a little bit about like the research problems that needed to be solved to actually get a model like this out.

• 8:03Speaker 01

  There are just so many. But, you know, the, well, where do we even start? I think the, in no particular order, and I'm only probably going to itemize like three of the problems that we've had to solve. I think one of the first ones is just figuring out how you can even build an organization and a process to be able to reliably churn out models. Like, one of the things that I always, that I always say to my team and people that I collaborate with, is that today, if you run a modern AI lab, your job isn't to build models. Your job is to build a factory that reliably churns out models. And if you think about, like, How you think about that totally changes what you invest in, right?

• 8:45Speaker 01

  And so until there was repeatability here, there was not a lot, in my view, there was not a lot of forward momentum. Like, we've just gone through this arc over the last couple of years of like going from alchemy to like industrialization in terms of how these things have been built. And without that, there was no substrate for these things to work. I think the next part, so like all these corollaries, right? Like, again, this is a space where you have to go slow to go fast. But so I think that was, that was one of the first parts. I continually believe that the, the thing that people always gravitate to because they think it's cool and sexy are the algorithms. But if we look at what actually drove a lot of this stuff, it's solving the engineering, right?

• 9:30Speaker 01

  Like, how do you do giant, like massive clusters that you can reliably keep up for long enough? And if a node goes down, you don't like end up wasting a bunch of time in your job, right? To be able to push the frontiers of scale. Now with this whole RL thing, we're going to be quickly moving to a world where there will be lots and lots of data centers, each of which are going to be doing actually a lot of inference on the base model and maybe testing it on new environments, maybe that, that a customer has brought to the, to bear, to like learn how to improve the model and then sending those new knowledge back to a centralized place where the model can learn how to be smarter. There are actually a lot of really tough engineering problems as well to be solved.

• 10:05Jacob Effron?

  Yeah, there's been folks like Jan Le Cun that have had some interesting like recent and recurring criticisms of like the limitations of LLMs. I'm wondering if you could kind of just like summarize that critique for our listeners and then, you know, what are your kind of thoughts on, on folks that say, look, it's going to be hard for these LLMs to ever have real original thinking?

• 10:22Speaker 01

  I think we just have counterexamples. I think AlphaGo was original thinking. We watch, if you go back even to the old OpenAI work where we were using RL to play flash games, right? If you're, if you're of a certain age, you probably remember like MiniClip and stuff like that. Great, great, great sinks of time in, in middle school. But, but it's so funny to watch that become the substrate of AI. And we were working on, we were working on, you know, just like using our algorithms to try to solve many of those games at once. And you just, you learn that they, they learn really quickly how to discover oftentimes like speedrun techniques by like glitching through walls and stuff to solve, to solve platformer levels. That's stuff that humans have never done before.

• 11:03Jacob Effron?

  And on the verification side, is it mostly just, you know, it's obviously just finding clever ways, I guess, to figure out verification for some of these different domains?

• 11:11Speaker 01

  You I think you use the model.

• 11:14Jacob Effron?

  I love it. I guess, you know, I'd love to shift to kind of the world of agents. And obviously, you know, you worked on computer use models at Adept. How would you kind of characterize where we are today with these models?

• 11:24Speaker 01

  OK, so I'm super excited about I remain super excited about agents. I still go back to, you know, 2020, 2021, when the first wave of truly powerful LLMs like GPT-4 were coming out. You'd go play with them and you're like, wow, there's so much promise. Like, it's it's made me a great rap song. It's like it's it does great roasts. Sometimes it does three-digit addition acceptably. And you're like, please order me a pizza. And it just cosplays being like a Domino's Pizza rep. Like, it just can't do it. It's obviously a major it's a major gap, right? In the utility of these systems. So even since then, you know, I was pretty sure that we had to go solve the we had to go solve the agent problem.

• 12:06Speaker 01

  And so we started working when I was at Google on problems that that actually still are called tool use, right? Like, how do you expose affordances to the LLM to decide when it should go should go do something? And back then, I think, well, the R L literature had always called it agents. But I think the general public didn't yet have a word for it. So we tried to come up with a new term called a large action model instead of a large language model. And that kind of had a little bit of traction. And then the world decided that it was going to be called an agent. And now everything's an agent and doesn't mean anything anymore, which is very sad. But but it does it was really cool to be the first, you know, first modern agent company. And when we started Adept, you know, there was the best open source LLMs were not good.

• 12:51Speaker 01

  So we were like, we have to train our own model because there was also no multimodal LLMs. Like, like image input LLMs like GPT-4V came so much later. And so we did we had to do everything end to end from scratch. It's kind of like starting an airnet company in like 2000 and having to go like call TSMC to build your own chip. Like, it's just insane. And so So along the way, what we learned was that really early on, what we learned was that LLMs out of the box, without any of the new RL stuff we're doing today, they're behavioral cloners, right? They kind of do what they've seen in the training data, and that means that they're really liable to go off the rails because the moment they end up in a situation they've

• 13:35Speaker 01

  never seen before, the generalization tends to be bad and it does something unpredictable. And so at Adept, we were always focused on useful intelligence. And so what did utility mean? It wasn't ship a cool demo that went viral on Twitter. It was put this in the hands of someone so that they don't have to do the shuffle things around on your computer grunt work that most knowledge workers have to do. And so those knowledge workers care about reliability. So one of our early use cases was can we go do invoice processing for people?

• 14:07Jacob Effron?

  Everyone loves invoice processing for these agentic models. It feels like a natural place to start.

• 14:12Speaker 01

  It's a great hello world. And so at the time, nobody had really done these things before, so like, let's choose an obvious hello world. So we chose, that was one of them. We did Excel, some other ones. But if this thing ends up deleting a third of your QuickBooks entries one in seven times, you'll never use it again. And reliability remains an issue. Even today, operators are super impressive, right? And seems to be a cinnamon cut above Claude computer use so far. But if you look at both of those things that are already out, and the biggest challenge is that they've both focused on end-to-end task performance, right? Like go in there and you type like, hey, I'd like for you to find me five vacation sites that I could go to this weekend, and it'll go do an approximation of that.

• 14:57Speaker 01

  But the end-to-end reliability of that is super, super low, requires lots of interventions, right? We're still not at a point where the real value of this is when businesses can really trust it in a fire and forget way. That's what we've got to solve.

• 15:12Jacob Effron?

  Maybe explain for our listeners, what do you actually have to do if you take a base multimodal model that's out there to turn it into a large action model? What is actually the work that's happening behind the scenes to make that happen?

• 15:22Speaker 01

  So I can talk at a High level about it, but basically there's two things you have to do. One is an engineering problem. The engineering problem is how do you expose in a model legible way? What it can do, right? So, you know, here are the APIs you can call. Here's a here's the UI elements you can call. Let's go teach you a little bit about how expedia.com works or how SAP works, right? That's kind of, it's a little bit of research engineering. That's kind of step one is giving it like a sense of what it can and can't do and basic abilities to go to go do stuff. The interesting part happens in the second component, which is how do you teach it to plan and reason and replan?

• 16:04Speaker 01

  And and follow user instructions and later on, actually even be able to infer what the user actually meant and go and go do that for them, right? That's a huge, huge research problem and it differs a lot from regular old LLM work. Because regular LLM work is like, let's go generate a piece of text. Even the reasoning work we're seeing today with math problems, right? There's an answer at the end. So it's like a single step, even if it's like thinking for a bajillion chains of thought. Like it's it's it's really taking it's really taking one step for you, which is like, yeah, I've given you the answer. With this, it's this whole multi-step decision-making process that involves backtracking

• 16:44Speaker 01

  and involves trying to predict the consequences of an action you take for the future and realizing, hey, the delete button is probably dangerous.

• 16:53Speaker 01

  You have to do all the work to teach the model that in a basic setting. And then you set it in sandboxes, set it loose in sandboxes to learn on its own terms, right? Like the best analogy for this, by the way, that I'll jam in, I forgot, I think, I think it was Andre Karpathy wrote this on the internet or something. It's like, it's like a modern AI training is kind of like how a textbook is organized, right? So first you have all of the exposition, I'm just cribbing this from him, but like all of the exposition of of some physical process. And then you've got some sample problems. So the first part's pre-training, the sample problems is supervised.

• 17:33Speaker 01

  And then the RL step happens when you have the open ended problems at the back that maybe have an answer in the back of the textbook. It's like we're just following that process.

• 17:41Jacob Effron?

  Yeah, I guess you know, you've obviously thought a lot about how this these, you know, agents actually get brought into the world. So I guess a few questions around that. The first is you mentioned, obviously, part of this is is kind of this engineering challenge of just letting the models know what they have access to. Like, how do you think over time, models will interact with like browsers and programs like it's going to be similar to how humans do? Is it just going to be via code like other other approaches that you've seen?

• 18:02Speaker 01

  If I were to ding the field right now, and one thing it is that there has been a massive lack of creativity on how people interface with these increasingly smart L L M's and agents. Like, we are in the, um, do you know, like, do you remember like when the iPhone came out, the App Store came out? People started making all of these apps like, um, hit this button to make the burp noise. And here's a beer that you can pour in your mouth by tilting the by tilting the phone. Like, our interfaces today feel like that, and that is so sad. Because chat is a super limiting, like, low bandwidth way to go get things done. In certain ways, it's it's easy, but in many other ways, right? Like, I don't want to have a seven turn conversation to decide what toppings I'm going to have on my pizza.

• 18:46Jacob Effron?

  Right.

• 18:46Speaker 01

  Like that. And I think that the lack of creativity there has been really bugging me. And I think part of the reason why is that the amazing product designers that could be helping us figure this stuff out. A lot of them don't yet like deeply understand the limitations of the models that people work in this changing quickly.

• 19:03Jacob Effron?

  Right.

• 19:03Speaker 01

  But and then conversely, so far, the people who have been able to advance the technology have always just thought about it as like, I'm here to go deliver a black box. And not, I'm here to go deliver an end to end experience. So when that changes, I'm excited to see things like, um, I'm excited to see things like, uh, like, uh, systems where when you interact with the agent. Like, it is actually itself synthesizing this like multimodal user interface for you to like, for it to best elicit what it needs from you. Right. And to have like shared context between the human and the A I. Like, instead of like, the current paradigm is like, you're chatting to each other.

• 19:39Jacob Effron?

  Yeah.

• 19:39Speaker 01

  It's like you are doing something together on your computer and looking at the screen more like parallel rather than perpendicular.

• 19:45Jacob Effron?

  I guess, you know, you mentioned, obviously, operator kind of works, sometimes doesn't, you know. What like, when do you think we actually get, you know, reliable agents?

• 19:53Speaker 01

  I mean, I think operators are super impressive, by the way. It's just that right now, the whole field is like, missing that last chunk, right? It's like self-driving, right? I forgot how many, it must have been over a decade, maybe even 15 years ago, we had amazing demo videos.

• 20:09Jacob Effron?

  I think, well, having done the self-driving podcast yesterday, I think it was 95 where they did the ride. They drove across the country like 99% confidently.

• 20:16Speaker 01

  Yeah, yeah, yeah. So like, that's a...

• 20:18Jacob Effron?

  So are we gonna wait, are we gonna have to wait 30 years for...

• 20:20Speaker 01

  No, no, I don't think so, because I think we actually have the right tools in our toolbox now. And I think that, yeah, I think this recipe for how to build a, build a AGI-level agent will work pretty well.

• 20:36Jacob Effron?

  I guess what like milestones are meaningful to you in the agent space? Like, what do you think the next, you know, the next thing you're looking out for?

• 20:42Speaker 01

  Okay, the main milestone I'm looking for in the agent space is, I can give, during training time, right? Like, I work at one of these main labs. I can, I have, like, the milestone I'm looking for is, I have a recipe where I can hand this agent in training any task and come back, end days later, and it's 100%ed it. Like, 100%ed it, like, not just like, you know, humans have given us like another, you know, five percentage points more reliability, but like, the agent has figured out how to solve the thing.

• 21:13Jacob Effron?

  You know, to your point, you started Adept like before there was, you know, really any open source models, and certainly like a multimodal open source model, and I think as a result, had to train a lot of things from scratch. Do you think like, you know, if someone was starting an Adept-like company today, like, could a startup be successful here? Or is it gonna be the foundation model companies and hyperscalers that like, you know, ultimately, or like, move the ball forward here?

• 21:34Speaker 01

  So I actually have a huge amount of uncertainty on this one, but, but my current view is that...

• 21:43Speaker 01

  Is that so I personally think AGI is really not super far away and

• 21:49Jacob Effron?

  And when you say AGI, like, how do you define it?

• 21:51Speaker 01

  A model that can do anything useful that a human does on a computer. That's one part of the definition. The other part that I also like is that is that it's a model that can learn how to do that thing as fast as a human can. Like a generalist human can. And I think I think either of those are really not that far away. But I also don't think that it's going to be, I think it's going to be deeply transformational, but I don't think it's going to diffuse through society really quickly. Because most, you know, as we know through AMDALL's law, right? Like, like, once you really speed up one particular thing, something else becomes a bottleneck and your overall amount of speed up is is less than you think.

• 22:31Speaker 01

  So I think what's going to happen is that we'll have the tech, but there will be those massive, a lot of my colleagues call it a capability overhang, right? Massive capability overhang where studies ability to actually use these things productively will lag quite a while.

• 22:47Jacob Effron?

  Do you have any like really thoughts on what the gating factor might be once we do have these capabilities?

• 22:51Speaker 01

  I think it's people. I think it's all it's people processes. It's figuring out how to how to co-design the interface with with the decisions that startups are making on how to use the models. It's going to be like social acceptance, right? Like, imagine you have this model that pops out tomorrow and says, hey, I have I have invented a brand new way of doing X. Like, everybody should go use this thing. Humans have to go make their peace with it and decide like, hey, is this actually a better solution? That's going to not be as fast as we think.

• 23:22Jacob Effron?

  Right. And I guess, you know, to your point, there might actually be an opportunity, you know, even if the labs are the first place to get to these models that can do this. There may be an opportunity for startups to actually be the ones to bridge the gap between these model capabilities and something that the end users actually want to interact with.

• 23:35Speaker 01

  I'm actually pretty sure that's what's going to happen is that.

• 23:39Jacob Effron?

  I'm obviously biased. I want that to happen.

• 23:41Speaker 01

  That's a good point. Well, I think I think I think that's a good bet because because at the end of the day, I actually still really believe that in a world with AGI, human relationships really matter. And and knowing and owning your customer and being more in tune with them about what they need is going to be more important than simply controlling this artifact that actually many other labs will have.

• 24:05Jacob Effron?

  What do you think it's going to look like for like, how am I going to use my computer in 10 years? When all these models are, you know, we've gotten to your definition of AGI. Like, am I going to ever sit down at the computer or like, what is what is your vision for like, the way humanity interacts with these things?

• 24:17Speaker 01

  I think that we'll just get new quivers in or rather tools in the toolbox for how we interface with computers. I think today, right, I mean, we've got people still use the command line, right? Like, people, that's a really important part of people's productivity. People still use the GUI. In the future, people will still use voice interfaces. But then also, I think also people will use more ambient computing as well. And also, they'll they'll they'll like have this generative UI thing we were talking about earlier. But I think the metric we should be looking at is just what is the amount of leverage per unit energy a human spends with computing? And I think that is going to continue to go up and to the right with these systems.

• 24:59Jacob Effron?

  Yeah, maybe talk through a little bit like this this future world of models and whether we end up with anything domain specific.

• 25:05Speaker 01

  Let's take the the hypothetical legal specialist. You probably want the hypothetical legal specialist to know some basic facts about the world.

• 25:13Jacob Effron?

  Yeah, so make people go do a general college degree before law school.

• 25:16Speaker 01

  Exactly, exactly. So I think like, I do think there will be specialized models. But I don't want to bury the lead by just saying there will be specialized models. I think there'll be specialized models, not for technical reasons, but for policy reasons.

• 25:29Jacob Effron?

  That's juicy. What's that mean?

• 25:30Speaker 01

  Oh, yeah, it's just like, you know, you have a couple of companies who just really don't ever want their data commingled with each other.

• 25:35Jacob Effron?

  Yeah.

• 25:36Speaker 01

  Or you've got you've got like some like, you know, imagine you're a big bank, right? And you've got your sales and trading division, you've got your investment banking division. The the AI employees or LLMs that power the stuff just Like how those employees today can't share information, should not be able to share information even remotely,

• 25:57Speaker 01

  even remotely through its weights, right?

• 25:58Jacob Effron?

  As you think about the key problems that still need to be solved in models, I mean, it seems like you have a lot of confidence that if we kind of just scale up compute in these approaches, like we're, we're going to get pretty close to solving what we need to solve. But, you know, are there any kind of like big technical challenge you see ahead in continuing to scale model IQ?

• 26:13Speaker 01

  So I actually, I actually don't believe that we take what we have today. Exactly. And then we just like pull ahead the cluster from two years from now and everything will magically work. I do think it's going to be a major, the scale is going to be a major factor. But my confidence actually comes from looking at what are the main remaining open problems? And trying to have an estimate for, for, for how hard they are. And I think that if a, like a super hard thing, like, you know, we need to go, we need to go,

• 26:52Speaker 01

  we need to go replace gradient descent, right? Or like, we need to go, we only can only do AGI with a quantum computer or something like that. I don't think that's in the cards.

• 27:03Jacob Effron?

  What do you do when new models come out? Like, how do you, you know, do you look at the evals? Do you vibe check them with a few like go to questions? Like, how do you get a sense of how good these new models are?

• 27:11Speaker 01

  There's two things that I do. One of them is that what I've learned is that, and this is what's so cool about this field. Sometimes you just look at a result and you look at, especially if there is a methodology that gets published with it, which is rare now. You just look at how they did it and you're like, wow, this is actually simpler than how we used to do this and the results are better. When that happens, it almost always becomes part of the deep learning canon. And so, and then you just have this moment where you're like, this is actually really beautiful. I think that's the main one. Then the other ones, like, you know, benchmarks are. Part of the hype of the field has been that has been that a lot of benchmarks that

• 27:53Speaker 01

  are good, but really like not that aligned with what people need from these models have just become so important in people's development processes. So they're all kind of gamed. I actually think I think that like evaluations are so hard. Measurement is so hard. Way more prestige and attention should go to that than actually in fact, many other things that we're doing right now.

• 28:20Jacob Effron?

  Yeah. And it seems like everyone kind of has their own internal evals that they don't release publicly. They're like, they trust way more. And it's like, you could see something like an OpenAI model perform better on like a lot of the coding benchmarks, but like everyone uses the Anthropic models anyway because they know they're better. And so it's interesting to see that that landscape evolve. Well, I'm curious to the extent you can talk about it. Like, I'd love to hear, you know, what you're up to at Amazon these days, how you think about Amazon's role in the broader ecosystem.

• 28:42Speaker 01

  Yeah, Amazon's a super interesting place. Actually, I felt like I've learned so much in a short amount of time there. Amazon is super serious about building general intelligence systems, especially general intelligent agents. And what I think is really cool about it is that I think everybody at Amazon understands that computing itself is changing from, you know, the primitives that we all know and love to a call to a large model or a large agent being probably the most important compute primitive in the future. And so people really care, which is awesome. And I think what's what's interesting about it is that at Amazon now I I cover agents. And what's been really cool is you get to see just the breadth of

• 29:26Speaker 01

  the breadth of everything that that that agents touches in a company as big as Amazon. And what's also awesome is that, you know, Peter Beal and I have started this new San Francisco based research lab for Amazon. And a lot of that was because folks folks in the highest levels of Amazon, I think, really believe that we have to make new research breakthroughs to solve those remaining problems we were talking about earlier on the path to AGI.

• 29:53Jacob Effron?

  Do you pay attention to like any of these like alternative architectures that folks are trying or like, you know, what are there areas of like maybe more out there research? Do you kind of keep your your eyes on?

• 30:03Speaker 01

  Let's see. So I always pay attention to things that are That look like they might help us better map model learning to compute. Can we use more compute more efficiently, right? It just gives us a huge multiplier over what we can do. But I honestly actually spend more of my time looking at data centers and chips. Because I just I just find it so fascinating. It's so cool.

• 30:29Jacob Effron?

  So cool.

• 30:30Speaker 01

  And there's some interesting plays being made there now.

• 30:33Jacob Effron?

  It seems like a big part of what drove this last progress in models was like data labeling. And like, obviously, all the labs were spending tons of money on that. Is that still relevant, like in this test time compute paradigm? Like, how do you think about that?

• 30:47Speaker 01

  I think there's two different jobs that have to be solved from data labeling. There may be more, but the two that come to mind for me. First is teaching the model, teaching the model the very basics of how to do a task by cloning human behavior. And if you have like super high quality data, then you can like use that to better elicit something you already see during loosely during pre-training. And then I think the second job is to teach the model what good and bad looks like for tasks that are fuzzy. And I think the I think both both of those will remain really important. But I think this middle chunk of just spamming human data labels to marginally improve models that already can kind of do a thing like that's going to be that's going to be the

• 31:32Speaker 01

  job of RL.

• 31:33Jacob Effron?

  You've obviously been on the frontier of this space for for, you know, decades. What's one thing you changed your mind on like in the AI world in the last year?

• 31:41Speaker 01

  You know, the one that I actually keep on coming back to is the importance of building the team culture in the right way. Like, I think we've always kind of known it, but I've become even more convinced that hiring really smart, energetic, intrinsically motivated people earlier on in their careers is actually like one of the best engines for for for a process. Like, I feel like this in this field, like every couple of years, the Oh, actually, yeah, the every couple of years, the optimal playbook changes. And so if people are too overfit to the previous optimal playbook, they actually slow you down.

• 32:23Speaker 01

  And so I think it's a lot better to bet on new folks coming in than I had previously thought. But the other one, actually, that I've changed my mind on, I used to think that building AI would actually have real, long-term, technical differentiation that you can compound on. I used to think, you know, all right, like, if you get really good at text modeling, it'll help you. You should obviously just become the winner in multimodal. If you're good at multimodal, you should obviously become the winner in reasoning and in agents, right? Dot, dot, dot, and these things should compound. In practice, I've seen like so little compounding. I think people are all trying relatively similar ideas.

• 33:02Jacob Effron?

  And I guess implicit in what you're saying is like, just because you were the first to break through A, doesn't actually mean that that then puts you in such an advantaged position to get to like, you know, breakthrough B. Like, basically, like, if you're ahead in LLM, and then we're talking about the reasoning side. I mean, it was OpenAI that happened to be ahead in both, but it's like, almost like that reasoning breakthrough could have come out of any of the labs. And just because they were first to, you know, kind of a GPT-4 level model, didn't necessarily mean they would inevitably be the ones to have the, you know, a one breakthrough.

• 33:27Speaker 01

  I mean, it's definitely correlated, but it's not like, it's not like deterministically true that you will then obviously win the next change.

• 33:36Jacob Effron?

  Well, I thought I want to hit on you, you know, you obviously got into this space originally through your love of robotics. And so I am curious, like, what do you think of where we are in the AI robotics space today?

• 33:45Speaker 01

  Similarly to like my belief about digital agents, I think that we have a lot of the raw materials. And I think, interestingly enough, digital agents gives us an opportunity to de-risk some of the hard problems in physical agents before you have to do all the costly stuff of like real-world items.

• 34:03Jacob Effron?

  Say more about that. So like, basically, solving the reliability problem on the digital agent side, well, you know, how does that actually end up bleeding into the the physical agent side?

• 34:10Speaker 01

  A simple example, a toy example. Let's say you have a warehouse you're trying to rearrange, and you have a physical agent, and you're asking it, hey, like, figure out the optimal plan for rearranging this warehouse. Right. If you were doing that by learning in the physical world, or even learning in robotics sim for that, that's kind of hard. But if you could do that in the digital space already, and you have all of the training recipes and the know-how, you've tuned the algorithms to be able to learn from simulated data on that, it's just, you've done the training wheels version of this already.

• 34:42Jacob Effron?

  And so it's funny, I feel like there's these polar extremes when people think about robotics. Some people look at it and they're like, oh, like the same kind of scaling laws we found in LLMs, we'll find on the robotics side, and we're on the precipice of this massive change. You hear Jensen talking about it a lot. And then there's other folks that are like, we're where we were in 95 with self-driving cars, where it's a great demo, but it's going to take quite some time to actually work. Where do you fall on that spectrum?

• 35:06Speaker 01

  I just go back to, the thing that gives me, would give me the most confidence is our ability to build training recipes that let us 100% tasks. If we can do that in the digital space, I think that it will be a challenge, but it will transfer over ultimately to the physical space, too.

• 35:24Jacob Effron?

  What's your timeline for when we have, like, robots in our house?

• 35:27Speaker 01

  Oh, gosh. Well, I think that's actually, it goes back to the thing I was saying earlier. I think some problems, actually a lot of problems, the bottleneck is not the modeling, it is the diffusion of the modeling.

• 35:37Jacob Effron?

  What about, like, video models? Obviously, there's been a bunch of folks going into that space. Like, it seems like the next frontier around that is really kind of like a world model and like, understanding of physics to allow for more open-ended exploration there. Maybe just comment a little bit about what you've seen there and your thoughts on that space.

• 35:54Speaker 01

  Yeah, I'm really excited about it. I think it solves one of the major remaining problems, which is, you know, we talked earlier about how today we're able to make RL work on problems where you have a verifier, right? Like theorem proving or something like that. And then we talked about how to generalize that to the digital agent space, where you have problems where you don't have a verifier, but you might have a solid simulator. Because I can go boot up my staging environment for insert app here and teach the agent how to try to use that. But I think one of the major problems left is what happens when you don't have a explicit verifier or an explicit simulator. I think world modeling is how we answer that question.

• 36:36Jacob Effron?

  Awesome. I want to shift gears a little bit to OpenAI and your time there. I mean, obviously, you were part of this very special time in the company and played a kind of seminal role in a lot of the advances there. You know, I feel like at some point, we're going to get this deluge of thought pieces about what made OpenAI culture so special in this era that developed, you know, the GPT-1 through 4. What do you think those pieces will say? Like, what made the organization work?

• 37:04Speaker 01

  Oh, I mean, I'm not sure those pieces will get it right, because I'm already seeing all sorts of bad, hot takes about why OpenAI succeeded during that period. I think what it is, is, you know, so when I joined, I was, because the research community was really small back in 2017. I think OpenAI was a little over a year old. And I knew a bunch of people on the founding team and some of the early employees, and they were looking for someone to, and one of the things I love about OpenAI, they got right from the very beginning, is blurring the lines between research and engineering. And they were looking for someone to run that. And so it was super fortunate, I joined, and it was 35 people, incredible folks on the team, like John Shulman and Chris Burner, who did a lot of our supercomputing stuff,

• 37:48Speaker 01

  and Wojciech, and there's so many people that I could name, who were just incredible folks around the table back then. And, you know, interestingly, at the beginning, it was helping OpenAI just build a lot of the infrastructure of how to scale out beyond like a tiny team that all fit one room, right? So a lot of basic engineering management stuff. But then, it started morphing into like, how do we define a differentiated research strategy that would enable us, basically, to take the right bets for this period of ML? And I think what that really boiled down to was, I think we realized earlier than everybody else, that the

• 38:30Speaker 01

  Previous way research worked of like you and your three best friends write a research paper that changes the world, like that era was over. And that we really need to be thinking about this new era where where we thought about like major scientific goals and tried to solve them with bigger teams of combined researchers and engineers, regardless of whether or not the solution was quote unquote novel as defined by academia.

• 38:51Jacob Effron?

  Yeah.

• 38:51Speaker 01

  And we would take the flak for that sometimes, like people when GPT two first came out, people said, oh, this looks like a transformer. We're like, yes, it is a transformer. And and that was something to be proud of.

• 39:01Jacob Effron?

  Yeah.

• 39:02Speaker 01

  Yeah.

• 39:02Jacob Effron?

  What did you think you were signing up for like when you joined OpenAI?

• 39:04Speaker 01

  Oh, I mean, I was just so stoked because I wanted to be at the frontier of research. At the time, it was OpenAI or DeepMind.

• 39:11Jacob Effron?

  Yeah.

• 39:11Speaker 01

  And and or I guess or also Google Brain, but I think I wanted to do something a little more speculative. It same lesson, as I said earlier, about betting on really intrinsically motivated folks who could be earlier in their career was just such a winning recipe, right? Incredible people like Alec Radford, like Aditya Ramesh, who invented DALL-E. Like, again, like a long list of incredible folks I can name who did like field defining things during that period, who did not have PhDs, did not have a bajillion years of experience.

• 39:47Jacob Effron?

  What like common traits have you noticed in like what makes these people so good? I mean, you you're one of the great AI researchers, you worked with a lot of the great AI researchers. Like, what have you know, what traits make these individuals so good? And then what have you learned about bringing them together like into teams to accomplish like what they're able to?

• 40:00Speaker 01

  So much of it is is intrinsic motivation and intellectual flexibility. I'll leave this person unnamed, but this person was so motivated and excited about what the research they were doing on on my team that about a month and a half in, I remember having a one on one with him and he just like just let drop that he had never bothered setting up wifi or electricity for his apartment. He had just moved to the Bay to join us. And and I was like, how is this completely okay? And it turned out he just spent all of his time at the office, just like running experiments, and it didn't matter.

• 40:31Jacob Effron?

  That's quite the level of of passion. I mean, I've heard you talk before about how it's kind of somewhat shocking that Google didn't have the GPT breakthrough, just given the transformer was. Invented there? How obvious at the time was it, you know, how game changing the technology was? And you know, that you talked basically that like it was hard for Google to coalesce as a full organization around this versus other research. But maybe just comment a bit about that.

• 40:51Speaker 01

  I mean, credit to Ilya. I remember Ilya was like, we gotta go. So Ilya was our scientific leader for especially for the the basic research part, which ended up spawning GPT, Clip and Dolly. And I just remember him going to work and being like, dude, I think this paper is really important. And poking people to try experiments they were running with other architectures with transformer.

• 41:13Jacob Effron?

  I mean, do you think there's a risk that obviously like now the foundation model companies are doing so many different things right at the same time? It almost feels like that there's ripe for another recipe maybe at some point.

• 41:23Speaker 01

  I think I think losing focus is really dangerous.

• 41:26Jacob Effron?

  You might be of like the biggest fan of Nvidia and Jensen of like most people I have in my life. And so I'm curious, there's obviously so much love in the ecosystem now for everything that that Jensen and the team have accomplished. What are some of the things that Nvidia has done that you feel like aren't talked about a ton, but like are actually a huge part of like what makes that company so impressive?

• 41:45Speaker 01

  I love Jensen. What a complete legend. Yeah, I feel like he's he's just made a bunch of calls here correctly over over such a long period of time. I think people know this now, actually, but I feel like the especially for the last couple of years, it has really paid off for them. But but bringing interconnect in-house and choosing to orient their business around systems was a really good move, I think.

• 42:12Jacob Effron?

  Well, we always like to interview with end our interviews with a just a quick quick fire round and so get your thoughts on some questions. I feel like I know how you're gonna answer this one, but do you think model progress this year will be more or lesser the same as last year?

• 42:23Speaker 01

  I think visibly it'll look probably about the same, but I think actually it'll be more.

• 42:28Jacob Effron?

  What's one thing you think is overhyped and one thing you think is underhyped in the AI world today?

• 42:31Speaker 01

  Overhyped is.

• 42:36Speaker 01

  Scale is dead. We're totally screwed. Let's not buy any more chips. Underhyped, I think underhyped is how do we actually solve extremely large scale simulation for these models to learn from?

• 42:51Jacob Effron?

  David, this has been a fascinating conversation. I'm sure folks will want to learn more about you and some of the exciting work you're doing at Amazon. Where can folks go to learn more about that?

• 42:59Speaker 01

  Yeah, so for Amazon, I would look up the Amazon SFAI lab. And I actually don't use a lot of Twitter, but I plan to get back on it. So you can follow me at Jelawn.

• 43:09Jacob Effron?

  Hey guys, this is Jacob. Just one more thing before you take off. If you enjoyed that conversation, please consider leaving a five star rating on the show. Doing so helps the podcast reach more listeners and helps us bring on the best guests. This has been an episode of Unsupervised Learning, an AI podcast by Redpoint Ventures, where we probe the sharpest minds in AI about what's real today, what's going to be real in the future, and what it means for businesses in the world. With the fast moving pace of AI, we aim to help you deconstruct and understand the most important breakthroughs and see a clearer picture of reality. Thank you for listening and see you next episode.