The bot Cicero can collaborate, scheme and build trust with humans. What does this mean for the next frontier of AI? With Noam Brown, Research Scientist at Meta

AGI can beat top players in chess, poker, and, now, Diplomacy. In November 2022, a bot named Cicero demonstrated mastery in this game, which requires natural language negotiation and cooperation with humans. In short, Cicero can lie, scheme, build trust, pass as human, and ally with humans. So what does that mean for the future of AGI? This week’s guest is research scientist Noam Brown. He co-created Cicero on the Meta Fundamental AI Research Team, and is considered one of the smartest engineers and researchers working in AI today. Co-hosts Sarah Guo and Elad Gil talk to Noam about why all research should be high risk, high reward, the timeline until we have AGI agents negotiating with humans, why scaling isn’t the only path to breakthroughs in AI, and if the Turing Test is still relevant. Show Links: More about Noam Brown Read the research article about Cicero (diplomacy) published in Science. Read the research article about Liberatus (heads-up poker) published in Science. Read the research article about Pluribus (multiplayer poker) published in Science. Watch the AlphaGo Documentary. Read “How Smart Are the Robots Getting?” by New York Times reporter Cade Metz Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Polynoamial Show Notes: [01:43] - What sparked Noam’s interest in researching AI that could defeat games [6:00] - How the AlexaNET and AlphaGo changed the landscape of AI research [8:09] - Why Noam chose Diplomacy as the next game to work on after poker [9:51] - What Diplomacy is and why the game was so challenging for an AI bot [14:50] - Algorithmic breakthroughs and significance of AI bots that win in No-Limit Texas Hold'em poker [23:29] - The Nash Equilibrium and optimal play in poker [24:53] - How Cicero interacted with humans [27:58] - The relevance and usefulness of the Turing Test [31:05] - The data set used to train Cicero [31:54] - Bottlenecks to AI researchers and challenges with scaling [40:10] - The next frontier in researching games for AI [42:55] - Domains that humans will still dominate and applications for AI bots in the real world [48:13] - Reasoning challenges with AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  We were trying to think of what would be the hardest game to make an AI for. We landed on diplomacy. The idea that you could have an AI that negotiates in natural language with humans and like strategizes with them, it really just felt like science fiction. I'm really glad that we aimed high at that point. I was a little afraid to do that, to be honest. It's a high-risk thing to aim for, but all research is high-risk, high reward, or at least it should be. We maybe get like two orders of magnitude more scaling and then we have a big problem. Do you train a hundred billion dollar model? This is why I'm interested in the reasoning direction.

• 0:41Sarah Guo?

  This is the No Priors podcast. I'm Sarah Guo.

• 0:44Speaker 02

  I'm Alad Gell.

• 0:45Sarah Guo?

  We invest, advise, and help start technology companies.

• 0:48Speaker 02

  In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions.

• 1:01Sarah Guo?

  This week on the podcast, we're welcoming Noam Brown. Noam's a research scientist on the Meta Fundamental AI Research Team. Noam co-created the first AIs to defeat top humans in two different types of poker. He also recently did an important project called Cicero. In this podcast, we'll dig into how this AI works, what makes for great AI research and engineering, and how AI games tie into AGI. Noam is considered one of the smartest engineers and researchers in AI. His work has deep implications for how humanity and AI co-evolve. The new bot Cicero can lie, can scheme, it can read a human's intentions and build trust. Cicero demonstrates these skills by performing better than the average human at a classic game called diplomacy. Noam, welcome to No Priors.

• 1:41Speaker 00

  Thank you for having me.

• 1:42Speaker 02

  Thanks a lot for joining. So, you know, I think in the world today, when a lot of people think about AI, they think about it as basically you put a couple of words into a prompt and then you get out an image, or you have ChatGPT summarize James Burnham's professional managerial class for you in a rhyming essay in the voice of a cat or something. And I think you've pushed in really interesting directions that are very different in some ways from what a lot of people have been focused on, and you've been more focused on game theoretic actors interacting with humans and with each other. And in parallel, you're kind of known as, as Sarah mentioned, as sort of one of these true 10x engineers and researchers pushing the boundaries on AI. And so I'm sort of curious, like, what first sparked your interest in games and researching AI to defeat games like poker and diplomacy?

• 2:22Speaker 00

  Well, I think, you know, my journey is a bit non-traditional. I mean, I started out in finance, actually. So towards the end of my undergrad career and also like right after undergrad, I worked in algorithm trading for a couple of years. And I kind of realized that while it's, it's fun and it's, you know, exciting, it's kind of like a game, you know, you got to score at the end of the day, which is how much money you made or lost. It's not really the most fulfilling thing that I want to do with my life. And so I decided that I wanted to do research and it wasn't really clear to me in what area. I was originally planning to do economics, actually. And so I went to the Federal Reserve. I worked there for two years. Honestly, I wanted to figure out how to structure financial markets better to encourage more pro-social behavior. And so in the process, I became interested in game theory and I thought I wanted to pursue a PhD like in economics

• 3:05Speaker 00

  focused on game theory. And two things happened. So first of all, I became a bit, a bit jaded with the pace of progress in economics, because if you come up with an idea, you have to get it passed through legislation and it's a very long process. And computer science is much more exciting in that way, because you can just build something. You don't really need permission to do it. And then the other thing I figured out was that a lot of the most exciting work in game theory was actually happening in computer science. It wasn't happening in economics. And so I applied for grad schools with the intention of studying algorithmic game theory in a computer science department. And when I got to grad school, there was conveniently a professor that was looking for somebody to do research on AI for poker. And I thought this was like the perfect intersection of everything that I wanted to do. I was interested in game theory. I was interested in, you

• 3:48Speaker 00

  know, making something, interested in AI. I had played poker when I was in high school and college and, you know, never for high stakes, but always just kind of interested in the strategy of the game. I actually tried to make a poker bot when I was an undergrad and it, it did terribly, but it was a lot of fun. And so to be able to do that, you know, for research in grad school, I thought this was like the, the perfect thing for me to work on. And also I felt like there was, there was an opportunity here because it felt doable. And I kind of recognized that if you succeed in making an AI that can play poker. You're going to learn really valuable things along the way, and that could have like major implications for the future.

• 4:24Speaker 02

  That's really cool. And did you have a specific end goal of your work when you started it? Was it just interest? In other words, you know, you talk to a lot of people in the field now, and they say, oh, our end goal is AGI, and it's always been. And I think sometimes that's sort of invented later as sort of an interesting story for what they're doing. Did you view this as just doing primary research and it's just personal interest? Did you view it as like there's a path leading to agents that function on behalf of people? Or was there some other sort of driving motivator?

• 4:48Speaker 00

  Well, so I started grad school in 2012, and it was a very different time in 2012. The idea of AGI was really science fiction. There were some people that were serious about it, but very few. The majority opinion was that AI was, if anything, it was kind of a dead field. I actually remember emailing a professor and having this conversation where I was like, "Look, I'm really interested in AI, but I'm kind of worried to pursue a PhD in this because I get the impression that it's just a dead field, and I'm worried if I'll be able to get a job afterwards." So conveniently, like a couple of years into grad school, things changed pretty drastically, and I happened to be in the right place at the right time. I think it was really fortunate in that respect. So the original intention wasn't to pursue AGI. The original intention was, you know, you learn interesting things about AI and game theory,

• 5:33Speaker 00

  and you build slowly. And it was really only a couple of years into grad school that it became clear that the pace of progress was quite dramatic.

• 5:42Speaker 02

  Was there a specific moment that really drove that home for you? I know for some people, they mentioned, oh, AlexNet came out, or, oh, you know, some of the early GAN work felt like a wake-up call. I'm just sort of curious if there's a specific technology or paper or something else that came out, or was it just kind of a continuum?

• 5:55Speaker 00

  I think it was a slow drip. I mean, I think for me especially, it was the AlphaGo moment. You know, like when you see that, it's just very clear. I mean, AlexNet too. I mean, before I started grad school, actually, I took a computer vision class, and they were talking about like, you know, SIFT and all this stuff. And then you get something like AlexNet, and it just like throws all that out the window. And it's just like mind-boggling how effective that could be.

• 6:16Sarah Guo?

  Nam, can you explain actually like why AlphaGo is so important? And like just size of search space and how you might contrast that to previous games?

• 6:24Speaker 00

  Yeah, so a big milestone in AI was Deep Blue beating Garry Kasparov in chess in 1997. And that was a big deal. It's kind of downplayed today, I think, by a lot of machine learning researchers. But we learned a lot from that. We learned that scale really does work. And in that case, it wasn't scaling, you know, training in neural nets, it was scaling search. But the techniques that were used in Deep Blue They didn't work in a game like Go because the pattern matching was just not there. A big challenge in Go was figuring out like, how do you even evaluate the state of a board? How do you tell who's winning? In chess, it's like difficult, but you can kind of write a function, but you can handcraft a function to estimate that, right? That you calculate, oh, each piece is worth this many points and you add it together and you can kind of get a sense of who's winning and who's losing. And in Go, that's just almost impossible to do by hand.

• 7:08Sarah Guo?

  It's essentially too big to do that.

• 7:10Speaker 00

  It's too big, it's too subtle, it's just too complicated and there's too much nuance. And if you asked, you know, the difference is also if you asked a human, you know, who's winning? They could tell you who's winning, but they couldn't tell you why. And so, you know, one of the things that people assumed was that, you know, humans are just better at pattern matching. And to have an AI come along and like demonstrate that it can do this pattern matching better than a human can. And even if it's in this constrained game, that was a big deal. And I think that was a wake up call to a lot of people, not, not just me, but I think across the world.

• 7:40Sarah Guo?

  I remember as a former like Go nerd, just trying to understand the moves that AlphaGo made to try to figure out how to play better. Cause it was like a, such a mind blowing moment.

• 7:50Speaker 00

  Yeah. And you know, if any, if any of your listeners haven't seen the AlphaGo documentary, I highly recommend watching it. You can, I think it's on Netflix and, or YouTube. And you can see just how significant this was to, to a lot of the world when you watch that.

• 8:03Speaker 02

  How did you end up choosing diplomacy as the next thing to work on after poker? There, there's obviously like a wide space of a variety of different types of games. And so what, what drove your selection criteria there? And how did you think about choosing that as the, the next sort of interesting research problem?

• 8:15Speaker 00

  So basically what happened, we succeeded in poker. And when we were trying to pick the, the next direction, it became clear that AI was progressing very quickly. Like much quicker than I think a lot of people appreciated. And there were a lot of conversations about like, what should the next benchmark be? A lot of people were throwing around these games, like Hanabi was one. Somebody was talking about like Werewolf or Sotou's Katan, these kinds of things. And I just felt like. This was 2019, and in 2019, you had GPT-2 come out, which was just mind-blowing. And then you also had DeepMind beating Grand Masters in StarCraft II. You had OpenAI beating human experts in Dota II. And that was just after like a couple of years of work, of research and engineering.

• 8:57Speaker 00

  And to then like go to a game like Settlers of Catan, it just felt like too easy. Like you could just take a team of five people, spend a year on that, and you'd have it cracked. And so we wanted to pick something that would be truly impressive, like that would require fundamentally new techniques in order to succeed, not just scaling up something that already exists. And we were trying to think of what would be the hardest game to make an AI for. And we landed on Diplomacy. The idea that you could have an AI that negotiates in natural language with humans and like strategizes with them. It really just felt like science fiction. And even in 2019, knowing all this success that was happening in AI, it still felt like science fiction. And so that's why we aimed for it. And I think that was the right call. I mean, I'm really glad that we aimed high at that point. I was a little afraid to do that, to be honest.

• 9:42Speaker 00

  It's a high-risk thing to aim for, but all research is high-risk, high-reward, or at least it should be.

• 9:48Speaker 02

  Do you want to give a quick minute overview of Diplomacy? So people can understand what it is and why the research was such a breakthrough?

• 9:53Speaker 00

  Yeah, Diplomacy is this game that was developed in the 50s. It was actually developed by this guy who saw what happened in World War I and kind of viewed this as a diplomatic failure. And so he wanted to create this game that would teach people how to be better diplomats, essentially. And so it takes place at the onset of World War I. There's seven powers that you can play as: England, France, Germany, Italy, Russia, Turkey, and Austria-Hungary. And you engage in these like complex negotiations every turn. And your goal is to try to control as much of the map as possible. And the way you win is by controlling a majority of the map. It's kind of like Hunger Games, where even though only one person can win at the end of the day, there's still this like incentive to be able to work together, especially early on,

• 10:35Speaker 00

  because you can both benefit and have a better chance of winning in the end if you work together. And so you have these like really complex negotiations that happen. And all the communication is done in private. So unlike a game like Risk, for example, or Settlers of Catan, where like all the negotiation is done in front of everybody else. In Diplomacy, you will actually like pull somebody aside, go into a corner, like scheme about who you're going to attack together this turn, who's going to support who. And then after you've negotiated with everybody, you write down what your moves are for the turn. And so then all the moves are read off at the same time. And you can see if people like actually follow through on their promises about like helping you. Or maybe they they lied to you and they're just gonna attack you this turn. So it has like some elements of Risk, Poker and Survivor. Cause there's this big trust component.

• 11:21Speaker 00

  And that's really the the essence of the game. Like, can you build trust with others? Because the only way to succeed in this game is by working together. Even though you always have an incentive to to attack somebody and grow at their expense. So yeah, that's the game. It's been around for a long time. Like I said, since the 50s, it was JFK and Kissinger's favorite game. There's research for this game from an AI angle going back to the 80s. But the idea that you could play this game in natural language with humans and beat them was just complete science fiction. Until a few years ago, like it was still science fiction, but we at least thought it was like worth pursuing it. And research really took off in 2019 when researchers started using deep learning to make bake bots for this game that could play the non-language version. So there's no communication.

• 12:04Speaker 00

  You just write down your moves and you kind of have to communicate non-verbally through the actions that you take. We were doing research on this. DeepMind was doing research on this and then also University of Montreal and a couple other places as well. And there was a lot of interest and progress, but we decided to take the risky bet of just like jumping to the endpoint. And instead of taking an incremental approach, aiming for full natural language diplomacy. And I'm glad that we aim for that.

• 12:30Speaker 02

  It seems like one of the pretty amazing things about what you all did is you basically created bots that other people, that humans thought were other people. And therefore they had to learn how to collaborate with each other, how to sometimes lie or deceive, how to sometimes think through sort of multiple moves from a game theoretic perspective. And so it's a it's a radically different thing than playing chess or or playing Go against another person and then just having almost a probabilistic tree of moves or something.

• 12:55Speaker 00

  Yeah, you run into this like human elements. You really have to understand the human elements. And what's really interesting about diplomacy, aside from just the natural language component, is that it really is the first major game AI breakthrough in a game that involves cooperation. And that that's really important because, you know, at the end of the day, when we make these AIs that play chess and Go. We're not developing them with the purpose of beating humans at games. We want to have them be useful in the real world. And if you want to have these AIs be useful in the real world, then they have to understand how to cooperate with humans as well.

• 13:24Sarah Guo?

  A lot and I were talking about centaur play and whether or not that would persist as an idea at all, given we've accepted that AIs are going to win games at this point. But I think the idea that AIs are going to take action by cooperating with humans, that needs to be a core capability seems obvious. And perhaps this is the making myself feel better story, but I am hopeful that that is a human skill that remains quite important, being able to cooperate with AIs.

• 13:55Speaker 00

  Well, from what I hear, centaur play is like, AIs have gotten so strong in games like chess that it's not clear if the human is really adding that much these days.

• 14:03Speaker 02

  That's what I told Sarah, too.

• 14:05Speaker 00

  Yeah, it's kind of a depressing thought.

• 14:06Sarah Guo?

  Yeah, I'm crying. I get it. I get it. I accept it. Yeah.

• 14:09Speaker 00

  I think the humans are still useful in a game like Go because the AIs are super strong, but they will also sometimes, like a few times in each game, make these really weird blunders. And in diplomacy, I think, yeah, it's super helpful to have an experienced human in addition to the AI. Though eventually, I'd imagine that these systems become so strong that it kind of goes the way of chess, where the human is just adding a marginal difference at the end.

• 14:33Sarah Guo?

  Yeah, I'm actually just wondering how long that window is for humans and centaur play in the game of life. Right? But it's okay. I got it. A lot was right.

• 14:43Speaker 00

  Hopefully, yeah, hopefully forever, but we'll see.

• 14:46Speaker 02

  Yeah. So do you mind explaining the work that you've done in poker and some of the breakthroughs that you made there as well?

• 14:51Speaker 00

  Yeah, my PhD research was really focused on how do you get an AI to beat top humans in the game of No Limit Texas Hold'em Poker? Specifically during my PhD, it was on Heads Up No Limit Texas Hold'em Poker. That's two-player poker. And this was a longstanding challenge problem. Actually, if you go back to the original papers written on game theory by John Nash, the only application that's discussed in the paper is poker. He actually analyzes this simple three-player poker game in the paper and works out the Nash equilibrium by hand. And then actually, at the end, he says, oh, yeah, it'd be really interesting to analyze a much more complex poker game using this approach. So I'm glad we finally got a chance to do that 60 years later. And it's interesting, I think especially after AlphaGo,

• 15:35Speaker 00

  this became a very popular problem. Because after AlphaGo, there was a big question of like, okay, well, AIs can now beat humans at chess. They can beat humans at Go. What can't they do? And the big thing that they couldn't do was be able to reason about hidden information, be able to understand that, okay, this other player knows things that I don't know, and I know things that they don't know. And being able to overcome that problem in a strategic setting was a big unanswered question. And yeah, so that was the focus of my research from basically my whole grad school experience. And there were a few different research labs that were working on this. And what would happen is every year, we would all make a poker bot, and we would play them against each other in this competition called the Annual Computer Poker Competition.

• 16:19Speaker 00

  And we actually won. So like, basically what happened is when I started my PhD, there had already been like some progress in AI for poker. And so the competition really turned into a competition of scaling. There's about like 2.5 billion different hands that you could have on the river, like the last round of poker in Texas Hold'em. And what we would do is cluster those hands together using K-means clustering and like treat similar hands identically. And that allows you to compute a policy for poker. Because now, instead of having to worry about 2.5 billion hands and like having to come up with a policy for each one of those, you can now like bucket them together. And now you have like 5,000 buckets or something, and you can actually compute a policy for that many buckets.

• 17:03Speaker 00

  And so this was like before neural nets. That's why we were doing these like this K-means clustering thing instead of deep neural nets. But you can kind of think of it as like, the number of buckets that you have is kind of like the number of parameters that you have in your network. And so in grad school, it kind of turned into a competition of scaling. How many buckets could you have in your bot? And first year is like 5,000 buckets, then we got up to 30,000 buckets, and then 90,000 buckets. Every year, we would have these bigger and bigger models. We would train them for longer, parallelize them, and they would always beat the previous year's model. And in 2014, We actually won the annual computer poker competition. And after that, we decided to take our bot and play it against expert human players. And so this was the first, what was called the brains versus AI poker competition,

• 17:48Speaker 00

  where we invited like these four top Heads Up No Limit Texas Hold'em poker pros. And we had them play 80,000 hands of poker against our bot. And the bot actually lost by a pretty sizable margin. And it occurred to me like during this competition that the way the humans were approaching the game was actually very different from how our bot was approaching it. So we would train our bot for like two months leading up to this competition, you know, on a thousand CPUs. But then when it came time to actually play the game, it would act instantly. And the humans would would do something different. Like, you know, obviously they would practice ahead of time. They would develop an intuition for the game. But when they were playing the game against the bot and they were in a difficult spot, they would sit there and they would think.

• 18:31Speaker 00

  And sometimes it was like five seconds, sometimes it was like a minute. But they would think that would allow them to come up with this better solution. And it occurred to me that this might be like something that we're missing from our bot. And so I did this analysis after the competition to figure out, okay, if we were to add this search, this planning algorithm, that would come up with a better strategy when it's actually in the hand, how much better could it do? And the answer was it improved the performance by about 100,000x. It was the equivalent of scaling the model, like scaling the number of parameters, scaling the training by 100,000x. Now, the three years of my PhD at that point, I had managed to scale things by about 100x. And, you know, that's like quite good. I was very proud of that.

• 19:16Speaker 00

  But when I saw that result, it made me appreciate that everything I had done in my PhD up until that point was just a footnote compared to adding search and scaling search. And so for the next year, I just worked basically nonstop, like 100-hour weeks, trying to scale up search through as much computation of the problem at inference time as possible. And then we did another competition in January 2017.

• 19:39Speaker 00

  Where we played against four top expert poker players again, $200,000 in prize money to incentivize them to play their best. And this time, we completely crushed them. Poker players were literally telling us they did not think it was possible to beat expert poker players by that kind of margin. Yeah, and so that's the story of like, you know, my grad school experience working on poker AI. That was for two-player poker. We ended up, after that, working on multiplayer poker, on six-player poker. Again, the big breakthrough there was that we developed a more scalable search technique. So instead of always having to search to the end of the game, it could search just a couple of moves ahead. And what was really interesting there is the bot, we did another competition, the bot won, and that bot cost under $150 to train, if you were to run it on like a cloud computing service.

• 20:24Speaker 00

  And I think that shows that this wasn't just a matter of scaling compute. It really was an algorithmic breakthrough. And this kind of result would have been doable 20 years ago if people knew the approach to take.

• 20:37Speaker 02

  If you look at a lot of other games, those sorts of big shifts in performance from a bot relative to people then shifts how people play, right? They learn from the bot or they adapt their game from watching games that the bots play. How did that play out in terms of poker?

• 20:51Speaker 00

  Yeah, that's a great question. So, you know, the competition, it was really interesting because, you know, so kind of like as a last-minute thing, we added this ability. The way the bot works, we give it different bet sizes that it can use. The game that we were playing, there's 20,000 chips, $100, $200 blinds, 50 $100 blinds, actually. And so it can bet any amount it wants from like $100 up to $20,000. And so there's not much value in like being able to bet both $5,000 and $5,001. And so we would discretize that action space to constrain it to like only considering a few different options. And so there's a question of like, okay, well, what sizes do you give it the choice between? And, you know, towards the end, when we were developing this bot, like, we just had room for extra computation. And so we just like threw in some extra sizes, like 4x the pot, 10x the pot.

• 21:36Speaker 00

  Like, it doesn't cost that much more, so why not just give it the option? I didn't think it would actually use those sizes. And then during the competition, it actually ended up using those sizes a lot. And it would sometimes bet like, you know, $20,000 into a $100 pot, which was completely unheard of in professional poker play. And you know, I was a little worried about this because I thought it was a mistake at first. And I think the players that we were playing against also thought it was a mistake at first. But then they they found that they kept ending up in these like really tricky situations. And you know, they would just really struggle with like whether to call or fold. And that's that's how you know you're playing good poker. If you see the other person like really struggling with the decision, that is a sign that you're doing something right.

• 22:17Speaker 00

  And at the end, they told us like, yeah, that's the one thing that we're going to try to incorporate into our own play. Adding these like what are called over bets into into our strategy. Typically, the the strategy was like, oh, you bet between a quarter of the size of the pot and one times the pot. And now in professional poker play, it's actually really, I wouldn't say common, but it's a part of the strategy to bet sometimes like 5X the pot, 10X the pot. If you can pull it off in the right way, it can be a very powerful strategy. And I should also say like, the way professional poker players train now, they all use bots to assist them. It's a lot like chess where you play the game and then you have a bot analyze your play at the afterwards and see like, okay, did you make mistakes? Where did you make mistakes? How could you do better next time?

• 23:03Speaker 00

  The game really has been demystified and become a lot like chess. I kind of describe poker as essentially high dimensional chess. It's like chess where you have to reason about like a probability distribution over actions instead of just like discreet actions.

• 23:19Sarah Guo?

  Yeah, it's really, it's really interesting because I don't think people really believe there was fully optimal play in poker before. Like they understood the probability distribution, but if you're playing live poker, like there's social cues, right? And social play and that has clearly been swept out. Not as an activity of like enjoyment, but in terms of a strategy that actually wins.

• 23:41Speaker 00

  Yeah, I think that's surprising to a lot of people that this idea that there is an optimal way to play poker. You know, there, there's this thing called a Nash equilibrium where if you're playing that strategy, you will never lose. It guarantees that in the long run, you will not lose an expectation. And the reason for that is because like if you're playing against somebody else that's also playing the Nash equilibrium, like obviously you can't both win, one of you is going to lose or you're going to tie. And so in expectation, if you're playing against each other against somebody else that's playing the Nash equilibrium, you're going to end up tying. But in practice, what ends up happening is if you're playing the Nash equilibrium in a complicated game like poker, the other person is going to make these small mistakes over time. And every mistake that they make is money in your pocket. And so you just play the Nash equilibrium, wait for them to make mistakes and you end up winning.

• 24:22Speaker 00

  And that is now the conventional wisdom among poker players that you start by playing the Nash equilibrium. If you're, if you're really good, you can look at the other players, see how they're deviating from the Nash equilibrium, playing suboptimally, and maybe you can like deviate yourself to capitalize on those mistakes. But really the safe thing to do is play the Nash equilibrium, let them make mistakes and every mistake that they make costs them money and puts money in your pocket.

• 24:46Speaker 02

  What was the most unexpected thing that come out of working on diplomacy in terms of, you know, what Cicerone could do?

• 24:52Speaker 00

  I mean, I think the most unexpected thing was just honestly how it didn't get detected as a bot. We were really worried about this leading into, into the human competitions because, first of all, there's no way to like really test this ahead of time. Like we can play with the bot, but we know that it's a bot and we can't really like gather a bunch of people together and stick them in a game. And, you know, have them play with a bot without having them realize that something's up, right? Like if this company is hiring them to like play a game like, and they know that we're working on diplomacy, like clearly they're going to be playing with a bot. And when people know that they're playing with a bot, they behave very differently, right? We didn't want to turn this into a Turing test. And so we had to enter the bot into these games where players did not know that there was a bot in the mix. That was the only way that we could get like meaningful results.

• 25:33Speaker 00

  And just to be clear, the reason for this is because like diplomacy is a natural language negotiation game. And so you're having these like really complicated, long conversations with these, with these people. And it's kind of hard to get away with that as a bot, but not be detected. And so our big concern was like, we stick this bot in a game and within like, five games, maybe even two games, they figure out it's a bot. Word gets out, all the diplomacy, the diplomacy community is pretty small, so they all talk to each other. And then in all the future games, everybody's like asking, you know, Turing test questions, trying to figure out who the bot is. And our experiments are just like meaningless. And so we figured like, okay, maybe we get lucky and we manage to get like 10 games in before they figure this out. But at least we have like 10 games worth of data.

• 26:15Speaker 00

  But surprisingly, we managed to go like the full 40 games without being detected as a bot. And that was surprising to me. And I think that's a testament to the progress of language models in the past couple of years especially. And also that maybe humans aren't as good at talking as we might think. Like, I maybe appreciate also that, you know, If somebody is saying something a little weird, because the bot does say weird things every once in a while, their first instinct is not going to be like, oh, I'm talking to a bot. Their first instinct is going to be like, oh, this person is dumb or distracted, or they're drunk or something. And then way down on the list is like, oh, this person is a bot. So I think that we got pretty lucky in that respect. But also, the bot did manage to actually go these 40 games without being detected. And so I think

• 26:59Speaker 00

  that is a testament to the quality of the language model.

• 27:01Sarah Guo?

  I think Meta is actually planning to release the data, which is going to be so interesting. But can you just describe an interaction from the bot you thought was interesting in these negotiations?

• 27:12Speaker 00

  Oh, yeah. I think one of the messages that was really, honestly, kind of scary to me was just when it was talking to another player and the player was saying, hey, I'm really nervous about your units near my border. And the bot honestly was not planning to attack the player. It was planning to go in the other direction. And it sent the player this really empathetic message where it was like, look, I totally understand where you're coming from. I can assure you 100%. I'm not planning to attack you. I'm planning to go the other direction. You have my word. And it really felt like a very human-like message. And 100%, I would have never expected that to come from a bot. And when you see stuff like that, it makes you appreciate, like, yeah, there's something really powerful here.

• 27:53Speaker 02

  How do you think about the Turing test in the context of all this? Or what's your updated model of whether the test is still relevant or how to think about it?

• 28:00Speaker 00

  So there was actually a New York Times article that came out from Cade Metz at the New York Times on the Turing test and what it means. And he actually talks about Cicero in the article. And basically, his view is that the Turing test is kind of dead. And I kind of agree with that. I think the Turing test is no longer really a useful measure, the way it was intended to be. Certainly, just because we have bots that can, I wouldn't say they can pass the Turing test, but they're getting close enough that it's no longer that useful of a measure. It doesn't mean that we have General intelligence, I think there's still a long way to go on that. There's a lot of things that these bots can't do well. But yeah, I think my view now is that the Turing test is not that useful of a measure anymore. That doesn't necessarily mean that it was always a useless measure.

• 28:44Speaker 00

  I think it just shows like how much progress we've made. We're not 100% there, but you know, the progress really has been staggering, especially in the past few years.

• 28:51Speaker 02

  What measure do you think or measures do you think makes sense to use? And then also, what do you think is missing on sort of the road to general intelligence?

• 28:57Speaker 00

  I think there's a few things that are missing. The big thing that I'm interested in particular is reasoning capabilities. You have these bots and they're all doing next word prediction, right? Cicero is a bit different, actually, in that it's actually conditioning its dialogue generation on a plan. And I think that's one of the really interesting things that distinguishes Cicero from a lot of the work that's happening in language models today. But a lot of the research that is happening is using next word prediction. And when it's trying to do something that's like more sophisticated in terms of reasoning capabilities, it's a lot of chain of thought where it's just like rolling out, you know, the kind of reasoning that it's observed humans do and they're in its training data, then seeing where that leads. So I think there's a general recognition among AI researchers that this is a big weakness in the bots today.

• 29:41Speaker 00

  And that if we want truly general, artificial general intelligence, then this needs to be addressed. Now, there's a big question about how to address it. And that's actually why I really like this direction because it's still an open question about how to actually fix this problem. There's been some progress, but I think there's a lot of room for improvement.

• 29:57Sarah Guo?

  What do you think are the most promising possible directions?

• 30:00Speaker 00

  That is, that is the trillion dollar question.

• 30:02Sarah Guo?

  You're the trillion dollar man.

• 30:04Speaker 00

  I think there's like clear, clear base. I mean, like first of all, a chain of thought really was like a big step. And it's kind of shocking just like how effective that was, given how simple of an idea it is.

• 30:13Sarah Guo?

  I tell myself every day when I wake up now, let's think step by step.

• 30:16Speaker 00

  Yeah. Yeah. So for those of you that don't know, it's just like you add to the prompts like, oh, let's think through this step by step. And then the AI will like actually generate a longer like thought process about how it reaches its conclusion. And then that actually leads to better conclusions. But you can, you can kind of see that as like just rolling out the thought process that is observed in human data. And so there's a question of like, okay, well, instead of just rolling that out, could you actually improve it as it's going through each step? And so I think things like that, I mean, I'm kind of keeping it like very abstract because you know, it's an important question. Also, I think there's not a clear answer yet. So I don't want to speculate too much, but I think that there is like room for improvement in this direction.

• 30:56Speaker 02

  What was the actual data set that was necessary in order for the training here? And maybe to take a step back, you know, I've been having a series of conversations with people about data and sort of like, when do we run out of data that's easily available? And when do we have to start creating either large scale synthetic data or RLHF data? Or, you know, do you literally pay bounties to people to just record themselves all day? So you can start collecting interesting data off of them to do different things with over time, right? As these models scale to a certain point where, you know, you've used up the internet and you've started using up all the video content. You start running out of stuff. I'm just sort of curious like how you thought about data in this context and what's necessary to really take things to the next level from a, you know, self-driven agent perspective like this.

• 31:36Speaker 00

  Yeah, it's not clear that data really is the bottleneck on performance here. And I've talked to AI researchers about this, and I think there isn't as much of a worry about this as people might think. Partly that's because there's a lot more data that's out there than people might realize and that people are using right now. And also it's because I think there are going to be improvements to sample efficiency as the research progresses. And so I think we'll be able to stretch the data more.

• 32:03Speaker 02

  Why do you think it's a bottleneck?

• 32:05Speaker 00

  So I think the bottleneck is going to be scaling. I mean, so you look at the models that exist today, like they probably cost $50 million to train. You can probably easily 10x that, you know? I wouldn't be surprised if there's a $500 million model that's trained in the next year or two. You can maybe even go another order of magnitude and like train a $5 billion model. If you're like the US government or something, or like a really big tech company. But what do you do beyond that? Do you train a $100 billion model? You'll probably see some improvement, but at some point it just becomes like not realistic anymore. And so that's, that's going to be the bottleneck. Like we maybe get like two orders of magnitude more scaling and, and then we have a big problem. And people are focused on like, okay, how do we, how do we make this more efficient? How do we train this cheaper, more paralyzed?

• 32:50Speaker 00

  But you can only squeeze so much out of that. And I think we've, we've squeezed a lot already. And this is why I'm interested in the reasoning direction, because I think there is this whole other dimension that people are not scaling right now, which is the amount of compute at inference time. You know, you can spend $50 million training this model ahead of time, like pre-training this model. And then when it comes to actual inference, it costs like a penny. And you know, what happens if instead of it returning an answer in a second, it returns an answer in like an hour or even five seconds or 10 seconds. You know, sometimes if people want to give a better, a better answer, they'll sit there and they'll think a bit and that leads to a better outcome. And I think that that's one of the things that's missing from these models.

• 33:35Speaker 00

  So I think that that's one of the ways to overcome the scaling challenge. And that's partly why I'm interested in working on that.

• 33:41Sarah Guo?

  Going back to related to what Alad said, diplomacy problems specifically didn't have like, you know, internet scale data. Right. As you mentioned, it's like a relatively small community. Can you talk about what you guys did in terms of self-play and the data that actually was involved?

• 33:57Speaker 00

  So diplomacy, the problem was interesting because yes, there's actually not a ton of data out there. I mean, we had a relatively good data set about about 50,000 games with dialogue. We did.

• 34:07Sarah Guo?

  This is from like web diplomacy.

• 34:08Speaker 00

  Yeah, this is from a site called web diplomacy.net. It's been around for like almost 20 years where people play diplomacy casually on the site. We were very lucky to get this data set. I mean, honestly, I was scouring the internet, trying to find like all the sites that have available data. And this was basically the only sites that had a meaningful amount of data. Like there was another popular site, but they periodically deleted their data, which was, you know, just mind boggling to me. It's just you're sitting on a goldmine here and you're just deleting it because it's taking up server space. I guess they didn't appreciate that like AI researchers will one day be interested in that. And then other sites just like refuse to hand over their data. And so I'm really glad that we managed to like work out a deal with web diplomacy.net because otherwise the project would have just never happened. Now that's about 50,000 games of diplomacy, about 13 million messages.

• 34:53Speaker 00

  And that is a good sized data set, but it's not, it's not enough to train a bot from scratch. Fortunately, we are, we're able to leverage like, you know, a wider data set from the internet. So you kind of like have a pre-trained language model and then you fine tune it on the diplomacy data. And you get a bot that can actually communicate pretty well in the game of diplomacy. Now that helps with the dialogue, but there's still a problem, which is that the strategy isn't going to be up to par. And that's partly because. You can't do that well with just supervised learning. You can't learn like a really, really good strategy in these kinds of games just with supervised learning. And it's also because the people that are playing these games are not very good at the game. Like, the most of the dataset is from fairly weak players. You know, that's just a reality. You have a bell curve.

• 35:35Speaker 00

  The actual strong players are like a relatively small fraction of any dataset that you have. And I should say, this is not limited to diplomacy. Like, we also found in chess and Go, we actually ran this experiment. If you do just pure supervised learning on a giant dataset of human chess and Go games, the bot that you get out from that is not an expert chess or Go player. Even if it's like conditioned to behave like a chess grandmaster, it's not going to be able to match that kind of performance because it's not doing any planning. That's really what's missing. And so, in order to get a strategy that was able to go beyond just like average human performance, or even like, you know, strong human performance, to something that's like much better, we had to do self-play.

• 36:20Speaker 00

  And this is like how all these like previous game AIs have been trained, right? Like, you look at AlphaGo, you look at, especially AlphaZero, the latest version of AlphaGo, and you look at the, you know, the Dota 2 bot. The way they're trained is by playing against themselves for millions or billions of trajectories. That's also how our poker bot was trained, for two-player and six-player poker. Now, the difference is like when you go from those games to diplomacy, suddenly, there is this cooperative aspect to the game. Like, you can't just assume that everybody else is going to behave like a machine, like identically to the way you're going to behave. And so, in order to overcome that, we had to combine self-play with a recognition that

• 37:03Speaker 00

  humans are going to behave a lot like how our data suggests. And so, using the dataset that we have, we're able to build up this model, a rough model of how humans behave, and then we can improve on that using self-play. And so, we're figuring out a good strategy, but basically a strategy that's compatible with how humans are playing the game. So, to give some intuition for this, like, because it's not obvious why this changes when you go from a two-player zero-sum game like chess to a cooperative game like diplomacy. I mean, also, I should say, like diplomacy is both cooperative and competitive, but there is a cooperative, a big cooperative component. Like, let's say you're trying to develop a bot that negotiates. If you train that bot from scratch with no human data... It's going to, it could learn to negotiate,

• 37:48Speaker 00

  but it could learn to negotiate in a language that's not English. It could learn to negotiate in some gibberish robot language. And then when you stick it in the game with six humans, that's a negotiation task like diplomacy. It's not going to be able to communicate with them. And they're just going to all work with each other instead of with the bot. That same dynamic happens even in the strategy game, the moves in the game, the non-verbal communication aspect. The bot will develop these norms and expectations around like what its ally should be doing this turn. Like, I'm going to support my ally into this territory because I'm expecting them to go into this territory. And I don't even have to talk to them about this because it's just so obvious that they should be doing this. But the humans have their own meta game where like, oh, it's actually really obvious that I should be supporting you into this territory.

• 38:32Speaker 00

  And if you don't understand the human norms and conventions, then you're not going to be able to cooperate well with humans. And they're just going to not work with you and work with somebody else instead. So that's what we really had to overcome in Cicero. And we managed to do that by using the human data to build this model of how humans behave. And then adding self-play on top of that as kind of like a modifier to the human dataset.

• 38:54Sarah Guo?

  That actually has some really interesting implications, right? Like, if you believe in the long term, we are going to have bots that take action in the real world, interacting with humans. And humans are perhaps not very good at optimal play in the game of life. And you're interacting with them. Like, it just brings home the point of how important reasoning could be versus learning and pattern recognition.

• 39:16Speaker 00

  Yeah, I think you're absolutely right that like this matters a lot if you want to make AIs that interact with humans in the real world, right? Like, if you have a car driving on the road, a self-driving car, you don't want it to assume that all the other drivers are machines that are going to act perfectly optimally in every step of the way. Like, you want the self-driving car to recognize that these other drivers are humans and humans make mistakes. And somebody could swerve into my lane. And yeah, and also like, you know, just like day-to-day interactions, understanding like the non-verbal cues of humans and like what that means. These are things that, or even the verbal cues, these are things that an AI has to be able to cope with if it's going to like really be useful to humans in the real world and not just beating them at chess.

• 39:59Speaker 02

  Games have been used for a while now as a way to measure AI progress. And you've worked on poker variance and diplomacy variance. And you mentioned before, other work people have done in terms of chess and Go and things like that. What do you think is the next frontier in terms of games and sort of research on them in the lens of AI?

• 40:14Speaker 00

  Yeah, so there's a long history of games as benchmarks for AI. This goes all the way back to like the very foundations of AI back in like the 50s. Chess in particular was held up as this like grand challenge for AI because if we can make an AI that was like as smart as a human chess grandmaster, then like imagine all the other smart things it could do. Of course, that turned out to be like kind of a false promise, right? Like you get an AI that plays chess and it turns out it doesn't really do anything else. But we've learned a lot along the way. And games are useful as a benchmark because you can compare very objectively to top human performance. Like it becomes very clear when you're surpassing human ability in this domain, even if it's a restricted domain. And you also have this benchmark that's existed before the AI researchers came along.

• 40:57Speaker 00

  Like AI researchers, it's really easy for them to come up with a benchmark once they have the technique already created. You know, you come up with the, you come up with the technique and then you're saying like, okay, well now it's really easy to come up with a benchmark that this technique will work for. And you don't want that. You want the problem to come first and games give you that. But I think we're reaching a point now where individual recreational games are just no longer that interesting of a challenge. You know, I said earlier, we chose diplomacy because we thought it would be the hardest game to make an AI for. And I think that's true. I, I can't think of any other game out there where like if somebody made an AI that could play that game, I would be like, wow, that's super impressive. And I did not think that that was possible.

• 41:41Speaker 00

  And so I think going forward, the field needs to move beyond looking at individual games and starting to look at, first of all, going beyond games, but also looking at generality. The approach that we've used in diplomacy is very different from what we previously did in poker and what others have done in chess and Go and StarCraft. And now there's a question of like, okay, well, if we really want a general system, a general AI, can we have it play all of these games at a superhuman level and also able to do things like, you know, image generation and question answering and like all these tasks? And if we could accomplish that, then that becomes incredibly impressive. And so I think games will continue to serve as this benchmark, but it's not, instead

• 42:26Speaker 00

  of serving as a benchmark that the research kind of overfits to, my hope is that it will serve as a benchmark that we use along other benchmarks outside of games, like, you know, image generation benchmarks and language Q and A benchmarks and these kinds of things.

• 42:40Sarah Guo?

  Given that the AI is already one in these restricted domains that are challenging in specific ways, like, how do you think about the domains that are going to be human skill dominant? Like, are there going to be domains that like that?

• 42:51Speaker 00

  Well, certainly, anything in the physical world, I mean, humans still dominate. I mean, when it comes to actually like, you know, manipulation tasks, these kinds of things, robotics is really lagging behind. And I think I'm trying to avoid doing anything in the physical world for that reason. Software is just so much nicer to work with. I think reasoning, there's still things that you can't, that humans are definitely better at, even in restricted domains. You look at something like writing a novel. I don't think you can get an AI to output like the next Harry Potter just yet. That might not be that far off. Maybe it's like five years away or something, but I don't think it's happening just yet. It's kind of scary that it's, I'm really struggling to come up with domains where I'm like, oh yeah, AI is not going to be able to surpass humans in this.

• 43:33Speaker 02

  Yeah, I was about to say, like, I feel like people often talk about areas where humans will always have an advantage just because they're humans, they want to feel good about the future versus because there's necessarily something that shouldn't be tractable from a, at least sheer logical perspective, right?

• 43:47Speaker 00

  Yeah, it certainly is. I mean, I think the big advantage that humans have, and it's not clear when AI will surpass humans in this, is generality. The ability to learn from a small number of samples, to be able to like, you know, be useful across a wide variety of domains.

• 44:00Speaker 02

  But isn't that generality overstated? Because I feel like in the examples that you mentioned, you said everything from like image gen to diplomacy and like a single architecture or AI or something. And often it seems like, you know, if you look at the average person, if they're very good at one thing, they're usually not good at everything, right? And so I kind of feel like the bar that we're using. In terms of generality for AI, sometimes it's higher than the bar we'd use for generality for people in some sense, or is that not a true statement?

• 44:26Speaker 00

  I think it's not just about generality, it's really about sample efficiency. Like, how many games does it take for a human to become a good chess player, or a good diplomacy player, or a good artist? The answer is orders of magnitude less than it takes for an AI. And that is going to pose a problem when you're in domains where there isn't much data. Now, that seems like a problem that could be overcome. I'm just saying, that's a problem that hasn't been overcome yet. And I think that that's one of the clear advantages that humans have over AIs today.

• 44:51Speaker 02

  When do you think we'll see the emergence of AIs in financial or economic systems? And obviously, we have like algorithmic trading and other things like that. And then we have things like crypto, where you effectively have programmatic approaches to effectively money wrap this code, right? And the ability to interact with those things in reasonably rich ways through smart contracts. You know, do you think we're, there's any sort of near-term horizon of people experimenting with that, or just interesting research being done in terms of the actual interaction of a bot with a financial system?

• 45:15Speaker 00

  I think it's already being done. If you look at financial markets, I'm sure there's tons of trading powered by deep learning. I've actually talked to a lot of finance companies about this. I used to work in finance, and also, like, a lot of finance companies love poker. And so I've given a few talks at like various places on AI for poker. And I've talked to a few places about like, is reinforcement learning actually useful for financial markets, for trading? And the answer I get is usually no. I think the major challenge with using things like reinforcement learning for trading is that it's a non-stationary environment. So you can have all this historical data, but it's not a stationary system. And it's gonna, like, the markets respond to world events, these kinds of things.

• 45:56Speaker 00

  So you need a technique, ideally, that really understands the world, not just treating everything like a black box.

• 46:03Speaker 02

  But could that at all feed into what you're saying about spending more compute on inference versus training? In other words, incorporating real-time signals at the point of decision-making? Or did you mean something else by that in terms of model architecture that would enable you to update weights in certain ways or things like that over time?

• 46:18Speaker 00

  Well, I think it goes back to the sample efficiency problem that humans are pretty good at adapting to novel situations. And you run into these like novel situations pretty frequently in financial markets. Yeah, I think it's also a problem of generality that you need to understand so much about the world to really succeed. Now, that said, I mean, I think that the AIs are successful in financial markets in like fairly limited ways. Certainly, if you want to like break up big orders, these kinds of things. Also, I should say like, I'm not an expert in this. Like, this is kind of outdated knowledge from me. Because I'm sure like, there's a lot of cutting edge stuff that's happening that people are not telling me about because it's making money. But I can tell you that this is like, kind of the perspective as of like, maybe five years ago. That I think that there's, it's being used in limited ways, but I don't think it's fully replacing humans yet.

• 46:59Sarah Guo?

  Do you think we're going to get bots that negotiate with humans soon? Or I guess, let me premise that is, we are eventually going to get them. What do you think the timeline is or the use case?

• 47:09Speaker 00

  That seems doable. I think it depends on how constrained the domain is. I think if you were to look at constrained domains, certain negotiation tasks, I think that AIs could probably do better than humans in that today. I mean, I'm trying to think of like, specific examples, but things like, you know, if you wanted to negotiate over the price of a good, it could probably do better than a human in a lot of those, in a lot of those situations. I think if there's things like salary negotiations, it might do better than humans at that also. I think it depends on how much you need to know about the world. And I think contract negotiations, for example, would still be difficult because there's so much subtlety, there's so much nuance to like, every contract. And it's not going to replace like, a professional negotiator for that kind of task just yet.

• 47:50Speaker 00

  But kind of the things that are more constrained, don't require as much like, outside knowledge about the world. I think AIs are probably up to the task already.

• 47:59Speaker 02

  So, a friend of mine who used to work with you says that one of the things you're really exceptional at is you tend to pick a neglected research domain with lots of promise. You commit to it long term, and then you become the best at it. And many people in the world kind of get attracted to shiny things instead and kind of distracted by, you know, whatever is in vogue, but then it turns out to be less interesting research. What are you thinking about working on next? Or what interests you as sort of the next wave of stuff to do?

• 48:22Speaker 00

  I think the, the big thing I'm interested in is the reasoning problem. And this is kind of motivated by my experience in this game space. You look at things like AlphaZero, the latest version of AlphaGo. And like AlphaGo in particular is held up as this like this big milestone in deep learning. And to some extent, it is like it was not doable without deep learning, but it wasn't deep learning alone that enabled that. If you take out the planning that's being done in AlphaGo and just use the raw policy network, the raw neural network, it's actually substantially below top human performance. And so we have with just like raw neural nets, we have all these things that are incredibly powerful, like, you know, chatbots, image

• 49:07Speaker 00

  generation software, but the raw neural net itself still can't play Go. It requires this extra planning algorithm on top of it to to achieve top human performance. And that planning algorithm that's used in AlphaGo, Monte Carlo tree search, is very domain specific. I think people don't appreciate just how domain specific it is, because it works in chess, it works in Go, and these have been like the classic domains that people have cared about for investigating these kinds of techniques. It doesn't work in poker, it doesn't work in diplomacy. I think because I've worked in those domains, I kind of like recognize that this is this is a major weakness of these kinds of algorithms. And so I think there's a big question of like, okay, how do we get these models to be able to do these

• 49:48Speaker 00

  like complex reasoning, planning tasks with a more general system that can work across a wide variety of domains? And if you can do that, if you can succeed in that task, then it enables a lot of really powerful things. Like, one of the domains that I'm thinking about is theorem proving. You know, it doesn't seem crazy to me that you could have a model that can prove the Riemann hypothesis within the next five years, if you can solve the reasoning problem in a truly general way. And yeah, you know, maybe maybe the inference cost is huge, like maybe it costs a million dollars per token to generate that proof, but that seems totally worth it if you can pull it off. And maybe you can do other things with it, too. Like, maybe maybe that's maybe that allows you to like,

• 50:30Speaker 00

  you know, write the next prize winning novel. Maybe that enables you to come up with like lifesaving drugs. But I think

• 50:36Sarah Guo?

  Just for context, the Riemann hypothesis is like considered the the most important unsolved problem in math, where, I don't know, the first X set of solutions have been checked, but we we don't know for sure yet.

• 50:47Speaker 00

  Yeah, and I think the key is that that I'm really interested in is the generality. Like, we can solve this problem in domain specific ways. But then it always ends up like kind of overfit to that domain. And so I think what we need is something as general as what we're seeing with transformers, where you just throw it at any sort of problem and it works surprisingly well.

• 51:08Sarah Guo?

  And I guess you're implying that there are ways to frame the problem to make progress that are more general, but really interesting to making progress in reasoning. And that could be around math or possibly code. Is that the right understanding?

• 51:23Speaker 00

  My hope is that the techniques are general. I mean, I think it's important to also look at a wide variety of domains in order to like prevent you from overfitting. And yeah, one of the domains that I think would also be a good fit is code generation, because I think to write good code, like next token prediction is going to, is getting you surprisingly far, but I don't think it's going to get you all the way there to like replacing engineers at big companies.

• 51:43Sarah Guo?

  Yeah, maybe one piece of just context for listeners is Copilot is amazing, right? But what we are doing with code generation today is very local context specific.

• 51:52Speaker 00

  Yeah. And so if you want to like plan out like a whole product, like that's, doesn't seem doable with existing technology. And, you know, I think, I think the perspective of a lot of people, when they, when they hear me say this is like, well, you know, but you just scale it, you know, you scale up the models, you scale up the training, and that's always worked in the past. And the example I like to give is you look at, okay, you look at AlphaGo, like, yeah, you could, in theory, scale up the training, scale up the model capacity, and you don't need planning then. You just have like a really large, you run this reinforcement learning algorithm for a really long time, you have this really big network, and it will eventually learn, in theory at least, how to beat expert humans and go. But there's a question of like, okay, well, how much would you have to scale it up? How much would you have to scale up this raw neural net, the capacity and the training in order to match the performance that it achieves with multi-college research?

• 52:37Speaker 00

  And if you crunch the numbers and it ends up being 100,000x. Now, these models are already costing like $50 million. Like clearly you're not going to be able to scale them by 100,000x. And so then there's a question of like, okay, well, what do you do instead? And the answer in AlphaGo is like, well, instead of having all of that computation be during training,

• 52:59Speaker 00

  You also have it spend like 30 seconds to figure out what move to make next when it's actually playing the game. And that shifts the cost burden from having to like pre-compute everything to then being able to think on the fly. And so that's why I think that avenue seems like the piece that's missing.

• 53:16Speaker 02

  A really random question, because if you look at the human brain, you have these very specialized modules with very specific functions, right? You have the visual cortex for visual processing. You have like different things for emotion in terms of specific modules. Like there's specific parts of the brain that if you ablate, you remove certain emotive or other capabilities, right? There have been accidents where like poles have gone through people's heads and ablated a very specific place and the people have survived. And so you see this sort of very specific ablation of function through the ablation of specific modules. Why is it the correct assumption to think that there should be a generalizable architecture versus you just have a bunch of sub-modes that are all running together that collectively enable a wide range of behavior, which is effectively what we see in the brain?

• 53:54Speaker 00

  That's a good question. I don't think that we need to be tied to a specific technique. The answer might be that we need to have like more specialized systems instead of just like one truly general architecture. But I think what I'm thinking about is more the goal rather than the approach. So we want something that's able to succeed across a wide variety of domains. And having to come up with like a unique approach to every single domain, that gets you part of the way. But I think that eventually that will be superseded by something that is truly general.

• 54:20Speaker 02

  Yeah, that makes sense. And I guess, you know, one big domain is just reasoning, right? So I didn't mean to imply that it's different subtypes of reasoning will require different approaches, but more there may be really big things that fundamentally may function in a very different way. And again, that may be incorrect, right? The brain is a evolved system, which means it has enormous limitations in terms of where it came from and how it got created. And you often end up with these local maxima when you evolve a system, right? I was sort of curious about how you thought about that.

• 54:45Speaker 00

  Yeah, there's certainly a risk always with research that you could end up in a local minimum. And it's like hard to, people like overfit to that. And I think, I think actually like machine learning was an example of this, like. Deep learning, not many people were focused on this because they kind of assumed it was this dead end and there were only a few people out in the, you know, like, Canadian wilderness that were working on this. And that ended up like being tremendously successful. And so there's value in diversity, there's value in diversity of approaches. And so I think, I think it does help to try to think outside the box and try to do something that's look a little bit different than what everybody else is doing.

• 55:19Sarah Guo?

  Noam, you are gonna go work on this really interesting area. I'm sure there are other problems you think are interesting, especially given the practical limits of how much money we're willing to spend on scaling up beyond another magnitude or two. What do you think other researchers or teams should be working on that they're not paying enough attention to?

• 55:39Speaker 00

  Well, I think we're in an interesting place now in AI where, given where things are at now, there's already an opportunity to build out products that can have a big impact on the world. And so it's great to see that there are people that are going in that direction and trying to like bring these, this research into the real world and have a big impact there, make people's lives better.

• 55:58Sarah Guo?

  For what it's worth, both ALOD and I got emails from multiple people telling us that they're building price negotiation agents as they, as we speak.

• 56:06Speaker 00

  Well, that's, like I said, I think it's doable. So I think, I think it's the right call. Yeah, I, I think on the research side, there's, there's still a lot of interesting questions about like, how do we make these things more efficient? Are there better architectures we can use? I mean, I think there's just so many questions across the board that are interesting. I think the, the big thing I would, I would recommend to researchers is not about like which area to focus on, but just like the style of research. I think there's a tendency to play it safe and not take big risks. And I think it's important to recognize that research is an inherently risky field. You know, there's a high probability that what you're working on is not going to be useful in the long term. And you have to kind of accept that and be willing to take that risk anyway. I mean, this, this

• 56:49Speaker 00

  happened to me, like by the early research in my PhD, in the grand scheme of things, really wasn't that useful. Like it didn't make as much impact in the long term as I, as I would have hoped. And, and that's okay, because, you know, I had one thing that ended up being quite impactful. And so I, I think it's important to, to be able to take those risks, kind of like go into the field, recognizing that you are taking your risk already by going into research.

• 57:12Sarah Guo?

  You heard it here first. Be like Noam. Work on things that make you nervous. I think that's all we have time for. Thank you so much for joining us on the podcast, Noam.

• 57:20Speaker 00

  Yeah, thank you very much for having me.

• 57:23Sarah Guo?

  Thank you for listening to this week's episode of No Priors.

• 57:27Speaker 02

  Follow No Priors for new guests each week and let us know online what you think and who in AI you want to hear from.

• 57:32Sarah Guo?

  You can keep in touch with me and Conviction by following at Sarah Normas.

• 57:36Speaker 02

  You can follow me on Twitter at @AladGill. Thanks for listening.

• 57:41Sarah Guo?

  No Priors is produced in partnership with Podpeople. Special thanks to our team: Synthel Galdea and Pranav Ruddy, and the production team at Podpeople: Alex McManus, Matt Saab, Amy Machado, Ashton Carter, Danielle Roth, Carter Wogan, and Billy Libby. Also our parents, our children, the Academy, and Open Google Soft AI, the future employer of all of mankind.