⚡️How Claude 3.7 Plays Pokémon

Special lightning pod with David Hershey from Anthropic, the person behind Claude Plays Pokémon. Sonnet 3.7 is currently trying to complete Pokémon Red live on Twitch thanks to a special harness that David built so that it can see the screen, navigate through it, remember facts about the game, and more. (Since recording, it has successfully escaped Mt Moon! You can follow along on Twitch: https://www.twitch.tv/claudeplayspokemon) This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Alessio Fanelli?

  Hey everyone, welcome back to another Late in Space Lightning Pod. This is Alessio, partner and CTO at Decibel. There's no swigs today, we got a special co-host, Vibo, which if you're a part of the Late in Space community on Discord, you've definitely seen. Welcome Vibo as a co-host, first time.

• 0:19Speaker 01

  What's up guys?

• 0:21Alessio Fanelli?

  And then we had David Hershey from Anthropic today, who's the person behind Claude Plays Pokémon. It's funny, I saw we had first DM'd about playing Magic the Gathering together in a SF.

• 0:31Speaker 01

  I really...

• 0:31Alessio Fanelli?

  And then people are like...

• 0:32Speaker 01

  On all of the different nerd angles you can get me. I'm glad to...

• 0:36Alessio Fanelli?

  And then people were like, "David is the person doing this." And I was like, "Okay, I'll DM him." And then, yeah, it was cool, we already had a touch point. So, welcome to the show, this is our second Anthropic episode. We had Eric Scholz from the Sweet Agent before, so welcome.

• 0:53Speaker 01

  Thank you, glad to be here, excited to talk Pokémon.

• 0:55Alessio Fanelli?

  Yeah, so let's give a little background on this. So, Sonic 3.7 came out a couple of weeks ago? I don't know, time goes by so quickly.

• 1:03Speaker 01

  Monday.

• 1:03Alessio Fanelli?

  This week? I don't know, man, it feels like two weeks ago. And then you had this Claude Plays Pokémon thing that kind of went viral, where if people remember, there used to be this thing called Twitch Plays Pokémon, where people could go on Twitch and kind of type in the chat and then busy like figure out what the next action that the emulator would take us. What you've done instead is given it to Cloud and basically have Cloud figure out how to walk through it. I'm looking at it right now, so far it's been stuck in Mount Moon for 52 hours. Poor guy, it probably met a 15,000 zoo vets. So yeah, let's talk about what gave you the idea for it, kind of the origin story that we can go through the implementation.

• 1:41Speaker 01

  Totally. Yeah, so I actually started working on it in like June of last year for the first time. And for me, so I work with customers at Anthropic and I just like really wanted to have some way for myself to be able to like experiment with agents like in a real way. Some framework, some harness where I could actually just like go to town and try some different things and see what actually worked to get Cloud to do like pretty long-running tasks in general. And so I like had that in one hand and then I was like, "Okay, what is the thing that will make me the most..." Addicted to making this work like how will I grind the hardest actually trying this and Pokemon was like a pretty clear answer. Someone else that in topic could actually like tried once to hook it up. So I had a little bit of like the the shell

• 2:21Speaker 01

  of what I needed to to actually put it together and to like kick off what what became an obsession a little bit in the coming months. So yeah, like I played with it in June in this like trying things out. This was like Sonic 3.5 came out in June of last year was just when I started kicked it around. It's very good, but like, you know, you could see like kind of signs of life, but like not much really happened. Um, and then ever since then, as we've released new models, it's sort of been like the way that I get to know one of our new models a little bit, right? So we released the new version of Sonic 3.5 in October and like use this to like really kind of see like what's it better at? And it got better, like you could see it start to like, it could get out of the house somewhat reliably, which was not always true. And it got a starter and I like even named it sometimes like it was like doing stuff, not great, but like it it could move.

• 3:05Speaker 01

  Um, along the way to like, I'm just like, we have a quad place Pokemon Slack channel. Like I'm sort of just like giving people updates. So over time, as I'm like posting GIFs and upparies updates, like I'm this is like slightly growing a popularity of a call following internally of people who are somewhat interested. But then like, uh, you know, a couple of weeks ago, I was bashing an early version of Sonic 3.7. And it just like, you could just tell it had like, it was a little different. It's clearly not still good. As you said at the top, like it's it's in my moon for its 50 something hour. This is a little bit worse than average from what I've seen so far by now, but like this is like, you know, about on brand. It doesn't really have a great sense of direction. It's pretty bad at seeing the screen, stuff like that.

• 3:46Speaker 01

  But like, it plays the game, you know, like it gets Pokemon, it catches Pokemon, like it caught its first Pokemon, it got out of variety of the first time, like a whole bunch of stuff happened for a first timer, like could squint and see a thing play in the game. And yeah, like posting updates, obviously internally, it was very fun. Like people were just like kind of going wild at the fact that this was actually happening finally. Um, and it was like entertaining enough that I could kind of see it. And the other side is like, we kind of just like got finally a sense that this was like an actually useful way to measure what was going on with this model. You know what I mean? Like, there's one thing that it's like fun and fun follow along, but like internally, like I think we got more of a sense that like, you could actually use this as a bit of a measuring stick for what's going on in the model.

• 4:30Speaker 01

  I've spent, you don't want to know how many hours I've spent staring at Quad Play Pokemon. I've so much, I have to have seen and read like millions of words that Claude has generated in the course of playing Pokemon over the last eight months. So like you can kind of get a feel for like what's actually going better or what's it getting better at and that kind of thing. And with this particular release, like I think the fact that it got this much better at this kind of reflects a lot of things that we wanted to be true about the model to begin with. And, uh, and those sort of lined up, we're like, okay, maybe this is like an interesting way to actually tell people about what's going on here for a crowd that maybe doesn't like quite know as much about software engineering and all the other ways we've told people about agents in the past.

• 5:05Alessio Fanelli?

  Yeah. Uh, were there any other games that you consider? To me, it seems like Pokemon is good because it's like, you know, isometric, you know, it's kind of like flat, so you get any scored and it's, it doesn't have too many hidden facts about objects, you know, kind of like everything is described. Did you consider anything else or was Pokemon just kind of like by far and away the first choice?

• 5:26Speaker 01

  I didn't, but it's mainly because like Pokemon was the first game I ever got as a kid, right? And so this is like purely coming out of my own nostalgia. Um, but also like the Twitch Play Pokemon, like I, I was also something that I cared a lot about a decade ago or whatever that was, um.

• 5:40Alessio Fanelli?

  At least tell me it's not a decade ago.

• 5:41Speaker 01

  I think it's actually a decade ago. I'm sorry. Yeah. Painfully, um, and.

• 5:47Alessio Fanelli?

  Eleven years ago.

• 5:49Speaker 01

  Yeah.

• 5:49Alessio Fanelli?

  That's insane. February 2014?

• 5:52Speaker 01

  Yeah.

• 5:52Alessio Fanelli?

  That is nuts.

• 5:54Speaker 02

  Pokemon Red is 20 years ago.

• 5:56Alessio Fanelli?

  Oh my God.

• 5:58Speaker 01

  20, 25 at least.

• 6:01Speaker 01

  So yeah, for me, it was that like since then, there have been a lot of people in the room like, oh, we could do this. We could do this. We could do this. I think there's like a lot of fun things you can do. Pokemon is actually really nice because like, if you don't do anything for five seconds, like there's typically not a consequence by the nature of like doing inference on a model every, like snapshot of time. It's actually a pretty good game to be able to do this with, but, uh, but yeah, it was mostly just like, uh, my, my love for Pokemon coming through here.

• 6:28Alessio Fanelli?

  Um, you put together a very nice architecture diagram. Do you want to screenshot that so people on YouTube can follow along and then we'll put in the show notes?

• 6:35Speaker 01

  Yeah.

• 6:36Alessio Fanelli?

  If you are just listening, um,

• 6:38Speaker 01

  You got it.

• 6:38Alessio Fanelli?

  I know that Vibo had a bunch of questions on the, on that too.

• 6:42Speaker 01

  Yeah, let's do it.

• 6:43Speaker 02

  Very, very straightforward question. Basically, can we just double click into all of it?

• 6:47Speaker 01

  Yeah, yeah, yeah. Easy.

• 6:48Speaker 02

  I found it off Twitch and like no one was talking about it. So I started sharing it around and I, I lost the original source, but basically everything in here is like pure gold. The memory is a little interesting, but yeah, if you want to just go through high level.

• 7:01Speaker 01

  Yeah, you got it. Yeah, I, I want to like preface that I do not claim this is like the world's most incredible agent harness. In fact, like, uh, I explicitly have like tried not to like hyper engineer this to be like the best chance that exists to be Pokemon. I think it'd be like trivial to build a better Peter program to be Pokemon with Quad in the loop. Uh, this is like meant to be some combination of like, understand what Quad's good at and benchmark like and understand Quad alongside a simple agent harness. So what that boils down to is this is like a pretty straightforward tool using agent from my perspective is how I would frame it. So at the end of the day, like the core loop is just like having a conversation that rolls out.

• 7:46Speaker 01

  And it's essentially like, you build the prompt, including like everything we've had up till now. You call the model, it sends back some tool use typically, you resolve those tools.

• 7:58Speaker 01

  And then talk about summarization, but like basically some, a few different mechanisms to maintain the information you need to do something long running inside the context window.

• 8:10Speaker 01

  So like what this boils down to is like when you think about what an actual prompt looks like. It rolls out kind of like this. You've got tool definitions, which describe three tools that I'll get to in a second. A short system prompt. It's like pretty boring. It basically tells the model how to use the tools.

• 8:27Speaker 01

  And like, there are about six facts about Pokemon that I give it. And like a few corrective things that I've seen it do like really horribly wrong. And I'm like, hey, you might want to consider doing this a little bit better. But it's like, really not a lot of system prompting going on. We have that knowledge base which referred to you. I'll talk about. This is the main way it stores like long-term concepts and memories as it's operating over time.

• 8:50Speaker 01

  And then the bulk of things is this conversation history, which is, it's like a chain of tool use. There's no like user interjections at all for the most part. So it's like, go, and then the model uses the tool, and then it gets a result back, and then it uses another tool, it gets a result back. So pretty straightforward.

• 9:08Speaker 01

  Feel free to like cut me off too, if you've got questions along the way, but otherwise I'm gonna keep rocking.

• 9:12Alessio Fanelli?

  Yeah, yeah, go ahead.

• 9:13Speaker 01

  Cool. Okay, so most of the money of this is just like in the tools themselves. When you think about what's going on,

• 9:22Speaker 01

  it's really like, it can press buttons and it can like mess with its knowledge base, and that's about it.

• 9:30Speaker 01

  I'll talk about navigator separately because that's like a patch for how it actually can deal with some of its vision deficiencies. Using the emulator is basically like execute a sequence of button presses. It'll say like press A, B, left, right, whatever.

• 9:45Speaker 01

  It gets back a screenshot and screenshot overlaid with coordinates of the game. These coordinates are used for this navigator tool that I'll describe in a second, but it's just basically like, help Quad get a slightly better spatial sense of what's going on on a Game Boy screen. I've been through it a lot.

• 10:01Alessio Fanelli?

  With the, sorry, does it come with the emulator or are you adding this in?

• 10:05Speaker 01

  I had that in. I have somewhat extensively reverse engineered Pokemon Red by this point to like extract roughly every bit of possible information from it. I don't use most of it, but like I have essentially everything you could know about the current state of the game. I have exposed programmatically to be able to tinker with it at this point.

• 10:23Alessio Fanelli?

  I was just reading this diagram like, yep, you just get what spaces are walkable based on what's stored in RAM. And I'm like, oh, you definitely reverse engineered this little.

• 10:31Speaker 01

  Yeah. Good news is we also released Claude code this week, if you saw that. And that has been, this would all not be possible without the help of having Quad also go figure out how to do all of this for me. Cause I could have done it, but there's a lot of like tedious. Here are addresses in memory, map that to a Python program that I had no interest in doing. So thank goodness for Claude code.

• 10:55Speaker 01

  So yeah, it gets the two screenshots. It gets like a small blurb of state, which I read straight from the game.

• 11:03Speaker 01

  There's a lot of this here. Actually, like funny enough, the thing that matters is location.

• 11:09Speaker 01

  Quad will like pretty aggressively hallucinate that it succeeded in transitioning between zones. If you don't like tell it, it did not. Uh, this just comes down to like literal vision issues. And, and so like most of the patching of extra help I've given it been like attempts to make it so that I could still play despite not being very good. Seeing Game Boy screens in particular, um, and then it gets like a handful of like reminders. This is, this reminder system, a decent amount of work, but it's like. Things like, yeah, remember to use your knowledge base occasionally and. We tell, we tell if it gets like stuck, for example. So if you detect that it like hasn't moved in 30 spots or 30 time steps. I once saw it see like a red box on the screen that was like the doormat and think it was a text box.

• 11:53Speaker 01

  And spend 12 hours pressing A overnight to try to clear the text box, which. You see that happen once and you add in some, some helpful reminders to not do that.

• 12:04Alessio Fanelli?

  How much knowledge does the model have about the game itself, you know? So for example, types, right? Doesn't know about types, weaknesses and things like that, or how much are you trying to put into it?

• 12:16Speaker 01

  Yeah. If you go to quad.ai, like it, it will tell you about like some stuff. I have not yet decided if. The knowledge that it has about Pokemon is helpful or harmful towards it playing the game. Um, like half of the time when it's like, oh, I know this about Pokemon, it then like uses that to hallucinate something. So for example, beginning of the run on Twitch, you saw it like go out of the lab. And see like this NPC in the bottom of Palat Town and be like, it's Professor Oak. I found him. And it's like. Very much not Professor Oak, but like the fact that it has like indexed on this concept is like a little. It's stuff like that, that it's like unclear to me where it is, but it clearly has some information about it.

• 12:56Speaker 01

  Uh, there's like a million game guides about Pokemon sitting on the internet. It's unsurprising that like, there's a decent amount of information there. I don't really give it a lot of extra information. It, it picks things up. I watched on the stream the other day, like it tried to use Thunder Shock on a Geo dude and it failed. It's like, hmm. I forgot about that. That does not work. And so like clearly there's like, it knows some stuff. It's not perfect. It picks some stuff up as it goes through the run. Ideally for me, like, I think it's just interesting to see like what it actually learns as it's playing. So the more it does that is the more I'm like actually interested in it.

• 13:30Alessio Fanelli?

  Yeah. The, one of our Discord members, not Jung, he had a good question about the sense of self.

• 13:36Speaker 01

  Yeah.

• 13:36Alessio Fanelli?

  Like sometimes it gets confused. Who is the actual playable character in the, in the scene? Like, how, how do you steer that?

• 13:44Speaker 01

  Uh, yeah, I, I think like sometimes it gets confused. Uh, it can be applied to many things in quad playing Pokemon. In particular, when it's trying to like, look at the screen and understand what's going on. So.

• 13:57Speaker 01

  I have like attempted to prompt it all sorts of ways. Like you are at this exact coordinate and you're in the middle of the screen and you're wearing a red hat and things like that. And like, that's all neat, but Quad doesn't particularly understand like the middle of a Game Boy screen and a whole bunch of concepts like that, which means like, you can prompt. All around everywhere, but like this kind of like spatial awareness and where something is with respect to something else is something that Quad's still just like not. Great at in its current incarnation. So one of the side, it's just that it sometimes loses track of who it is on the screen and thinks there's something else there.

• 14:29Speaker 01

  I will keep tracking through this. So. I hinted at this like other tool that I give it called Navigator. And this is just like the only other patch that I have for the, the vision issue. So Navigator, basically what it does is like Quad can say it wants to go to one of these coordinates. Uh, that we provide in the screenshot. And then we like automatically press the buttons to get there. Uh, it has to be something on the screen. Like I'm not trying to let Claude just like navigate a whole map by asking too politely. But one thing you'll notice if you run it without this tool is if like Quad wants to get from one side of a wall to another side of the wall, it like happily just tries to walk through the wall repeatedly because it doesn't quite have the concept of like what's between it.

• 15:14Speaker 01

  And I've spent a lot of time like prompting around this and it just like isn't, it's just not, it's one of those things I'm not very good at. So in order to make it somewhat fun to learn from Quad playing Pokemon at all, we use this navigator tool, which like helps it actually get around a little bit better.

• 15:28Speaker 02

  So since we covered a bit about the different tools, the prompting and the strategies, I'm curious how many tokens all this is using. Like there's a part to conversation history and truncating parts of the messages in state. But like, yeah, at a high level, how many tokens is this using? And then can we kind of go into where those are coming from, what's being truncated?

• 15:47Speaker 01

  Yeah, you got it.

• 15:49Speaker 01

  Uh, when you like, think about the, the prompts here, potentially like every step, something that looks like this gets sent. So if we just go through what each of these looks like, everything in the system prompt is probably like a thousand tokens, pretty small, like handful of paragraphs. Knowledge base, I let get up to like 8,000 tokens. Right? So I put some like arbitrary cap on it. So it doesn't go to like Quad will write, put a whole bunch of BS in there if you just let it keep writing stuff. So like the cap helps constrain it to like, try to think about what's actually important a little bit. And then the conversation history, I have like kind of finicky, but it basically rolls out

• 16:32Speaker 01

  um, 30 messages. That's actually like something you can tune. I've tuned it to be 30 messages about like the best performance I've gotten. And so what that means is it basically like use the tool, get a response back, use the tool, get a response back. It's allowed to do that 30 times. And then at that point, it triggers the summary, which takes that conversation history, summarizes it, makes it the first user message, and then we kind of roll back out again.

• 16:58Speaker 01

  So the bulk of the tokens end up being in the conversation history. Once it's as longest. In fact, like this, the bulk past that ends up being these screenshots, which are scaled up a decent amount to fit in. I do actually like, I allowed to see a number of the previous screenshots, but not all of them, because you start like, it ends up being a ton of context. If you'd let it see you like even 30 turns worth of screenshots. So I'd trim out a few.

• 17:24Speaker 01

  That's where the bulk of the actual tokens are. So in practice, this rollout ends up like at max ending up around 100,000 tokens. I think is where it is like the longest message you ever send to the API on one of these turns. And it will, it will fluctuate in like summarization and depending on state of knowledge based probably between like 5,000 and 100,000 tokens.

• 17:44Speaker 02

  And is that like per action state of the game? And roughly, do you have like a high level ballpark estimate of how long this, how much and how long it costs to run this? Like, let's say people want to compete and yeah, yeah, like how much?

• 17:58Speaker 01

  I think you'd really want to think about running this as a side project in terms of the impact on your personal wallet and how much you care about Pokemon. It's not clear to me that without the blessing of anthropic, I would have decided to take on, take on this project for my own wallet sake, especially if you want to like experiment and like try 10 different things. I mean, it's, it's costly. I don't know. Like I haven't spent a lot of time on the exact number. It's not that hard to estimate. If you like, I just told you a bunch of numbers, you can kind of back it out. But like, I think to like, do a lot of experimentation, there's like at least thousands of dollars of tokens being consumed. So it's not a, it is not a, a, a cheap rollout. Yeah. But yeah.

• 18:44Speaker 01

  In the scheme also how some people use tokens, it's not terrible.

• 18:48Alessio Fanelli?

  How many turns are you keeping in memory before you summarize?

• 18:52Speaker 01

  It's thirty right now. Yeah, I've tried more and less. I think like one thing you see a lot when you talk to people building agents is there's like some effective context length that actually like how's the model be the smartest? Um, and that seems to vary slightly model by model, but but for this model, for whatever purpose, like this thirty message worked better than twenty and better than forty. So uh, kind of plot in between those that it worked pretty reasonably.

• 19:20Alessio Fanelli?

  Yeah. Does that change based on location? Like, how many would you want to give it to get it out of Monmoon? So I say, hey, we got it. We got to bring it and plot home. We can't let him stay for another fifty-seven hours.

• 19:32Speaker 01

  I actually am not sure it does. Like I I've tried posting, like I can have a ton of screenshots, like twenty or thirty screenshots at a time be able to see. And it's like not obvious that like that temporal concept is actually super relevant relevant to it.

• 19:49Speaker 01

  And again, this is just like,

• 19:55Speaker 01

  trust me as someone who has spent like a lot of hours obsessing over this, uh, you can try to prompt Quad a lot of different ways to understand how to navigate better. And anything short of telling it exactly what to do does not improve. It's like actual navigation. It's just like not a skill it's great at. It's like good enough to to like random walk its way through some of the complex mazes and in like good, easy areas, it's pretty good at bopping around. But yeah, I think I can tell you if there was like a way to prompt this slightly different that uh, would navigate better. And I would believe there is something, but it is not like, it is not an easy lift.

• 20:29Alessio Fanelli?

  Yeah. Yeah. I asked the, I just asked Quad AI right now, how do you get through a Monmoon in Pokemon Red? It does have, it does have a plan, but I don't, I don't, I don't know. I don't know if it's the right, I don't know if it's the right plan.

• 20:41Speaker 01

  I have seen it come up with a lot of answers to that question, and most of them are right. Uh, this is part of the pain. When I talk about, I'm not sure if its knowledge is better or worse, like you usually fix it like, oh, I know the exit is on the eastern wall and it just like spent twelve hours trying that.

• 20:58Speaker 01

  Um, AI, yeah, it's like unclear to me that that we're actually not just like harming it by having it think it knows the answer. Yeah.

• 21:05Speaker 02

  Yeah, I think that's the interesting part, right? Like, you don't want it to just know the answer.

• 21:09Speaker 01

  Yeah.

• 21:09Speaker 02

  Well, model clearly knows a lot about the game. There's like EV IV maxing. Pokemon was very, very extreme. But like, if that's what you wanted, we could just hook it up to a knowledge base, like hook it up to a guide if you know how to be Pokemon, right? But the interesting piece here is actually like, can it figure out what to do without just memorizing the path through?

• 21:29Speaker 01

  That's exactly right. Like, that's part of why, uh, you know, I don't know, part of what I've realized putting this out in the world is people will draw their line of where purity is anywhere on the spectrum. Like, is it, is this cheating? Like, yeah, maybe, um, who knows? Um, like frankly, like, I don't particularly care. The, the main insight that I have is like when we put this out, like, you learn a lot about what the model's good and bad at by staring at it. And that's kind of what I like about it. So.

• 21:58Speaker 02

  Evaluating the model is kind of separate than your emulator and how it can use an emulator, right? Like, we can always improve those things. I'm curious, um, as you switched from 3.5 to 3.7 and sort of reasoning models, were there any degradations there? Like, did it, did it kind of get worse at anything? And was the prompting somewhat consistent? Like, a lot of what we've seen with different reasoning models is like, you kind of prompt them differently, right? You tell them what to do, let them figure it out, but, um, yeah, any, any insights there?

• 22:27Speaker 01

  Yeah. Yeah, that's a good question.

• 22:31Speaker 01

  One thing that's nice about three words on it is we, it's like this hybrid reasoning model. So like, it kind of can do the old thing and the new thing. And it's actually pretty good at just like being an out of the box model and having this like thinking mode where it can spend time reasoning. So I, I didn't like really run into any like serious degradations. The one thing I'll say is like literally every model that has come out with Pokemon, like the, the main change that I have made to this agent is deleting prompt stuff. Like, there's a whole bunch of like Band-Aid-y prompt stuff I've added in the past. It's like, Trying to like steer it away from doing a lot of the things that it got horribly stuck doing in the past.

• 23:11Speaker 01

  And as the models get better, I found that just like making sure it's as simple as possible and giving them as much sort of like free rein to try to solve a problem as possible. Is useful and like the way I think about this is I'm like less confident over time that I understand exactly how a model is intelligent, right? Like it's capable of all of these like ridiculous things. It does PhD level stuff in some ways and like is unable to screen see a screen as well as a four year old in other ways. But like my confidence in like exactly what I need to tell it to do to be smart at playing Pokemon is actually like really small.

• 23:46Speaker 01

  If I tell it, this is the way you need to solve this problem, that might not actually be the best way for 3.7 to solve this problem. It's like just different than I am in terms of how it thinks about these things. I found that just like kind of like pulling some of the unnecessary instructions where I tried to like use my intuitions about what would make the model better. Out of the prompt over time is the thing that just like sort of consistently as models got smarter. Gotten more juice out of this.

• 24:13Alessio Fanelli?

  I was watching the stream yesterday or the day before and it was a very tense battle. I think they were like down to like 2 HP each and like the opposing Pokemon like missed. A scratch or something and it didn't die and like you could tell like I was like, wow. It was like very dramatic and I was talking about the game. How I yeah, is there any thought being put into like trying to have it more like, do you prompt it to be more rational to let it know that it's not a real life? That it's a game is like, it feels like it gets very distressed when they're actually, the Pokemon's are actually gonna die.

• 24:51Speaker 01

  It's funny, they, um, it knows it's Pokemon. It's like you're playing Pokemon Red, like it does know that and it has a sense of that, but it clearly grows some attachment. I'll tell you a fun story.

• 25:03Speaker 01

  We tell it to nickname its Pokemon now. It will occasionally do without it, but it's like more fun if it nicknames its Pokemon. So that's like in the prompt is like, it's fun if you nickname Pokemon, you should consider it.

• 25:13Speaker 01

  And one thing we found when we started doing that is it got more protective of the Pokemon it nicknamed. Like, it's pretty obvious, like when it catches a Pokemon. Now that it has a nickname, it will like go heal it right away if it's hurt. And that did not ever happen before.

• 25:28Speaker 01

  Which is pretty like, so there, there's some cute little things, cute quirks about Quad who really wants to protect its precious nicknamed Pokemon, which is great.

• 25:36Speaker 02

  So I will say it's kind of normal, like, like when I was five playing Pokemon Red and, you know, I had two HP in the midst of scratch, that meant everything.

• 25:44Speaker 01

  That was existential. I agree. I agree completely.

• 25:48Alessio Fanelli?

  How about skilled transitioning? So one question that I had. So you're playing Pokemon Red, right? So you want to play Silver or Gold next? Um, have, have you thought about how models can kind of learn from these games and like store these learnings and then use them again in the future? I'm sure it's not part of the project today, but curious your thoughts.

• 26:08Speaker 01

  I've thought about it only a little bit, which is like, I think there's some like, when you actually read one of the knowledge bases that it has gained, like on some of the longer rollouts, when they're good, like there's actually some like pretty decent tidbits about how it should act and try and do things and like some of the ways it's succeeded in. And actually, like, one of the things that's most unique about 3.7 Sonic that I've seen is like, it will have like meta commentary on what it's good at and bad at and its knowledge base. Like, I, I misperceived this thing. And so like, I need to be careful doing that again. You occasionally see show up there. Which is, um, which is pretty cool. So like, I could imagine there being some way to like, translate that knowledge base from one game to another.

• 26:49Speaker 01

  Uh, I think my knowledge base is frankly like kind of kluge of an implementation right now. Like it's like more or less a Python dictionary that's appended to the prompt.

• 26:59Speaker 01

  And I think like you could, you could find better ways if like your goal is to transfer across games and things like that to manage a knowledge base that Quad can actually use more or well in different scenarios.

• 27:12Speaker 01

  But there's definitely pieces there that like, I think it would get, be off on a better foot on the next Pokemon game if it had that, or even if like I were to restart the stream or like have some, some tidbits that it would probably like, uh, speed up if it like had access to things that I learned in the past, then it's interesting.

• 27:29Alessio Fanelli?

  Yeah, yeah, I, I always think of that in card games, you know, like you have the idea of like temple in a card game and it's like, you know, it's the same magic as it is and you know, Star Wars, flesh and blood, all these different things. I feel like games is similar, where like learnings you get from Pokemon, you can bring over to similar, kind of like open world games in a way.

• 27:50Speaker 01

  I think it's also like particularly interesting for some of the things that are like how Quad learns how to play a game in general, where it's like, pressing too many buttons at once is a bad idea. Like I lost track of what's going on, that kind of thing.

• 28:03Speaker 01

  Like definitely is stuff that it has learned that is like interesting in a meta way, uh, that it's like hard to give it that sense of self necessarily in training. I think sometimes like it's hard for it to know like what it's getting bad at in some scenarios, but it's interesting to think about how it can learn across things.

• 28:18Speaker 02

  Well, like, uh, some of this also is due to a simulator, right? So a lot of what's learning is how do I use a simulator? What am I good and bad at? But the model internally should know quite a bit about Pokemon, right? Like if you've played Pokemon, going from Pokemon Red to Emerald to Diamond, having played the first one doesn't help you that much in the second, right? You kind of get the general concept, you get what types are good against other types. And the model, model knows a good bit of this, right? But it's still interesting to show, this is more so like, it's, it shows that knowledge bases kind of help with understanding how to use the emulator, right? Like it struggled and then it figured it out. So you know, with Pokemon, it's like this thing can now learn how to use.

• 28:59Speaker 01

  Yeah, which is pretty cool. That has been like part of what's been fun, seeing Maul's made progress on this thing.

• 29:06Speaker 02

  I had a bit of a follow-up question to the last one with Lassio. So if people want to blow thousands of dollars and want to, you know, improve this a little bit, is there anything else that you'd want to see done? Whether that's like improve emulator, try different stuff? Is this just anything that like anyone watching this, you'd, you'd kind of hint them towards what you'd want to work on, what they'd want to work on?

• 29:27Speaker 01

  Yeah, no doubt.

• 29:30Speaker 01

  If I had to guess, like, the biggest lift that exists around this is probably

• 29:38Speaker 01

  something around the memory, which I don't think is like hyper optimized right now.

• 29:43Speaker 01

  The nice thing about the memory is like, it's always in the prompt, like, it doesn't go away. Like, sometimes if you leave it up to Quad to try to like read and load and save to memory bases, like, it will underutilize it or forget things.

• 29:55Speaker 01

  But I think there's probably something there. I will say, all of the many, many hours I've spent tweaking around the edges of this thing, nothing quite does it like a new model, though. Like, fundamentally, I think the limitations right now are like some smarts things like, I've seen, and I mean this in the kindest way, but I've seen a lot of people in Twitch tell me about ways that they can fix the navigation capabilities for the better prompt.

• 30:21Speaker 01

  The people would be welcome to try, but I would guess that would be like a somewhat fruitless avenue. I don't think, I think it's just not very good at understanding. The first time, I'll give you a very quick anecdote, which I think is like my favorite for like, why this is particularly hard. I have this clip of Quad leaving Oaks Lab and being like, great, I left Oaks Lab. Now I need to go up to the north end to go to Route 1. And it just like hits up on the D-pad and goes straight back into the lab. And it's like, shoot, I'm back in the lab, I need to leave. And it hits down, it's like, great, I'm out of the lab. Now I can go up to Route 1, it's straight up. It just like goes up and down 12 times. And it's like, you're not, you're not fixing that with a prompt, it just literally doesn't get it. It doesn't understand. And so,

• 31:05Speaker 01

  it's pretty hard to make like little around the edges changes that like make a huge, huge difference.

• 31:09Alessio Fanelli?

  Yeah, I mean, I've always been fascinated by the fact that Twitch Plays Pokemon actually beat the game.

• 31:14Speaker 01

  Yeah.

• 31:15Alessio Fanelli?

  From a, you just look at it and you're like, this cannot possibly work. Because you have people trying to sabotage it too in the chat. Not everybody's trying to solve it. What, what do you, so I just looked that up, it took 16 days and 7 hours for Twitch Plays Pokemon to be read. How, how close do you think we are to a model that can beat it in less than 16 days? And do you think it needs like some core, like model really big jumps? Or like, do you think it's like, we're close?

• 31:44Speaker 01

  I think, I think there is model stuff, at least from Quad. Like, I am confident there's model stuff that needs to happen for it to be like really capable. I have like four spots in the game stuck in my head. It's like, I think there's literally no hope it's gonna get through that. So I think there's like a gap that's mostly around like its ability to like see and navigate and remember visually like what's going on that I just don't think is like we've figured out yet. So to me, that's like a pretty big gap.

• 32:14Speaker 01

  I do expect like, I think it's gonna keep getting better. Like I have no reason to believe that this is not just like a fundamental like ability to scale, learn and understand problems thing. That I think is getting better as we train models to be more capable. It's sort of these like long horizon tasks like. I actually do think this is like a pretty reasonable proxy of that. And I think they will continue to get better for a little while. I don't know if there are like affordances around images and videos and stuff like that that we need to figure out to make it work. It's like unclear to me if that's true or not. Um, but yeah, I think we have a little ways before we can beat the game in 16 days. I do not have a lot of faith that the current stream is gonna gonna be standing in victory road in 13 days.

• 32:49Speaker 02

  What's been your favorite moment from like building this to thinking of the idea that just seeing it play any, any like major highlight?

• 32:57Speaker 01

  Uh, I think like the, the hypest I have been is when it beat Brock the first time where I was just like, you know, I've been doing this for eight months. And then like a few weeks ago, like I kick off a run, wake up the next morning and it's like, oh my God, oh my God. And I, and it was the, the other good thing about it is like, I woke up at 8 a.m. And I checked my, I, I have it send me updates to Slack. Um, this is like ridiculous things, but, um, it's like literally like about to start the Brock battle. Like I open my phone. It's like, oh, this is like happening right now. And it was like a pretty hype way to start a day. I think that was my, uh, highlight. I have a lot of like other cute things, like some of the cute nicknames that's done over time and things like that are, are endearing, but, but that was like the peak hype for me was like,

• 33:41Speaker 01

  we beat a gym leader, like we've got a badge, like Quad's doing it, you know?

• 33:45Speaker 02

  A bit of a follow up. So I noticed that you mentioned it eventually started beating multiple gym leaders. Were these all the same run? Was it different ones? Was it?

• 33:55Speaker 01

  Yeah, I have like, I have the, the run that you saw that's like on the graph we put out alongside like in our research blog is like a single run that I have watched like, get through.

• 34:07Speaker 01

  At least Serge's gym and then it got a little past that. And the reason that that's where we stopped reporting is because that's like the physical amount of time that occurred between when I started it and when we launched the model. That's like, uh, that was a very hyper, uh, hyper up to date graph on, on the best run we had. So.

• 34:23Alessio Fanelli?

  Awesome. Um, I know we're running out of time. My last question is, are we gonna work on magic? On Cloud Play's magic next? Or maybe we can do like the magic arena intro challenge?

• 34:34Speaker 01

  Uh, funny story. There was a project I did right before I joined Anthropic that was like training, uh, an open source model to like slightly be better at picking draft or cards in a draft. Like I was training it on like the 17 lands data that exists to like learn how to, how to pick cards out of a, on a packs a little bit better. Uh, and I, I did talk about that in my interview to get hired at Anthropic. So. So if I've put time into this, I'm ready. I am ready for that project too. That I have that code sitting around as well somewhere. I really get in it all my nerd, her nerd ML slash, uh, slash gaming hobbies here.

• 35:12Alessio Fanelli?

  Yeah, no, I'm ready. I don't know if you're planning on open sourcing any of the Pokemon stuff, but if you wanna work in open source on the magic stuff, I'll be happy to, to collaborate.

• 35:23Speaker 01

  Awesome. We've talked about it. I don't, I don't know yet what the plan is. I, there's like a certain amount of like, this is not my day job that I have to figure out how I want to, uh, Yeah. deal with that, uh, we'll see.

• 35:33Alessio Fanelli?

  Yeah. Um, awesome, David. Any parting thoughts? Anything people have missed?

• 35:39Speaker 01

  No, I, I think like the one thing I do like to drive home when I, I've been talking about this is like, I really do think like this is just demonstrating like a thing that is going to make agents better with this model, you know? Like, this is a very fun way to see it, but like, I think the thing is that it like has some ability to like course correct update and figure things out a little bit better than models have in the past. And even if there's like stuff it's dumb at, like it tends to have an ability to like power through it in a new way. And so I think what it's exciting to me is just like, I think there will be some real world stuff that comes out of this model once people play with it. And I'm pretty excited to see like, uh, how people take the skills we put on display a little bit here or, or lack thereof in some cases and, and figure out how to turn them into actual agents that do stuff.

• 36:21Speaker 02

  Have a quick last question on that, actually. Is there any guidance or any way that you like quantitatively measure the evals of this system? Like, a lot of it is vibes, a lot of it is how far it gets, where it gets stuck. But like, are there, are there any lessons or any specifics about how you measure how it actually does?

• 36:39Speaker 01

  So I've done a lot of like little, small tests of like, put it in this scenario and see what it does. But I like, frankly, the best test I have is just like, run it 10 times on this configuration and like, see how quickly it progresses through milestones of the game. I mean, it's the best thing about games, right? Like, it's why the games are such a useful thing. There's literal like benchmarks of gym badges that are moments of progress in a game, which are like ways to evaluate what happens. And so I think like how quickly it's able to make progress is actually a pretty recent, or a reasonable like eval, if a slightly expensive one to calculate. It's an integration test, not a unit test.

• 37:14Speaker 02

  Awesome, David. Thank you for joining. Thank you, Vibo, for filling in on the whole side, too.

• 37:20Speaker 01

  Yeah, my pleasure. Thank for having me, guys. I appreciate it.

• 37:21Speaker 02

  Awesome.

• 37:23Speaker 01

  Good to see you.