⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins

Watch the full show on YouTube: https://www.youtube.com/watch?v=UySWl-3LfKg

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Alessio Fanelli?

  Hey everyone, welcome back to a Late in Space Lightning Pod. This is Alessio, partner and CTO at Decibel. And swyx is running late, so we'll just get started and maybe you'll join. Today, I'm joined by Jack and Mart, who are the two researchers behind the Factorio learning environment. Since we had multiple guests on the podcast before bring up Factorio as a both as a hobby and a way to think about building startups. We thought we would do this episode today to figure out which models are best at Factorio and some learnings in heavy model learn through games. We talked about the Voyager paper with Minecraft in the past, so welcome to the show.

• 0:37Alessio Fanelli?

  Hello. Hello. Excited to be here.

• 0:40Alessio Fanelli?

  Were you guys Factorio players before, or did you just pick Factorio post fact as the best environment for this?

• 0:46Speaker 01

  Although I played Factorio for years and about 18 months ago, I was looking into this idea of the paperclip maximizer. And that kind of cautionary tale where if you set a model to maximize paperclips and optimize a factory, what kind of bad stuff do they end up doing? In furtherance of that goal, and it got kind of got me thinking about the applicability of Factorio as an environment for benchmarking models. One of the kind of motivations for it is that when you compare Factorio against Minecraft, for instance, the complexity scale is drastically different. In order to complete Factorio, you know, complete and launch a rocket, you need to mine maybe about a million more resources to get to that point. Whereas the same kind of point in Minecraft, you have to just get maybe 200 resources.

• 1:28Speaker 01

  So you know, we have orders of magnitude difference in in Factorio compared to some of these other environments. And if you want to go beyond just launching a factory and you know, see some of the huge factories that real players have managed to make. I think the biggest one at the moment is generating something like 19 million resources a second. So we have factories that you know, stem from, you know, making maybe one resource every couple of seconds, all the way to 18, 20 million a second. And that range gives an enormous amount of signal for comparing models, uh, you know, from tiny models that aren't very good all the way to superhuman models. And enable us to differentiate them kind of at any step along the way. And that, and that I guess was the core idea.

• 2:09Speaker 01

  And you know, I kind of set out on a Friday night about 18 months ago, uh, you know, as a, as a casual weekend project, you know, in and out operation, get it done. And 18 months later, we kind of got it into a position to actually release. Mark had joined working on it about nine months ago or so, and we really started focusing on this about three months ago. And I'm just trying to clean it up and get it, get it into the public, basically.

• 2:33Alessio Fanelli?

  I was, I was on the complete opposite side. I had never played Factorio before, before joining. But I was kind of sold by Jack because of all the reasons that he just brought out. And then I remember I had to do the full tutorial before even starting going, actually building this thing. Um, and already during the tutorial, I could kind of see the complexity and the, the opportunities for this. And it was then really cool to kind of see as we build this thing, where, where these models get to and, and where do they, um, where did they kind of fall short? Yep. Um, it would be great if you start sharing your screen, we can kind of run through your FLE blog post, paper slash, uh, whatever you, whatever the right nomenclature is nowadays.

• 3:17Alessio Fanelli?

  So yeah, at a high level, I guess, yeah, for people that have never played Factorio, they're watching this on YouTube. You can kind of see what the environment looks like. It's kind of like a cyberpunk industrial game. I don't know how else you guys would, would describe it. Let, let's start from the game harness. Was it easy to plug a model into the, the game? Like, does it already have like an API? I know Minecraft has very good support for kind of like shell commands and all of that.

• 3:45Speaker 01

  So it does to an extent. Exactly the same as Minecraft. Factorio exposes a low level Lua API to interact with the game. The, the way you mod Factorio traditionally is you, you know, you write some Lua scripts. You hook that into the game and then when events happen, your script runs. But this doesn't really work for doing what we were interested in, which was kind of running this at scale and getting, you know, a cluster of servers running. So we had to develop a, an entirely different way to mod the game by exploiting.

• 4:23Speaker 01

  The admin console of multiplayer factorial servers. So you know, you want to run a factorial server with your friends, one of use the admin. And as an admin, you have a little console where you can ban players, you can, you can change the permissions of different players and just basically modify the state. And what we realized is we could hook into this admin console over the network using a protocol called Arcon, you know, carried by TCP. And this basically enables us to execute actions in the game remotely and kind of shard them across dozens or hundreds of factorial servers, which is necessary if you want to do training or any of the other workloads you might need at scale.

• 5:07Speaker 01

  So going back to your question, you know, what's the harness like? It was very easy to get a low level harness working, you know, tell the character, move one step to the left, move one step to the right, that kind of thing. But that type of approach is probably a lot, much better suited to training models from scratch. And I guess a Markov decision process architecture where, you know, you give it a bunch of these low level actions and you train a network just to, you know, maximize the order in which these low level actions are executed in order to get an outcome. But such an approach would prevent you using any of the fine, of the pre-training that large language models have got. So that forced us to think a bit more carefully about what interface we could expose to these models.

• 5:51Speaker 01

  And we ultimately settled on a code synthesis approach where these models would write Python code. And those Python code, in that Python code would invoke certain high level actions, such as move to this distant point or place an entity next to another entity. And you'd leave it up to the model to think about how to abstract these tooling, abstract these actions into a structure that would give it.

• 6:18Speaker 01

  An ability to manage factories of arbitrary size. You want the model to be able to come up with classes, with functions, with data structures, such that when it has a factory that might be producing millions of resources a second, it might be sprawled over a huge amount of the game space. It needs to be able to reason over these different parts of the factory efficiently. So we decided Python, because the pre-trained models are pretty good at Python, and that was a reasonable choice. So yeah, we kind of set up this harness, the models write Python. They execute a Python in the game, which results in factory segments like you can see on screen kind of coming into being. And then the models get given a score, which is basically the production statistics, you know, what it's producing a second.

• 7:03Speaker 01

  And we also keep track of milestones, such as technologies the model was able to come up with, or new items that they were able to create for the first time. So we have these kind of two reward signals that complement each other. And the reason why this is necessary is to avoid certain, I guess, behavioral collapses, where a model might choose, for example, to mine coal, and mine a billion or a trillion coal, and that's all it does. And that would give a good score, like that would have a high production score, but it would probably indicate something has gone very wrong in the evaluation or training process. So we have a couple of different ways of measuring model performance.

• 7:44Alessio Fanelli?

  Yep. And then you have, can you talk about the lab play versus open play structure of the two tasks here?

• 7:51Speaker 02

  Yeah, I think one of the biggest questions for us was to figure out, what do we actually want to evaluate in this environment? Because lab play is simpler. Lab play is you give the model a task to create a factory of some entity, and then the model just needs to create a factory using our API.

• 8:21Speaker 02

  That gives you an easy way how to evaluate the upper bound on the, on the, on the performance because we have 24 target entities and

• 8:32Speaker 02

  each, each new entity is more difficult than the previous one. And then we quickly saw essentially the upper bound of like, how large factors can these models create? Because you need to, you know, for every input requirement, you need to create another subsection and then connect that section, all that. So LabPlay is simple in that sense and it very specifically measures the spatial reasoning of the models. Can they use the API? And how well can they work in constrained environments? Now, OpenPlay is the unbounded, create the biggest factory that you can think of type of, type of environment.

• 9:15Speaker 02

  LabPlay is, is in that sense, a subset of OpenPlay where OpenPlay, in OpenPlay now, the models have to actually create their own objectives as well. Because if you just give it, if you just give it the goal of create the biggest factory, it needs to make that large goal into sub-goals. You know, first, let's create drills, then let's create inserters, then let's put them down. Then let's create then a mine of this resource, that resource, all that. And um, so you have these two, let's say, abilities at play there of, one, is the model actually good at using the, the API and working in, in Factorio? And two,

• 10:00Speaker 02

  what objectives does the model set for itself? So for instance, one of the interesting things we saw was that DeepSick was quite decent at LabPlay. It was kinda on the level of GPT.

• 10:12Speaker 02

  But in open play, DeepSeek was terrible because what it often just decided to do was, let's just create more chests.

• 10:18Speaker 01

  Yes, let's let's create 250 chests and put them in a nice little line. And that is my factory.

• 10:23Speaker 02

  Exactly. So you can really see the myopic and short term objective setting at play in the open play because a lot of the times the weaker models, they didn't think four or five steps forward, they just created like fell down to just creating crafting entities, making putting down one drill versus actually creating an automatic setup.

• 10:49Speaker 01

  But integrating all the components.

• 10:50Speaker 02

  Yeah, exactly.

• 10:53Speaker 03

  I'm curious, what is the real world analogy to lab play and open play? Um, you know, I I think it's very significant that Claude is so much better than DeepSeek in open play. And is it just, are they just missing a post-training step? And what is the task that that this corresponds to?

• 11:14Speaker 01

  So I think that the the thing that

• 11:18Speaker 01

  the task is long-term planning, basically be able to reason over things, figure out an appropriate objective and maintain that objective for an extended context without kind of losing the plot and diverting into different areas. Whereas the lab play is much more about spatial reasoning and the kind of low level figuring out how to compose stuff that that has the appropriate space and precision around it such that it works. As you might know from Factorio, much like writing a program, if you put one thing in the wrong place, the whole system grinds to a halt. So that's like a slightly different skill set that we're evaluating than the long-term planning stuff. It's not obvious to us why Claude is so so much better than the other models in both settings. I mean,

• 12:02Speaker 01

  our suspicion is it is better trained for for this kind of long-term.

• 12:09Speaker 01

  Planning aspect.

• 12:12Speaker 01

  One thing we have found in preliminary results is that the reasoning models don't seem to do as well as you'd expect in this setting.

• 12:21Speaker 01

  And I think that's probably because the way we set this up, it's a bit like we're already making it do reasoning traces over an extended period of time. Asking the model to do reasoning over reasoning kind of eliminates a lot of the performance improvements you get from applying the reasoning models alone to problems like Mathfrontier, where there's a lot of benefit from just writing stuff down and figuring out what works.

• 12:44Speaker 03

  Sorry. Oh, go ahead, Sean. Sorry if I missed this, but in the original publication, there was, there were no reasoning models in there. Is this just results you haven't published?

• 12:52Speaker 01

  This is unpublished. This is our preliminary results, but we're gonna, our plan is to publish additional results with the reasoning models. And we've also added in a, an adapter to support VLMs. Now, a lot of the models we did evaluate do have vision. In our, in our published results, the models are interacting symbolically with the environment. It doesn't actually see the pixels on the screen. So one thing we want to bring out in our next work is how much, how much improvement can you get from allowing the models to take screenshots of part of the factories they want to debug?

• 13:29Speaker 03

  A quick, quicker response to that and then, and then Alessio. Typically, the results here, not having seen your results at all, typically everyone's very excited about adding vision and then they add vision and they find that it doesn't add anything.

• 13:43Speaker 01

  Yes, I mean, that, that was our, our hope as well. We were, no, no, truly, truly, because we, we had this great idea about adding vision right at the beginning. And then you, you see those experiments where they have two lines crossing each other in an X and you ask the model, you know, what is this showing? And it can't even say that the, the basic geometry of the picture, it can't, it can't intuit that two lines are, have an intersection. So to kind of get around this, we've scrapped in, in the vision component, we've scrapped using the tile maps and the graphics of Factorio and have instead fallen back to a very simple geometric renderer, which the models can kind of choose to configure when they invoke it as a tool.

• 14:24Speaker 01

  And the hope there is that we can stumble across, stumble upon the right vision mode that these models are reasonably good at. But I, I think your intuition is right. I would be surprised if we get that much from it at this stage. But it would be it would be cool if we do.

• 14:40Speaker 02

  We did. We did try it out. We did try out sending in the screenshots at some point. And the the problem is because you have so much going on in larger factories that the monsters start to hallucinate things. They don't correctly understand what is broken, why a factory doesn't work. It gets very detailed, very fast. And in our testings, when we when we made this decision, as you said, adding visitors didn't improve things. And it sometimes made it even worse because they started hallucinating entities that weren't there and started doing often went on tangents that weren't actually real in the game state.

• 15:25Speaker 02

  Right now, at least, the the vision actually wouldn't improve. Now, of course, as we go on, the models get better and better. It might improve in the future, but that's at least the current state what we found.

• 15:36Alessio Fanelli?

  Yeah. Um. Then if you guys can scroll down to the environment part, there's a couple thing that compared to the Voyager paper were interesting. So in Voyager, there was this idea of like self-discovery of actions where the model could like then save skills in a skill library and then reuse them later. It seems like here you have memory, but it's unclear if the model can save actions or if they can only use the tools that you specified. What was the design decision there?

• 16:05Speaker 01

  Sure. So apologies for not being clear in the in the publication, but the models can define classes, they can define functions and then execute them later.

• 16:16Speaker 01

  Only Google models tend to do this, which is quite interesting. Another finding we we discovered was that each of the models we evaluated have had fundamentally different coding styles. Claude, for instance, the Sonnet 3.5 was very much fire and forget. It would it would write codes in a in a kind of Pythonic way, just like let it fail, don't be careful about it. Whereas G P T four would use defensive programming, use self assertions, so basically asserting facts about its internal state.

• 16:47Speaker 01

  And what else was it doing? Is that fair?

• 16:50Speaker 02

  Yeah, I think G G P T four had that. And the same with Mini as well, if I remember correctly. They had a much more, more defensive check before we.

• 17:01Speaker 01

  Check before we do anything.

• 17:01Speaker 02

  Check before we do anything. But funnily enough, often those checks were also incorrectly set up. So it tripped itself up with its kind of overly defensive nature sometimes.

• 17:11Speaker 01

  Yes, so you know, we have, we have all these capabilities in the environment. Agents can define skills, they can incorporate the skills. This gives agents a lot of reach as the factories get bigger. You could imagine they could create, you know, quite sophisticated, like, for example, they could create hypothetically an N S A T solver in the kind of a namespace and then use that for a correct item and entity positioning in the game if they were able to do this. As I say, models are fairly rudimentary at this at the moment, but part of the motivation for this environment was to enable us to evaluate superhuman models and the evolving state of the art and not become irrelevant in six months

• 17:54Speaker 01

  as happens to a lot of existing benchmarks when they kind of max out.

• 17:58Speaker 02

  It is, it is also a bit, a bit different compared to Minecraft because in, in factorial, a skill is very because it's very dependent on the environment. Where do you have your resources? What, what is the current state of the factory? It's a lot more difficult to create generalizable skills. And that, that would be useful actually in multiple, multiple factory settings apart from the easy, you know, connect X to Y, which we, which we exposed as an API already. So we did also give it a go at the early stages, but we found that often the skills weren't that reusable,

• 18:42Speaker 02

  that you would hope at least because of exactly the reason that as soon as your environments change, if you make a skill over creating a factory, then that skill must be really general and like cover all the edge cases to actually correctly be usable later on.

• 18:59Speaker 03

  Maybe it's a little bit more like. You know, Minecraft, the rules don't change that much, or the the environment doesn't change that much. And with Factorio, you are literally changing environment every time you build.

• 19:13Speaker 01

  Yeah, yes. I mean, what you kind of hope to see, I guess.

• 19:19Speaker 01

  Who said, there was a kind of a principle, not the Peters principle, but it basically says that in an organization, the technology you develop tends to mimic the communication structures that you find in the organization. And I was wondering, I was kind of hoping to see a similar pattern emerging here, where the the abstractions and the code structures that the model invented would align quite closely with the topology of the factory it had created. Let's say it had one section which was just about mining, and then one section which was just about manufacturing. I was kind of hoping to see that it was able to bring that out and abstract that away to enable it to generalize better in the future. But we haven't yet seen this. I mean, this is something we hope to see when we start training. But, you know, that's an open question.

• 20:05Speaker 03

  Did you ever try adding web access, like, search?

• 20:09Speaker 01

  No, we haven't. I mean, we we kind of considered this. I mean, there's a couple of sources of data, I think, inspired by Mind Dojo, maybe Mind Dojo, where we we pulled in a lot of data about Factorio on one hand, and then we pulled in a lot of factory blueprints that people created on the other hand, and tried to use RAG, basically. Sorry?

• 20:34Speaker 03

  Yeah, yeah, yeah, that's right. That's right. There's a there's a there's a huge community of these pre-made things that that we know the properties of. Yeah.

• 20:41Speaker 01

  Yes. So so I mean, that we are, I guess, we haven't really, we still want to try this, but it became a bit of an adjunct to the core work, doing this extra stuff. One of the issues we found with blueprints is that First of all, they're specified declaratively, much like you get a terraform, you know, or a cloud formation, infrastructure as code kind of blueprint, which said where everything had to be. But it didn't provide the imperative steps necessary to get there. So on one hand, you know, we have our Python code that the agent writes, which are the steps, one after the other, to get to the factory. And on the other hand, we have blueprints to just say the final state. So we spent a fair amount of time trying to bridge this gap.

• 21:24Speaker 01

  And we came up with some, I guess, procedural factory generation techniques, which could take a declarative blueprint about what this factory needs to look like, and then

• 21:35Speaker 01

  deduce the steps necessary to get there. The downside of this is that it's very low level. You know, you're not getting the abstractions that you'd like to see. But on the plus side, we have a lot of that data. We have tens of thousands of these factories. So I'm thinking that this is the kind of thing that would be really useful just to train the model to do from scratch and just give it, you know, 10,000 examples of factory units that people have made, learn on it. But when you try and move that into a benchmark setting with already pre-trained models just using in-context learning, it's just not that helpful. A thousand lines of Python telling you how to make this kind of factory unit, which it may not be directly applicable to the agent's immediate objective. It didn't really give us any uptick in scores.

• 22:19Speaker 01

  So I think we kind of put that in the bucket of what we're going to do when we, when we start really throwing compute at it. So for example, when we did all these experiments, we kind of accumulated about a million programs that agents have written with scores attached. So we kind of got this big bucket of programs and traces, trajectories through the game. And we want to take these and just kind of throw it at a model and just see what happens if we really try and get it to learn how to maximize score directly, like a paperclip maximizer.

• 22:46Alessio Fanelli?

  Is there any mechanical advantage to being an AI in Factorio? I remember when DeepMind was building AlphaStar, the StarCraft thing. One of the big advantages that it had is that you can, again, like, you know, so precisely click on units and manage them versus a human kind of has some margin of error. If you were to put like, you know, the top Factorio players versus your FLE, is there like a big skill gap still? Or do you feel like at some point it's more about just the long-term planning is really what what comes into it?

• 23:20Speaker 01

  So, yeah, there is a big skill gap. As mentioned, because they execute Python programs, they can kind of come up with quite complicated policies, which a human player would have to kind of develop on the fly. So in that sense, the agent has an advantage. But when we kind of track the number of ticks it took to execute these policies against the number of ticks a human player would spend to get somewhere, the gulf between models playing this and humans playing it is absolutely enormous.

• 23:53Speaker 01

  So, for example, the the current record for speedrunning is about an hour and 20 minutes to get to a rocket launch. Whereas when you measure the ticks, it took models something like eight hours or so to get to the point where they have a kind of working factory that could make a few things, a few, let's say, iron gear wheels or electric circuits or maybe some science and maybe start progressing through the tree. So, you know, there's there's probably 100 times difference in the competency of these models untrained than a human expert. But, I mean, again, maybe a better experiment would be to take someone who'd never played the game before.

• 24:32Alessio Fanelli?

  Yes.

• 24:32Speaker 01

  Put them in the game and then compare it with the models. Because the models, you know, they have never been trained on this stuff. They only know a little bit about Factorio. They they have the manual of how to interact with it and they just they just have to go and and try their best. So maybe that's what we should do. I should try and recruit my fiancee and kind of make her sit down for for 10 hours in front of the game and see where we get.

• 24:52Alessio Fanelli?

  That was me nine months ago with the tutorial.

• 24:53Speaker 01

  That was, yeah, fair enough.

• 24:55Alessio Fanelli?

  Yeah.

• 24:55Speaker 01

  First time, my first time I.

• 24:57Speaker 02

  Yeah, to like, I don't think there is a mechanical advantage per se. I think, I think the largest improvement, yeah, as Jack said, is purely what type of a structure are they able to come up with? And if you do start to train against that objective, then probably we will, we might see some wacky structures which maybe humans haven't thought of in the future. But that's, that's the, the, the, the gap towards there, the road towards there is, is long.

• 25:29Alessio Fanelli?

  Awesome, guys. Well, this was great and there's a lot of Victoria fans. As we wrap up, a couple of things. Are there any games that you wish people would build similar experiences on just to see how the models perform? Maybe it can be, you know, different, different types of games that are not more builder driven?

• 25:48Speaker 01

  Um, I mean, personally, I would really like to see some agents made for playing Age of Empires 2, uh, because it's such a well-designed game and the community's kind of come back over recent years. But the AI of the models in the game is, is really terrible. And I think it'd be really fun to see how good we can, we can train these models to play that game.

• 26:09Speaker 02

  I was thinking about the exact same thing.

• 26:10Speaker 01

  Age of Empires 2?

• 26:11Speaker 02

  Yeah, yeah. I was also thinking Age of Empires along, um, which I think is more like a Starcrafty type of theory rather than, rather than Victoria, but that, that would be.

• 26:21Speaker 01

  Before we, before we wrap up, I just want to quickly scroll down just to kind of bring out some of the results, very briefly, just to kind of emphasize that we found that this is on a log, log graph. So we see that Claude right here has almost got twice the score of the nearest best model. Um, and then you have your Geminis and your GPT fours and all the way down here, you have GPT four mini, which was so bad upon occasion, it actually begged to be turned off. It, it gave up and said, I can't do this anymore. Please reset me. And then it would, and then it would refuse to do anything else for like a hundred steps.

• 26:58Speaker 02

  It's really smart.

• 26:59Speaker 01

  So maybe the kind of ambition or.

• 27:03Speaker 01

  Determination is some kind of latent factor in the competency of these models as expressed in the game.

• 27:07Alessio Fanelli?

  That's like when Doctor Strange just sees all the futures and it's like, I just, I just mean.

• 27:13Speaker 01

  One more chance, I'm out, I'm out guys.

• 27:16Speaker 03

  I think getting models to acknowledge that they cannot do something, actually, I think it's an interesting form of refusal. Or, um, or knowing the limits of their competence, because I think often we're incentivizing these models to say anything.

• 27:33Speaker 01

  Yes.

• 27:34Speaker 03

  And so, to me, that's almost a positive result, and he's just saying, we're not going anywhere. Like, give a positive reward for that.

• 27:42Speaker 01

  Thanks for trying.

• 27:44Speaker 03

  Yeah, I mean, just to give you the floor, please highlight, you know, anything else that you got a lot of responses on this when you, when you, when you launched it. Any other popular debates or misconceptions that you want to address?

• 27:58Speaker 02

  Options.

• 28:03Speaker 02

  I don't know if we have any that we didn't directly already touch upon.

• 28:10Speaker 01

  Well, I can say something then. So, um, our plan, this was basically phased, this, this work was phase one of a phase two plan. Our goal now is to start training these models directly on unbounded objectives. We think that there is a pretty strong application to this in terms of, to the field of model alignment. Specifically because in this instrumental convergence hypothesis, the paperclip maximizer hypothesis, where we think that certain behaviors are always going to emerge regardless of whether it's a human doing it or whether it's a machine. One really stands out, which is goal content integrity, specifically the reluctance for people or maybe models to change their goal once that goal has been set.

• 28:54Speaker 01

  So we really want to train these models, basically build a paperclip maximizer and see whether this behavior emerges. Because if it does, that I think might force us to change our attitude towards alignment, which is heavily based on this kind of incremental approach. But if we get to a point where the models are very clever and they don't want to have their objectives changed, it kind of means we only have one really good shot to set their objectives ahead of time. Because otherwise it's going to become quite hard to do it down the road. So this is a kind of an open project.

• 29:24Speaker 01

  Appreciate support from anyone who is interested in this work or has had experience in either alignment or in training these models. My email address is in the link next to my name. So, you know, we're currently based on the GPU mode Discord in a kind of private channel, but, but send me an email. We'll get you involved. And if you want to help out, we've got, you know, literally dozens of, dozens of things that we think would be really exciting directions to take this work.

• 29:53Speaker 02

  Cool. Yeah. Thank you guys for joining.

• 29:55Speaker 01

  Thank you very much. Thank you for having us. Have a nice day.