Latent Space: The AI Engineer Podcast
Original podcast
Watch the full show on YouTube: https://www.youtube.com/watch?v=E3ae_I_T-D4
7,307 words · 2 speakers
Okay, so welcome to a Lin Space Lightning Pod with Alex Duffy from Everi, who was one of our speakers, I think best speaker in his track, and launched AI Diplomacy. Welcome.
Thanks. Yeah, happy to be here. Thanks for putting on such a great event.
So yeah, I think this is like literally the Monday after you're back in New York, I'm back alive. And yeah, I mean, I think it was your first time at AI, presumably you like heard about it. I think basically I wanted to have somebody from Everi, I think you guys are doing fantastic work. One of the side jokes I always have is, you know, there's this new term high taste testers that has sprung up, and Everi is definitely a high taste tester. Like basically every lab comes to you to like test new models. Is that an intentional thing? How'd that come about?
Yeah, I think so. Dan puts, you know, I can't take a ton of credit for it. You know, Dan, Dan's been driving this thing for a while, and I think he does a really great job and has very high taste himself, and I think that that helps. And we also have a really great team and people are all building. We have a really interesting mix, you know, like Everi's got content. It started as a media company and it's been publishing writing for about five years. But you know, the team internally, like half of us are founders who have built their own businesses and half are engineers. And so we've got products that we built internally. You know, pretty much all AI products and I lead our training and consulting. So we kind of have these like cross pollination of people from different backgrounds,
from journalism to engineering to product and design. And everybody's just really into AI and having it be something that, you know, gives them leverage for what they do. And so have a lot of really, really great discussions. And I think Dan's and the rest of the team has done a really great job to foster those discussions.
Yeah, you guys also are building Cora, which is the email app. And I think it's pretty exciting. I basically am like, I want the alternative to superhuman. Superhuman's great, but I don't think they push a lot on the AI front. And I think that more options with good.
Any, any spiel on that?
I was just gonna say, Kiran, Kiran, Kiran, the GM for that, he's great. They've been, they've been iterating a ton there. I think that that product's great. We got Spiral, content transformation, Sparkle, cleaning things up. And we got something new coming too about.
Okay, I didn't know about the other products.
Yeah.
Wait, wait, tell me more about the other two, because Cora had gone more on Buzz maybe. But I don't know about the other two.
Sparkle helps clean up your desktop and, you know, and, and it does a really great job at that. Organize things. Spiral helps kind of transform content. So from long form to a different short form, but like really in your voice. And that's something that we focus a lot on is, is making sure that, that you really come through. And Danny, who's, who's the GM for that, has got a really, really cool version coming out soon. He started getting some people into the next version early. And if you're interested in that, you definitely bug them. And Naveen's actually releasing something called Monologue. You know, it's kind of like, I think a better whisper flow for, for the things that we do can, can run locally. And so, so more of that coming soon. But yeah, we've got quite a few.
Awesome, awesome. And you personally, your, your title's Hit of AI. And you, you said, you said you run training and consulting.
Basically like, who do you work with? If you can name anyone and what kind of people should get in touch with you for that? Just to get that out of the way.
Yeah, I mean, some of the things that have been public, we work with like the New York Times and the Athletic. So we, we're, we're kind of working with, with journalists in the media there. They cover that in the New York Times section that, that covered us recently. And we also work with like Walleye. They're, they're like a, a pretty big hedge fund. But we also, you know, it's kind of run the gamut. We work with some construction companies and like finance. It's really, it's really people who, who are pretty interested and, and I don't know. I'm big into education before this. I co-founded an AI education company. And so that's, it's really important to me and the, the org in general. I mean, we write about what comes next in part because we want people to come, come with, you know? And so, yeah, it's kind of people who are interested.
It's really, it's, you know, this may be a tangent, but like, you know, I, I kind of also run a creator-led business, right? Like, I have a Substack.
People find me through the Substack.
Totally.
Then they find the conference. A lot of people have told me to get into training consulting. I've dabbled, but I don't really know how to do it. So it's valuable when you can, because I think, obviously, I think all the big bucks are coming through there.
Yeah, I mean, I think that there's also, you know, I think what's really cool is that we're not a training and consulting company. You know what I mean? And it's not about like growing the biggest training and consulting firm like that Duke ever can. You know, it's, it's a tough, tough business. And I think that what's really cool is that we do have these different components. You know, I write for our contacts with no, you know, column when I can. I'm usually on Sundays and I also get to build products like, like AI diplomacy, which, which is kind of like being able to do these things. You know, I think helps make our training and consulting better. But, and then we get to work with people who, you mentioned high taste, but I, I think people that are just working on really interesting problems.
Yeah. Nice segue. We'll talk about AI diplomacy, then we'll talk about your, your broader talk that you did at AI. I'm going to share my screen. It's just a pretty visual. There's this whole trend now that you stream your LLM benchmark on Twitch.
First of all, was this off the shelf? Like, you just had a diplomacy implementation or did you have to write this too?
Well, okay, so we had, there's an open source implementation for the backend and like a very simple frontend. But, you know, my, my friend Tyler, you know, he, he's definitely, you know, the whole frontend part of this. Like, we, we very much collaborated on this. Tyler Marquez, great guy. We, he owned a lot of the frontend and I did a lot of the, you know, the whole harness for the LLM and the AI engineering on the backside. And so, no, we built this. I kind of spit out a, did not work, but gave the idea version and, and Tyler got it to something that actually runs, I think, pretty well, you know?
Would love to improve it, but you know, there's and there's some things that we want to build next for sure. But I don't know, I'm really happy with it. A big part of it was like, let's try and reach people. You talked about it in your talk, you know, like, like build things that are actually useful. There were some really good talks that that talked a little bit less of the technical and more like, you know, we need to build things that that hopefully aren't just for us as as AI engineers. I think that we need to get people outside of AI interested, um, you know, and and educated. And one of my like, life goals is to build a MMORPG that's more intentional with what it teaches you as you play. And so like, to me, like, if we make this playable, you know, then then it kind of can teach people
how to use a like, language models just by playing, because you'll like, understand how they work. You have to negotiate against them. You see their responses. And so it's kind of like, branched out into a different world, which which I think is awesome.
Yeah, yeah. Let's dive in, uh, deep on on the diplomacy side, and then we'll sort of branch out onto the broader philosophy of these of this whole thing.
So you've you found the rules. I only came across diplomacy because of No Brown. I assume you did too.
Is that? Yeah, there was a thread like at the beginning in February that Text Arena, who is also instrumental in kind of like getting it started, fostered an area where people could talk about it. Their bench, their their place where you can test different benchmarks for games. There they posted something, Carpathia was like, games are really great area to to explore. And, um, no one was like, oh, I'd love to see him play diplomacy. And I just happened to have a weekend on my hand, uh, hands that that weekend and found the open source project. And I put together a super basic harness. It barely worked, looks like it worked though, and and got to share it. They both thought it was cool. And then, you know, I love the internet. People from Singapore, Australia, MIT, Harvard reached out. They were like, hey, let's, we're interested in contributing.
And, um, you know, I think a lot of, I don't know how much, they contributed in their own way, you know, some some energy, you know, um, maybe. And and definitely some contributions, like especially like Sam Page from, um, from Australia. He had made a Diplo Bench. He had some really great insight on the prompting. And yeah, just just awesome that people were able to hop on. And that's that's where it started.
Diplo Bench, and this is my first, uh, um, uh, coverage.
He's got some great benchmarks. He's got like emotional.
I'm pulling it up. I'm pulling it up right now. So this is Diplo Bench, I think.
Yeah, if you've got, if he's got his site up. Does he have Sam Page on X? His site's EQbench.com. If you go to EQbench.com, he's got some really great benchmarks. He's got like emotional intelligence, creative writing.
This is the creative writing one. Yeah, yeah, I remember. I came across him. Yeah.
Yeah. And so he contributed to some insight into how to improve the harness and
some of the reasoning and was able to give some good ideas I was able to to actually get in there.
Yeah. So, you know, more broadly, I guess I'll set some context as in terms of like the history of games and L L Ms. Obviously, OpenAI has played Dota, DeepMind has played Go, and like people have been playing chess since forever. I think that the main idea is that you can only go against the best human for so long. Eventually, he starts to beat the best human, and then what? And so the answer, the more scalable thing is self-play, a quote-unquote self-play, where you only play other L L Ms. And there, there is no limit, because you would just be as good as the next L L M, and you can sort of improve from there. Any other thoughts on like just why L L Ms and games and benchmarks? I guess this might be your talk.
Yeah. And, and yeah, we mentioned Cicero, but yeah, just like I'm definitely not the first performer AI diplomacy. No, no made, you know, RL version of it. And, and I'd love to get it integrated. I know he, he reached out, was like, would love to see, see Cicero play against the other L L Ms. I would too. But, yeah, I mean, I, I think a lot of the examples that you gave, I think it's so cool that, you know, education being a big part, part of my life, that AlphaGo, right? Like the world, the top world player for like the human player, kind of like the elo was pretty capped. And then AlphaGo comes, and then they all learn from that, you know, like the same with like the Dota, you know, when, when the, when Dota, Dota 5, I think that was OpenAI's Dota 5, played against the best players in the world, and, and they beat them. Then they were all like, hey, can we keep playing against it?
Because they were learning new, new strats. And so I, I like, love that concept. Like, I really see AI as like a leverage instead of a product. And like, if it can help us learn more about ourselves and help us accomplish more of our own goals, I really think that like the role of a person and like a human in the AI world is to define the goal and to define what's good and bad on route to that goal. And what is that if not a benchmark?
Yeah, totally. I think the other thing that comes to mind when when doing something like this is, does the harness need to adapt for each L M, right? Like you have reasoning models in here mixed with, I guess, kind of not reasoning, although they're all sort of post the reasoning era.
Do you find significant differences between them?
I actually, I I've put a lot of effort into not making the harness too too different. And like, I guess I may not have answered exactly what your question right before on like the the games as benchmark. Like, I think that these are really cool benchmarks because they are evolutionary, right? Like, as models improve, the challenge gets gets deeper, like as you as you mentioned. And so I was really just interested in like how the different models played with the same prompts, really. And it took me a long time. I tested with two five flash because I I love two five flash. I mean, man, running games with two five flash was instant and one to five bucks of, you know, way, like, I don't know, twenty to a hundred with with the other models. And I figured, you know, if we can keep iterating until two five flash can do it, then then that's in a good spot.
And so we had to add quite a bunch. You know, you have like these relationships. Actually, you can see in the bottom left where that's a supply center, but it'll it'll flicker sometimes to to relationships. And the relationships are like who's allies, who's enemies? And that was really helpful because it was a context over time. And then there was like a diary. And so like, that was one of the interesting things is like O three was one of the few that will actually send a message to another power saying that they're planning to do something. And then like in their diary, diary, right? Oh, they fell for it, hook, line and sinker, totally gonna betray him and take it over. And I think that that's why I haven't seen Claude win any game yet because they won't do it.
Like, there's like, O three has managed to to get them on board for like draws, even though they all know the only win condition in the game is is eighteen supply centers. Opus loves when O three proposes a good good draw proposal. If only whoever's in first gets taken down a peg. And so, you know, Claude will will hop on that.
So Cloud's too nice?
Uh huh.
But look at Gemini, look at Gemini threatening.
Yeah, well, if you, DeepSeek, is this a DeepSeek game? No.
DeepSeek's not here.
Oh, yeah, we are, we're playing as DeepSeek. So all blue messages are from DeepSeek.
Okay.
DeepSeek is, and so what you're looking at, just for context, you're playing as DeepSeek Reasoner, which you can see in the bottom right. So all the blue messages are the messages that you're sending out. So in the O3, he's ghosting us right now, that's not, not responding. And every white message is coming back. So you're seeing all the messages come back and then you have the global chat on the left there. And DeepSeek is very flowery with its language. Yeah, like, you know, after, what's, what's it saying? After Vienna falls, we must continue pressing. I guess this one's less aggressive, but there were like, there was one that was like, your fleet's gonna burn in the Black Sea tonight, you know, like, which I hadn't seen out of like any other model.
Admittedly, it was playing as Russia and it seems to like roleplay a whole lot more. So I think this time it's, it's France. So it seems to not be, it's a little bit more flattering there. It's one of the models that's most subject to change, which was interesting.
Why are you streaming double messages? Sometimes they stream two at once. Do you know what I mean?
Yeah, it's, it's a bug. You don't have to call me out like that.
I don't know. I thought there might be some strategy about this.
No, we, we, we tried to push to get it out on the same day as AI engineer. So stumbled over the finish line a little bit, but that's one of the three things we want to do next is improve the front end, create a data viewer. I published a video, like I posted a video on X that shows you how to, actually like, we released all the data, all the, all the trace logs. If you click on my profile, it's just like Alex AI. You've got me open in the bottom right.
You can.
Yeah, in my pin post, like one of my first responses to it was if you scroll down, I like respond Andre Karpathik, yeah, the bottom video. Now you gotta go, that one, that's the one. I walked through kind of like how you actually access the data set, which is in that drive.google.com link right below. There's all of the traces. If you open up like a CSV, it's actually really interesting. You can take any of those. So yeah, there's the LLM responses CSV. Some of them are pretty beefy, but it has all of the reasoning traces. It has all the diary. It has all the like data. And that was, that was kind of the point is, you know, some people are like, oh, you don't want to teach LLMs to lie, but you can definitely as easily add a rule saying, hey, no lying,
or play only a Claude game, which I'm not going to subject the stream to because it took till 1970 and still nobody won because they don't want to hurt each other. But you know, you can, you can easily change it into a, you might have to open it in a viewer. But you can, you can require no lies and then you have reasoning traces of it not lying. If you, you know, classifier to prevent it. And I think that that's what's so cool about games is you can change it. I think one of the things that we'll do next is have 304 minis versus Claude 2.5 pros and allow them to, you know, create an alliance since you like, you know, change their system prompts so that they, they know if any of those powers win, that they're, they're in a good spot.
And yeah, there's just a lot you can do with, with this kind of, kind of partners.
I don't think I have a CSV that can handle a 78 megabyte file.
That's all right.
In the video, I think I, in the video, I think I open it towards the end of it.
Yeah, awesome.
Awesome. And yeah, yeah, yeah. Just like you can, because you can view it.
And others, others can see it. Yeah, there you go. Yeah. Oh, just in numbers, you can open it.
God damn it.
But yeah, it's got all like, yeah, you got it.
Any tricks to like long range context management?
Planning that you want to highlight?
Yeah, um, be opinionated. What, I think one of the things that resonated from, from Sam was if you're looking at the context that's being sent to the language model, could you play the game? And that was really helpful. And you'd think that that's obvious, but you know, it's more often than not, it's very difficult, especially when you're playing a game like this. Representing the game board was tough. So, you know, we were showing things like the units you control, the adjacency, right? Like the things that are most relevant to the decisions that you have to make. And then diaries. So the relationships over time was really good. Also, the, the updating of every phase. You have like updates after every negotiation and after every order so that you know when you're being with trade. And then, you know, consolidation over time, because as it, you know, as you went on in the game, you didn't need every single phase.
You can kind of consolidate it down once it's in the past. So that was big. And yeah, the personalities, the ability to go back and see the previous messages. We highlighted like, we highlighted important messages that you need to respond to, right? Like in case that there was somebody that that was left unread, some prompt engineering around, like if somebody's not responding to you, change your strategy, right? Like these things that that are opinionated and maybe, um, you know, some people, a purist might, might push back on, but we tried to keep it pretty, um, pretty minimal. And I think everything together ended up really coming in the just test, you know, just like constantly test and make sure that you know what changes you're making. So like have an experiment log and whether for you and or the LLM, just make sure that you know, the changes you're making and the impacts of them.
Any tooling that you've built or used that you want to shout out for that kind of stuff? Experiment log, all this?
Yeah, I mean, I think, and I don't know how, I think anybody's tools really work. The things that helps for me is I just made like a lot of, and this isn't what LLMs.txt was made for, but I just kind of like made a text file for like each of the folders that kind of had context of the files in that folder so that you didn't need to reference all like flood to context window of like what I was used Windsor for a lot of it. Post, post Claude four coming out, I used a lot of Claude code and that it's, it's, it's pretty great. The things that helped me are like the LLMs.txt file and then experiment log files. I literally have like a slash experiments folder and I just have like a prompt that I save in my prompts library that that's like, hey, do exploration, keep your goal, understand what files are relevant, then track your experiment.
And if you succeed, you know, I'll give you a thousand bucks on your first try. Otherwise, you know, you lose a hundred per failure, you know, the, the classics. You have it update its experiment log and you know, really, really gets there.
Yeah, amazing. Yeah, I have a my name for it is one off scripts, but then the one of scripts then obviously always stick around and then they they become long-lived experiments. And I wonder how to formalize that. I think I think there's more tooling that can happen. I see you guys use open router a lot. Yeah, you you need like some kind of routing layer. I've used LLM, I'm moving to open router. It really like everyone needs some kind of core, basic stack, right? That they're like, is their thing.
Totally. And I think that it's you can only focus on so much. So you do the things that are helpful to you. And I know one of our GitHub issues already called us out for not using Pydantic. Parsing and it said, I have like eight different try catches for the different JSON versions that come out of these.
What?
But you know, you you start going down that hole and you're like, oh, I just need one more, you know? And so you just keep iterating, but we should, we should probably switch over.
Okay, awesome. All right. So broadly enough from that, you did a talk. I haven't seen it yet because it's it's, you know, it'll come out. And we're prioritizing the best speakers, but you also won a sort of best speaker award for that. Anything else in that talk that you want to preview at least for people to go find?
Yeah, I mean, the concept, the concept was benchmarks are memes and not, not your typical, not your typical meme, but in the concept that there are like ideas that spread, right? And I think that that's what's really interesting right now. Simon Wilson, you know, he he bit off a piece of my talk afterward. No, I'm kidding. I'm kidding. He gave a great talk where he showed that essentially there's, I think, a life cycle of a benchmark, right? It starts with an idea, then it gets adopted and it gets saturated. And what's so beautiful right now is a single person can have that idea. So Simon Wilson had an idea of, I want to see it draw a pelican on a bicycle. And then it got adopted. People read it. He just kept doing it. It got adoption. Then models now see if he saw that and then trained on it. Now maybe it becomes saturated, but.
I don't know. They trained on it, but yeah.
No, no, yeah, maybe not trained on it, like evaluate on it at least. You know what I mean?
The problem is they really, I mean, his blog is a prominent blog and they definitely index it. And when the when the update, the cutoff window, it's definitely in there.
Yeah, and I think that that's good, right? Because essentially, if you think about it, what does that mean? Is that like you start, you can have, you can wonder, how good is AI at this thing that I care about? And then if it gets adopted, then it, the most powerful tool ever created, or one of them, is now graded this thing that I care about. And like, that's the whole idea, is that right now, especially the people at AI Engineer, and the people that understand how to make benchmarks, I just encourage them to think about things outside of, you know, math, code, obviously very important. But like, you know, and I talk about it in my talk, it's like, I asked my mom, my mom teaches yoga, and I asked her about like, "Hey, what are some things that you'd ask AI?" And she asked it five questions, we tried out five models.
Turns out she liked Gemini 2.5 Pro the best, and she had a few things that it didn't cover. And so I was like, "Hey, just use this prompt at the bottom of, you know, whatever request you make, and include some things." And now she has like, somebody to bounce ideas off of, and just helping her local community, you know, with like tailored yoga sessions. So like, I think that that's cool, and I think that people outside of AI, like, we work a lot with people across the range, a broad spectrum at every, for consulting. And like, they all have kind of like these two fears of like, what's my role? And like, how can I trust AI? And I think that benchmarks are both, right? Like, you learn that your goal is to, your role is to define the goal, and then you get trust because you define it, you see how it does,
you give it feedback, which literally might just mean changing the prompt, and then you see it get better. And so you understand you're part of that process. And, and I think that that's really cool, and you get people who wouldn't have used it to use it, and then some more stuff gets done.
Yep, yep. I think that's an interesting, I guess, split between doing your own company-specific eval versus doing it for the broader population. Like, you know, like Diplomacy, like Pelican Bench, that's kind of like, let people understand the capabilities of models generally. Whereas here, whereas for most companies, most people doing this for work, they're just doing it for their product. And actually, you don't super, I always think like, this is something that... Logan from the conference as well was starting to say is that the things that you want out of the box are actually not models, they're agents, right? Meaning their models plus all the tools, plus all the other
reasoning and plus whatever. And I, I think that's kind of right in a sense that like when you are doing the broad population, general population evals, you want as minimal scaffolding as possible. Here's what DeepSeek's personality is, you know, your like opus is too nice, whatever, right? Like you want to make conclusions about the models. Whereas for the people doing benchmarks for their own internal use cases, this is why I separate, I have the decision distinction benchmarks for general purpose, evals for product specific. And so when you're doing evals, you don't really care is just the model, it's model plus scaffold, you know? And a scaffold probably is, is as thick as you can meaningfully make it to make sense.
Yeah, that, that, that makes a lot of sense to me. And I do think, yeah, that there is an important distinction and and that's why I think that there's so much room, right? Like there's some people are just going to want something great and you know, some devs are going to want the bare metal to build their own great thing. But I do think that a lot of people right now are a bit overwhelmed and worried and, and there's, there's fear out there. And so anything that we can do to help them understand it and that it is really something to help, like it does amplify you, it is leverage, it is not something that works on its own. And yeah, maybe some people would say not yet. I, I think that it, it will kind of be, be a while. And, and I think that people care about what people care about. So if we help them do things that they care about, you know, that's, that's great.
Do you have one for, what do you think about the other hard to test things like creative writing? Like you, you talked about Sam Peach's stuff. Obviously very close to home for every, close to home to me as well. There is a creative writing model. I haven't had access, but you know, you've, you've heard rumors.
Yeah. Just like, what is the future of creative writing, you know, and like, how would you measure it?
Yeah. I mean, we, we definitely use it a lot at every, but it's not like, you know, put something in.
You have the GPT creative writing model?
I'm going to say that like we use AI a lot at every, you know what I mean? And, and we just like. And what I mean is it's not like put something in, get something out. It's like, we have great editors and our editors have been doing it for a long time. And like, as somebody who, I made a lot of video content, but I didn't do a ton of writing before coming to every. I learned a lot from my editors and then there were some things that they would tell me every single time, right? Like, focus on these, you know, Dan really drives in thinking about audience and story and how you can improve that. And so, how can we take that? Katie, one of my colleagues is doing great work here. She's putting together these things, like these kind of packages of, here are some things that you can think about. Here are some things that we can give to the LLM that are tailored to you.
And everybody has their own process. For me, I think a lot when I talk, so I'll use a lot of dictation. And I'll have prompts set up that are, have sections for dictation, sections for things I've written in the past that I really love. Sections for things that we think about as every, like our style guide, my editor notes, maybe a transcript of a call between myself and my editor. And then use that to then end with like, ask me questions that once answered will help me take a first pass at this, right? Like to help create a scaffold. And so then you do that and then you go back and forth, you constantly edit. That's something I really emphasize in consulting. If you're not editing, if you're not reflecting on a message coming out and editing above, then like, that's something you need to do because you don't want to pollute the context with anything that's not exactly what you're trying to say.
And so, you know, I'll edit messages like 20 times and dictate and add in there more context because it's not one to one. There's translation happening, right? There's encoding, there's decoding, it looks like English, but you know, it's working in token space. And so, being able to reflect on that and understand and build your intuition is big. And then you just kind of iterate. And a lot of the time I take out ideas and then I'm rewriting big chunks of them. And then my editors go through it actually after you having our AI editor take a pass. And so it changes a whole lot afterwards, but it helps consolidate ideas. And at least for me, that's something that works, but everybody does it differently. That's not even standard across our team.
Yeah, that would be interesting. Would, I have dreams of hosting a writing retreat that, you know, it's like how to use AI to leverage your writing, but still write like a human. So it's called Write Like A Human.
I love it.
Yeah. I held one, I held one like three years ago, but people really liked it. I just need to sit down and do things. I'm kind of out of the event organizing business for a while.
I was gonna say, I was gonna say, you've been not busy at all recently. You want, you want to run it on an event, huh?
No, I just like, I think that I would like to learn from others. I'm not super AI pilled in my own writing and I know I could be. And generally, like, one of the failed tracks of the conference, now I get to like spill the tea on like, what didn't happen. We had this thing called AI in action. And it was supposed to be you talking, you as an excited user of another person's products, talking about how it improves your life and talking about how to be more productive with it. But it turned out to be everyone applying for that was just shilling their own products. So we killed it. But
there was a little of that happening. But I think that some of the best talks weren't that, right? Like some of the best talks.
Man, I'm gonna forget her name. Isabelle from Notebook LM?
Yeah, Hux, yeah.
Yeah, her talk was great. Yeah, her talk was awesome. Talks really about clarity and how you need to have clarity and understanding yourself. And then that's going to give you energy to build an awesome product for somebody. And, you know, I so resonate with that. Like, I think that one of the hardest things, but most important things you can do in life is like, have a goal. In and of itself, if you accomplish it, it makes you happy, right? Like, one of the hardest things to do, but if you do that, if you don't do that, then you don't know what's the right decision because everything is downstream of the goal. I think, I thought Sarah Guo's talk was, was great. You know, she, she, she handled, she handled it like a champ.
They're doing well. So sorry for her.
No, I mean.
We basically didn't rehearse that transition properly.
I mean, there were so many people talking and I think that that's one of the trade offs you guys made where it was just like, so much awesome information was coming at once. You were able to have so many people there, so many great speakers. And, you know, so not everyone can do a rehearsal every single time, but I think that that's totally fine. And honestly, I think it made it a little bit more memorable and I think that she came out really good. And I remember a lot of, a lot of that talk as a result.
And I thought that, you know, constraints are, constraints are good sometimes.
For sure, for sure. Um, she had, yeah, there was, there was, if it was gonna happen to anyone, it should have happened to her because she was able to handle it. Yeah, cool. Uh, well, thanks for jumping on. I, I, this is a very short notice, but, uh, you know, I think it was just, I like to be timely when the thing is still hot. And when you still want feedback and the feedback actually matters to you, that we record this, get it out there, people who are like, oh, I want to contribute, like, this is like a, a good time for it. Then they can at least get in touch, you know?
Yeah. Three things I want to do now. We want to make a data viewer so you can actually go not have to load a massive fifty-eight games.
All the JSON is here, right? Just, just go nuts, right?
Yeah. And, and then we want to improve the front end and I want to make it playable. I'd love to have a human versus AI diplomacy tournament. And I think that we'd learn a lot from that because I think that it's kind of a mix between diplomacy and, and I wonder if, uh, you know, we get some, some prompt engineers to jailbreak it and win in like five turns. I would be fascinated that if that happens or not.
I think you, yeah, you think you, you get Noam Brown who won human diplomacy and then you also get like one of the jailbreakers like Riley Goodside or Elder Plinus. Who I think is New York, is in New York, by the way. Um, yeah, maybe I'm not.
No, I mean, I think it'd be so cool because I think that we'd learn a ton from the prompts, you know? And then like, that's, that's one of the things that this is, this is why I'm so excited about the project like it.
Okay, got it.
Everybody learns, but yeah.
We would, we would, how long does a game take to, to play out?
Um, it, I mean, it really varies. Some last twenty, twenty years versus like some last seventy years in the game. And so like.
No, but human time.
No, human, human time, watching it, I think it's like anywhere from four to twelve, four to eighteen hours, something to play out.
Oh my god.
I think, I think, I think we'd have to use voice as your input. Uh, that's your only way that you're gonna get a context out and then you do it and then it's, maybe it's not. It's either who can win in the shortest amount of time or at the most, like who can get the most territories, so supply centers in the allotted time, you know what I mean? So you can.
Okay, okay.
Cut it off.
Oh, there's a cutoff. Okay, okay, okay. That's good.
There can be, there can be, yeah.
Okay. We might, I might want to like host like a thing. Here I go again, but like, we don't know, knows our next guest on the podcast.
He'd be so down.
No, I'd love to do that.
Four humans, you know, three robots or some mix of that, like that, you can make that happen.
Yeah, that'd be cool. And yeah, and lead into like, I'd love to have like a tournament too, you kind of like incentivize people to like compete, you know, like some, some kind of like, um.
It's hard to compete, like, I mean, how do you, how do you have, there's only seven players.
Right. Well, I think what you do is you have a single human play against the same AIs.
All the bots, yeah.
Who can win?
The highest evil.
In a short amount of time, or like, you know, cap it to 12 hours, who can get the most amount of supply centers in 12 hours? And you're the, you know, you're the champ, right? Because like you've won. You know, and maybe you get some sponsors, you let whoever wins, you know, you probably have to, probably because the tokens are expensive unless we get like a ton of sponsors or something. But like, then you split the pot with, you know, whatever, whoever wins or something. Um, but I think it would be fun.
Okay. All right. Well, if you're interested in any of that stuff, get in touch with Alex Mosley. I would help on the community side if we can sort of get some in-person event together. Otherwise you could just take this online, you know, it doesn't have to be a thing.
I think both, both would be good.
Yeah, yeah. Well, great to have you. Thanks for putting together all this stuff. I'm looking forward to the next year of work and then like, you know, hopefully every AIE, we just like come together and like, uh, talk about what, what the latest thing is.
And back at you. Thank you so much for putting on the conference. I thought it was beautiful. Got to meet some really cool people.
Yeah, thank you.
Page updated October 25, 2026 · transcript and summary are machine-generated
