⚡️ Podcast Crossover: AIE, AGI, frontier lab strategy with ​ ⁨@matthew_berman⁩ and @swyxtv

Watch the full show on YouTube: https://www.youtube.com/watch?v=bIOCKXcxloQ

• 0:00Speaker 00

  I don't even know how to start this. You have thirty minutes, right?

• 0:03Speaker 01

  Yeah.

• 0:06Speaker 00

  This conference is incredible. There are so many people here. I want to start with the origin. How did you think to start an AI engineering conference? How did it come about?

• 0:16Speaker 01

  Yeah, I've been a speaker for many years before that. And I also had seen people seize a moment where you can see like an industry shift. And I thought it was pretty clear. I had seen basically front-end engineering become its own professionalized field with dedicated conferences, dedicated influencers and tech stacks and all those things. And I've seen the same thing for cloud engineering and data engineering and all that. And I was like, it's very obviously going to happen to AI. And so I bought the domain ai-engineer and planned this conference and I wrote the blog post. And it really kicked off when Andre Karpathy also endorsed it.

• 0:58Speaker 01

  And I think it's gotten more true over time, which is the kind of way I like to see it. Gartner put it at the peak of its hype cycle two years ago. It's still hypey, still going.

• 1:08Speaker 00

  Had you, had you started a conference before? Had you ever done a conference before? You were just like, okay, I'm just gonna do it?

• 1:14Speaker 01

  Well, I didn't do it alone. I partnered with a friend who had done previous conferences. So basically from my conference speaking career, I just took the best of every conference I had been to prior. And Ben was running Reactive Thought, which is the best React conference in the West Coast. So I just had him start AIE and I would do the content, he would do the logistics. And that's how it goes.

• 1:34Speaker 00

  What, what was the most difficult part first during your first AIE and then this one?

• 1:39Speaker 01

  Yeah, I think the first one was just convincing people that this is even worth showing up to because like a first of anything, like you don't have any history. Yeah, like now people have word of mouth, they have the YouTube channel, it's all, it's all good. I think the, the very first one, you got nothing. And the people just had to take on a leap of faith of like, well, my reputation, Ben's reputation comes together. Do let's do something interesting. I don't know, right? And it was also not that much in terms of like the tech, right? Like at mostly, we were still talking about prompts frameworks and maybe a bit of rag. But there just weren't there weren't any agents like it was very, very primitive at the time.

• 2:20Speaker 00

  Yeah.

• 2:20Speaker 01

  Um, I think also the last thing was getting a big lab to come. And for us, it was Logan and OpenAI at the time. And I think a big part of our value proposition now is that we can get all these labs to show up. At a neutral place where they all compete on the same grounds, which is maximally beneficial for engineers. And most competitive for the labs. And it turns out the labs like it. They like winning on a in a sort of like an even playing fields. But you're just not going to get that in like a Dev Day or a Code with Claude.

• 2:49Speaker 00

  You know, you know what I really appreciate? Uh, there seems to be uh, the entire stack of complexity here. Everything from kind of really practical, everyday user, kind of standard user. And then all the way down to highly technical, deep in the way.

• 3:01Speaker 01

  I just did a one hour chip session uh, with Etched, who just came on a stealth this week.

• 3:06Speaker 00

  So good. Yeah, yeah, I just saw that. Um, you, what do you think about Etched and are they gonna disrupt Nvidia?

• 3:12Speaker 01

  Um, I don't think they want to disrupt Nvidia. I think the better framing is that all of inference is just so goddamn big. That of course you're gonna have ASICs for inference.

• 3:22Speaker 00

  Yeah.

• 3:22Speaker 01

  You're going to and Nvidia is gonna be continuing to build GPUs. Are going to.

• 3:28Speaker 00

  Is it is Etched more akin to a Grok or a Cerebrus?

• 3:32Speaker 01

  Yeah, I have been pitching them as or not pitching, but framing them to friends. As next generation Cerebrus, next generation Grok, right?

• 3:39Speaker 00

  Right.

• 3:39Speaker 01

  Cerebrus started ten plus years ago, not their IPO. Um, that that process in form factor is relatively well established. What Etched and Maddox and the other new gen chips are doing are much more dedicated to like basically everything that we know. Uh, post transformers, post chat GPT and optimizing for that workload. So that completely makes sense that the Cerebrus, Grok, Samenovas of the world. Didn't optimize for it because it didn't exist back then.

• 4:04Speaker 00

  Yeah, but so you know, when you, when you make custom chips, you run the risk that the model architecture trains changes drastically due to some kind of innovation, right?

• 4:12Speaker 01

  Yeah, they don't care.

• 4:13Speaker 00

  They don't care.

• 4:14Speaker 01

  Yeah, um, the chat GPT like GPT three point five ish architecture. Has mostly stayed the same this entire time.

• 4:21Speaker 00

  Yeah.

• 4:22Speaker 01

  It's a pretty good Batman, like even if there is a new model sometime in the future, the current workloads on existing models, like 4.0 is still being used, right, by some, by some folks. Cause they, you just don't move, like, once the thing works, it works, like, don't, don't touch it.

• 4:37Speaker 00

  Well, alright, I wanna talk about something new, and you've been obviously quite busy this week, but Fable 5 is now back, have you had a chance?

• 4:45Speaker 01

  They released it for us.

• 4:46Speaker 00

  They did, they released it just for you, I don't think, maybe you tell me, have you even had a chance to?

• 4:50Speaker 01

  Yeah, yeah, so there's a, there's an Easter egg on the website that is built by Fable. It's, it's, it's a, it's a game that, that's you.

• 4:57Speaker 00

  AI dot engineer.

• 4:59Speaker 01

  Yeah.

• 4:59Speaker 00

  Okay, uh, you wanna give any hints as to what to look for, for the Easter egg?

• 5:04Speaker 01

  Uh, look for a globe icon that's, that bounces when you hover over it.

• 5:08Speaker 00

  Alright, so you, you've played around with it, uh, does it feel the same as those couple of days when it was first released? Cause a lot of people are talking about it being, you know, nerfed and there's a kind of a lot of floating around on X right now.

• 5:19Speaker 01

  I see, I see, um, yeah, people are very quick to complain about Anthropic. I think the, I don't know if, I, look, I don't know if it's nerfed since the, the first Fable launch. I haven't, I don't, I haven't, you know, I don't have hard numbers on that. I know that there are false refusals happen and still happen a lot, and those are really annoying when they happen, right? The, the downgrades, the opus.

• 5:42Speaker 00

  Um, I haven't had that happen to me once.

• 5:44Speaker 01

  Okay.

• 5:44Speaker 00

  I haven't had a reroute happen to me once yet, which is.

• 5:47Speaker 01

  Maybe you're not spicy enough.

• 5:48Speaker 00

  Maybe, I'm, I'm definitely ripping tokens though, so if it, if it were gonna happen, it would be there.

• 5:53Speaker 01

  But maybe you need to talk about chemical weapons or something, and then it, then it'll go.

• 5:57Speaker 00

  Not a lot of my projects need that.

• 5:59Speaker 01

  Uh, so, so, no, I, I, I think, uh, like, look, when it works, it is extremely smart, uh, but it's also very slow. And I think, uh, definitely I don't see myself using it for everything, uh, because of that, right? Like, I, you only use it for the smart problems, and that's probably the way they want it anyway. But I, I think, I think maybe, uh, the other part about. You know, there was a narrative like a month ago where like people were like, oh, like Anthropic is limited on chips. Uh, limited on compute capacity. That's why they're not rolling out Mythos in Fable. That's clearly not true.

• 6:30Speaker 00

  Well, now it's not true, right? I mean, they

• 6:33Speaker 01

  It probably wasn't true a month ago. It didn't change that much. Like, they know their roadmap.

• 6:36Speaker 00

  Maybe not a month, but like, you know, the cerebrus, uh, sorry, not the cerebrus, um. Uh, they, they just partnered with, I think it was AWS and they, they, they landed a bunch of chip deals, a bunch of compute, uh, bandwidth deals.

• 6:47Speaker 01

  But you, you think they like, weren't planning these for like months before, like, they knew this, they, you know, that's the thing, like.

• 6:54Speaker 00

  XAI, that was the big one, right?

• 6:56Speaker 01

  Uh, anyway, I'm just saying like, these, these things don't happen just overnight. And like, they probably were already like, the Mythos rollout was genuinely safety related. Uh, that's why it was like limited. It wasn't because there's some secret conspiracy to like limit or ration compute.

• 7:12Speaker 00

  I still do think they were bandwidth limited or they were compute limited because if you look at their quota. And, and how aggressive in reducing it and using it, uh, it's gotten much better, but especially two months ago. I mean, you would burn through your quota in a matter of minutes. Um, it's happening again, to be fair with Fable, because it just uses so many more tokens, but two months ago, it just seems like. They, they could not get enough compute to serve the models that they have.

• 7:39Speaker 01

  Um, yeah, well, only, only they know.

• 7:41Speaker 00

  Yeah. All right, let's talk about the other frontier lab, one of the others. Let's talk about OpenAI. I woke up, I think it was this morning.

• 7:49Speaker 01

  Yeah.

• 7:49Speaker 00

  To the news, I guess rumored, I haven't, I gotta actually read more deeply into it, but. OpenAI is offering 5% equity stake to the US government.

• 7:59Speaker 01

  Yeah.

• 8:00Speaker 00

  Like, what do you think is going on there?

• 8:02Speaker 01

  I saw that headline, uh, I don't know where, I don't know anything about the sources or anything. I, I think it kind of tracks, like, it's not within the outside range of possible things to do. I think, uh, OpenAI has been relatively more friendly with the White House than with other frontier labs. That's, that's like factual to say.

• 8:21Speaker 00

  Yeah, yeah, yeah, yeah.

• 8:22Speaker 01

  And, um, I also, I also just generally do think the people of the United States or the, the people of any country where. Um, you know, like all such like frontier intelligence is created. They do need to have a share or a say in some upside. Otherwise you have like. A permanent underclass in significant social distress. And I don't know, you know, I I've played around with the idea of like universal basic AI. Like everyone gets like a chat G P T pro subscription or something. But like giving the government five percent to have the government in have a stake in the success of OpenAI as a leading lab in the U S does tend to make sense.

• 9:07Speaker 01

  Uh, I come from a country where this is normal. I'm from Singapore. Tamasek and G I C own significant portions of the Singapore economy and it works just fine. And actually, a lot of our pensions and our savings and our insurance and whatever is all invested in governments, uh, in like nationally critical important companies where the government owns a stake. What's so different?

• 9:31Speaker 00

  Do you think they're gonna treat it more like a utility? Because

• 9:35Speaker 01

  Oh, yeah.

• 9:35Speaker 00

  Uh, so there's like a few different kind of flavors of how this plays out. The government owns five percent. They give out distributions to U S citizens, or maybe they treat it much more like a utility and they start heavily regulating and can step in at any time. What do you, obviously, we're speculating now. What what do you think is most likely to happen?

• 9:53Speaker 01

  I think this thing is too volatile to treat as a utility, right? Like, imagine if like Edison was like, you know, working on his like electrical stuff and like and lighting and all that.

• 10:03Speaker 00

  Yeah.

• 10:03Speaker 01

  And he immediately tried to regulate it like, no, you'll probably wait fifty years first. So it's like a little unfair to like in twenty twenty five, you know, like for three, four years after chat G P T to be like, no, like we should regulate it now like a utility because it's there's no more innovation to be had and you should just leave it to natural monopolies and charge like cost plus. Like, I don't think that's we're there yet. I think we need to wait another few decades.

• 10:26Speaker 00

  So what what is it? Is it pay to play? Is it bend the knee? Is it, hey, we're gonna give you five percent and you're going to allow us to release our our models more quickly? You're gonna step back on regulation? Speculate with me, Swig.

• 10:37Speaker 01

  Yeah, so like I'm not a citizen, so I I do tend to be careful about what I say politically. Um, I do think that, uh, it is interesting to think through people optimizing for this administration versus, oh, there's an election every four years. And like, actually, this is a multi-turn game, not a single turn game. So maybe don't only think in one turn.

• 10:57Speaker 00

  Yeah, um, okay, so then like, let's continue down this track a little bit. The government, especially in the last few weeks, has been very hands on deciding what models are publicly available. The timing of models, right? So GPT five point six, which is announced, but they have to wait a few weeks to actually release it. What like, do you think this is the new standard? Do you think they're kind of scrambling to come up with a framework that's consistent across? All of the AI labs? What do you think is going on?

• 11:25Speaker 01

  I think the White House thinks it is consistent and uh, it is probably better than what? The states tried to do themselves a year plus ago with SB forty seven, all those things. So yeah, I mean, yes, on a national level, do it. Um, and beyond that, I think the best, uh, best you can say about this is this isn't new. In terms of tactics in politics, and this is the art of the deal.

• 11:53Speaker 00

  Yeah.

• 11:54Speaker 01

  But I think I honestly, I want to read the book.

• 11:56Speaker 00

  And it's an initial gambit that is quite extreme, then you roll it back, but you still get.

• 12:00Speaker 01

  It's happened before?

• 12:01Speaker 00

  Yeah, I mean, it's literally, he wrote about forty years ago, however long it was, thirty years.

• 12:06Speaker 01

  It works.

• 12:06Speaker 00

  It works.

• 12:07Speaker 01

  And I would be, would be funny if like, this is actually how humanity is saved from Skynet. Is the art is the art of the deal. Because there was no other, uh, there's no other mechanism under capitalism to control companies.

• 12:24Speaker 00

  But there, I, uh, do you think the government is more worried about Skynet? Or do you think they're more worried about other countries doing distillation hacking? Uh, or using the model for cyber warfare themselves?

• 12:35Speaker 01

  I think there's, uh, there's, there's multiple reasons that all align, so.

• 12:38Speaker 00

  What are you most afraid of?

• 12:39Speaker 01

  When they, when they all align, it's hard to tell motivations because they all point the same answer.

• 12:43Speaker 00

  Yeah.

• 12:44Speaker 01

  So like, there's no, like, there's no point ranking them because they're all aligning. Uh, well, I am, I am, I am a doomer in like, in terms of like, people talking about P doom of like, let's, I've met people who have P doom of zero, P doom of less than one percent. Mine is like closer to ninety on a scale of fifty thousand years. Um, I, you know, I think people, P doom should be attached to timeline. Um, and so I think.

• 13:09Speaker 00

  Fifty thousand years.

• 13:11Speaker 01

  Which is a rough approximate time of the Homo sapiens, right? Fifty thousand to four hundred thousand is like our span and like, you know, like, we don't have a right to exist anymore than anything else that went existing before us. And if we're birthing a new life form, um, it is reasonable to expect that by accident, you know, we, we wouldn't, we may not do so well compared to it. I mean, how are trees doing versus human civilization? And like, we, we move much faster than trees. These things move much, much faster than us. I think absolutely bears some thought.

• 13:43Speaker 00

  Yeah, I mean, I mean, what you're saying is basically the chance of actually aligning these models in the long run is very low.

• 13:51Speaker 01

  I think our default assumption should be that it's low. And if we miss in the right direction, we will have missed in the safe direction anyway.

• 13:57Speaker 00

  Yeah, that's fair.

• 13:59Speaker 01

  So I think, um, one thing that I often don't comment about is where A I E is in relation with E A C and the, the D cell movement. And, um, it's like right in the middle. So I, I talked about this at the last Wolfsphere keynote, where politically we want to be pragmatic, you want to be optimistic, but you also don't want to be unconstrained optimistic, which is what E A C is.

• 14:22Speaker 00

  Yeah.

• 14:23Speaker 01

  Um, and that's why putting guardrails and systems and fine tuning and, um, doing evals are important and in A I E versus with an E A C, you just talk about H I timelines all day long.

• 14:34Speaker 00

  Right.

• 14:35Speaker 01

  Cause you don't care. Um, I do think that some measure of having the engineers in control of things, watching things, monitoring, change, change of thought probably is the right path for humanity. So like the lichen of it, like, you know, we live as far as you know, we're the only life form in the, in the observable universe. Uh, it's, it's, life is very fragile and like, we shouldn't assume that by default, we get the right to continue existing unless you engineer it.

• 15:01Speaker 00

  If you had to shrink the timeline from fifty thousand to let, let, let's get your P doom on ten years and fifty years.

• 15:09Speaker 01

  Oh yeah, I mean, P doom in ten years is, is near zero.

• 15:12Speaker 00

  Uh, well, that's good. That's good.

• 15:13Speaker 01

  Yeah, fif- fifty is like, that's like, that's like the end of our lifetimes. Um, and like, I think.

• 15:21Speaker 01

  I don't know, I'm just, I'm just gonna throw out ten. No, ten's too high, five percent. Um, and, uh, again, I, I think on the timelines more of what you see in Foundation. You know, the TV show?

• 15:35Speaker 00

  Yeah.

• 15:36Speaker 01

  Um, or Dune.

• 15:38Speaker 00

  Great show.

• 15:39Speaker 01

  I think, um, when you, when you think about the long evolution of life forms and history, you shouldn't be so, sort of near-term focused. Because, yeah, probably L L M's are gonna run out at some point and they're not AGI. And, okay, we have maybe another thirty years of AI winter or something. And then like the next paradigm really is actually the thing. Or it takes another three thousand generations.

• 16:05Speaker 01

  Like, what, like, you know, who are we to say that we happen to live in the exact moment that everything ends? Like, that's very egotistical. It's like, you're the main character, really?

• 16:14Speaker 00

  Yeah.

• 16:14Speaker 01

  Like, you know, who's to say you're not like, you know, in prehistory right now, as far as the future people are concerned?

• 16:20Speaker 00

  Yeah. Do, do you think, uh, large language models are, uh, are enough to lead to recursive self-improvement, thus leading to some

• 16:30Speaker 01

  We have some form of it.

• 16:31Speaker 00

  next architecture that might be AGI?

• 16:34Speaker 01

  Oh, I see. That is a very nuanced question. I think actually my answer is no on that one. Um, we, yes, L L M's enable RSI. RSI is really just like, is it recursing? And yes, it is recursing on it. We have a whole auto-research track yesterday covering that stuff. Um, but it is limited in its recursion. Because it probably just explores things that have been explored before. So it's like, not that far off from the distribution of stuff you already know.

• 17:03Speaker 00

  Yeah.

• 17:04Speaker 01

  So discovering true unknown unknowns, uh, having real, like, real innovation, having a real sense of world models, that is still the domain of research. And, yeah, we probably need something else. We did have a world models track here. It is not very well-developed. Everyone knows it. But clearly, this needs to be fleshed out more. Because the stuff that Dorcas is saying is absolutely right. Uh, we, like, current learning paradigms.

• 17:31Speaker 00

  What part?

• 17:32Speaker 01

  Uh, data efficiency is the next problem. Um, we had a data track on the same thing. Um, and You know, like, learning over a trillion tokens in order to get to some form of human equivalent labor is very, very inefficient versus what humans do. Humans learn on the order of like millions and they can already do like very useful things. Billions and you're a full working adult.

• 17:57Speaker 00

  So, do we have to try to fit large language models, the compute that it requires into kind of the human box?

• 18:04Speaker 01

  No, we don't, no we don't.

• 18:05Speaker 00

  Why does it have to be, well, if it's not exactly how humans learn, then it's not the right way?

• 18:10Speaker 01

  No, I, I absolutely agree with what you're saying there, which is, they call this a sour lesson, right? Like every time you try to make a human analogy to machines, you probably fail because machines develop very differently from humans. Uh, so, no, I agree, I, I agree with that and I also think that it can still be an alien form of more efficient learning. Doesn't matter, we just know it is super inefficient. Like, that, that is something that we know is an unmitigated negative. So, let's make it more efficient and better. Uh, and that means that, you know, if in order, instead of two thousand examples to learn one thing, what about twenty examples? What about two examples? Um, and that scales a lot more, um, and that means, you know, we can, we can get, we can actually get to a point

• 18:52Speaker 01

  with continual learning that we can actually have uh, agents that adapt and, and build up a real world model. Otherwise, you're always stuck to the pre-trained, post-trained paradigm that is probably hitting some kind of limit right now. Like, it, like, I generally do expect Fable to be the end of this era of L L Ms. Because you can't, like, I already told you about the slowness. Do you agree with the slowness, by the way? That Fable's slow?

• 19:19Speaker 00

  I, I've actually, so I've, I've been using it in Cursor and it feels a little bit faster than when it first came out a couple of weeks ago.

• 19:24Speaker 01

  Yeah.

• 19:24Speaker 00

  Um, but it is slow. It is slow.

• 19:26Speaker 01

  I mean, look, and Topi's working on, uh, better inference. Yeah, now, train through history, but it is slow and like therefore you just won't use it for some things that you know don't require Fable.

• 19:35Speaker 00

  You also don't want to pay Fable prices for everything.

• 19:37Speaker 01

  Yeah, like, even, like, I'm, I'm trying to live in an infinite budget world. But the, the budget I don't have is time, right? Like, this, this, this is the limiting budget that everyone has. I don't care if you're working at Frontier Lab, you still, you still have time as a, as a, as a limiter. So, um, yeah, if that's like, if a twenty trillion model is it, there's no two hundred trillion, there's no quadrillion scale model, then that's kind of it as far as usability is concerned, living in an infinite. Budget environments. So therefore, you just need something different. Whatever is, maybe it's like thinking machines and stuff. Maybe it's together AI's SSM stuff. Whatever it is, we don't have it yet.

• 20:16Speaker 00

  I've, I've seen, I've seen a lot and I've thought a lot about this. The, the model capability overhang still seems very real. Even from the previous generation OPUS, uh, GPT five point five. The, the amount of value that can be extracted from those models still seems at least to me to be, uh, uh, critical. And, and now we have a whole new, uh, generation of model that we're even going to get more model ever hang from. What, what do you, do you agree with that? Do you think you just need to keep building the tools around the model?

• 20:48Speaker 01

  AI engineer exists in the white surface area between the peak capability and deploying it everywhere else, right? So the more model research peaks and spikes capabilities in one domain, but it's not evenly distributed in all products yet. That's where engineers have a job forever, basically. Um, so I'm very pro that. Uh, I, I, I think capability overhang will exist for a long time. I think it does keep have waves of consolidation where you're like, actually, all this stuff I build out, I don't need it anymore because the next model has got it from just like a single prompts. So I'm going to throw it out, but like, we do spring cleaning every now and then, like, that's normal.

• 21:29Speaker 01

  And we build it up on previous gen model assumptions that then go away. And I don't think we should feel any attachment to the code. At the end of the day, we're all just trying to like serve customers better, do work cheaper, faster, easier. Yeah.

• 21:41Speaker 00

  Let's, let's talk about that a little bit because I've built a bunch of cool kind of one-off projects that one model generation later, I was like, oh, well, that's kind of useless now. And then you, you know, you have anthropic moving in different directions. They're moving up the stack to the application layer. If you're a founder, what, or you know, what advice would you give to founders who are trying to build at the application layer? Where should they be spending their time? How should they think about the expanding model capabilities in each subsequent generation of model?

• 22:07Speaker 01

  Yeah, I my answer to this is two words, Agent Lab. I like I like the two word solutions. I like AI engineer, I like Agent Lab. Um, so I have a piece on this if people want to see, uh, it's on.

• 22:17Speaker 00

  Where do they find it?

• 22:18Speaker 01

  Uh, yeah, latent dot space slash Agent Labs.

• 22:22Speaker 00

  Okay.

• 22:22Speaker 01

  Um, and basically, the idea is that you always want to be the AI guys for your customers. Actually, don't pick the solution, pick the problem. And if you're like, okay, I'm like, whatever it is in AI, whatever the hot thing is, whatever the new trend is, what a new model is, I will be the AI guy for dentists or for lawyers or for finance people, uh, or for coders, whatever. Um, then you will just be build that lasting brand of we like, we will build the products that do the last mile for you and fold in all the new functionalities and features that people discover into that. That is the sustainable thing.

• 23:00Speaker 00

  Isn't that, isn't that, isn't that betting against the generalization of the models and the capability of the labs, frankly?

• 23:06Speaker 01

  Uh, no, yeah, so let me think about this. So no, but maybe yes. So I I don't know where I stand in regards to your question because the models do generalize and sometimes wipe out entire product categories. But I do think that if you are smart enough to like, I literally, I like, I am the AI layer for lawyers. I'm your outsourced tech team for lawyers. Okay, that my my old business model is gone. Fine, I'll make a new one with like whatever new capability overhang is created. What you're betting against is capability overhangs ever existing in the future. And I think that's a pretty safe bet to make.

• 23:46Speaker 00

  Yeah.

• 23:46Speaker 01

  Um, so like there will always be capability overhangs. They may not stay still. And so you gotta be nimble. But the Sierras of the world, the cognitions of the world, the curses of the world, the Decagons and um, Harveys, these are all Agent Labs for their field. They can be trusted, brand names to always apply the latest AI to their thing. And maybe they're a little bit behind the the frontier labs on. Like the the latest models, but in terms of solving user customer feedback, you know, I've I've been inside of cognition for the last six months. There's no one else. The the labs do not have 200 people dedicated to like, you know, being on call with you with Goldman Sachs going like, okay, guys, what do you need?

• 24:28Speaker 01

  We got it. You you need the Microsoft Teams zero integration? Got it. You don't use GitHub? You use this like weird org thing that? It's like a fork of Alasian's Bitbucket when it was open source? Got it. Claude's not gonna do that.

• 24:40Speaker 00

  Look, I mean, let's continue on that. So you you've you've been at cognition for six months.

• 24:45Speaker 01

  And you had the food. It scores well on food bench.

• 24:47Speaker 00

  It is very good. High on food bench benchmark.

• 24:50Speaker 01

  We didn't score better on other benchmarks, but food is a good start.

• 24:53Speaker 00

  You just have to invite me back. That's how you gotta run the benchmark again. Um, do you do you think, um, uh, sorry, I lost my train of thought. Um. Oh.

• 25:02Speaker 01

  Agent abs.

• 25:02Speaker 00

  Yeah, so a lot of discussion lately has been around token budget and and really token maxing being unattainable to most people. Do you think that's actually a huge value to the non frontier lab? Uh, agent lab companies, so the cursors, the cognitions of the factories of the world are, isn't this kind of like prime time for them? Because first of all, they're model agnostic. They actually have incentive to do model routing really well, whereas the frontier labs don't. Do you think this is the era of kind of the the uh, uh, model agnostic companies?

• 25:33Speaker 01

  Yeah. I think that is a reasonable conclusion to make, which doesn't mean that I think it. I think that is a lot is that is currently what everyone is saying because of this current debate. Uh, I do observe that in general, the the big, big wins have just been going all in on one thing. Um, and I'm old enough in tech to know, have seen this before with the cloud wave where.

• 26:03Speaker 01

  There've been companies that were all in on AWS or all in on GCP. And there've been other companies that were multi-cloud. They're always, you know, like I'll use the Terraform, the whatever. And the simple argument is that if you route, if you pride yourself on routing, you will never exploit the full capabilities of one because you're not all in on one. You're not.

• 26:22Speaker 00

  Right.

• 26:23Speaker 01

  So it's a really good way to be lowest common denominator of every model out there and completely miss all capabilities.

• 26:29Speaker 00

  Yeah, you also don't have the economies of scale that in AWS has going all in owning the entire stack.

• 26:34Speaker 01

  Yeah, so yeah, Agent Labs get discounts from every model provider. And that's also very interesting when people compare public pricing of, I guess, discounted cloud code from Anthropic versus what Anthropic does with Model Labs, with Agent Labs. And I think that's also an interesting discussion on that front, too. Anyway, all which to say, I do think that this routing thing is a marketing line. I wouldn't necessarily believe it so much as when I talk to the top-tier agent builders because they're all about maximizing and fully exploring like the, and complete prompt surface area and optimization area and tool use and caching and God knows what else in there, right? Like exploit everything they give you.

• 27:16Speaker 01

  Or do you want to just stay superficial and only do chat completions across 100 different models?

• 27:20Speaker 00

  Yeah.

• 27:21Speaker 01

  Right? Which is more likely to win as an Agent Lab.

• 27:24Speaker 00

  All right, well, Swix, I want to say thank you very much.

• 27:27Speaker 01

  Yeah.

• 27:27Speaker 00

  Appreciate the conversation.

• 27:28Speaker 01

  It's good to chat, man.

• 27:29Speaker 00

  Yeah.

• 27:30Speaker 01

  I'm glad you could see me on my biggest day, on my biggest show.

• 27:33Speaker 00

  Hell, yeah.

• 27:33Speaker 01

  I think you've seen me in like the podcast, the office and all those things. This is, I'm an in-person community guy.

• 27:40Speaker 00

  I am genuinely impressed with what you build here. You have a bunch of great, excited, enthusiastic people, not only giving talks, but also learning a ton and, and yeah.

• 27:50Speaker 01

  Making connections, getting co-founders. I need your help on our YouTube. Our YouTube sucks.

• 27:55Speaker 00

  Let's do it.

• 27:55Speaker 01

  Uh, so, yeah, lots of learning to be had.

• 28:00Speaker 00

  Thanks, swyx.

• 28:00Speaker 01

  Thanks.

• 28:01Speaker 00

  Very nice.