Latent Space: The AI Engineer Podcast
Original podcast
Watch the full show on YouTube: https://www.youtube.com/watch?v=JYaGAqvPfYA
5,344 words · 2 speakers
Alrighty, so we are in a remote studio with a lightning pod with Dylan Davis, or D Squared. You're a big member of the community. I've interacted with you over the years. Actually, I don't know. How did you first come across LateinSpace?
Well, I first came across you. So I found your working in public blog post, book and conversations probably like seven years ago. I don't know, like seven, six years ago.
Damn.
From there, yeah, I'm an OG. I'm a swyx OG fan. Well, yeah, I found you through that and then just basically followed you and saw LateinSpace and then followed there.
And how do you introduce yourself these days? What do you work on?
Yeah, so I'm the founder of Gradient Labs. It's a company that helps other organizations, usually mid-sized companies, implement AI internally to help automate their processes, saving time, money and sometimes generating revenue.
That's very tight. I'm sure you say that a lot. Awesome. So I put out a call on Small AI for, you know, this comparison thing between Cognition and Anthropic on their multi-agent articles. And you answered it with like, I think the best answer, which was you made your own video. You're trying to wrap up your YouTube as well. And it was like a really good analysis. So I just figured we would sort of rehash it a little bit. People can obviously watch the full thing on your YouTube, but this is meant to be more of a discussion between the two of us. What's going on?
Yeah, no, I appreciate it, man. And I'm glad that it resonated. So do you want me to get started?
Let's get right into it unless you have anything else you want to sort of say as context.
Yeah, I mean, I guess the, and you probably have a better insight into this, but there's, I guess, some discussion around the fact that both of these blog posts, and context for the viewers or listeners, is Anthropic and both Cognition posted blog posts one after the other. I think it was like June 12th and June 13th. One basically pushed the fact that multi-agent architectures are beneficial and useful. And the other one basically stated the opposite, saying that single-agent architectures is more superior for different use cases. And if you actually read into both blog posts, you realize that they're both right. So it's not either or, but yes, and if you're a, what is it?
Like the stand-up comedy thing.
Improv, yeah.
There you go. That's it. So I created this image. It looks really cool, but it's also kind of convoluted and not correct. So what's what's happening is they're actually holding hands instead of fighting each other.
True love.
All right. So what I'm going to show on the left-hand side, we'll start with Anthropic, go to cognition, and then we'll end with, I think, kind of like overall insights and how to think about when to use which architecture.
So for the Anthropic blog post, they focused mainly on their use case for deep research. So I'm sure if you're aware, if you have access to Claude, I think it's Pro and Max and all the paid plans, they have access to this new deep research feature that was released a few weeks ago. And with this research feature, I tend to compare a lot of the models. So when I do deep research, I do Gemini, I do perplexity, I do opening, I do all of them when I ask different questions. And time and time again, over the last couple of weeks, I found that Claude has by far outperformed the others. And I guess the definition of good for me right now is not just length, but also the number of sources and diversity of response.
My guess is part of the reason it's so highly effective is the way they've structured their agent. So they took a multi-agent approach with this feature. And they gave us a beautiful little graphic in their blog post that kind of walks you through exactly how this functions. So on the left-hand side, we have a conversation from an AI that's basically, or from a human, asking a question. So this question is, what are all the companies in the United States working on AI agents in 2025? Make a list of at least 100. For each company, include the name, website, product description of what they do, the type of agents they build, and their vertical industry. Now, at the surface, this is obviously a pretty beefy question. And the
I would say the thing that's happening here that's useful for this specific structure for multi agents is that it's a broad question. They're asking about a variety of things and they're asking for multiple perspectives on that specific thing. So what happens is they send the question over to their agent architecture, which is where they have a lead architect, so a lead agent at the top that's orchestrating everything. And they'll then send off this question broken into sub-questions. So they'll send over the question, they create a plan, they create a bunch of sub-questions, and they send off these requests to multiple agents and they do it in parallel.
And that's kind of the unique thing here that I found. It's, and maybe, so if you haven't answered this, or maybe you have insight, is I've, I don't necessarily know, I can't confirm this is happening, but I know in their documentation it says that it is, where they'll send off a request as the lead agent to multiple sub-agents to then do the research. I'll pause and, yeah.
I mean, I think that's what they're saying they're doing. I don't know why you wouldn't doubt it.
Yeah, well, because I'm a skeptic, all right?
That's good. All right. So basically, all these sub-agents are doing their work and they'll go off and research a specific sub-question and that parent question. And as they're doing the research, each one of these sub-agents have the 200K context window. And that means that all of them are going to have tons and tons of insights and research for that sub-question. They'll synthesize that, once they've synthesized the responses, they'll feed that back through the lead agent, and the lead agent then will put a variety of insights that are relevant for answering the question in its memory. After that's done, it'll then consolidate the memory insights and then create its report back to the user. Another important caveat here is they have another citations sub-agent that's working as these sub-agents are responding to validate that the citation and the piece that it sent
back is connected to the source that it referred. So that's kind of the overall flow of how this works. Pause and see, swyx, what are your insights, feedback, questions?
I think it's a relatively straightforward implementation so far. I think the thing about the nuance here is the communication back and forth. You know, dare I call it agent to agent? And like when they interrupt each other or you know, how they how the lead agent orchestrates them, that can get into a lot of complexity. So this image actually looks simpler than it really is in real life.
Yeah, and and it's funny that you say that because when I when I initially made this video, I thought to myself, I was like, oh, maybe maybe I'll show this one. And this chart is more detailed as to what you're talking about of how the multi-agent structure works and how the conversation happens. And it is more detailed, but it's also more overwhelming. So yes, it is definitely more complex than background. So after we have a good understanding of multi-agent, I wanted to quickly show what a single agent architecture looks like. And this is a chart from Cognitions blog that walks through a basic flow or basic architecture of what a single agent could look like. So in this case, we have the task at the top where it's getting an input from a user. So the user's input comes in and we go through the same process. So we're going to have the large model at the beginning. So it's going to be a big beefy model like O three Gemini two dot five pro, cloud, sonnet,
opus, sonnet, or yeah, cloud, sonnet, four, cloud, sonnet, four, opus, etc. And when they input the information, it's going to create a plan, break down that plan into subtasks, sub questions, etc. And it's going to continue to research. The difference here is instead of doing it in parallel with subagents, they're going to do it in sequential order. So they're going to have the agent, it'll first research the first question and it'll then consolidate the insights from that and it'll go to the next question and it'll go to the next question and it'll go to the next question. And this order works for some use cases, but when it comes to research, there are some downfalls. And one of the downfalls is the fact that the context window is going to get bloated pretty quickly, which means that the variety of insights and diversity of insights is
limited based off of what it can pull within those iterations when it runs linearly. And that's kind of one of the benefits of the multi-agent structure where you can have a multitude of agents doing research from different perspectives, synthesize that without loading the context window too soon.
And that's kind of the process and how this chart looks and flows. Groovy. And some things on performance. So in the blog post that Anthropic posted, they ran some tests and they noticed that the multi-agent structure outperforms the single-agent structure by 80% based off a different variety of variables they measured. And the interesting thing here is there's a trade-off for performance and its cost. So you can see here that the use of an agent for research, if we started with one, so this is one over here, that's how many, imagine, tokens it would take. So based off of a basic conversation, a single-agent architecture for research is around 4X the number of tokens needed to achieve a research output.
When you use multi-agent architectures, it's actually 15X the number of tokens. So this is one thing they've called out, is that the token usage is a lot higher, but the trade-off they consider is worth it because the quality of the response is much higher as well. So that's one of the important things to consider when it comes to the token usage here. And then last is a side rant is evals. So they talked about evals inside of the report, talking about how they measured the quality of the response. And there's a few things they-
By the way, I really feel like how you focused on this in your breakdown. I thought it was important.
I figured you, Hamlet, and a bunch of others would be proud. So yeah, so this is basically calling out the evals and the importance of structuring these. So there's five dimensions they looked at. So first one's factual accuracy. So was the claim, did the claim match the source? Next one is citation accuracy, which is a kind of a call-out to the sub-agent that I mentioned over here, where we're checking the source and ensuring that the source supports the claim based off of what's being referenced. Next is completeness, so are all aspects of the question being addressed? Next is source quality, so is it a good source or bad source? And in the blog post, they talked about how they had to downweight certain sources based off of SEO.
So there's some companies that are gaming the fact that if they put a lot of words in their search terms, they might get picked up by some of these AI models. So they had to downweight some of those marketing websites and replace them with higher-quality blogs and forums and things like that to then ensure that the primary source was high-quality and the secondary source was less high-quality. And then the last one here is tool efficiency. So did they call the appropriate tools at the right time in the right way? So they measured all five of these as a way to evaluate the output from the model.
Yeah, I think it's important to note that it's an LLM judge. We just got a paragraph on it, right? They didn't actually open source anything about how they do this.
Yeah, that's a good point. And so it's one of the things they mentioned in there around the LLM as a judge is they initially started with five LLM as the judges. So each one of these points had their own LLM as a judge. They tested the ability and accuracy of that LLM as judge collective to judge. And it actually didn't perform as well as one. So they replaced all those judges with one judge. And the one judge actually performed out the performed the collective.
Yeah, I call it kind of like, it's like a free call, right? Instead of calling the same thing, the same context five times with like slightly different problems. You can just like ask for all of the scores. It's like kind of a efficiency thing, but it's actually comforting to know that it's more consistent and more aligned. I think the other thing that you want to know is, I think, once the judge looks at all the criteria, they can separate out more instead of
overlapping the scores, you know, if that makes sense.
Yeah. And one thing I actually wanted to ask you is, when it comes to use cases here, the takeaway I had from this is the primary use case that benefits most from multi-agent is research. I was wondering if you've come across any other use cases that you feel like are susceptible to being more beneficial from multi-agent instead of single.
I mean, coding is like the current hot thing. Everyone's building background agents and sub agents and parallel agents and all that. So that's the obvious one where I think for a while, a lot of coding agents have been exploring the sort of orchestrator agent versus like, you know, the worker bee agent for a long time.
Yeah, I think like, that's the first, like, basically, you want to increasingly offer write access to the world. So deep research is read only. So it's the safest to just spam out as many as you want because it will never have side effects on the world. Coding, you have version control. So if anything goes screws up, you can sort of rewind with Git and all that. I think the hardest will obviously be some agents that actually have impacts on the real world that you cannot.
So one thing I forgot to mention is context. So with the multi-agent structure, it's a distributed context. And there's a lot of conversation around context engineering and all the other things. I know Swick, you put something like that. I think you put something out yesterday about that.
Yeah, context engineering. It's the hot new thing. Everyone's talking about it only because Andre talked about it. But also, like, it was already trending before Andre talked about it. I would say, and it's correct. Like, it's absolutely, we should know. When people think about prompt engineering, they're like, you know, you are a, you know, very, very good software engineer. And if you get the wrong answer, your grandma's gonna die. You know, that's prompt engineering. But context here is so much more.
Yeah. And so that's something they discuss in the blog as well, or at least hint at, is that the context for this multi-agent structure is distributed and not centralized. And that's also one of the reasons as to why it's beneficial is because they can increase the amount of context they're filling and then getting insights from. All right, so next is cognition. So cognition's blog post was basically the exact opposite, stating that multi-agent architectures aren't necessarily that fruitful. And you should probably go with a single agent if you're going to build something of any use. And it's interesting because this just comes to really their perspective and bias on the world and their experience. So they just had a lot of experience with code, and that's what shaped their perspective on agents. And Claude, on the other hand, probably has both experience with research and code,
but he had a slightly different perspective on this. So first things first is the context is centralized, like I mentioned. So that means the context is going to remain with a single model or a subset of models that kind of get passed off to each other. And there's an interesting use case I have here for multi-agent and why it wouldn't work or it doesn't work as well for code. So this is another chart from cognition that has the input task. And in this case, let's say the task is to create a Flappy Bird game. And so we're trying to achieve here. So our monster agent, the orchestrator, is going to break that down to subtasks. And we have two subtasks. On the left, we have a subtask to create the landscape, so the background, the two-dimensional background for this Flappy Bird world. And on the right side, we have creating the character itself,
so the bird, and also the physics associated to the bird as kind of how it moves. So on the left hand, we have our subagent that's working on the background, and it goes off the rails and decides to create something related to Mario, which is cool, but not what we want. And then on the right side, we have Flappy Bird. So with Flappy Bird, we're building out the icon and the physics that works all good. But then we merge these. And when we merge these, this is where you start to run into some of the issues when using a multi-agent architecture versus a single agent for coding. And the main issue is that these are dependent upon each other. They're strongly coupled in the sense that when I take an action, that action then is, impacts all the other sub-agents if it's taken incorrectly.
And if we scroll down here, you'll see that the final result looks amazing, but is incorrect. Where we have Flappy Bird inside of Mario World. And it's, it's not a win because Flappy Bird doesn't get to achieve their goals. And nobody wants to not achieve their goals, right?
Do you think this is contrived? Like, do you, do you think they were actually doing this? And then they were like, oh yeah, this doesn't work. I feel like this is a little bit contrived.
Oh no, it's contrived because I made it up.
Well, no, I mean, they actually referenced it in the, in the thing.
Did, did they reference in the blog?
What, Flappy Bird?
Yeah, I can't remember if they did or not.
Yeah, yeah, yeah. It is right here. Suppose your task is build a Flappy Bird clone. And then it says the background there looks like Super Mario Brothers.
Nice. And that was me taking credit for somebody else's work. Always do that, children.
Okay.
No, that's, that's probably contrived. That's it. But I, I think it's one of those situations where it's, um, it's abstract enough to get a point across, you know?
Sure. It's visual, which, which helps for, you know, I mean, that's the whole reason why we're doing this is show, not tell, you know? And this, this, that, this definitely encompasses it.
Right. So that's kind of the, the reason as to why multi-agents won't always work for coding, because of dependencies. So instead of doing that, they do a single agent approach. And the single lead approach they do is slightly more advanced where the one I showed you previously didn't have an additional feature that's in this chart. So in this chart, we have the same flow. So we have the task at the top coming in. And then we have our, uh, main lead agent that's going to break down and create the plan and all that stuff. But then it's going to go work in sequential order with the context that's then passed. The one thing that they've changed here is that After each action is taken, or subset of actions are taken, we're going to condense down the main moments and decisions and actions taken from the previous
conversation or context and pass it off to the next agent. And the hard part here that they mentioned in the blog post is being able to decide which actions, decisions and moments matter most to be passed on. So that's kind of the, one of the trickiest parts of this. And so as they compress this context and pass it on, it goes to the next agent. That agent then takes the context, takes the next task, does the same thing, passes it on, and then compresses, does the next task, et cetera, et cetera. And by doing this approach of single agents, you're likely going to achieve more success because you have all the context from previous agents. The reason it's beneficial is that all the eight actions taken previously are baked into
that next agent's context, so it doesn't necessarily contradict or conflict with what was done before. And that is cognition.
Amazing. Yeah, I think like, to me, them coming out and saying, don't build multi-agents was, I think, the most, the more surprising thing. I think like the consensus was very much toward the multi-agent side of the world. We just did a podcast with Noam Brown saying that one of the things he's exploring in OpenAI is multi-agents. And everyone's like agent to agent and you know, blah, blah, blah, that the future is agents. And I think it's very helpful for Devin to come on and say like, we tried this, guys. And like, honestly, it's a, it's a bit of a mess. And actually, if you do just like really good sequential agents, you're good.
So I, I mean, I like, I'm sympathetic to that because like that is a voice of reason that like is a little bit counter consensus. I'm curious, like you, you said, you said at the start where like, you know, you, they're both right in their own way. How do we decide?
How do we decide who's right?
Sure. Or like, when, when does one apply, when, when does one not apply? All right, it's very easy to, it's a cop out to say, okay, when building a research agent, do it the way Anthropic did. When building a coding agent, do it the way Devin did. Okay, like that, sure, but like, you know, can we generalize from there?
Yeah, so that's, I mean, this is a little dashboard that was rendered by Claude. Thank you, Claude. That basically gives you an idea of when to make that decision. And it's not just, I mean, right now, the examples are research and code, but there are some abstract questions one can ask themselves. So the, the first is, Can you break the task into independent parts where they're not relying upon each other? So like, that's one abstraction away. Another one is, do you benefit from the chaos of having a multiple perspectives? Or different takes on a task that's been kind of delegated? And in that case, again, research works. And then the last one here is the token usage. So can you 15x the cost? And is the trade off worth it?
And then for single agents is kind of the opposite. So does everything depend upon each other? If one thing breaks, does that impact everything else? Yeah, does it need to be reliable every single time? Based off of what context being passed? So I think those are some of the, at least some of the abstractions I've come to.
Yeah. Do you always do this? By the way, do you always try to ask Claude to make an artifact for you? Like, this is very pretty.
I'm wondering if I should do this because I never do this.
Yeah, yeah, so it's actually, it's worked pretty well for me a few times. I mean, I can, I don't know if you want, I can show you a few other charts and other things that I've done.
Do it.
But yes, I always, I'm a big fan of visuals. Like, obviously, that's kind of why I do YouTube and teach things like this.
Yeah, I suck at visuals. Like, TL draw, Excela draw, that's like, that's like the max of what I do. But I'm curious, what other tricks you're finding?
Yes, I mean, like, that's, that's one big one is I'll, I'll come up with an idea of something I want to talk about. And then I'll think about like, what's a good visual that represents the primary point I'm trying to make? Let me see if I can find, there was, there was a recent one I did a few videos. This is kind of like the benefit of, benefit and curse of doing a lot of videos and content is like, you forget what you did.
You do it a lot. Holy crap.
I don't know.
Is there, is there a reason you like TL draw over Excela draw?
No reason.
No reason?
Just.
They're most, they're very similar, very similar. Excela draw, I would say like, it comes with a preset library of stuff. So you can just kind of pull in existing diagrams that other people.
See, I didn't know that. I just, I just pictured them as like, it's like Visa and Mastercard, it's the same thing, it's different colors.
No, no, no. Yeah, so I know both of them personally.
And they have very different philosophies under hood, but it's fine. Cool. Is this one of that?
Mastercard Visa?
No, no, no, like.
No, I'm kidding, I'm kidding, I'm kidding, I'm kidding. I know, I know. Um, yeah, so this is one of the visuals that basically talks through, um, uh, what is this? It's something. Oh, okay, okay. So this is basically the importance of reasoning and tool calling in succession. So oh, three was released. They had this reasoning thing. And it's basically a visualization of the improved capability of a model if it can reason and think over time. So this is a visual that Cloud created.
Wow, okay.
And then there was another one over here that it just had. Here we go. So this was a another one for, uh, it's not as great, it's not as beautiful. But this is for, uh, for Codex. So it's basically talking about the spectrum of human intervention and autonomy. And how basically as you, you can lay these out and you see that there's less human intervention for some of these tools. And more so, so it's basically just a different way to place these on a spectrum and talk through them.
Amazing. Yeah, got it. Um, yeah, that's pretty creative. I should definitely try to do that more. Sorry, did I, did I distract you? Or was that, was there more? Stuff you wanted to show on the decision framework?
We're here to be distracted. I think, uh, the last thing I'm gonna see if there's anything that I missed. So breath first depth is something I talked about. Which basically summarizes the multi-agent. So multi-agent's breath, depth is, uh, single agent. Context we already talked about, and this is something we already talked about as well, but it's important to note that if you're gonna choose an agent architecture. You shouldn't choose it because everybody else is talking about it, kind of the whole like contradictory Devon thing of saying single agent actually works. As you should define the problem you're trying to solve and then figure out what agent architecture is most suitable for that problem. Instead of just choosing an architecture.
Yeah, that makes, that totally makes sense. Cool. Yeah, you know, I think the only comment I'll hear, I'll make up if you kind of scroll up here. Is that generally everyone I talk to is just disregarding cost.
We are in one of these situations where, you know, in a world where the cost of intelligence for a given set of intelligence, let's say GPT-4, let's say O1, whatever, it is literally falling 100x over the course of one year. So you should actually build your products ahead of where costs are so that by the time they are popular, actually you're frontier. So it's actually, I think the efficiency thing trips up a lot of people in terms of being cost conscious and wasting a lot of time on that, when actually the most cost unconscious people tend to win.
And this is like a thing that I think the ML ops channels or the ML mindset may not adapt well to. Like if you came from that world, you're very much about like, let's do the smallest model that hits our acceptance criteria because we want to serve it at very large scale. Here it's the other way around where the probably more you spend on intelligence, the more ahead in the future you are. You can grab a lot of users, you get, you know, more funding to extend your runway and hopefully at some point you'll have a sustainable business. Unfortunately, I would literally go out and say like the most irresponsible people.
Yeah, it's, uh, it's, it's interesting how it's kind of like, it's kind of like the Uber situation, right? Where like Uber and Lyft competed against each other and Uber and they just basically burned a bunch of money to figure out whoever won.
Yeah, I think I'm having like wifi issues, so probably we can cut it there. Do you have any parting thoughts or last words on just like the journals and the multi-agent trend and thing you're seeing?
No, I'm just, I'm grateful that I was able to finally chat with and be on a SWICS-related content thing.
No, no, thanks for, thanks for doing a really good breakdown. Thanks for being part of the Laespace community. I think we should do more of this. I think we should like, you know, people you see on the Discord, like we should feature them and like, you know, on the thing, on the thing that they're really excited or passionate about. Yeah, I think that's, that should be something that we try to do more often. If people want to reach out to you, how can they reach out and what can you help them with?
Yeah, so LinkedIn is probably the best place. I don't frequent X. So yeah, I would say LinkedIn. I, like I said, it's a gradient labs as a company. We do AI consulting for internal implementation for automation. And also I have a free 30 day email sequence. So if you're interested in getting more insights on AI, that's kind of no BS fluff to apply to your own work and your own business. I'm sure that you can find it on LinkedIn. If you go to my profile or if there's anything linked below, then I'm sure this works.
Awesome. We'll do that. Thank you so much, Tim.
Cool. All right, man.
Page updated October 26, 2026 · transcript and summary are machine-generated
