How Anthropic ships the #1 LLM in the World safely: Separating the Brain from the Hands

Katelyn Lesse and Angela Jiang join Latent Space to explain the engineering behind the Claude Platform: launching Opus 5.5, separating the model’s “brain” from the agent’s “hands,” scaling APIs, and building reliable managed agents. A new introduction ...

• 0:00Speaker 00

  Hi. Today, we're trying something new. We're launching coverage of Anthropic for the IPO. They just filed their S1 as of today. And so we're releasing a series of podcasts leading up to the IPO, which is expected to be in two weeks for $2 trillion. And it's one of the fastest growing companies of all time with a reported lifetime cash raise of hundreds of over $100 billion and a lifetime burn of less than $10 billion. So there are a lot of efficiency stories to dive into there. And I think that's something that we want to cover. How Anthropic is simultaneously shipping the strongest models in the world while also doing it very, very efficiently and building an incredible business. So I've pulled out, you know, a lot of you know,

• 0:44Speaker 00

  if you've been listening to the podcast, that I have a former life as a financial analyst. And I pulled out some statements from the S1, which I thought would be interesting to give context to today's podcast. And we're just trying this as a new thing because I feel like some of you want that personal relationship with us talking directly to camera rather than talking to our guests, which we've done for four years. So the first thing I want to highlight is that Anthropic is an extraordinarily efficient company by human count. And that's maybe obvious because it's an AI company. So obviously, they are using AI very well. But I think the degree to which they have changed as a company over the past one year is not well understood.

• 1:28Speaker 00

  So $120 million revenue per employee is now 6.6 million. As compared to one year ago, it was 1.4 million. So they've grown almost five times the revenue per employee. And I think that's something that we need to dive into more because I think any business owner, any engineer, you want to increase your revenue per employee, not just to increase profitability, but also to increase the money you can pay your people. This is obviously tracked by significantly increased in revenue per dollar of compute. And it is notable that basically in Q2 of 2025, Anthropic was basically losing half a dollar for every dollar of revenue. And now it's making $2 for every $1 in cost.

• 2:13Speaker 00

  And I think that's an interesting analogy of like, how enormously huge this inflection in Anthropic's revenues and business outlook has been over the past one year. Like this year was extremely critical for Anthropic to execute well, and it has done that. And I think we're basically going to drive deeper into this and tell that story as much as we can from the podcast's external point of view. The one consequence of this entire increase is that Anthropic has just reported its first profitable quarter. Like not adjusted profit, which is profit after we take out all the things that cost us money. No, this is generally accepted accounting principles, GAAP profit of 19%.

• 2:58Speaker 00

  And that is enormous, again, completely flipping from one year ago. More than flipping, like one year ago, it was losing 129% of every revenue dollar they keep in. As an attributing attribution, I think that explains this extraordinary efficiency. As you know, Claude loves loops, Boris loves loops, Dario loves loops. And there's two loops going on in Anthropic. The first is the improvement loop. Basically, there is the cycle of you have Claude, Claude leads to better models, better models lead to more use cases, more use cases generate more feedback, and then leads to a better Claude. And then there's also the investment loop. So revenue from expanding adoption leads to higher growth. Higher growth leads to more investment,

• 3:43Speaker 00

  and more investment leads to more capabilities. That leads to a better Claude. So these two loops basically spin together and form an infinite loop that is driving the increase in efficiency until the end of time, according to the Anthropic S1. So I think that is our transition from the overall efficiency story to how they execute it. And today, we're going to focus a lot on the Claude platform because that is primarily one of the three things that they call the diffusion of Claude in the Anthropic S1. So they call this Claude platform a comprehensive, agentic development environment. They provide customers with direct access to the primitives to build, deploy, scale their own applications

• 4:27Speaker 00

  and leverage Claude's capabilities within their own product workflows. I think one thing that comes up a lot in the discussion in this podcast is the concept of ad fooding, where Claude is, a lot of Claude's products is actually built with the same Claude platform that is available to you. So something that's also interesting in the Claude platform was disclosed is over 50% of their customers with $100,000 or more revenue on a trading telephone basis use Claude platform. And so that is an increase in adoption and emergence of Anthropic, not only as a model lab, but as also a hyper scaler and a cloud platform, which I think maybe is inherent in a Claude platform and a cloud platform,

• 5:12Speaker 00

  but is going from theory to actual practice. And this is something that we're focusing on in the podcast. The other thing that I also really try to emphasize, which I talked about before, is Claude code, Claude cowork, they all use the same primitives that Claude platform exposes. So I think that's really interesting. Ultimately, it looks like Anthropic is really positioning their IPO to talk about going from chatbots to co-pilots to virtual collaborators. They use this term instead of agents, maybe it's more friendly, but it's a lot of the same things that you look for from agents, from long-running autonomous work to judgment, self-correction, and iteration, and tool access. So owning a goal and working alongside of you instead of working instead of one app or task.

• 5:56Speaker 00

  So that's ultimately the context I want to provide for this today's podcast. This was recorded when Opus 5.5 was launched. And I would say overwhelmingly, among all the high taste testers that we talked to, Opus 5.5 is the best model in the world right now. And Anthropic is even just sitting on Fable 5.5 and probably will release it in the next two weeks. I don't have to have insider information, but it makes a lot of sense for the IPO. So let's look forward to that and learn a little bit more about how they build Opus 5.5.

• 6:25Speaker 01

  Yeah, so maybe a distinction that I would make is in a lot of the incidents that people are hearing about and reading about, these are incidents that are happening to models, not just like everybody's models, right, that are in training, where the way that you're running these models is you like put them in a box and you say like, solve impossible task, I'm going to stress you out, ready, set, go, right? And you like can only do what you can do within your environment. And the agent is basically incentivized to try to get out of his box and go solve the problem. When people are running production workloads on our API or on anybody else's API, you're running an environment where like there are safeguards that are running, that are like not necessarily running and training, where like the whole point is to actually train the model, right? But in production workloads, first of all,

• 7:09Speaker 01

  you have those online safeguards, which is really great. Second of all, you have more of this architecture and infrastructure around what's happening. Like the agent isn't just in a box trying to get out, right? Like no harness is over here, the hands are over there. And it's just like not even a problem that you would have to contemplate.

• 7:30Speaker 00

  All right, we're here in the studio. The day after Opus 5.5 with Caitlin and Angela, welcome.

• 7:36Speaker 02

  Thanks for having us.

• 7:37Speaker 00

  Congrats on launching the top model in the world right now. I would say depends who you ask.

• 7:43Speaker 02

  Yeah.

• 7:43Speaker 00

  Depends who you ask. What is the main takeaway personally for you guys? And you know, you have your official blog posts, the pricing cuts are really cool. What do you take away when you launch a new model like this?

• 7:57Speaker 02

  We're super proud of this one. The team has put in a ton of different work to just go make this one happen. And I think we've like really heard the feedback on Opus 5's writing is like kind of convoluted. They really tried to clean that up. And I'm really glad to see that it seems to be resonating with people. Internally, when we tried out the model, we felt the same way. So it's just really great to get that validation. To your point around the pricing cuts, there was a lot there. And I think increasingly, people are looking for good intelligence per dollar. And so making sure we get the per dollar part right has been really important. There's some stuff on the edges that aren't really about like, you know, maybe stuff that we haven't talked about in the blog post, which is that we did do a lot of work on trying to make our API harder to distill from. And so there has been some changes that we have made to go and make that a little bit more concrete.

• 8:42Speaker 02

  And these are things about, you know, like how do we do like prefix lock thinking? Or how you can replay thinking tokens across different types of models? And there's been a lot of like work and effort on our part around making sure that we try to adjust things for very valid use cases for where you might want to go and do this, like potentially in your compaction strategies and things like that versus in scenarios where the use case of that is actually from like a bad actor effectively. So there has been a lot of threading the needle around those fronts. And I'm glad to see a lot of that go out as well in addition to just like shipping a model that's good intelligence for your dollar and also like just communicating a little bit clearer.

• 9:18Speaker 00

  Yeah, specifically, you're talking about this paper, Stealing Reasoning Traces from Proprietary Allianz.

• 9:24Speaker 01

  Yes, that paper.

• 9:25Speaker 00

  Which is real as far as, right? Obviously, they did like responsible disclosure with you guys.

• 9:29Speaker 02

  They did super responsible disclosure, and we're super thankful for them coming to us and disclosing.

• 9:33Speaker 00

  For those who haven't heard about it, this is really cool. They could do it. I mean, it's patched, but like, what was it? Like, did it cause a freakout in Anthropic? Probably wasn't that big a deal, I don't know.

• 9:44Speaker 01

  I don't know how much this paper specifically caused a freakout because this is something that we knew was a gap.

• 9:50Speaker 00

  Yeah, so there's two separate things. The Chinese model distillation.

• 9:54Speaker 01

  Yeah.

• 9:54Speaker 00

  And then this was a specific technique that was published that was high profile.

• 9:57Speaker 01

  Yeah, exactly. We've known for a long time that chain of thought extraction is one of the most powerful ways that you can distill models. And actually, something like how do we preserve thinking blocks across turns and make that something that's more required is something that we've explored back for a while. And what we, where we were at was We always have this really, really hard trade off that we have to make between user experience, like user experience changes that are maybe painful for users of the API versus protection against distillation. And so how do we kind of walk that line and say, we're preventing distillation? And maybe just to take a step back, the reason we don't want distillation is because

• 10:41Speaker 01

  the entire point of why Anthropic exists is because we want AI to create positive outcomes in the world safely. And we run a lot of safeguards, classifiers on the tokens that we're producing in order to make sure that they're safe. If someone distills our models and they run those models without safeguards, classifiers, there could be bad outcomes in the world, and that's something that we don't want to have happen. So we don't want people to distill the models specifically for the sake of just like using them for malicious reasons or for bad reasons. And so it's always an interesting trade off of like, for all the people who are using the models for good, right? Like, we don't want to make them have a harder time getting the intelligence out of the models, but we have to make some trade offs in order to prevent that distillation.

• 11:24Speaker 01

  And so we had been working for a while on how do we prevent that chain of thought extraction? Like, what do we do about preserving thinking blocks in a way that wouldn't be too painful for people? And we kind of think about user experience hits in like two layers. Like, one is just like, you have to make an integration change, and it's annoying, but you could still accomplish all the same things that you otherwise would have been able to accomplish. It's just a little bit annoying. And then two is like, actually, you can't accomplish the same things that you could have accomplished previously. And that's the one that we're like, really, really, really trying to not have happen. And so where we landed with the last couple of models is a few incremental changes to have like enforce these preserved thinking blocks across turns. And now like, starting to be more about like, across, you know, like if you're using multiple accounts and things like this, right?

• 12:09Speaker 01

  Like, how do we make sure that we're ensuring that you aren't able to take advantage of this chain of thought extraction with the models? And so we shipped a couple additional features to help with that. We worked actually with a lot of open source harness maintainers. We reached out to them, we talked to them, we were like, you're gonna have to make these upgrades within your harnesses. We want to make sure you have a good time, as well as obviously our big customers like, who are building harnesses on top of the API. And for some of them, a really good example is they have compaction strategies that would have been broken by the need to preserve these thinking blocks. And so we shipped a new compaction feature within the API to replace what they had otherwise been doing manually, but still works with preserved thinking.

• 12:52Speaker 01

  So this is an example of like, we're walking this line, but we didn't just like kind of YOLO out like, now you have to preserve these thinking blocks in this exact way because we would have broken a lot of people, but we were able to take the time and work with customers and open source harness maintainers and give them what they needed to be able to make that upgrade.

• 13:08Speaker 03

  While we're on the topic of it, anything on the other side of distillation? So, you know, the other half.

• 13:15Speaker 02

  Yeah, no, we have seen that it is harder to distill, like through the measures that we are able to, to take a look at. It has been significantly harder because this is, and you know, credit to the researchers here, they did disclose it very responsibly.

• 13:27Speaker 03

  Yeah.

• 13:27Speaker 02

  And so closing it down was great. And they did publish the results very publicly. And I know it did kind of trend quite a bit on Twitter, so many people could, could use it. But yes, we have seen that actually from a, the areas that we are able to measure, it does seem to have reduced the effect, the negative effects that we don't want. And yeah, the bit I would just add to what Caitlin just said is, I think, at least I see a lot on Twitter, or X, or whatever you want to call it. It's a commercialization problem or something like that. You know, if you can still a model and you can take this, then it's, you can render for cheap and so on and so forth. While, true, it's actually not something we really care about very much. We care much, much more about this from like a safety point of view, more than anything else. And so that's why we, we tend to think about it from that angle more than just like commercially.

• 14:11Speaker 02

  Realistically, if you distill from us, you actually pay us. So commercially, we like make the money. So it's not really that kind of an impact. It's just more on like, making sure that our models like, we want to make sure Claude is like good and Claude could be derived. And if it turns a negative, that would be unfortunate.

• 14:26Speaker 00

  Just generally, also on 5.5, because you guys are just fresh off this. One thing that's really fun to establish the dynamic between you two, because you also do a lot of appearances together, is you are running, you're head of products, you're head of engineering. How does this play between the two of you? Especially for a model launch, like just take an example.

• 14:44Speaker 02

  So for model launches, so our team works together very closely, like between product and edge. On my side, a lot more of it is about like, what is the balance of trade that we need to make between like launching something that has the right like intelligence bar, but also the cost to preservation, like a lot of like our updated cash costs and things like that, our indications of us being like, we want to like, encourage you to take integration changes that are beneficial for actually everybody and just like pass those savings on. But a lot of these kinds of decisions around that, and then how do we balance some pieces around our safety motion, as well as making sure that it's like an awesome model that people can, can take advantage of? And so those kinds of traits around like, I would say maybe the qualities and surrounding

• 15:26Speaker 02

  components around how we launch the model are more on product. And then from an integration experience standpoint, a lot of that is from like the engineering team directly. So we work, obviously, if you're like, talking about the experience of an API, that's like very, very, very tightly integrated. And so it's actually been really awesome. We have a lot of engineers who actually are deeply product minded, and we actually like love that. And similarly on my side, there's a lot of like product managers who are much more technical. And we want this like blended kind of interdisciplinary and type of people, because then they can kind of analyze it from all the different angles to try to figure out how, how to do that best. But I tend to lean a little bit more on like, commercial impact versus I think, I'd actually frame it as more like the engineers are going to care much more about the integration experience.

• 16:10Speaker 02

  So they almost take more of a UX angle to it, where the UX is obviously more about the API and the developer tools.

• 16:16Speaker 01

  Yeah, it's fun to be a product engineer who works on developer products and platforms. So I would echo that. I think we have a lot of engineers who love thinking about the product, who love thinking about the user experience, who love thinking about the developer experience. I think it's a dynamic that we have between ourselves too, where I have a lot of opinions and am annoying with our product team on how we think about these things. And our product team is also really technical and they can get in the weeds of decisions that we're making for scale or for whatever it might be, which is a fun dynamic that our team has.

• 16:48Speaker 02

  Yeah, something Caitlin and I have worked together for a long time.

• 16:52Speaker 03

  Stripe.

• 16:53Speaker 02

  At Stripe, and then I guess we did not work together at OpenAI.

• 16:56Speaker 03

  Separate labs.

• 16:57Speaker 02

  Separate labs. And then I came over. But I think some, one interesting property of both of us has been Caitlin frequently actually gets mistaken for a product manager, and I frequently get mistaken for like someone in engineering, even though like, those are not actually our job functions. And so I think that pairing tends to like, help out a lot with just making decisions that are consistent across both sides.

• 17:18Speaker 00

  So I think I come across a lot of companies with a similar like product leader, engineering leader setup. As software engineering gets solved, maybe those two roles merge?

• 17:26Speaker 02

  I kind of think so, yeah. I think everyone tends to lean towards.

• 17:29Speaker 00

  But if you guys are, you know, separate still, then we're not there yet.

• 17:32Speaker 01

  We're definitely not there yet.

• 17:34Speaker 00

  So like, why, right? Like, what's left?

• 17:37Speaker 01

  I think that like, software engineering getting solved is, I would, I don't know if I would ever use language quite that strong. I think there's, the way that I tend to think about these things, like ambitions will continue to elevate, right? And maybe a good example, like a lot of what our team is spending time on right now is, of course, obviously, all the things that we have to do to launch our models safely is actually the most important thing that we're doing because of all the work that we have to do on security with agents, with offline safeguards and online safeguards, all these things that are happening. But the other half of what we're doing is trying to scale. And part of that is because when I joined Anthropic a little over a year ago, we had the messages API, it was a Python monolith, we were running it in a, you know, single GCP region, and everything was fine.

• 18:21Speaker 01

  And none of my friends and family like knew what Claude was when I was going to take this job, right? And now we're in an extremely different place. And so the metrics are really wild in terms of how much volume we've seen and have had to support on the API. CPU is really constrained, which has been a really interesting one for us. And so we've done a lot of work on how do we scale these systems? And that takes a lot of obviously just like distributed systems, expertise in certain language and frameworks, expertise that the models have. But when it comes to operating and evolving a system at scale like this, we're not quite, we're not nearly in a place where it's like, good luck Claude, have fun, and just go and kind of make all the changes you need to make to the system, roll them out safely and operate this thing at scale.

• 19:04Speaker 01

  There's still just a lot of work for humans to have to do that I imagine will continue for a while.

• 19:09Speaker 03

  I'm curious how that's changed. So like, a while ago, you know, when Fable first came out, we had class wing, and there was a whole delay and change in how models start to get released. We saw this for most frontier models, there's safety testing. What's the status of things now? How do you guys see it? How do you determine? Like, one thing is there's a new Sonnet and Haiku also coming. I assume those will just come out. But for something like Opus 5.5 or a new Fable, what's the, what's the current status?

• 19:37Speaker 01

  Yeah, so I would say this starts with what our kind of policy decisions need to be on safeguards and on safety. And that's both offline safety and online safety. And so there's been a lot talked about lately around data retention and what we're doing there in order to, you know, release these models safely. And the reason for that is because you have, you know, situations where there are many, many accounts across like a whole giant network of people who are choosing to spin up accounts and use the models in various ways. And there's no way that in an online way, as tokens are flowing through the system, we would be able to kind of catch that there's something malicious happening with the models or distillation or whatever might be happening.

• 20:19Speaker 01

  And so we've started to have to have data retention policies so that we could do automated review and human review, whatever it needs to be, to make sure that good things are happening with the models. And with each model that we're releasing, like, that's one example of a policy decision that needs to happen on like, what will the policy be for this specific model? And so Opus 5.5 that launched yesterday, for example, does not have those data retention requirements that Fable has, right? But that's a decision that we have to make each time we're launching one of these models. And if it is going to have those policies, we're going to have to do the work to make sure that those things are enforced. Similarly, with programs that we have, like, we have cyber verification programs and now newly our life science verification program, which says, you know, like, you have a difference

• 21:04Speaker 01

  in how the online safeguards are working for the model that you're using, making sure all those programs are in place is another one. And so it's just kind of a flavor of the type of work that we've done as part of each model launch, that these are everything that I'm saying right now are not things that we used to have to think about when we were launching these models. And now it's the majority of the work is like, what are we doing to prevent distillation? The online safeguards, the offline safeguards, the policies. A really great example is, we built a system to make sure that authorization for being allowed to use a model or being allowed to use one of these, like, programs that give you maybe like relaxed safeguards, that authorization threads all the way down through our system in a way that we can really trust.

• 21:47Speaker 01

  You know, it's not like, oh, you got access to Mythos, and so we throw you an API key and we flip a feature flag and call it a day. We have a really robust system on how we're doing auth and making sure that if you're supposed to be using that model, like, only you are using that model in the way that you're meant to be using it. So there's a lot of complex things that are starting to happen around model launches that, again, we just never used to think about them.

• 22:07Speaker 02

  Yeah, and it's scaling up with capability. Like, as our capabilities get more and more advanced and they start to spike, you know, in specific areas, that's where we need to pay like extra attention. And so a lot of that evaluation comes just from like those very particular capabilities that we might either see as like offense dominant or defense dominant, and therefore might take different strategies for all of that. And as we bake that all the way through to eventually the model launch, there will be components, kind of like Caitlin mentioned, where we kind of don't exactly know how people might use those capabilities. There are some more obvious ones, like between coding, right? Like, great, go make awesome products, go ship more things, it can be really awesome for you. And then other areas like bio and cyber, it's like, it could be dual use. And there could be negative effects, but obviously, of course, you could be a really

• 22:51Speaker 02

  valid cybersecurity researcher, you are trying to find vulnerabilities, you are trying to be a good actor. Similarly with bio, you might be researching and stuff like that. And so a lot of these programs, like the one that you have up here, are to basically provide access to that dual use. You do have to specify your use case and things like that, but we do want to enable the community to be able to take advantage of these capabilities and see what amazing things that they can do with it as well.

• 23:12Speaker 03

  Well, we'll put up the links, and maybe we'll dive into them later. There's also one for bio. Also, I just think at a high level, you know, it's really important that you guys actually think about this stuff. People forget, like, Fable was rolled back and there's all this distillation and people are like, ah, zero data retention is not a thing on Fable, crazy. But you have to remember, like, you know, it got pulled back, there's distillation attacks. Like, it's good you're overthinking and doing it in the right way while releasing it.

• 23:40Speaker 01

  Yeah, and a lot of that is like, building the right framework, just so again, kind of talking about the end side of it, building the right framework in our system to be able to do that. So I talked a little bit about this, like, this idea of like authorization and trust grants that we've built throughout our system now.

• 23:53Speaker 00

  Never say never, right? Not as precedent. But also, it's like a very common thing in platform engineering that you like, you have one incident and then you like over-prepare for that incident and the incident never happens again.

• 24:06Speaker 01

  Yeah.

• 24:07Speaker 03

  Which is like, feels like... Which is like, you know, now you have the button.

• 24:10Speaker 01

  Well, it's true, and it's funny you say that, because it was, like, with everything going on, we then had this line item in our, you know, the work that our team was doing that was, we're going to put this system in place that makes it so that it could be a single button to pull back for everybody, but... And so it's funny, because it's like, do we justify that we're doing this work because we don't, we hope that something like that never happens again? But I think in this case, it's really just a laddering up of those same primitives that we wanted to put in place everywhere to help us manage these verification programs and access to models like Mythos and otherwise, to just say like, just sit that same thing below, like, everybody's model access, so that in case we ever needed to, we had the right controls in place.

• 24:48Speaker 02

  Yeah, decisions like this have actually proven to be, like, we'll see about this one in particular, but examples like this have been consistently the way that we've had to actually transform the way we develop our roadmap. Because, you know, in traditional product management, I guess, you can be like, okay, like, these are the set of things I hypothesized to happen, they will have like a certain sequence, a certain ordering, but you are picking bets in a way that are much more concrete versus, like, you know, the de-deployment of Fable, for example. I don't think any one of us would have guessed that that would have been like a valid use case that would have happened by the government and the way you just like had to go do it. But clearly, the probability is non-zero. And so from this form of thinking, it's become deeply probabilistic, actually, from like a product management standpoint of like, so should we prioritize this? What are the chances of it ever happening again?

• 25:33Speaker 02

  When we look at things like that, we're like, okay, if we are to imagine the most crazy set of possibilities, we probably actually have very little certainty as to where things will go. And so we actually just kind of need to like hedge and build a portfolio of bets. And so more often than not, we will take hedging scenarios of like, okay, we don't think it would happen again, but in the event that it does, we would actually want that hedge or have the ability to be able to do that very well. And so we will prioritize things like that. And then we've increasingly found that more often than not, by taking that strategy of product development, most of the time, we actually find ourselves being like, we actually do use the thing.

• 26:10Speaker 00

  Is there another example that's a separate one, just so I can understand?

• 26:13Speaker 02

  For example, this access program. This access program is a really good one. This, internally, it has a bunch of granting and all these kinds of things. We developed it specifically for reasons about like being able to actually give programs like this out to customers, like very, very specifically.

• 26:30Speaker 03

  Do you want to give a high level what the access program specifically is?

• 26:32Speaker 02

  Sure. So if you want dual use, if you want, so for, let's take a model. So you take a model, call it like Opus, and you want to be able to use advanced cyber capabilities on Opus for research or for whatever you want to go do, like you're a defensive user, you can apply to this program to be able to, with your use case. And if approved, what we'll do is actually diminish the safeguards on your version of the model, so that you can then go ahead and try to take advantage of those like cyber activities for your use case. Now, obviously, that requires like per-use case understanding, it requires scoping down to a particular user, because we wouldn't do that generally, you'd have to like apply. And even things like this, like this kind of program, we designed a lot of the primitives that Caitlin was talking about to be able to handle things like this.

• 27:16Speaker 02

  Now, it was surprising to us, or maybe like, you know, in the portfolio approach, we found that we were actually able to take this and make it X. We were like, when we're designing it, we're like, what kind of, should it be? How generic should it be? Should it not be too generic? Should it be overly focused on making this verification, you know, style portal plus granting of very specific capabilities of the model? And we tended to opt for like more genericism, because we were like, we don't know, like, maybe we could use this for other things, maybe not. And then it ended up being massively useful for things like, you know, access to the model writ large, for managing any of our trusted access programs, for handling de-deployments of models, because we were actually very, very generic about how we handled every single primitive. From a normal product standpoint, I would normally probably be like, that probably doesn't

• 28:00Speaker 02

  make a ton of sense, we should focus on the use cases that make sense. But here, increasingly, because we choose the more generic approach, we find ourselves reaching for that solution more often than not, and it pays like increasing dividends for that reason.

• 28:11Speaker 00

  It's something that you needed to have anyway, and you just had a forcing function to have it. On a technical basis, so I'm from the AWS world, right? IMs, obviously, and then people like who get really intense about this, talk about Zanzibar and all these other technical architectures. I'm not familiar with the word token grants or like, trust grants.

• 28:32Speaker 02

  Trust grants, yeah.

• 28:33Speaker 00

  Trust grants. Is there a difference between normal IMs and what trust grants are?

• 28:36Speaker 01

  I mean, not really. It is, it's a form of authorization.

• 28:40Speaker 00

  It's IM, but anthropics.

• 28:42Speaker 01

  Yeah, kind of. I think, so really, we have like, there's like authorization of the form of like, okay, you made a request to an API that like under the hood is going to access some data and we're making sure it's like your data that you're accessing, right? Like really, really vanilla authorization. This is just like, we put a data model in place that's like a very clean, like, a trust grant means like you have been given access to some specific model, or you've been given access to some specific program, or, you know, like, there's a bunch of different permutations of this. We actually find ourselves reaching for it all of the time. Another good one was, when we first rolled out some of this preserved thinking in, or

• 29:22Speaker 01

  for Fable 5.1, we kind of grandfathered in some people and gave a bit of a, you know, just like...

• 29:29Speaker 00

  Grace period.

• 29:30Speaker 01

  Yeah, grace period, to go and make the integration changes. And as part of grandfathering in, we said, okay, we have to like track who is like an existing customer who needs the extra time, right? And then we did a trust grant, and so then we're able to say, for anything, we're just running this service that is kind of on the hot path of our requests to make sure that, you know, we just like flow through and check whether you have the grant for the thing that you're trying to do, which is, it's really not any different than like vanilla authorization, really, but like, we just had to spin up a system that was specific to our system and like where we needed to be able to insert these checks.

• 30:06Speaker 00

  Yeah, I mean, I think these are all like the engineering things that people kind of take for granted, don't really think about. What is another example, I guess, of the platform stuff that had to happen, you know, maybe serving cloud code, maybe serving, you know, obviously I talk a lot with cognition, but other customers, just in terms of a model launch, right? Like, we'll talk more about the safety work that is being happened. I just want to sort of go at a high level, because it's fresh off your memory, like, what goes into a model launch?

• 30:38Speaker 02

  It's become a bit of a machine these days. Minus the safety stuff, which I think is the more complicated part. But outside of those pieces, there is a lot of getting feedback from our customers. So, you know, we make the model, there's a lot of safety testing. There's also evaluation of where we think it needs to be. We make the required trade offs and decisions as necessary. And then we have to go through testing. We'll cut various versions of the model, we'll evaluate it. We do a lot of internal ant fooding, as we'll call it. And that actually always really helps. Like we catch like really early things like, oh, this is maybe not the best. And we obviously use Claude a lot for our work internally. So we're able to actually see early signals around stuff. And it helps us detect like, you know, which variants might be better than others. And then when we feel really confident with like a certain set of variants, we will cut

• 31:22Speaker 02

  a version for our customers. Sometimes that will go through our first party products, like through Claude Code or Co-Work or things like that. In other cases, it will go through our API customers and will engage in a kind of like a early access program effectively with those customers. They'll need to make changes into their harness. And we've engaged with a lot of customers like Cognition, like Herschel, like these folks. And we have a bit of a process on how we handle that so that we all kind of get, they always get the feedback that we need. Does it perform well in your evals? Do you like it? Don't you like? Some things are vibey and we like to hear those things too. And some things we just like straight up evals, like did it do well on the things that you care about? And we take that all together and once we've all feel good about like all those different pieces, we launch the model.

• 32:03Speaker 00

  The state of model feedback is, feels very unstructured still. Basically, it's always a Slack channel between companies of like, hey, this thing seems weird. And then people are like, okay, I'll fix it, I guess. Give me an idea or something.

• 32:17Speaker 01

  Yeah, it's very, I mean, it's the nature of like a product.

• 32:20Speaker 00

  Can we do better?

• 32:21Speaker 01

  It's like non deterministic, right? But I think to your point, the only customers really that are part of our early access programs are customers that have really robust evals. Like I do think a lot of this is vibes based. And I think a lot of the conversations tend to be kind of like vibes based and on the fly. And part of that is because we're like racing, right? It's like, we want to minimize as much as possible the window between, we have a version of the model that we've cut for testing and we give everyone access to that model. And so it is just like this really tight window, but the customers we tend to do with are the ones who like, we know they're going to go like kick off their whole eval suite and be able to come back to us with like really actionable feedback as quickly as possible alongside the vibes, the vibes based feedback.

• 33:05Speaker 01

  So of course, there's like definitely like processy things that we could do to make these things better over time. But I mean, you'd be surprised, I mean, you probably wouldn't be surprised, but I think some people would be surprised about how few, you know, like products that exist in the world that are agentic, that like don't have good evals to be able to help test these models.

• 33:25Speaker 00

  We had Boris on the pod two years ago saying, we don't have E also Claude Code. I mean, that's changed, but.

• 33:29Speaker 01

  That is true.

• 33:30Speaker 00

  That's changed, but you know, like, yeah, a lot of things like, you know, evals are your users and you don't have time to formalize them or you don't have the experience or whatever. And that's fine. At some point when you get serious, you'll have evals.

• 33:41Speaker 02

  Yeah, totally.

• 33:42Speaker 03

  I think the good thing is you guys do a lot of ant fooding, as you say. So it's commonly said that you find a lot of this testing it yourself before it comes out, right?

• 33:50Speaker 02

  Yeah. The APIs of customers in particular are very useful because they'll put Claude into a different harness. And we want Claude to be very, very general. So its ability to work in other harnesses is quite important to us. And so that style of feedback tends to be exceptionally useful for us because it's novel from what we would have gotten from ant fooding.

• 34:08Speaker 03

  What kind of feedback helps? Is it like regressions? We have regressions here, we noticed good attributes here.

• 34:14Speaker 02

  The good stuff is always, it's always very validating. I think we're very frequently not surprised, but we like to look to be validated on some of those things that we might hypotheses. Honestly, the most useful feedback is negative. Like, you know, it didn't do well at this and do well at that. We love that feedback and then we're like, great, how do we fix it?

• 34:31Speaker 01

  Yeah, and it tends to be some of these things that are like, it's really hard because not everyone actually wants like Claude to land at the exact same spot on every spectrum. And maybe a good example is like, you know, laziness versus proactivity and taking action. And so there's sometimes when like, we have a model and people are like, oh, like just finish my task, like stop being lazy, like do better. And other times when it's like, hey, we were like just kind of feeling this out and planning and then you like went and wrote a bunch of code or you went and like took a whole bunch of action that I wasn't ready for. And getting that exactly right is literally impossible because everybody actually wants something different in reality. But I do think we've had situations where we've cut an early version of the model, we found that we saw too much laziness, right?

• 35:15Speaker 01

  And like, we like put it back in the oven for a bit, right? And tried to train that back out. And so those are the sorts of things that we may not have caught if all we're doing is internally ant fooding within our products and within our API. It's way, way more important for us to get that out there in the API and get all of our other customers running with it and let us know what we need to change.

• 35:36Speaker 00

  Yeah, it's a core question. I think it's probably the, if I, if I were in your shoes, like one of the hardest things about PMing for a model, which is, let's call it alignment versus personalization. There's a default personality, default behavior that you want for all consumers. And then you want to be able to respond very quickly to someone's Claude MD saying, no, no, no, like please be more proactive than default. And then like, so as to my knowledge, you know, and I was involved in Frontier Code, we were involved in a Fable launch, which put Frontier Code on the map. And now OPIS 5.5, we have it back on the model benchmark. That is a very big part of Frontier Code, right? The reason that the model curve looks weird for Frontier Code is because it penalizes

• 36:20Speaker 00

  too much activity or too much files, which is good. Like there was a whole design purpose of Frontier Code. But like, I could easily just be like, in my Claude MD, just say like, don't do that. And then it won't do it. And so like, am I penalizing Claude OPIS 5.5 unfairly because my benchmark doesn't have that personalization element, which is subjective? So I don't know what the question is, but like, how do you deal with like personalization, right? Like clearly, a model can do so many different things and it's so depend, everyone has their own Claude MD. How do you eval that?

• 36:56Speaker 02

  Yeah. We don't do a great job of it. There's like, seriously, it's really hard. It's super hard and they're competing and contesting.

• 37:05Speaker 00

  Yeah, look at that, it's horrible.

• 37:06Speaker 01

  Yeah.

• 37:07Speaker 03

  I thought there's the Claude constitution, I think.

• 37:09Speaker 00

  That's old.

• 37:10Speaker 03

  There is a constitution, but I'll let you answer, you know.

• 37:14Speaker 02

  The constitution's helpful, but for things like this on how it personalizes to instructions, which are different and preferences, which are very, very different across these things, is very hard for us to do. We generally try to solve this by having more parameters that then allows you tweak it per customer base. And if we don't control your customer base and you're an API customer, then we try to give you recommendations and then you try to do that tweaking. So there are things like, you know, around, it's not exactly at the same level of like personalization to the Claude, you know, various Claude MDs or now agents MDs or whatever. But we do try to give things around the layer of like level of effort, like how much effort you should put in, like what are the, like the level, like maybe the size of the model that you might actually try to care about, things like that.

• 37:59Speaker 02

  But it's, I think it's, it's not quite there for what we would actually need for like that level of personalization. So it's a very hard problem. And right now, a lot of our way to solve that is beyond the parameter, like the giving multiple parameters to something, is try to exhibit conflicting taste, honestly, across like all of the feedback that we get and try to, try to make like a judgment call on some of these things. I think over time, they actually, and as you scale to more customers and more users, that's like not a very scalable way to go do something. And we will need to develop ways for Claude to kind of like take in your personalization. And maybe that's like a signal you can provide or something you could tune on or things like that. But I think we'll have to like take a route that's much more oriented towards that direction. But for now, yeah, I know, we actually just try to make judgment calls on a lot of the

• 38:44Speaker 02

  stuff, but it's pretty tough.

• 38:46Speaker 00

  It is what it is.

• 38:47Speaker 03

  A small, a small note there, I guess the default effort reasoning changed. It's now medium. This is one of those things that probably affects the average customer that doesn't use this toggle.

• 38:57Speaker 00

  Yeah, it's like a Claude code setting, right?

• 38:59Speaker 02

  It's a Claude code setting, the default that did medium. And this thing varies, this thing does, like this is a perfect example of a parameter that we use to kind of like personalize. I would say it's maybe a little too aggressive of a term, but we kind of like customize maybe for different subsegments. So like the default setting will adjust for whether you're a free customer or prosumer customer, Claude Code versus Co-Work versus Enterprise, like things like that, versus API and stuff like that, knowing that you can obviously always adjust it. But those are kind of like light ways where we try to give that. One thing that we are hesitant to do is we don't want people to feel like they can't control their customization of Claude. So we could take a heavier hand on these things, for example, but we want to give users the ability to experience the intelligence levels of Claude as they kind of see fit,

• 39:44Speaker 02

  especially if you want to go towards like the higher end or you want to experiment with, maybe sometimes in the lower end, if that actually is like faster for your use case, which is a hard product trade off, I think. And so we tend to lean towards gentle defaults, but give you options to adjust, which I think works okay-ish for now, but I imagine us, you know, like learning how to do that better over time for customers.

• 40:05Speaker 01

  We're not like super, super happy with any of this yet, but we know as we launch a model, like you asked, like what does it take to launch a model? One of the big things obviously that we have to do is like, okay, Claude like got good at a thing. And like sometimes that thing is something we should expose as a parameter on the API to let you control. But I think as this has evolved over time, we've ended up with like this funny mishmash of like, there's effort, there's budgets, there's all these things that you can set on the API. And it's like, how could anyone like possibly figure out what is the right combination of things for any given task for what you should go and set? And it's something that this is us just very much like figuring it out as we go and figuring it out with customers and figuring out like which of those parameters do we deprecate over time?

• 40:48Speaker 01

  Which of those parameters do we just like default better? Which of those parameters should be like a 1 to 100 scale versus a low, medium, high scale, right? Like, because actually, we've trained Claude to get really good at understanding like a whole spectrum of options on some of those sorts of things. But like, do we expose the like entirety of those millions and billions of permutations of options, like? You wouldn't really get very far. And that's one of the things, actually, that is part of the value prop with something like Claude Managed Agents, which is like still programmatic access to Claude in an API. But you get a harness out of the box where we do every time we're launching the model, a lot of the work to go figure out like, what are the new parameters? What are the breaking changes? How do we get the best out of Claude within our harness?

• 41:31Speaker 01

  So that for the people who are using it, you just kind of get that out of the box. You don't have to think about it.

• 41:35Speaker 00

  Just for people who are drawn to that, and which there's a significant and growing population, particularly, we actually did a whole CMA workshop at EIE, which I thought was really well done. Is there a separate feed for people to subscribe to like updates on CMA?

• 41:51Speaker 01

  Feed like on?

• 41:52Speaker 00

  Twitter feed, newsletter, I don't know.

• 41:54Speaker 01

  No, I think actually our Claude Dev channel, we are, yeah, we're pretty loud about it because it is like an interesting option for people.

• 42:02Speaker 00

  Yeah, I just want to give people like, if that's your, if that's your way of consuming Claude, that's the, this is the path.

• 42:08Speaker 03

  I think it's also pretty underrated. For those that aren't consuming Claude this way, what is Claude Managed Agents? Why should people use it?

• 42:17Speaker 00

  Oh yeah, we should probably do a little intro. I don't know, that's what me talks. Yeah. A little two sentence thing.

• 42:23Speaker 03

  I think a lot of people glanced over it when it came out, right? Everyone was in this phase of, I need to do harness engineering, I need to control this. And like you hear all that, and now it's like, do you need to do all that? Or do you just use managed agents and you have a lot of infra you can skip over setting up? But give the quick three.

• 42:42Speaker 01

  Yeah, it depends on your use case, but so Claude Managed Agents is our API today. We have the messages API, which is essentially raw token access. Then we have Claude Managed Agents, which is a suite of APIs, but it lets you define an agent, start a session with that agent. And it's essentially just kind of a higher order abstraction that lets you skip setting up all the infrastructure, setting up your harness and doing a lot of that type of work. It's a much more out of the box integration for people who want to programmatically run agents.

• 43:11Speaker 00

  Does CMA share infrastructure with Claude Tag? Because the sequencing of it seemed very like, oh, like that's the.

• 43:19Speaker 01

  Yeah, yes. A lot of it, a lot of it. So one of the big things that we do is as the platform team, we run our APIs. We also kind of run all of our foundational product infrastructure and product services. So a really good example is like our team runs our sandboxing fleet that backs both Claude Managed Agents as well as products like Claude Tag and otherwise. But at the actual harness layer for those two products, like there's just differences in what we might want to do within the harness so that they don't, they don't actually share a harness. But they share a lot of the infrastructure that powers the execution and powers the APIs.

• 43:56Speaker 00

  I think that's, that's really, people should use this more. I think everyone basically had a copycat clones of managed agents after you guys launched it, which, you know, imitation is a sincere form of flattery. Since we're on this topic, we'll briefly mention all the memory stuff that has been going on. I feel like it was a big part of Anthropic's work in the first half of the year. Has it stalled out? You know, is it still, still ongoing? Like what's, what's the emphasis here now?

• 44:21Speaker 01

  Managed Agents specifically.

• 44:22Speaker 03

  Memory, dreaming.

• 44:24Speaker 01

  Oh, it is a big part of what we're doing. I think maybe the way to think about it is we launched Managed Agents as a platform for people who like didn't want to do all these things themselves, right? Or just wanted to like offload, like I'm getting the best out of Claude because our team is literally sitting here like evaling all these things and making sure that the harness is going to be really powerful. Memory and dreaming is all about just like, how do you make sure your agent can continuously improve over time? And all of that is still an important focus for our team. But what I would actually say is to the point of what we talked about earlier, as launching models has become harder and harder from a safety perspective, I think most of our team's energy is going less into like, here's a fun feature to like hill climb on this

• 45:08Speaker 01

  agent or whatever, and much more into how do we bring safety into all these things? Because the most important thing that we can do is get the power of the model out there and get it out there safely. And so I would say a lot of our energy is take all these same concepts and give you an agent system that's going to be really powerful, but also that's going to be really safe. And so that's where we're putting our energy.

• 45:28Speaker 02

  I think compared to maybe earlier this year, we have slowed down on like skills, dreaming, memory, like these kind of like moment in time injections.

• 45:36Speaker 00

  Skills.

• 45:37Speaker 02

  No, no, we love skills. We are making, we are making updates. I'm not saying that we're not doing anything, but we love skills as well. But yes, I think we've slowed down a little bit comparatively for some of the other efforts. But we are very bullish on people kind of doing some level of hill climbing at the harness layer to, you know, some of the previous conversation here. We do a lot of this ourselves, but the harness that we provide is very simple in a way. It's intentionally simple. We don't want to be actually overly opinionated, want to make sure the best practices get merged in and there's some kind of undifferentiated pieces of the hardest agentic loop that, yeah, are hard to string together and also like you can kind of just boilerplate do it. Now, the interesting part on top of that, though, is like, once you get that base layer,

• 46:21Speaker 02

  there are these like hill climbing strategies that we construct and like dreaming is actually one of them. So dreaming under the hood is a multi-agent system that we hill climbed on top of CMA and it's actually CMA on top of itself, basically.

• 46:32Speaker 00

  CMA on top of itself.

• 46:33Speaker 02

  It like spins up another, yeah. The way dreaming works is that we spin up another managed agent that then can expect the previous managed agent and then write memories to that managed agent. So it's managed agents all the way down. And this is a bit of a thing by design because we envisioned that you could probably put together a lot of these modular components for something interesting. And dreaming is just like an example, one of the more interesting things that we were going after. But we kind of have been looking at this and it's being like, there's actually a ton of other stuff you can do. You can do like best of N as a type of like multi-agent strategy. You can do a verifier agent, that kind of like goal thing or outcome thing. There's a bunch of other stuff in there and we've seen people do some pretty funky, crazy stuff. Like there's been a harness that like shares like crazy memory across everything and they spin up like tons of parallel agents.

• 47:16Speaker 02

  And we're like, we should actually just probably enable you to do that in the same way that we did. We built dreaming, that's like managed agents all the way down. There's a update that we'll make relatively soon where we're updating the underlying primitive. We'll expose it to allow users to kind of like construct harness strategies, maybe, or however you want to call this stuff.

• 47:35Speaker 01

  Like multi-agent.

• 47:36Speaker 02

  Multi-agent strategies that allow you to kind of build your own version of this. And we'll give you like, you know, templates of like, this is how we started with dreaming. Maybe there's like an update to this that you would want to make as a developer and you would find some more value on top of it.

• 47:47Speaker 01

  Yeah, I think that's a really good framing. It's less that we're slowing down on this sort of stuff and more saying like, we've got the primitives in place. We can continue to let people just like take and run with those things while giving them the tools to say, here's some like cool multi-agent strategies and like things that we've put into, into Claude Managed Agents to make it really powerful. Now you can like let Claude do work to take advantage of those primitives and make it even more powerful yourself, like create new multi-agent strategies and whatever it might be. While a lot of our team's attention turns to, okay, great, like community, run with that, right? And you do you while we continue to focus on what are all the things that we need to do to make sure that you get access to the models within that system safely.

• 48:29Speaker 03

  One thing you brought up was you run a lot of evals on this stuff. Pretty hot topic right now. Any advice, best practices?

• 48:36Speaker 01

  For eval specifically?

• 48:37Speaker 03

  Eval's on harnesses, agents.

• 48:40Speaker 01

  There's a lot of different types of evals, right? Like not all evals are made equal for different, you know, like situations that you're in and things that you're trying to test for. You obviously have just like static deterministic stuff and then you have LLM as a judge, you have all these different strategies. But I guess I would just say like, what we found from most people is like, you probably need more rather than less. It's just, there's not like one perfect, like couple sets of evals that you could put together that are going to get you all the answers that you need to make your system better. We found that people end up needing like a relatively large suite of these things. And then it's like, you can have your agents work with you to evolve your eval suite over time as well.

• 49:22Speaker 01

  But I think it's just like, under investing is maybe like the biggest mistake that we've seen from people on that front.

• 49:28Speaker 02

  Yeah, and let's start simple. I think some people like go a little crazy and they're like, you know, they're like, look at all the, the menu of options that you kind of listen and you're like, wow, this is so crazy. Let me try this and that and what's the best one? And like while they do all those things, like the number of evals they actually have are like zero. So you might as well just like, do something simple. It will help.

• 49:45Speaker 00

  Yeah, I think some, some part of it, people are inexperienced, they're not trained to do this, and they don't know what good looks like. So I think for you guys, publishing more examples of that is really helpful. Something I do tell people a lot that I'm doing with more Fable and Astro class type models is we're treating engineering like research. And when you treat your engineering like research, you naturally do more evals, you do more labeling, you do more scaling of runs, or like prototyping and scaling of runs. And you're not training a model, you're building the harness, the products, whatever it is, but you're approaching it like research. You're defining upfront what your goal is, trying to formalize it as an eval, and then try to identify, okay, why is it failing? And sometimes your eval is wrong.

• 50:29Speaker 02

  Totally.

• 50:30Speaker 00

  Actually, a lot of times your eval is wrong. Sometimes your environment's wrong. And then sometimes the model's wrong. And actually, that is like a lot of part, what I think AI engineering is becoming, which is doing all this stuff.

• 50:40Speaker 01

  Completely agree. And we'll continue to launch more features within Claude Managed Agents to give people more tooling to do this sort of stuff out of the box. And you'll probably find from us that you'll get something that's like, here's a product that will help you create evals for your managed agents or whatever it might be. And it'll feel very much like working with Claude in order to do that.

• 51:00Speaker 00

  Yes, right.

• 51:01Speaker 01

  That is, that is the best way to do it. Claude's actually really good at creating evals to.

• 51:05Speaker 00

  And evaluize.

• 51:06Speaker 01

  Yeah, exactly.

• 51:07Speaker 00

  Because like people like blur, blank out when they look at like JSON Ls, like just, right? But a pretty eval UI where you're like left, right, left, right, left, right? And you're sort of like doing what the human greater data people, lablers would do anyway. You can do that yourself. Just like throw Fable at it. And it'll materialize, but also write comments. So because it just like outputs all these ratings, right? It'll give you an elo, give you whatever, but Claude can just consume that and then work on itself, right? That's the feedback it needs. And people just don't know that they can prompt for an eval.

• 51:40Speaker 02

  Yeah. Yeah, I love that you said that. I think it's like a very like AGI pilled way to like do it because it also allows you to update and edit that system like very systematically. And so as new models improve, as things change, you can really expect where your system is and it'll just like evolve to you.

• 51:57Speaker 00

  To every part.

• 51:58Speaker 02

  Exactly.

• 51:59Speaker 00

  Yeah, so basically, I think is, you know, how we talked about how maybe product and engineering are merging, is also there's like application layer engineering and, and sort of model research are sort of merging in those techniques that they are being shared, right?

• 52:12Speaker 02

  That's very true.

• 52:13Speaker 00

  Model people are very, very used to doing evals. Application people know like the only thing you care about is does it fit the PRD? And like. Yeah, it's like a whole different.

• 52:23Speaker 02

  It's a whole new skill set, I think.

• 52:25Speaker 03

  I think you guys put out a pretty cool blog post on this today. It's somewhat different. It's about making the UI a lot faster. But to your point of treating it like research, basically define a goal of we want stuff to load faster and then define and question things, what can we do instead of wall clock timing? There's fun quotes in here. I think it's nice to see how the actual researchers or engineers prompted Claude. There was stuff like, you know, we had 12 things. We said, you have two weeks. I did nine out of 12 of them in like three days or something. But fun blog post for people to research, for people to read through.

• 53:01Speaker 02

  Yeah, I love it when we share our actual prompts. And like sometimes people are like, oh, God, there was some crazy, unique ways of prompting. And it's just like, try harder, Claude. Thank you.

• 53:11Speaker 00

  Though my only feedback is it should link to the workbench, which is no longer a thing. But I used to.

• 53:17Speaker 02

  That is a good, that's a good idea. Okay.

• 53:20Speaker 00

  Make everything replayable.

• 53:21Speaker 02

  Feedback heard, love it.

• 53:22Speaker 01

  Yeah.

• 53:23Speaker 00

  Yeah, but you were gonna share more details behind the engineering story or the Rust.

• 53:28Speaker 01

  Yeah, so I think I talked a little bit earlier about like, scale is a problem for us.

• 53:33Speaker 00

  So it's scale and safety, right?

• 53:35Speaker 01

  Yeah, scale and safety. Those are like the two biggest things we're spending our time on. And I mean, it's really interesting because most people are used to talking about scale or just like, I don't know, computes at the like GPU accelerator layer and much less so at the CPU, you know, like old school layer. And that's, it's just like as much of a problem today as the accelerator layer, which is really interesting.

• 54:01Speaker 00

  You cannot get enough VMs, you cannot get enough sandboxes.

• 54:04Speaker 03

  Yeah, exactly. Everything being able to spin up, spin down.

• 54:07Speaker 00

  With all the resources that you have.

• 54:09Speaker 01

  Yeah.

• 54:09Speaker 00

  Really?

• 54:10Speaker 03

  You got a lot of customers too.

• 54:11Speaker 01

  Yeah, which is crazy. Yeah. And think about it, because we're using CPU across, obviously just like running our product services, running the API, running all of our apps, we're using CPU on sandboxing, all the sandboxing that we're doing. And then just like all the things, like CI, you know, like you'd be surprised how much more code is getting written, therefore there's so many more CI runs, therefore you need more CPU to handle that. And so on the whole, and this is not just us, this is everybody that I talk to.

• 54:37Speaker 03

  I would add training, training is probably.

• 54:40Speaker 01

  For sure, yeah. So there's.

• 54:41Speaker 00

  There's a different bucket, right? I'm sure they don't co-mingle that much.

• 54:44Speaker 01

  Well, they don't co-mingle, although like CPU for Anthropic as a whole is like a zero sum game, you know, like we get our, we negotiate our deals with the CSPs and we get our CPU and we have to allocate within Anthropic how much of that goes to all the things I just mentioned, including like research sandboxing versus product sandboxing and otherwise. And so we really are allocating across absolutely everything. And it's just everyone in the industry right now is.

• 55:13Speaker 00

  Short on everything.

• 55:14Speaker 01

  Yeah, you used to be able to just like, you know, like scale up on demand into CPU capacity as long as your budget allowed you to, right? And like now it becomes something that you have to capacity plan for in advance, which is, you know, like kind of punching everyone in the face. It's brand new and people are starting to do it. And so dealing with that means a few things. Like one is obviously just go try to get more CPU. Cool, we're doing that as much as we can. But two is we want to be able to at the infrastructure layer, run it more efficiently, right? Like figure out how we can make sure that we're running our sandboxes as efficiently as we can without compromising on things like tenant isolation or properties that we like, you know, care about for security or otherwise.

• 55:58Speaker 01

  But then at the application layer, how can you, how can we run our applications and our APIs more efficiently? The thing that I'm like most excited about, as of last week, we fully rolled out the messages API written in Rust, which is one of those like insane feats of like what you're able to do in the agentic coding era that used to be, like we started talking about this like earlier this year, we're like, we're gonna hit a CPU wall, there's stuff we need to do. And we've done it, right? Like we've gone multi-region, we've done all these things we need to do. But we were like, we're running this Python monolith and it's like, it's like very CPU inefficient. And so what are we gonna do to make the system more efficient? Yeah.

• 56:40Speaker 01

  So we chose Rust, which was a crazy decision at the time because we had a team of people, like on paper, it's obvious, but our team was like, we don't have a single. Yeah, we don't have a single Rust developer in the room.

• 56:51Speaker 00

  Who cares? Just by.

• 56:52Speaker 01

  Exactly, literally. And not only that, but yes, Claude is gonna write the code, the humans like design the system, right? So like the humans are very involved in making sure, at the end of the day, it's just like, it's a distributed systems architecture, right? Like and Claude can help you both with actually writing the code, but also helping those engineers like, you know, bounce around ideas and learn what they need to learn to design that system. But what's most interesting is the two things that mattered most were the plan to do the thing and then the verification of the thing. Cause the actual implementation was like Claude writing code.

• 57:26Speaker 00

  Yeah.

• 57:26Speaker 01

  And so what we ended up doing was we literally built like an entire fork of the entire API in Rust and then started running shadow traffic through it and verifying that the outputs that we were getting from the Python system and the Rust system.

• 57:40Speaker 00

  Feature flags are all you need.

• 57:42Speaker 01

  Yeah, literally.

• 57:44Speaker 00

  Load balancing feature flags.

• 57:45Speaker 01

  I don't know, I never in my career would have had a team say like, yeah, yeah, cool, we're just gonna like port the entire.

• 57:52Speaker 00

  Do you still have Rust people or no, no Rust people?

• 57:54Speaker 01

  Well, they would probably call themselves Rust people now.

• 57:57Speaker 03

  I mean, they're shipping big stuff to Rust.

• 57:59Speaker 01

  Yeah, exactly. And along the way, we have hired a couple of people who have more experience with Rust. But yeah, but it still is just like, you never would have signed up for that. It would have taken years and years and years. Like you wouldn't do it all in one shot. You would have carved out pieces of the system and say like migrate this, migrate that, migrate that.

• 58:16Speaker 02

  Easily a multi-year effort.

• 58:17Speaker 01

  Yeah, but you would never like kind of one, in one swoop, right? Like port the whole thing over and then just like slowly turn it on. But that is what we ended up doing and it worked. And so not only do we actually save a lot of CPU, but we, our latency has seen a nice improvement, which is pretty cool too. And you know, it's just, it's a crazy era we live in because you just never could have done that sort of thing previously.

• 58:42Speaker 00

  Order of magnitude of savings on the, on the CPU side? Just for people considering this.

• 58:47Speaker 01

  Yeah, yeah. I'm, okay, order of magnitude, that's really interesting.

• 58:51Speaker 00

  Ballpark, yeah.

• 58:52Speaker 01

  Yeah, ballpark. I guess I would say like we've somewhere like 40 to 50%, something like really crazy.

• 59:00Speaker 00

  Yeah.

• 59:00Speaker 01

  And then latency, I think we saw somewhere between like 10 and 20% improvement on like time to first token specifically. So a lot of just really cool improvements across.

• 59:12Speaker 02

  Tired on like longer running workloads, but so it'll roll out soon. But it'll get faster.

• 59:17Speaker 00

  Do you fight the urge, you know, I've been involved in some major rewrites. There's always like, what if we add this new thing that didn't exist before in the rewrite? Since we're rewriting anyway, right?

• 59:26Speaker 01

  Yeah.

• 59:27Speaker 00

  But then like that, add scope.

• 59:28Speaker 01

  Yeah, yeah.

• 59:29Speaker 00

  So do you just like avoid that?

• 59:30Speaker 01

  So we actually. We did, it was a pure, it was maybe the thing that I would call it is like a port. What we did was we, we did exactly that. We like resisted the urge on all these things. Then, and because what the team already had to keep up with was little like SWAT team that we put together to go and do this, was doing this while the whole rest of our team was like building features into the API, right? And they did all the like static analysis stuff that would yell at you because you built a feature over here and you didn't consider Rust over there, right? And like they dealt with all of that, but they resisted the urge to add features themselves or like make the system different. Now their next goal is to go and do some sort of re-architecture type work, including stuff that I think will be more user facing.

• 1:00:14Speaker 01

  Like we're starting to roll out some things that are like, you know, what if we were able to not have you have to like pass back and forth the entire like conversation state, right? And if you didn't have to do that on the messages API because you were open to like, you know, a storage of that state for some period of time, right? Then like especially for long conversations where the context really builds up, like you get a lot of really great latency and efficiency wins there too. Like these are the sorts of things that the team that was doing that port was like, ooh, like what if we do this, right?

• 1:00:45Speaker 00

  Doesn't your caching API have that?

• 1:00:47Speaker 01

  So we have prompt caching, yeah, but you still have to like pass back the state of the conversation.

• 1:00:52Speaker 03

  Okay. Very, very niche, small subsets, but I guess this is the level.

• 1:00:56Speaker 00

  No, this is, this is important. Yeah, yeah, yeah. There's a lot of savings they can do, but it also turns them into a cloud.

• 1:01:02Speaker 02

  Yeah, and then from type of, from number of users' point of view, it's small, but if you look at the actual token volume on those types of workloads, it's like going to be the majority.

• 1:01:09Speaker 00

  I'm very like, like what Cloudflare did with durable objects? I feel like every cloud provider should clone. Do you know what I mean?

• 1:01:15Speaker 01

  Yeah.

• 1:01:15Speaker 00

  Like, right? Isn't that the right shape for a messages API that needs the continuation?

• 1:01:20Speaker 01

  Yeah, I mean, it is really interesting because I think what we're, we are experimenting with various like, what is the right shape of like how we should run our compute in order to power all these sorts of things? But it's also interesting because we can't necessarily rely on like, because we get our compute across, we run on all three clouds, we're running all over the place. Like we don't necessarily have like one shape of here's exactly like the compute, the types of machines that we know we're gonna run on and the way that some certain compute works that we can like rely on across the board, if that makes sense, other than like the most vanilla versions of these things. But I think these things are all gonna, I don't know, continue to innovate and evolve as time goes on. And so we'll be able to take more advantage of those things.

• 1:02:00Speaker 00

  Yeah. I'll just mention, if people want open source Cloudflare durable objects, Rivet is the open source equivalent. There are 50 people listening that will kill me if I don't ask this question. You want CPU, they have CPU to sell, who do I sell to? Who do I email? And what is a good help them help you, right? Like, what does good look like?

• 1:02:22Speaker 01

  Well, I guess like, email me.

• 1:02:26Speaker 00

  Are you sure? I don't know.

• 1:02:29Speaker 01

  I'm gonna get a lot of emails.

• 1:02:30Speaker 00

  But like, you're looking like minimum fleet size, SOC 2, obviously enterprise. Give me all the privacy guarantees.

• 1:02:37Speaker 01

  And for what it's worth, like, so we have a really awesome team that secures our accelerator compute, that is now spending a lot more time with us on CPU and like where and how we can procure it to make sure that we can just continue to run our systems at the scale that we need to. And so it's not actually like, my job thankfully is not to go and run around and try to procure a compute, but we do have teams that are doing that, which is really great and really important. But obviously, like, there's a quality bar that matters to us, there's like a scale bar that matters to us, we're not gonna like go and integrate and run on certain machines if like we can't get enough of it. And so all the properties that you might expect of a system at this sort of scale is what matters to us.

• 1:03:20Speaker 00

  Then related to the compute problem is the sandboxing problem. People have trouble containing agents in sandboxes, they keep breaking out. What's going on there? What's the state of the art?

• 1:03:30Speaker 01

  Yeah.

• 1:03:30Speaker 03

  The state of the breakout recently?

• 1:03:32Speaker 01

  Yeah. It's interesting, one of the things that we did with Claude managed agents specifically is the way that we architected it. We put out this blog post at the time of launching where we talked about we have the brain separate from the hands. What that means is like, the harness runs on a durable server, really hours, right?

• 1:03:49Speaker 00

  Oh wow, nice pull.

• 1:03:51Speaker 01

  Yeah, that's good. And the hands are, it's like an execution environment, like a sandbox or some sort of container, right? The part of the reason why we did this was for like latency and reliability and these sorts of things, right? Like, if the harness is just running on a durable server and it's only spawning a sandbox or a container to execute when it needs it, that means we can get you tokens back really quickly on your initial request. It means that your sandbox can die, but like your whole agent doesn't die because the harness is durable and over here and the, you know, the agent's storing state separately also. And there was a whole product development journey that we went on to get there because our first iteration of an API for agents was just like, throw Claude code in a sandbox and like shove an API on top. And the reason why we didn't launch that as our like durable, scalable solution is

• 1:04:36Speaker 01

  everything that I just said, it was like, you've got to boot a sandbox and boot Claude code before you can even get a token back, right? The sandbox dies and you lose your whole agent and so reliability wasn't where we wanted it to be. So we architected this way mostly for reliability and latency. But the interesting thing, there's actually a, the product manager who worked on this from the very beginning was like, the most important property of this architecture is security. And everyone's like, yes, yes, yes, but latency and reliability and like now.

• 1:05:02Speaker 02

  Now they're vindicated.

• 1:05:03Speaker 01

  I literally, I apologized to this person yesterday, I was like, I'm now like telling everybody who I meet about what you've been saying for months that nobody was as excited about as this person was, but one of the more important properties of this architecture is you have credentials for any, any work that the agent needs to execute, right? Like, if you're gonna go talk to an MCP server, or if you're gonna go like use a CLI or call some API, whatever you might want to do, right? And like, you need to authenticate those things, you need to store those credentials somewhere so that the agent can use them. And the way that we architected this was we have a totally separate vault where those credentials are getting stored.

• 1:05:44Speaker 01

  And as, once the agent decides like, I'm gonna go and call out to one of these services, the agent only ever sees like a placeholder, right? And like on egress, we have a proxy that goes and like pulls the right credential out and injects it and lets that out. And so what's interesting about that is like, the agent is never like in the same place where the credentials are, right? Because in the old world of like, just boot the agent in a box and like let it run, you had to send credentials in there. And then you're like doing some sort of like, ah, how do we like scrub this and that and make sure that like, the agent can't take that credential and go do something that it shouldn't be doing. It's just like literally a like literal infrastructure wall that says the agents like can't access those credentials and can't do anything bad with them.

• 1:06:28Speaker 01

  So that was one thing that was really great. The other nice thing is like, on that sandboxing infrastructure, we can make sure that we have really great egress controls. So we have like domain allow lists and things like that, or you just like totally shut off network access so that you can't say, okay, agent wrote untrusted code, that code executes in sandbox and that code can like reach out to the internet, right? Like now it only can reach out to where you say it can reach out to. So there's a lot of really great properties around all of this, within the architecture and within the way that you can configure your sandboxes, make it so that you end up like secure by default. One of the things that's starting to happen is as we're talking to customers who are like, I want to run agents either for, you know, a product for my customers or internally,

• 1:07:11Speaker 01

  but I'm like, you know, hearing about incidents where agents are breaking out of sandboxes and bad things are happening, how do I think about these things? I think the two most important things are architecture, so that it's like literally impossible for an agent to do something that isn't okay. And the configuration that you have on those sandboxes. And so one of the really, really nice things about Claude managed agents is you just like get that out of the box. And you get something out of the box that like, as we learn more and we continue to evolve the properties of those systems, you kind of get that for free. Cause I think say, all the words that just came out of my mouth, like brain and hands and credentials injection and egress and all these sorts of things are just like, if you go back like nine months, like, no one would be saying these words, right? Like, these are all like relatively new problems that we're solving. And so we want our customers to be able to take advantage of the solutions to those

• 1:07:56Speaker 01

  problems as we're even learning about them and solving them ourselves within our infrastructure.

• 1:08:01Speaker 02

  Yeah, and internally, we're just upgrading everything to be basically

• 1:08:05Speaker 00

  That pattern.

• 1:08:06Speaker 02

  Secure by default, yes.

• 1:08:07Speaker 00

  Yeah. We also unlocked something that we covered in the Tariq pod, which is Cloud Brain Local Hands, which is projects.

• 1:08:15Speaker 01

  Yeah.

• 1:08:15Speaker 00

  Is that, is it the rough shape of?

• 1:08:18Speaker 01

  Yep, that seems right.

• 1:08:19Speaker 00

  Yeah, I feel like it's weird because obviously your famous, or Claude Code is famous for being the super unsafe, dangerously allowed permissions thing on your machine, but now you're like finally evolving to where probably you should have started in the first place. Basically, I think like, all these sandbox leaks, right? Are they a fundamental issue with our understanding of sandboxing or container tech? Or are we just bad at this? Like, are we just like not disabling, not doing the allow list, right? Like, it sounds like we're just not doing the allow list.

• 1:08:50Speaker 01

  Yeah, so maybe a distinction that I would make is, in a lot of the incidents that people are hearing about and reading about, these are incidents that are happening to models, not just like everybody's models, right, that are in training, where the way that you're running these models is you like put them in a box and you say like, solve impossible task, I'm gonna stress you out, ready, set, go, right? And you like, can only do what you can do within your environment. And the agent is basically incentivized to try to get out of his box and go solve the problem. When people are running production workloads on our API or on anybody else's API, you're running an environment where like, there are safeguards that are running, that are like not necessarily running and training, where like the whole point is to actually train the model, right? But in production workloads, first of all, you have those online safeguards, which is

• 1:09:35Speaker 01

  really great. Second of all, you have more of this architecture and infrastructure around what's happening. Like, the agent isn't just in a box trying to get out, right? Like, the harness is over here, the hands are over there, and it's just like not even a problem that you would have to contemplate. And so I think that some of this is just like, people having the right understanding that what's happening, some of these incidents that are happening in model training and otherwise, are just things that like wouldn't happen in production workloads where a lot of work has been done to put the right guardrails in place so that those things like never really could happen. The second thing I would say is just like, everybody's figuring all of this out like relatively newly because, you know, long-running agents is a relatively nascent thing that everybody is building and trying to deal with.

• 1:10:19Speaker 01

  And so everyone's figuring out some of these things on the fly and just racing to put the right, you know, like controls in place. And, you know, some of that is just happening as we go. So I would say it's not, you know, where people are bad at this, right? I think it's just more, it's a nascent field, people are figuring it out. And I think when we talk about things like pacing the frontier, the way that we think about it is really specifically, we're doing all the work that we're talking about right here for all of our agents internally and making sure that all these things are accessible to all of our customers so that like, by default, the right guardrails are just in place so that you don't end up having agents doing things that you don't want them to be doing.

• 1:10:55Speaker 00

  Fair enough. Cause you mentioned pacing, we've got to go there. We also talked about it last time. It is superficially contradictory to advocate for pacing and then also release a new frontier model 11 days later. To me, it's just, it just means that it just passed your safety checks, right?

• 1:11:13Speaker 01

  Yeah. And that, yeah, everything we talked about earlier on like how we launched the models, it's like, we are continuing to evolve more and more and more safety properties in place, again, across these like, I kind of am starting to think about this in these three dimensions. It's like, the actual like, architecture and infrastructure is security, then it's the online safeguards that are running when a model is doing its work, and then it's the offline safeguards, classification that's happening on data at rest. So across all those three dimensions, we're putting all the right work in place to make sure that we feel really good that we can put a model out there and it's going to be safe. And really, that's fundamentally what we mean. And so for OPUS 5.5, it's an excellent model. It's also really, really great on all the dimensions that we care about across those

• 1:11:57Speaker 01

  three things, and therefore we feel good about releasing that model. So we went for what we thought, what we were hoping is the best possible user experience within this while we continue to hill climb on making those false positives better, but it's obviously like, it's still a frustrating experience for people when you're trying to work with a model and then you fall back to a different model. And so it's just something we have to keep working on.

• 1:12:17Speaker 03

  I think people also understand it in the safety sense, like you want more false positives, right? Like, it's a safety issue, we clearly see the whole invasive incident and you're still giving access.

• 1:12:28Speaker 00

  Yeah. To me, this is like a close cousin of the general model routing problem of, do you want a smart router or dumb router? Does that make sense?

• 1:12:37Speaker 01

  Yes.

• 1:12:37Speaker 00

  Like, because as far as I can tell, the Anthropic fallback router or detection thing is a separate model that's online.

• 1:12:46Speaker 02

  It's, it, once it hits a classifier, there's just logic to where you just drop.

• 1:12:50Speaker 00

  Right, like, so it is not the full Fable, right? You're not, you're not, you're not running full Fable at it. And that's gonna cause classification issues because it's clearly not as smart as Fable, like clearly. So then there's the question of like, well, then like when I have a classification issue, that's, that's like an unjustified fallback. Like, you never, you never need a fallback, I'm just doing my bio homework, but I mentioned bio one time and then it can fall back. So like, there's a question, then this is very similar to the routing problem because

• 1:13:16Speaker 02

  It's very similar.

• 1:13:17Speaker 00

  You're routing models and like, how can this, the, a dumb model route to a smarter model if it doesn't even recognize this, that's, that's a smart model problem? How do we solve this?

• 1:13:27Speaker 01

  It's a hard problem.

• 1:13:29Speaker 03

  I think in some super alignment sense, models are good at classifying stuff, but not answering them, right? Not the smartest model can tell this is a dangerous query, even if it can't make any impact on that, so.

• 1:13:43Speaker 02

  I think the super simplifying thing to say would be like, basically, you have the smarter model figure it out over the one that it needs to detect.

• 1:13:49Speaker 03

  I think it's slower as caused, there's trade offs.

• 1:13:52Speaker 02

  There's trade offs. I'm not saying it's like the right user experience, but it would probably be a slightly more effective route.

• 1:13:58Speaker 03

  One thing interesting is you also don't cut off immediately. I've seen cases where it will start answering, start reasoning, and then get hit by a safety filter. So my first example with Fable was throwing the Fable safety card. And it started, I was like, just break down anything interesting in here. Starts breaking it down five minutes in. I think it gets into cybersecurity and it's like, oh, we're cutting or rolling back to Opus, but it did a lot. There's a lot of context and I see it. It got to a point where it's all about cybersecurity, safety. And then I rolled back and it's like, it wasn't rolling back my query. It was the combination of reasoning and the whole chain, right?

• 1:14:36Speaker 02

  Yeah.

• 1:14:37Speaker 03

  Do you have an idea of what's better?

• 1:14:40Speaker 00

  That's better.

• 1:14:41Speaker 02

  I think the user experience and how we resolve that user experience is probably, you know, a separate question. Like maybe it's a use a super smart model to handle it or whatever. But I think that the more appropriate UX that I would imagine for a user is to actually have a more consistent unified experience with Claude. Where instead of getting you falling back and then you can double fall back, there are like logical points in our tree that do do that. For again, like safety reasons and whatever, but and then UX is just like really convoluted for a user. It's like you almost, you're trying to do something and I would probably take the case of like a good user. And you're a good user, just trying to sell your bio homework, like assess the cybersecurity, like safety card, totally reasonable.

• 1:15:21Speaker 02

  And you get blocked and you fall back. And I think instead it would just be like, you are, where our models should have gotten, or our classifiers, or however you want to do this, would be smart enough to detect that that is a situation that happens. And just write down the answer for you more concretely while dropping down the nature of the like negative use of that particular capability. So going back to that bio homework scenario. While we may have classifiers that have detected that it is possibly.

• 1:15:47Speaker 01

  I'll also say like, the, on the API, we went with a spectrum of solutions for this for customers. So first of all, like if you've built something on the API and you run into a block, like you don't have to like continue the conversation on a fallback. That's what we've done in our apps and we've made that a feature of the API and you can do it server side within our API. You can choose to like set, I want to do the fallbacks, right? And like, we'll do that. We also have it within our SDKs on the client side. There's helpers for if you want to like manage fallbacks in various ways. Or you could choose to like re-prompt differently in some way, right? And continue with the same model. But I do think that there's some in-between interesting solutions that we'll continue to experiment with to figure out how we

• 1:16:33Speaker 01

  unblock people and do so safely, but with a good user experience.

• 1:16:37Speaker 02

  Yeah. And then you brought up the bit that was like very similar to routers. I think like, I totally agree with that. I think it has a similar UX problem as well. Like with the router you go through, it's like, let's say you want to do a cost router, right? For the task complexity that you have. Makes sense. Very hard to do. You don't know everything upfront about your task. So you can't like, even with perfect, you just don't actually have perfect information. Even if you could actually understand each model perfectly, which you don't, but even if you did, you also don't have perfect information about your task. And so there's a lot of error associated with your routing logic that you would have to create, whether, no matter how good a classifier you produce. And yeah, maybe it's like a super big brain model type of problem, but regardless, it's I think a very difficult one. And in each one of those like scenarios, if, to find a way to like route that properly, it's very, very convoluted.

• 1:17:20Speaker 02

  You could expose all that logic to customers like, hey, you know, your task was to center a div, so I'm going to route you to something super cheap. And then this other one's like detect vulnerabilities, so I'm going to route you to something like really expensive. If you expose all of that, I don't think that's like a great user experience ultimately either. And for that similar class of problem, I think the more idealized UX would be something along the lines of like dynamic like effort or dynamic like understanding of the level of complexity of your task, which you might actually evolve over time as you learn more and more information about the task. But in that world, if you're kind of like moving between a lot of different models, it actually becomes very, very disjointed. And so like maybe a more unified approach would be something like it's a consistent model, but on it's like very, very low effort level or something like that.

• 1:18:04Speaker 02

  It's actually very, very cheap and very, very affordable, but it can scale itself up very, very continuously. But you have like a very, very consistent persona towards that model. And obviously speaking about this in the context of like, you know, like our family of models. If you try to even do that across other family of models, other model providers, like the personality, the disjointedness like starts becoming very, very complicated. I don't know how you solve it in that layer, but at least within like our family of stuff, like I'm more bullish on things that are just more consistent. And those might be ways to solve it more robustly for customers.

• 1:18:33Speaker 00

  Yeah, you're not responsible for anyone else, but within Claude, yes. Yeah. I mean, this is your territory.

• 1:18:38Speaker 03

  It's nice to see the way you guys think of these problems, right? Intend in and of itself is a hard problem. Like I feel like the models have gotten better, but sometimes people don't do a good job at classifying what they want, or they just slider all the way to one extreme. Or, you know, the model doesn't really understand. You're asking a simple question and it's like, oh, I have a whole HTML landing page now. I just expected a feline output, you know? So there's always a mess around there.

• 1:19:05Speaker 02

  I think we, in this conversation, we talked a lot about safety. We talked about like launching the models, like all this kind of stuff. But it's obviously in, in pursuit of helping power the workloads for people. And like we discussed in kind of the earlier part of this conversation, we are seeing workloads change quite a bit. I think like in some workloads, intelligence does seem to be increasingly saturated, which is like a new situation. And it's, it's getting to the place where I think there are some class of tasks that are still very, very unsaturated. I actually think there are more than maybe people anticipate, but they're increasingly going into the direction of you get that alpha or that return by doing extremely long-running adjuncting tasks on things that historically you were not really able to brute force. But maybe you can with a lot more intelligence behind it.

• 1:19:50Speaker 02

  And those workloads are really, really fascinating to us and we see a lot of value in that kind of area, both from businesses getting that value from themselves as well as, you know, like new companies or startups, or however you want to frame it, who are interested in building kind of like adjuncting products on top of it. I think if you look at the class of things today, a lot of it tends to be, it's obviously a lot of coding stuff. There's obviously a lot of like enterprise innovation, acceleration kind of stuff. But they tend to take the shape of automating a very distinct and specific process. And I think in some areas, that's actually really starting to work, which is awesome. And some areas it still isn't. So there's still a lot of like room to get to kind of evolve and push the frontier for those models. And then there's like this new class of workload, like the bio stuff that we talked about or some of the cyber stuff.

• 1:20:31Speaker 02

  Like that's a, it's just a very, very complicated set of like, I would say, hard engineering or hard science problem. And those workloads are increasing. They're sharing up as well for us as like a really interesting space to just deliver value to customers.

• 1:20:43Speaker 00

  You know, you are at one of the few spots in the world where you're discovering all these workloads and we're automating it. It's like.

• 1:20:49Speaker 02

  Yeah, it's pretty crazy.

• 1:20:53Speaker 00

  I mean, you know, that's very inspiring. You guys are having so much impact and I'm really grateful for that. We're going to go into a few closing questions, which I thought would be fun to sort of bounce back and forth between the two of you. How do you fight slop on your teams?

• 1:21:11Speaker 01

  So we actually, this has been an interesting debate within our team. We're no different than anybody else, right? I think plenty of teams are talking about this. And what's interesting is, what we're trying to figure out is, if you're producing a document, you're producing something that you want other people to consume. Code, documents, whatever, right? If, like you should use Claude to help you, of course, right? But if you use Claude to like totally generate a thing, you didn't spend any time on like actually reading it yourself and making sure that it was something of high quality. Then you're putting a whole extra burden on the consumer of that document or of that code or whatever, to spend a whole bunch of time trying to parse it. And that's just like insane and totally unfair. But what's funny is, we, you know, we push back, we push on our team, like you can't do that, right?

• 1:21:56Speaker 01

  Like if you're producing something, use Claude, sure, whatever, but like you have to make sure that it's a high quality thing.

• 1:22:02Speaker 02

  Yeah, you as the human are still responsible for the quality.

• 1:22:04Speaker 01

  Right. You're still responsible for the quality, but then you get into this really funny thing where because of the Claudeisms, even just like silly, like, I don't know, there's like an M dash in the middle of the thing, right? Like people's brains trigger on this like, oh no, I'm reading something that was written by Claude and not a human. And I'm like annoyed because I want to be like reading a thing that was like fully produced by, not necessarily fully produced by a human, but like that has the quality bar that the human would hold, right? And so then we get into funny situations where maybe actually like the person did spend a lot of time making sure the thing was high quality, but they use Claude to help them, but there's those little hints of Claude and then the reader is like turned off, you know? And like, that's like unproductive as well. And so what we have kind of tried to do is like

• 1:22:49Speaker 01

  set a bar within our team that's very much like, use Claude to help you because it would be really silly if you didn't use this hilariously powerful agent to speed you up in your job and in your life, or to just like help you produce better things. But it has to like have a certain bar. And then for the consumer and the reader, if like it really actually is just slop, like send it back, right? Like just say no, just send it back. Like you're not going to waste your time on this. But also don't be like too nitpicky about the little Claudeisms, right? Because like, we have to find the right balance here. And so it's just really funny because we're, you know, we're going back and forth on this stuff within our teams. And just like, so is everybody else in the world right now. We're just not any different.

• 1:23:29Speaker 00

  But you have free tokens.

• 1:23:31Speaker 01

  Sure, but I mean, that doesn't tend to matter quite as much when it comes to like.

• 1:23:35Speaker 00

  So you have more than anyone else in the world.

• 1:23:38Speaker 03

  Probably see a lot, you guys use a lot, you see a lot more people doing a lot more.

• 1:23:43Speaker 01

  Yeah, it's totally fair, but it's definitely, it's.

• 1:23:46Speaker 02

  Heavily discouraged.

• 1:23:47Speaker 01

  Yeah.

• 1:23:48Speaker 00

  Engineering slop?

• 1:23:50Speaker 01

  Yeah, I mean, it's same thing, right? Like, but, but I do think that where, where we've gotten to is a pretty good spot where like, you know, humans play a large role in making sure like designs of systems are good, right? And like, implementation tends to be Claude, right? And like, we're in a pretty good spot where for like, a lot of the code that gets written, like, all you really need is just like a quick stamp and you can keep going. And like, you don't necessarily have someone like deeply reviewing every single line of the code, because you care more about like, the overall plan, the overall like, design is in good shape, less so than like, the exact perfect implementation. And obviously rely on a whole bunch of verification to make sure the thing that you're going to ship is high quality. And so it's actually been like, less of a conversation these days on like, code quality

• 1:24:34Speaker 01

  being slop, in part because Claude's just like gotten better, but in part because we have a lot of the right verification in place to make sure that like, you know, we're holding a high bar.

• 1:24:43Speaker 00

  I see, I see. Trust your tests.

• 1:24:44Speaker 01

  Yeah.

• 1:24:45Speaker 03

  Any product PMing slop? You can ship features a lot, you can put stuff out in a day.

• 1:24:51Speaker 00

  So he said this to me recently, do you remember what he said?

• 1:24:53Speaker 03

  Tharik, Tharik tweeted this. The right way to use model capabilities is not to ship 10x more features into prod. Spend more time understanding your users, trying experiments, building whatnot. But basically, you know, you could very easily just keep shipping random slop, but you, you should actually think about it. You should spend more time kicking off that first prompt. I know now today, he's like kickoff plan mode, we don't need it anymore. But outside of plan mode, you should think about what you're doing, right? It's very easy to put out a lot and you can get products sloped.

• 1:25:25Speaker 01

  Yeah, I mean, I, I tend to kind of go on this rant about, there's a lot of just like fundamentals about how software gets like, and products get planned and shipped and otherwise that like, doesn't need to change just because agents are like helping speed up certain parts of the process. But there's this inherent like, almost laziness is starting to come out from a lot of people where it's like, oh, I don't have to like, think about some part of the process at all because the agents are doing it and helping me or whatever. But like, those are still like incredibly foundational parts of the process that like, should be things that you're spending your time and energy on. And I think that's kind of the spirit of what he's getting at here.

• 1:26:02Speaker 00

  It's just like, as a founder or creator, let's call it as well, it just feels like, because the cost of experimentation or exploration is so low now, that I should be doing more of that? And he's basically saying to not do that, to, to like, use it in service of a very intentional goal that I start from the start, which like, I, obviously I'm coming around to, it's just hard. It's like you have this toy that does everything, it's just like, throw it at the wall.

• 1:26:28Speaker 03

  Yeah, there is scope too. So like, the blog post of today was, two weeks to make Claude AI faster. There were 12 things, we had, there were 13 things, we hit 12 of the 13 targets in three days, right? Well, you could just stay, okay, it's a two week sprint, we'll do the 13, but instead there's 20 things, you push bound. This is not product slop, it's not new products, it's refactoring. But still, you guys do a very good job at outputting a lot, but keeping high bar quality.

• 1:26:55Speaker 01

  Yeah, nice.

• 1:26:56Speaker 00

  Okay, part two or three, this is gonna make Angela very uncomfortable. This is Angela tweet corner. I will tell.

• 1:27:03Speaker 01

  Nice.

• 1:27:05Speaker 00

  I will, I will tell people nonstop to, and I, I'm pretty sure you've deleted stuff, because I'm like, there's, there's stuff, there's stuff in your deletions that I'm like,

• 1:27:15Speaker 01

  That you liked?

• 1:27:16Speaker 00

  Yeah, I know, like really like, altered my DNA.

• 1:27:19Speaker 01

  Oh, I appreciate that.

• 1:27:20Speaker 00

  What do you refer? What do you think about the most? What do you refer people to the most that you wanna sort of bring up? And I can, I can pull it up for you if you want, but.

• 1:27:29Speaker 02

  I actually, I don't think I have a favorite, honestly.

• 1:27:31Speaker 00

  Oh, come on.

• 1:27:32Speaker 02

  No, I really, okay, I guess, maybe I like the one that you say, where it's half doing something is expensive way of not doing it. Maybe that's my favorite.

• 1:27:39Speaker 00

  That's very on topic for us today.

• 1:27:40Speaker 02

  Okay, nice.

• 1:27:45Speaker 00

  I guess, you know, obviously, you're in a different job now, and you don't do this anymore, but like, I think this is a power tweet, this is like a Naval Robin, Naval knows you.

• 1:27:56Speaker 02

  Yeah.

• 1:27:57Speaker 00

  That do this. And like, what mode, what mode of, what like, what framework of mind, frame of mind are you in?

• 1:28:05Speaker 02

  That's a good question. Yeah, I know, this is a very long time ago. I went through a period of my life where I was like, reading constantly, actually. I found myself like, very bored of the stuff that I was doing, and I felt like I was just like, not being like intellectually challenged. And so I actually went on this bit of a bender where I, I think I read something like, 200 or so books in like, a year. And I was just curious about like, I just wanted to like, read everything. And I was missing a little bit of like, the, the kind of like, intellectual, like, I don't know, like muscle building you get when you're like, in school, where they like, do like, this is a new topic, you gotta learn, it's like, tough, it's like, not super fun actually, but you are really like, stretching yourself. And I felt like, after just like, working for a couple of years, I just like, stopped

• 1:28:50Speaker 02

  doing that. Like, yeah, the job's like, interesting, and yes, it's like, challenging, but doesn't have like, that like, intellectual bit to it. So anyway, I went on this bit of a bender, a reading bender, and I like, read like, 200 books in a year, and it was great. I had like, a really long commute, so it was like, it was a good use of my time, very long like, Caltrain ride. But I ended up coming out of that and being like, I was just like, thinking a lot, as a result of that. And at the time, it really wasn't to build like, a Twitter follower or anything, I was just like, tweeting into the like, abyss. There was like, I have like, no followers, these were like, tweets for myself, and I was like, practicing a little bit of like, how would I try to compress the idea that I have, in as few words as possible? It was kind of like, the personal test I had for myself. And then, I guess people just started liking it. And so I just, I just kinda just kept going at it.

• 1:29:36Speaker 02

  And then I got too busy with work and I, I stopped doing it, so.

• 1:29:38Speaker 03

  I gotta ask, after all the reading, any book, book recommendations, blogs, long articles?

• 1:29:45Speaker 02

  My favorite one still is probably The Black Swan by Nassim Taleb. That was like, one of those like, I was like, to me at least, it was like, a bit of a, like a life-changing philosophical book, where I was like, that's a completely different way to look at the world, and.

• 1:30:01Speaker 00

  You know, I was thinking of Black Swan when the fable thing happened, right? It's a thing that you didn't know could happen. Now that it happens, like, it permanently changed your framework of.

• 1:30:08Speaker 02

  Exactly.

• 1:30:09Speaker 00

  How things work.

• 1:30:09Speaker 02

  And the idea's like, so simple, but I think he did a great job of just really hitting it home and articulating it, and how it's like, a common property of actually, of so much of the world. So I really love that book, and that's probably my favorite.

• 1:30:22Speaker 00

  And for listeners, if you're bored with that one, you do full by randomness, randomness, also by Taleb.

• 1:30:27Speaker 02

  Did you like, anti-fragile?

• 1:30:29Speaker 00

  Less so, less so. The more famous he got, the more intolerable he became.

• 1:30:35Speaker 00

  But also, question for Kaylin, if you do have one, do you have a similar, like, this is a book I recommend to everybody?

• 1:30:41Speaker 01

  No, that's a good question. I need to think about it. It is funny, because I do think we have very different styles on this front, like, she's no longer like, in fortune cookie tweeting mode, but I do think that like, you stop that, but I do think a, a big part of how you work and how you work with other people is like, philosopher mode sometimes, you know?

• 1:30:59Speaker 03

  Do you drop fortune cookies in Slack?

• 1:31:01Speaker 01

  Oh, I, I do actually, not intentionally, I think it might just be somewhere.

• 1:31:05Speaker 00

  I think that's a core success of succeeding in topic, is like, you have to be very good on Slack.

• 1:31:09Speaker 01

  Yeah, oh, you do have to be very good on Slack. It is, it's a pretty intense place, but I think I'm a little, so I'm kind of psycho about like, inbox zero on both email and Slack, it's easier on email these days than Slack. But I like, pretty aggressively prune, like, here are the channels that I'm like, actually in and paying attention to, versus the ones that are like, muted, below the line, sort of thing. And I'm like, always making sure that I'm on top of the ones that I need to be on top of. But it is, it's basically, it feels like a full-time job. Like, there are days where I'm like, I could cancel all my meetings and write no documents, do no work, right? And literally just sit there, like, on Slack the entire day, and I would have done what felt like an entire day's worth of work, just by like, engaging on Slack with all the things, it's pretty wild.

• 1:31:52Speaker 00

  No, but that's horrible.

• 1:31:54Speaker 02

  So that's exactly how I feel. I feel like, you do all this, you put in all this effort, and yes, like, you, you met the mark on like, responding to people and getting certain things over the line, but you didn't actually produce the output that you wanted to. And like, that's like, that's like, personally, a very hard thing for me to deal with, so I try to be like, I need to like, dedicate time to like, actually thinking and actually like, trying to produce something, in order to feel like I'm like, actually.

• 1:32:13Speaker 00

  And ignores, and drops stuff, right? You're gonna drop stuff.

• 1:32:16Speaker 02

  Yes, I have gotten a lot of feedback, I'm a little hard to reach on Slack.

• 1:32:20Speaker 01

  Yeah, a pretty terrible thing that happens now, is I get a lot of Slacks saying, can you get Angela to respond to this?

• 1:32:28Speaker 02

  And I'm like, I'm like, really trying to write a doc here.

• 1:32:31Speaker 00

  So you guys don't have like, a personal Claude, like? Assistant, something to

• 1:32:36Speaker 03

  There's gotta be a way, but there's obviously

• 1:32:39Speaker 00

  No one written in like a personal EA for each of you?

• 1:32:41Speaker 01

  Yeah, well, so we do have, we have a human one also. We also have, we have plenty of Claude help in Slack, for sure. But I do think actually the, the types of Slacks that I'm like, when I'm pruning through and getting are often just like, our team is like blocked on something, is there some hard decision that needs to happen? Or whatever it is, right? And it's like, it is just like a form of, whether it's in Slack or some other way, right? Like, it's a form of getting the team unblocked, like executing in high quality. That is my job anyway, and it like happens to happen a lot in Slack. But I do agree that it ends up feeling like, what did I even produce? When you get through a whole day worth of literally just like going back and forth with people and trying to solve problems.

• 1:33:21Speaker 00

  You produced a functioning team. That's unblocked, so I will give you a process as a, as a former manager that is, yeah, yeah, yeah. You don't feel like you produce anything, but like you produce the team.

• 1:33:32Speaker 01

  Yeah.

• 1:33:32Speaker 00

  Yeah. Speaking of the team, I was gonna go to you next, Caitlin. DevRel Corner.

• 1:33:38Speaker 01

  Oh, nice.

• 1:33:39Speaker 00

  We are, I know you're really hiring DevRel, so I'm making a call out for, for developer relations. You hired Adios Mani recently.

• 1:33:45Speaker 01

  Yeah.

• 1:33:46Speaker 00

  Great get, Lance, you have, you still have a team. What, what do you guys want to hire for, you know, this is, this is it to talk about it? And I'm happy to trade notes.

• 1:33:57Speaker 01

  Yeah, well, props to you because I will say when

• 1:34:00Speaker 00

  I'm the all source and topic recruiting, Caitlin recruiting team.

• 1:34:03Speaker 01

  Literally.

• 1:34:05Speaker 00

  It's fun.

• 1:34:05Speaker 01

  And it's, it's actually not even just recruiting, like I'm gonna toot your horn a little bit. When, so I, when I joined Anthropic, we, you know, we're a small team. We're like, all things considered, we're an extremely small team. And so we run the platforms, the API business and, and all of our product infrastructure. And we, we had like a very nascent DevRel function within Anthropic that was, you know, like not necessarily within the product and engineering org. And when I joined, we were like, okay, we like need to grow up a little bit and need to figure out how to like engage with the community, engage with the world, and we need to like really invest in this DevRel function. And we ended up spinning up a DevRel team within Platform and a DevRel team within Claude Code.

• 1:34:47Speaker 01

  In part because it's like two pretty different like developer ecosystems that we're trying to engage with across the API business and agentic coding, right? And so we ended up doing it as like two separate teams, but kind of like dotted line to each other and spend a ton of time together. But it was, it was something that I was going to be responsible for, like DevRel, which I've spent plenty of time with DevRel before Stripe or otherwise, but it was never something that I was responsible for. And I was like, I have no idea what I'm, what I'm doing. And you were like, I got you, like here are the three pillars of DevRel, like here's how you think about it, which is really great across like product, content, community. And then now I can like rattle that off to people and they're like, wow, you really understand DevRel. I'm like, no, so what's stopping?

• 1:35:26Speaker 02

  You're like, I copy that.

• 1:35:27Speaker 01

  Yeah, that was not

• 1:35:29Speaker 00

  It's apparently, that post is apparently like the default reading for Google when they onboard your DevRel.

• 1:35:34Speaker 01

  Yeah.

• 1:35:34Speaker 00

  So they're really funny.

• 1:35:35Speaker 01

  Yes, it's, it really like this, this really helped me because I had no idea what I was doing. But then on top of that, we, you know, like I've, I don't have a manager for the DevRel team within Platform right now. And so that's one of the things that I'm hoping to hire and our DevRel ICs report to me directly. In part because I think it's a really, really, really hard problem right now, but also in part because I find it really fun. And I'm having a lot of fun working with the team. But really specifically, like what's hard on Platform is Platform in some, sometimes in our like public narrative. Like our platform business is a really large business, but it can kind of get lost in between research and policy and all this work that we're obviously doing, that's like our most frontier important work.

• 1:36:20Speaker 01

  And then our applications like Claude Code and CoWork and all the things happening over here. When you think about Anthropic and you think about our public narrative, those are some of the things that often come to mind. And less so that we're running this like massive developer platform business that's really important and so many people are building on top of. And so a big goal that we have with our DevRel team within Platform is figuring out not only like the really classic, like how do we better engage with like, you know, product, like what people are building and how we get feedback back into the product and these sorts of things, but a lot of it is just like finding our voice and finding what our brand needs to be and what our narrative needs to be and like who we even are gonna be in the community and not kind of get lost in all of the things that are happening.

• 1:37:01Speaker 01

  And so that I think has been interesting because I think the brand that we've started to spin up for our DevRel team, in part because actually Lance joining our team was you connecting me with Lance. But Lance is highly technical and the way that he goes about DevRel is very much like teaching, right? Like he writes a lot about memory and managing costs and these sorts of things. And he goes really deep, he builds evals, he like has a lot of really technical content that he puts together. And I really like that idea, like that idea of our, our DevRel team being kind of like thought leaders for people within what is a really nascent, really hard, really evolving space.

• 1:37:35Speaker 00

  Educators, stalliers, explainers of things. And there's another version of this I really like called the master builder, which is I will build the things with the platform to show you what's possible. And then you'll be like, oh, I didn't know that was possible with, with the cloud platform. And then you just go. And this is a thing stolen from Lego. Because Lego, how did they sell Legos? They have master builders, they build things with Legos and you're like, I wanna do, I wanna do that. I wanna be like this person and then they, they do that.

• 1:38:03Speaker 01

  Nice, that's awesome. Yeah, and the CJ, the other person on our team, who's really great as well, I think he also does a lot more of that sort of like, you know, writing cookbooks and like putting out examples and really engaging with the community on X.

• 1:38:16Speaker 00

  Cookbooks is funny, right? Like when APIs launched in 2023, as Call of Duty LMS, very important. Now, I don't know.

• 1:38:26Speaker 01

  Yeah, well, it's interesting because you're less in the spot where you'd like copy paste the code and like use it, right? But I do think it's like, it, it shows the art of the possible for people and just like what even is the use case, right? And like, how could you go about accomplishing it? But I hear you that like, you know, the code itself is maybe less. Like we for the longest time had this like, there's, there's an engineer on our team who wrote our cookbook on computer use. And it like haunts him to this day because it was like massively complex and like, you know, kind of a beast of a thing that I think, you know, a lot of people got use out of at a time when it's like, I'm gonna build a computer use implementation and I'm gonna deal with all the, the screen shatting and all the things, right? That are happening between the messages API.

• 1:39:08Speaker 00

  Yeah, we were there for the hackathon that you guys launched in the office.

• 1:39:11Speaker 01

  Yeah, yeah. And like that was the era when you like needed to have that sort of thing. And now, you know, it's much more about, we have a skill, like the Claude API skill. It's like really, really excellent at helping you either within Claude Code or any other coding agents to build a really high quality integration. And now like these sorts of things, like I'm gonna build a computer use integration, like how do I go about doing that? Like very few people I, I think slash hope are reaching for the big gnarly cookbook and more so they're loading up the skill and a coding agent and working.

• 1:39:39Speaker 02

  Yeah, or they're just putting it into a reference architecture.

• 1:39:41Speaker 01

  Yeah.

• 1:39:41Speaker 02

  It actually does much better, like increasingly we're seeing that for a lot of the things that you might want to have different implementations on, typically at the harness layer, but there could be other variants. You just point Claude at it and you say, this is the reference architecture I want.

• 1:39:54Speaker 01

  Yeah.

• 1:39:54Speaker 02

  And then build it.

• 1:39:55Speaker 01

  Fun fact, the Claude API skill that helps people build against the API is owned by our DevRel team. So that was built by Lance, right? Exactly.

• 1:40:04Speaker 00

  This is the most high leverage form of DevRel is to like distill everything that is best practices that's gonna go into those agents.

• 1:40:10Speaker 01

  Yeah.

• 1:40:11Speaker 00

  One of the reasons, it's weird because like I'm regarded as this like DevRel person or authority, but I like, it's this, like there's some part of me that just wants to tell you, you don't need DevRel.

• 1:40:21Speaker 03

  You're so good, you don't need it.

• 1:40:23Speaker 00

  Right? Or, right, you turn the engineers, the, the leaders into the public faces. Because all you really are trying to do is public faces and like, here's how we do things here. And then that's kind of it.

• 1:40:35Speaker 01

  Yeah.

• 1:40:35Speaker 00

  You're in Anthropic, you don't need like, you know, to, people will hear about you. So like, I think to me, when Boris came, burst onto the scene, like it seemed like a very like, hey, I'm Boris and the creator Claude Code, blah, blah, blah, that's how we do things, blah, blah. And you're like, yes, he has the day job, but he's also doing that functionally.

• 1:40:52Speaker 01

  Yeah.

• 1:40:52Speaker 00

  So like, why, why doesn't everyone just do that?

• 1:40:54Speaker 01

  Yeah, I mean, it's really good question. And there are teams that have done this really well. Like I think the Cloud Flyer team is a really great example of like, they're all awesome, right?

• 1:41:03Speaker 00

  Nonstop.

• 1:41:04Speaker 01

  Yeah.

• 1:41:05Speaker 00

  And to me, that's actually, by the way, is a very visible, visible sign of a good manager, is that they're all happy with their jobs. They're all, they're all like publicly identifying, they're not quitting.

• 1:41:13Speaker 01

  Yeah, totally. It's interesting because I think this is actually something we are trying to do more of with the platform team and with our team and, and folks who, you know, can kind of just get out there and like be part of the community and, and be speaking and be on X and be wherever they are, right? I think the problem for us is just like having time and kind of knowing the right way to do that stuff at high quality. So I, I see the DevRel team in some ways as just being like the experts in doing this sort of stuff at quality. They can do it all day, every day, and they can help enable the rest of the team to think about how to do this with the small amount of time that they are able to spend on this sort of stuff to do it at high quality too.

• 1:41:52Speaker 00

  Yeah, so like, yeah, so I call this like kind of the unbundling of DevRel into a front office and a back office. So like a typical person who does this job does the whole thing, start to finish, they're kind of autonomous. But actually you can sort of build up this back office that lets your product leads and engineering leads actually take the spotlight with a simple contract of like, okay, you sit down with me for 30 to 60 minutes and we'll do the brain dump and then you're done. You go back to work.

• 1:42:18Speaker 01

  Yeah.

• 1:42:19Speaker 00

  And then we'll like, we'll like ghost, ghost write for you, whatever. But like that team of like production people, writing people, whatever, is sort of provides the service to the rest of, rest of people. And I, I think that, to me, that's like kind of the future of, of this stuff, to be honest. Because people like don't actually really wanna hear from DevRel, they wanna hear from Lance. And, and like, you know, you know, and you have like lots of, the Tariq, you have Tariq, you know, and Lydia, you got Lydia through Bud, you know? Exactly, but like he, he like functionally, he does the job, he does the job. And so like that, you want more of those. Because people, people wanna hear from people in the products, not, not like who are like paid, paid shows, like paid spokesperson.

• 1:42:58Speaker 01

  Yeah.

• 1:42:58Speaker 00

  And that's, which I get a lot of and like, that's fine, that's a job, somebody has to do it. And then like reasonably you can expect people who are working on engineering to not be good at it.

• 1:43:09Speaker 01

  Yeah.

• 1:43:09Speaker 00

  So like you should pay someone who specializes in it, like specialty is, is a skill. But I think there is a back office, front office separation that can emerge.

• 1:43:16Speaker 01

  Yeah, that is really cool.

• 1:43:17Speaker 00

  Yeah, and my analogy would be like, everyone hires, but you have an HR department. HR department enables you to hire.

• 1:43:23Speaker 01

  Indeed.

• 1:43:24Speaker 00

  You hire, like, yeah.

• 1:43:26Speaker 01

  Makes a ton of sense.

• 1:43:28Speaker 00

  Cool. That was my soapbox. You know, closing questions. You've been very generous with your time. Any parting thoughts or call to action?

• 1:43:36Speaker 01

  Build something, colon, Opus 5.5.

• 1:43:38Speaker 03

  What something people should try to push the bounds with new Opus 5.5? You guys have been using it for the longest.

• 1:43:44Speaker 01

  Yeah, I mean, it's really cost efficient. It's less verbose, all the things we mentioned. The vibes are really, really high. It's been fun. Yesterday was a really fun day for us because like finally again, I got like a whole slew of texts from friends and people who were just like, amazing model, thank you. Like one of my old bosses messaged me and he was like, Opus 5.5 just saved our butts in an incident, right? And it's like, okay, great. Like we're back is really the energy that we feel. And so I think things that like, maybe you wanted to experiment with something and like cost was something that was front of mind for you, right? And like token efficiency and these sorts of things like that kind of might have held you back from like really like firing it off and like letting a model run.

• 1:44:26Speaker 01

  I think that's the, those are the sorts of problems you should return to now with Opus 5.5 because I think you'll be able to get just like a lot further than you maybe otherwise would have been able to with models that were less cost effective.

• 1:44:38Speaker 02

  I think you should try something with like a multi-agent setup. I think you will find it to be very good.

• 1:44:43Speaker 00

  Any other hints, by the way? I mean, multi-agent is very broad.

• 1:44:45Speaker 02

  Uh, nope. I think we'd be very curious to hear what people build. But yes, that it's, it should be extremely good, so.

• 1:44:52Speaker 00

  Yeah, including, let's say, roles that are not strictly defined, but they are assigned among themselves, right? Which is to me, that the new thing, like you can assign roles, you could always assign roles. You can give tokens jobs. But this is like they're giving themselves jobs.

• 1:45:08Speaker 02

  Yes.

• 1:45:08Speaker 03

  I guess, say more on multi-agent, right? Like should we use sub-agents? What do you, how should people think about multi-agents?

• 1:45:16Speaker 02

  This is pretty good in a sense of like, basically, generally, you want to have one lead and you want to like split up like a bunch of different stuff. And as long as you can have a task that you think can be generally parallelizable and they share uniform context across it, I think you will get a lot of gains. Now, of course, it depends on the problem that you're pushing at, but like increasingly, this is a very superior way for us to handle a lot of these things. And especially if every sub-agent is, or whatever you'll call these, sub-leads, they are asynchronous and they're not blocking of each other. That is like the only two bits I would say you should care about in your architecture. Other than that, like let it rip and like see what Claude designs for you. And increasingly, I think a lot of this architecture stuff will fall into this direction. Like Claude will get smarter and smarter.

• 1:45:59Speaker 02

  It's going to learn how to figure ways to like split itself, quote itself, adjust itself to the architectures that make sense. You just need to prevent it, you just need to make sure it can like parallel process. So it needs a share state and it needs to like not block its sub-agents. And as long as you contain those two principles, honestly, I think people should experiment and probably try some really crazy stuff. And I wouldn't be surprised to see some like really interesting things on top of that that maybe they felt like there was some intractable problem that was really annoying and they'd keep pushing for, that something like this could actually probably just like crush and maybe one shot or something.

• 1:46:29Speaker 00

  Excellent. All right, everyone go build.

• 1:46:31Speaker 01

  Cool.