Ep 77: Anthropic’s Dianne Na Penn on Opus 4.5, Rethinking Model Scaffolding & Safety as a Competitive Advantage

This episode features Dianne Na Penn, a senior product leader at Anthropic, discussing the launch of Claude Opus 4.5 and the evolution of frontier AI models. The conversation explores how Anthropic approaches model development—balancing ambitious capability roadmaps with user feedback, making strategic bets on areas like agentic coding and computer use while deliberately avoiding others like image generation. Dianne shares insights on the shifting nature of AI evaluation (moving beyond saturated benchmarks like SWE-bench toward more open-ended measures), the evolution of scaffolding from "training wheels" to intelligence amplifiers, and why she believes we're closer to transformative long-running AI than most people think. She also discusses Anthropic's distinctive culture of authenticity, the under appreciated benefits of model alignment for producing independent-thinking AI, and why the real bottleneck to AI agents isn't model capability anymore but product innovation.

• 0:00Speaker 00

  Opus 4.5 is a really impressive model. It's had amazing benchmarks. A bunch of companies are getting great outcomes from it. And on supervised learning, I got to sit down with Diane, the head of product for research at Anthropic, to talk about all things Opus 4.5. We hit on what's newly enabled by the models, as well as where Diane thinks we are with computer use. We talked about how Anthropic does research and gets to models like this, and how they use these models internally. We talked about what's next for models and the evolution of scaffolding over the past years. And we hit on how Anthropic's safety focus actually helps it on the product side. This is just a lot of fun to get to go deep into the research process, model capabilities, and a bunch of other things. I think folks will really enjoy it. Without further ado, here's our conversation. And thanks so much for coming on the podcast.

• 0:38Speaker 01

  Thank you for having me. Happy early Thanksgiving.

• 0:41Speaker 00

  Yeah, Thanksgiving Eve pod is like a real, a real treat to get to do. I feel like you guys dropped this, this big model as a gift to us all right before the holidays. I'm sure a lot, I'm sure a lot of founders will be saying they're grateful for it tomorrow at the Thanksgiving table.

• 0:54Speaker 01

  Love it. That's, that's the vibe we're going for.

• 0:57Speaker 00

  You guys already had some really impressive models. What does it look like when you begin to work for something like Opus 4.5? You know, how do you think about the, you know, what actually is required to improve these models? And like, what does that whole process end up looking like?

• 1:09Speaker 01

  I think like, in some ways, we have a pretty, I would say, ambitious and like, long-range roadmap around model capabilities that we care about and that we care about making improvements on. So this includes things like better instruction following for users, coding advancements, making the models better at memory, et cetera. And really, in some ways, every generation of Claude are like the vehicles by which these capabilities are expressed. And so like, even when we are designing new versions of Claude, it's really around what are the advancements overall that we want to like, make sure can be delivered? And, and how do we actually package it also?

• 1:53Speaker 01

  Position it, price it in a way that like resonates for the types of use cases that users have today? But also, they might not even be aware that like AI can help them take on.

• 2:04Speaker 00

  And so you kind of pick a set of like, problems or things that you want to improve the models on? And then, you know, to what extent is it kind of clear the, the research directions to go down to improve that? Or are you having like, a dozen different things? And you're kind of figuring out early on which, which ones help actually move the needle there?

• 2:18Speaker 01

  I think it's both. I think we have like, had a very...

• 2:24Speaker 01

  Generally, a strong sense of what we think like this technology can do, which is that like we can think, we think that is generally can be extremely transformative across a wide variety of like use cases, whether it's like engineering or others. I think like other things like surprises with like how users and also like builders discover them. So things like making Claude really good at Excel and PowerPoint. That was like a relatively small investment earlier on in the year. And like what we found is that it really resonated with financial services customers. And so we're doubling down and making Claude really good at that type of like general Excel work or like PowerPoint work that seems really beneficial. So it's both.

• 3:07Speaker 00

  What's the right way to conceptualize like doubling down? Is that like, you know, getting more data around a topic or doing more RL around that area?

• 3:14Speaker 01

  Yeah, I think it does for like practical applications does start with like users and both users who might be coming to us with a use case. But also when we think of like something like computer use, why should people care? Like, we kind of have to imagine almost a world that we want to go towards. And so there's like almost an imagination phase. As a PM, it's the closest thing I could think of is like product vision docs or PRDs, where you're trying to figure out like, what is this, so what? Why should somebody like common use this like solution? And then also then translating that into like practically what are evaluations? What are evals, we call them, right? Like, what are the evals we want to build to know the model's really good or not good at it?

• 3:58Speaker 01

  Maybe there's, it's already half the way there and like it needs to make certain improvements on data or RL for those like the last 50%. Maybe it's 20%. We actually need to figure out like much more significant changes we want to make. And so starting with like sort of that envision in mind is actually very similar to traditional product management, which I think might be surprising us to hear.

• 4:22Speaker 00

  It's really interesting. I think especially as like the kind of classic evals feel like they're more and more divorced from like what actually the value is in the real world or the ways that people are using these things. You have this combination of both. Customers coming to you with here and now problems that are probably really helpful evals. You obviously have the best seat to what these models might be able to do in the future and really imagining what the, what some of those use cases might be.

• 4:43Speaker 01

  Yeah.

• 4:43Speaker 00

  Were there, were there any, you know, things that you were imagining that Opus 4.5 is the first model to be able to do that maybe, maybe we're in the vision doc like a year, a year and a half ago. And you're like, maybe one day we could do X or, uh, you know, what were some of the model capabilities that really surprised you?

• 4:57Speaker 01

  Yeah, I think some of them have been like continuations of, you know, being able to do more complex agent to coding tasks. But also making work that is like more iterative and actually longer running. So like, that's been one thing like we're starting to see, I think, more of an inflection point on both like complexity and also, um, being able to iterate and continuously improve, um, some of the deliverables. I think computer use has been another one, like, um, we saw and have been investing in computer use for a long time. Last year in November, I think November, October, we launched a computer use API and since then we continue to invest in it.

• 5:40Speaker 01

  And like I use things like Claude for Chrome, which is like our browser use extension feature pretty often. And like, I saw like an improvement just in terms of the quality of that interaction because now Claude's vision is much better. So I think sometimes it's a combination of like multiple things kind of working together.

• 5:59Speaker 00

  I guess there's, you know, a lot of builders listening to the podcast that, you know, are curious about computer use. And do you have like a rough mental framework today of like where computer use works and where it doesn't and kind of where we are on the journey?

• 6:11Speaker 01

  Yeah, I think like computer use have gone from this like very early stage experimental feature when we launched it. Um, and we were initially thinking about it as like, probably the first stage is like a complement or a feature on top of something core like agent to coding. To now, I think more and more between Opus 4.5 and beyond, able to be more of a like end to end agent just by itself. And so like, can it not, not just like within a more constrained environment, look at the types of like, you know, QA testing that was like popular use case. But actually like be able to help monitor and be an agent in like a web browser. It's still more constrained than like an open ended, like, let's say your entire like, um.

• 6:57Speaker 01

  Laptop, but it's like very helpful to be able to have like an agent that like helps me reschedule my calendar on Google because that does get pretty complicated. And yeah, it's, I think that's kind of just been kind of the arc, like moving from more constrained environments to towards things that are a bit more open ended.

• 7:15Speaker 00

  Was there anything that you played around with with Opus 4.5 when you first had it internally? That like was, was surprising?

• 7:21Speaker 01

  It was actually really helpful from a product team perspective to debate like things like pricing and like how our positioning for the model.

• 7:31Speaker 00

  Yeah.

• 7:31Speaker 01

  Um, because it's such a great like leap in many ways. So from a product manager perspective, it was actually not just a great writer, but it was a great thinker. It came up with like, you know, different, different ideas around things like pricing and positioning, um, where it wasn't just refining my ideas, but it came up with alternatives more spontaneously than I had seen with other models.

• 7:54Speaker 00

  You know, traditionally a lot of the Opus models were like much more expensive. And this is both a really good model and like quite cheap, honestly. What, what like drives that? And I guess, how clear was it from the beginning of this work that like this Opus model would be able to be served this efficiently?

• 8:09Speaker 01

  From the beginning, we were hoping that this would be a thing for Opus models where we're able to have more efficiency gains and we're able to pass it then on to our users and customers and builders. We intentionally also made it possible in addition to like the core model training for things like the effort parameter, which I actually think is like underhyped right now, uh, where you could actually get like Sonnet 4.5 level of intelligence at a fraction of the price.

• 8:38Speaker 00

  Yeah.

• 8:38Speaker 01

  And I think this is something that like we as an industry haven't gotten really good about, which is like, just because a model has a certain token price and sticker price, that's not always a good measure of like the end to end cost to achieve a task. And what we want to do with Opus and what we want to do with things like the effort parameter is to make it more accessible, that like you could actually achieve a higher quality and a lower cost. And, you know, we're starting with that with Opus and I'm really excited about this area going forward as well.

• 9:12Speaker 00

  Yeah. Do you feel like most, most folks like get that intuitively or like what, how do you think about educating the market around, yeah, this idea that it's not just the per token cost, but obviously the amount of tokens it takes to, to complete some of these tasks?

• 9:23Speaker 01

  Yeah, I think it's something we as model providers could be doing more around. So one thing I hear from builders is the smaller models actually just take longer to do a task, or it might actually not even get the task done, but you've spent a bunch more tokens than if you just use an Opus model. Um, you know, for a lot of this year, we were traditionally focused around like Sonnet and having a very core flagship offering. And as we move into like different tiers of models, it actually is becoming more important for us to educate developers and users on it. And so like, I think this is an area we're going to be investing in from like a marketing perspective.

• 10:03Speaker 00

  You all have a very good intuitive sense of what these models can do. But then when you put it out there, I'm sure they start getting used in like a million different ways that you, that you couldn't possibly have conceived of internally. In these first days of release, like what have been some of the things that have surprised you the most?

• 10:16Speaker 01

  Yeah, I think it's like very early. We're also in the middle of Thanksgiving week. So I'm sure my answer will, will change a little bit as like, uh, people have like tested the systems. I would say maybe like two things. Um, one, in our early access with customers, um, you know, we invested in areas like I mentioned around, uh, making the models better at like office, like deliverables. Um, it was really surprising to see how big of an accuracy jump Opus 4.5 was for customers like Shortcut, which is like, um, a cell agent. I think customers were saying something like 20% accuracy improvement, just like without changing harnesses, without making other changes.

• 10:59Speaker 01

  So that felt like resonated a ton because then they can pass that intelligence improvement to their users. Um, I think other things that I've seen just like generally, um, in the early days, I think people tend to test the models on like gaming use cases. And so like 3D games have gotten better. In particular, so that's always really cool. Visualizing some of this has just been an easy way for people to kind of see intelligence bounds. And I'm really excited also for just like the feedback that we're getting around quality. So customers and users saying like, oh, it's actually helped me clear out my entire backlog of bugs.

• 11:39Speaker 00

  Now with with Opus 4.5, as you think about kind of the broader ecosystem of applications and and, you know, I'm curious, your mental model of like, what does have product market fit or like really works today on top of these models?

• 11:53Speaker 01

  I think the big ones are, you know, agent to coding is like very visibly something that's like here to stay. I think we continue to actually have more like enterprises reach out about solutions like cloud code and like coding capabilities. I think like synchronous agents have pretty strong

• 12:20Speaker 01

  reason for PMF generally beyond coding. I think it's more that like we've not figure out the, yes, like a like a general like industry haven't figured out the right like harness plus product features to build on top. Like what is a, you know, agent to coding, but for like, you know, web monitoring and like personal agents. A lot of use cases are still very like chat focused. And so I feel like we're kind of at an inflection point that we need a bit more intelligence to kind of really boost the neck like large set of use cases.

• 12:54Speaker 00

  And do you think that like we have that level of intelligence now for those use cases?

• 12:57Speaker 01

  I think people will be surprised by how good Opus 4.5 is. And I think there will be new things that are born from from Opus 4.5. There'll be new features and products.

• 13:07Speaker 00

  So more of these like proactive experiences or like agents going out and doing things on the on your behalf and then surfacing it up to you versus you prompting like via chat.

• 13:15Speaker 01

  Yeah, that was one thing we heard a lot from like internally where it felt like the model just got it without having to have explicit human instruction. And I think when you couple that plus the fact that Uh, one, the context is getting high, the quality of the context is getting higher. And also things like memory start working better. Really, you get a system that can do not just one deliverable, but like do things that are not possible before, like monitoring, maintenance, like those are actually things that are like really valuable. Uh, it's not really just about shipping an MVP. It's about also like, how do you maintain uh, something that you build?

• 13:56Speaker 00

  I'm curious, you brought up that example of shortcut and like really improving Excel. I imagine there's like an endless set of people that come to you and they're like, we would love you to improve your model on like X domain or Y task we have. How do you prioritize that? And like, do you think that the foundation models end up going different directions just based on what they prioritize here?

• 14:15Speaker 01

  I think a little bit, yes, right? Like there's, you know, we do have customers that ask us about things like image generation or video. And we're very intentional. Um, and I've been here about like two and a half years. We've been very intentional about like focusing on like expanding intelligence. And so I, I think you do see that a little bit with the labs already. Um, in terms of your question of like feedback, like, um, I think it's kind of a two-way flywheel, right? Um, sometimes there are like pain points or paper cuts. We hear from customers. What's really good about like our shipping velocity today is if a customer comes to us

• 14:57Speaker 01

  or a potential customer comes to us, we can make changes pretty quickly. Cause if it's uh, in 4.5, maybe it's in cloud five, right? Like you have like multiple chances to get your feedback heard. Um, and then how do we also automate more of that? Like, how do we build the systems on our side to like more organically? Figure out the right environments to build and like build that like loop ourselves. Um, and then how do we deliver like new advancements uh, more, more generally? So like things like computer use and beyond. So I think it's a little, it'll be bidirectional.

• 15:32Speaker 00

  Yeah, it's interesting. You talk about building some of these tools internally, like, uh, you know, more easily spinning up environments and being able to uh, lower, I guess, probably the amount of effort that is required to, you know, to improve the models in these specific categories. And I'm sure there's like internal tooling that really helps with that.

• 15:47Speaker 01

  Yeah, it's definitely a continuous, um. Investment area. We just have a lot that we could be doing. And so we have to be really intentional about like where we spend like to the point, how do we get Claude to help us?

• 16:01Speaker 00

  Yeah.

• 16:01Speaker 01

  We're done.

• 16:02Speaker 00

  I mean, it sounds like multimodal, like, you know, obviously, has been below the, the, the, you know, the focus area. Are there any other things that you think of of like, yeah, we could, you know, we've explicitly chosen not to, you know, spend too much time on X or Y?

• 16:14Speaker 01

  Um, I think those are some of the big ones. I think like we've also, from a product perspective, been very intentional about focusing on business use cases generally. Right? And so like this like means that like a lot of our focus and investment in product is around things like data security, privacy, meeting like enterprise requirements for them to be able to adopt and use Claude over like, let's say, like, consumer use cases.

• 16:44Speaker 00

  It feels like there's been this like really interesting discourse around enterprise agents over the last few weeks, I think largely driven by the podcast ecosystem. Where, you know, you had, I guess, Andre Karpathy going on Darkness's pod and being like, you know, hey, I don't know if these, this is really it. Like, it feels like we're still a decade away from, from real enterprise agents. And then I guess Ilya most recently came on and was like, this current paradigm only gets us so far. Like, we're, we're going to kind of hit a wall. You know, you then obviously square that with all the improvements in something like Opus 4.5. Where do you kind of, what's your opinion on all this discourse and like the, the feeling, I guess, maybe within Anthropic around it?

• 17:18Speaker 01

  I think, um,

• 17:21Speaker 01

  intelligence and improvements and model improvements. What I've learned is like improvements are not necessarily like always a smooth line, the way that you see in an eval benchmark of like, it's always like this. And I think they're more jagged edged. And I think depending on what eval or personal tests that you have, maybe it feels like a small jump, or maybe to someone else, it feels like a large jump. So I think there's like a level of like, what is the exact like framing you're looking at? Um, I think like

• 17:54Speaker 01

  from a customer and user perspective, what we hear from like a Rakuten or a lovable, etc., is that like,

• 18:02Speaker 01

  capabilities have actually still continued to to improve their team's productivity. And I think like really from like a is AI transforming technology and we talk about like less AGI, but more like transformative AI internally and the tropic. Like, are we making technology as that is transformative? I think we're very much on the path on that. If Claude is like transforming every generation of Claude transforms internally. How anthropic employees work. And so we feel that because we're adopting it in a different degree. And so maybe if you're not adopting it to the same degree, it might not feel like it's meaningfully like make making advancements.

• 18:42Speaker 00

  You've obviously gotten very far and I imagine some of the things that were on the board are off now. But what's on like the next board?

• 18:47Speaker 01

  Nobody can predict the future, but I think like some of the areas that I'm really excited about is like this move towards like longer running intelligence. So again, not just it. A human gives and delegates a specific task to Claude, but actually like Claude taking responsibilities that are more open ended. So not just like build me this like portion of my website, but maintain it. Refactor the code when you think it's correct and not needing so much of like handholding. I'm also really excited about capabilities like office improvements that I mentioned, things like Excel, PowerPoint. There's a lot more to go.

• 19:33Speaker 01

  And also computer use, I feel like computer use is actually on the way of like adoption and quality where like it could really be transformative. From like a enterprise and also just like general user perspective. The way that like a edge of the coding could be like computers, like being able to navigate a computer is how most of us interact with each other. We're on a podcast right now, right? So like being able to have Claude that like can interact in these environments where like people work makes them much more useful. So really excited about like where computer use will go.

• 20:12Speaker 00

  I mean, you mentioned long running agents and I feel, you know, it feels like this is certainly where the world's headed. And you know, maybe even having a fleet of agents doing work on your behalf. It feels like in many ways, you know, obviously the models will get better. But in many ways, part of this is solving the product side of it. And like, what does it actually look like to have a long running agent going on in the background? And what does it look like to have a set of a dozen of them that you're managing? You're obviously a product guru. Like, how do you think about what the product surface area might look like here? And, you know, have you seen anything that you thought was particularly interesting?

• 20:44Speaker 01

  Yeah, I think long running agent and like long running intelligence. Generally, it's like a, it's not a use case in itself, right? We need to like actually have like a user problem that it's like a good fit for.

• 21:00Speaker 01

  So I think things that are like really valuable are around like being able to maintain, iterally improve. Right? Like, let's say you have like a, like you're an investor and you're, you want to like, understand the latest like stock movements or like how you should adjust your portfolio. That's like a one time thing. And I think like what we kind of lack today is like having these like really long horizon tasks in a way that it makes it easy to eval quality improvements. So like the closest one that I think we're starting to like, um, track. It might not be the right one because I think, you know, no eval is perfect. It's like vending bench, which is the Claudius, like run a vending machine business, right?

• 21:45Speaker 00

  But I love that you guys do this.

• 21:47Speaker 01

  Or, or, or, you know, Claude plays Pokemon, right? Like these just, you know, Claude played Pokemon when we first.

• 21:54Speaker 00

  Yeah, I kind of missed the Pokemon eval. That was, I like, I don't know if I just missing it in the reports. I liked when that was like the, uh, one of the main ones people were discussing.

• 22:01Speaker 01

  Okay, I am, I will take a note for like potentially bringing it back. But like.

• 22:07Speaker 00

  Maybe the models are all too good and they beat the game really quickly, but.

• 22:10Speaker 01

  Well, well, it's also not just about completing tasks, but like, how long does it take? Right? And like, can you actually complete the task in much less steps? Because you're not brute forcing the problem. And like, you remember the fact that like Ash Ketchum was like in Palatown at this time and already talked to like some user, right? Like to nerd out for a second.

• 22:29Speaker 00

  We can go deep on Pokemon. I'm, I'm, I'm all game for that.

• 22:33Speaker 01

  Yeah, but like, I guess the, the, the part of what I want to know here is just like, you know, intelligence isn't just about like, can I complete a task and check the box? More and more, it's also the quality of the judgment and the quality of like how. What intuitions some models can have in a situation. And so can we actually dramatically reduce the time it takes or the amount of effort to get to a certain outcome? And sometimes this really shows up in things like around long running tasks, right? But really, I'm just like excited about this area. I think what we need more as an industry is better evals around it.

• 23:11Speaker 00

  And what are the evals that still kind of matter?

• 23:14Speaker 01

  Oh, it's a very good question.

• 23:17Speaker 01

  I think we would need to evolve beyond things like Sweetbench and like Talbench. I think where especially with Opus 4.5, I think we got to like 80.9% and like these DC valves are so saturated.

• 23:29Speaker 00

  Also loved the example of like finding a way to upgrade the ticket through like, you know, changing a fare category. And that was a very fun way to to to to get around things.

• 23:40Speaker 01

  Yeah, it's like, it's crazy because like when I shipped the tool use API last year, I think we were in like maybe below 50% on accuracy. It's just again, there's so much advancement. It's really how do we measure and how do we productize it? I think the evals that matter continue to be for us like, what are the areas where people are using Claude and how well does it work? So I think like end user measurements of quality continue to matter or feedback that we get from customers. I do think we will probably see more movement towards evals that are more open ended, like a vending bench sort. It's not going to look exactly like that because I don't know how realistic running a vending machine is all the time.

• 24:24Speaker 01

  But like that type of direction of like, it's open ended. There is some quantifiable way to measure quality, but it's not just like yes, no. Because how much of like most things in the world are yes, no tasks, right? If we're moving beyond coding into other types of like impact, like.

• 24:45Speaker 01

  It's hard to say, like, is there a yes, no, like, sweep bench equivalent in, like, biology?

• 24:51Speaker 00

  And you've kind of alluded a few times now to, you know, the harnesses that people are putting around these models. And obviously, you work very closely with teams that are at the cutting edge of using OPIS4.5. How would you, like, you know, do you feel like there's a typical kind of set of scaffolding people are building around models today?

• 25:05Speaker 01

  I think similar to model intelligence, scaffolds have evolved. I would say in like 2022, 2023, even 2024, a lot of the scaffolds are more like training wheels to keep like the model on distribution. And they tend to be of the form, do not do this, always do this, right? Just like instructions and like 20 rules. And I think more and more this year will be seen is scaffolds become much more around augmentations for intelligence rather than training wheels, or like the best scaffolds tend to be. And it really removing the parts of the scaffold that are no longer intelligence amplifying.

• 25:51Speaker 01

  So, for example, on like Claude Code, our scaffolds are relatively lightweight. The types of tools we give it are things like bash tools that are not like specific, very unique. And the point of that is that we want to maximize autonomy of like the work as it's being done by the model. And so the like types of scaffolds that I think will continue to be valuable, maybe things that are intelligence amplifying. So things that are like giving it like generic sets of tools, multi agents, I think, have started to be like more viable this year.

• 26:31Speaker 01

  Where it's like, not just like having one model, but like orchestrating a set of models to like amplify and improve on things like context, quality, etc.

• 26:40Speaker 00

  I think a bunch of builders are asking themselves, you know, what set of this stuff becomes obviated by the next generation of models versus, you know, continues to amplify intelligence for that next set? Is it obvious from the inside or is it also like, you know, we'll, we'll see when we get to those models?

• 26:55Speaker 01

  I think there is a bit of like, this is why having like a thinner layers of harnesses and like scaffolds is important.

• 27:03Speaker 01

  I think though, like, how much does it change? Like, it's a little bit hard to predict. But we do see improvements in quality by some level of iteration on scaffolds by model. Part of it's also users' requests also change, right? Like, what we see is like, you know, when I use like Claude Code or use Claude, if I see that it's really good at editing a document, I might give it a large set of things and like, hey, come up with an alternative strategy. Right? And so like, we're continuously also as users pushing the products that people build. And so it's not just in service of like new models to update your scaffold, but actually in service of like user behavior.

• 27:47Speaker 01

  And users will naturally push your product to be slightly more complex. And how do you as like a developer meet that demand? It's like, I think the right way to think of it.

• 27:59Speaker 00

  You've kind of been at Anthropic since the early days. And so I feel like it must just have been a fascinating journey these past two and a half years. I wonder maybe to start, like, how do you kind of compare the culture of Anthropic to like other places that you've worked?

• 28:13Speaker 01

  I would say a lot of how like, you know, our leaders

• 28:18Speaker 01

  show up in and how people talk about the mission and the goal. It's like very much actually how we internally, a big part of how internally we make decisions. Like, it's actually a lot of like walking the walk. And I think Anthropic has been the most like authentic. And as much from like the first day I joined to like now of a company that I've been in. I think it goes without saying, but like the talent caliber and talent density has been incredible. It's been the most like talent dense like environment that I've gotten to work with. And I think people are like just deeply take like radical ownership,

• 29:00Speaker 01

  deeply thoughtful, kind, but direct, and also in the service of making products, models, capabilities, research, better.

• 29:11Speaker 00

  I'm sure our listeners will be curious, like, what does your day to day look like?

• 29:14Speaker 01

  I think like every three months, my job changes because it's almost like we're a different company. I think we were like 150 people when I joined and I like two years ago around this time, I was setting up like our first A B tests like and emailing prospective customers to like try new models. Right. It was like extremely hands on, you know, like as a PM, you do whatever it takes to get this thing done. Um, I think now more day to day, as we have like more product managers, like more researchers, more engineers, a lot of my day is spent like helping the team and a lot of my PMs are more embedded with their research counterparts.

• 29:58Speaker 01

  And so like coaching, supporting them is a big part of my day. Maybe a third of the time with calls with customers, just better getting the sense of like all of the different use cases, particularly what's emerging, like what is not working today that like is really, uh, could be really pivotal. Um, so spending a lot of time with like either businesses or startups. And then, uh, I think, you know, especially in this environment, thinking about what's ahead.

• 30:28Speaker 00

  Obviously, what a wild last two and a half years. You know, I, I guess as you reflect back on that like journey, are there some like key decision points that stick out to you as like, oh, that, that really like tipped things, uh, or, or was, was pretty pivotal?

• 30:41Speaker 01

  I think we were very intentional, uh, in the early days. I would say in 2023, the most common user request we got was. You guys need to have an embedding model because we're doing RAG and RAG is all the rage. And I think we were in very intentional about what LLMs and AI could do and what Claude could be. And we focused on things like investing in agent to coding. And so this is a little piece of like being user led versus user centric. Right? Like, what is the thing that they're not even asking for? Like, nobody's coming down our door in 2023 being like, gave us engine to coding.

• 31:23Speaker 01

  So, so I think that's one, like just the early days of like being very clear focused around what is the bigger like opportunity. Um, I think another one personally is when we did choose to ship computer use. Um, on the API and we shipped it as an A, a beta feature. Um, we knew that it didn't quite work completely yet, but we thought this was really helpful for like showcasing what AI could do. Um, and how it's just a different form factor. I, I think we're still on that journey, but like bring that to the world. And there was a lot of discussions around like, hey, how do we make sure like

• 32:02Speaker 01

  everything is like very safe, et cetera. And so we did a lot of safety work, but it's impossible to capture every edge case. And so we had to put it out into the world a bit to like, then figure out what else we need to like make safe as the capabilities expand. And so I thought that was like, just like a very authentic um, moment, um, making a bold decision, um, that I think was the right one. And yeah, I think those are, those are probably like two of the large things. Um, there's a lot of other fun moments, but they're not like, it's like, I think pivotal i- i- in my mind.

• 32:38Speaker 00

  Well, what's, what sticks out as a fun one?

• 32:40Speaker 01

  Um, so my team and I worked on Golden Gate Claude, which was an

• 32:45Speaker 00

  Oh, yeah.

• 32:47Speaker 01

  Of our interpretability work. And that was very cool because um I think the company was still less than 500 people, but it was starting to feel a bit bigger. And we shipped Golden Gate Claude from model to the UI within less than a day.

• 33:07Speaker 00

  Did you know it was going to be this like kind of viral thing?

• 33:09Speaker 01

  It went a little viral internally. So I think like usually Anthropic employees have good like product sense and taste. So we were like, okay, if we like it, let's just try it. Maybe it's like, we get like maybe a couple hundred people who like geek out with us.

• 33:25Speaker 01

  And but we didn't know. But yeah, it was like very grassroots. Some engineers, researchers, PM and designer. We were like, okay, let's figure out how do we show this to users? And we needed to do it in the week because we had published a paper. So that was probably one of like the proudest moments.

• 33:45Speaker 00

  It's been a fascinating conversation. We always like to enter interviews with a quickfire around where I basically just stuff in as many questions as I can before we we run out of time. And so maybe to start, what's one thing you've like changed your mind on in AI in the last year?

• 33:57Speaker 01

  I actually think we are closer to transformative, long-running AI than I expected even starting the year. Like it actually feels like the building blocks are kind of there more. I feel that more now than than than in the beginning of the year.

• 34:14Speaker 00

  You obviously have a range of sophistication of end users of the Claude API. Are there things that the most sophisticated folks do that you wish that just all of our listeners would would do with these models to use them more effectively?

• 34:27Speaker 01

  I think there's like maybe two, two big things. I think the boldest like builders and users are constantly thinking about like, not just what's working today, but actually might have things off the shelf that like did not work before, or some prototype that doesn't quite work, that like they still kind of put together and test with new versions of like models that come out. So like we call this kind of like prototyping, like just having library, user-centric prototyping.

• 35:05Speaker 01

  And I think like that has been something that's like been really valuable to see of like, because a lot of times these systems are not. Planned, it's almost like you have to discover the capability. And if you don't have a prototype, then you're not going to discover the thing. You're always going to wonder, oh, I wonder when it's going to get really good at drug discovery. But if you don't have like some way to actually like test that each time, like it's always going to be too, too, too abstract. So I think like builders having very ambitious like prototypes, product ideas or features that might not work in the past. But just like having things like Hackathon where you could actually test these ideas is really important.

• 35:46Speaker 01

  And then I think like being willing, when you have a new version of a technology to like, invest and actually potentially changing your product experience to like meet the intelligence tailwinds.

• 36:00Speaker 00

  Yeah. What is model taste?

• 36:02Speaker 01

  I think of model taste as like, just like, you know, product sense and like product taste is kind of develop over time. It's really like a continuously honed sense of like model capabilities and a willingness to discover the capabilities by being hands-on. And also by understanding what users are trying to do and like continuously like iterating on that. What do you think model taste is?

• 36:33Speaker 00

  Yeah, no, I mean, I love the, I love the idea of like, you know, getting your fingernails dirty with the models, right? And just like, you know, developing some intuitions to both, you know, what they can and can't do. And also the right ways to push them or build scaffolding around them, you know, in order to get the most out of them. There's just some people that you give everyone the same tools and they always seem to just get way more out of them.

• 36:55Speaker 01

  Yeah, I think it's like your willingness to like experiment. Like, I think kind of creatively, problem solving is not the perfect word, but people who are willing to kind of like approach models as like a new discovery. And you're constantly like trying new things. That's kind of a way of developing model taste.

• 37:17Speaker 00

  Well, I'm struck by too, you talked about, you know, folks having, you know, being able to try new things as the models come out. You know, I imagine it's not as simple as just like plugging in OPUS four, five and seeing if it works because obviously there's a lot of scaffolding one has to build. And so I imagine part of the art is also being able to have some intuition as to, you know, I'm not just switching out the API key and up, it doesn't work, but I'm, but I'm actually have some idea of like things to tweak that that actually might make it work.

• 37:41Speaker 01

  This is why we actually internally have things like hackathons pretty regularly. I think we do a hackathon maybe every three or four months internally at Anthropic just because people have these like pent up ideas of like, I wonder if Claude can now do X. And you really need to give builders, engineers, product people, a place to do that creative work and do that discovery work outside of their day to day. Cause yeah, like, you know, nobody asks for a thing like agent to coding to work and it has to be a bit discovered. It has to be a bit, you know, hands on to, to, to figure out if like something is possible.

• 38:21Speaker 00

  Yeah.

• 38:22Speaker 01

  And giving people the place to do that is really important.

• 38:25Speaker 00

  I guess you, you talked about obviously long running agents, you know, being, being closer and all these things. What's like one thing you think of as a society, like we're not talking about enough or one of the implications of all this that maybe is under discussed?

• 38:40Speaker 01

  I think we don't talk enough about the benefits of safety, not just of the fact of, hey, it's to make sure the model doesn't do bad things, but more also like, what is the benefits of an aligned model? One of the key issues that is like an active research area is around things like sick or fancy.

• 39:04Speaker 00

  Yeah.

• 39:05Speaker 01

  That like models are just tell you what you want to hear. And actually a well-aligned, safe model. It's not, it's actually the opposite. So it's actually independent thinker, right? And independent thinkers are actually how we have breakthroughs and how we have better ideas. And I don't think we really talk enough about why safety is actually really good for higher value intelligence, too. It's not just a constrained AI. It's actually a way to like amplify quality of intelligence if it works well. Like coming back to my like example in the beginning, like I asked Claude, like, hey, how do we think about pricing?

• 39:46Speaker 01

  Here's two options and it comes up with a third that was really good that pushed my thinking. And if we had a very sycophantic version of Opus 4.5, it might not have done that. It would have just agreed with me, right? And so it's like, I don't think sycophancy is solved by any means, but just like by investing in like making AI that's aligned. We could actually get to a better intelligence. I feel like we don't talk about.

• 40:09Speaker 00

  I love that example. You know, obviously I think folks at Anthropic famously have like very aggressive ASI timelines. Like, do you, do you fit into that bucket or how do you think about that?

• 40:20Speaker 01

  I would say my timelines have probably moved up this year. Based on things of what I'm seeing with things, uh, models like Opus 4.5. I feel like the building blocks are actually closer than we think. And that it's actually more of like a product overhang or like product opportunities to express it. Um, and how do you like, again, build the right scaffolds or build the right ways to harness like the model quality? And so I'm more of like a barbell. I think like there's like a very large probability in short term. And then like very large probability in long term.

• 40:57Speaker 00

  Well, this has been fascinating. I feel like I will delay the, uh, the launch of the future generation of models if I keep you any too much longer here. But, um, I just want to make sure to leave the last word to you. Like where can folks go to learn more about Opus 4.5? About you, about anywhere you'd like to point them. Uh, the mic is yours.

• 41:13Speaker 01

  Yeah, I think our blog and our website, I think that, you know, we are pretty like, we like to like the words speak for itself. So I think following on our website is the right place.

• 41:27Speaker 00

  Yeah, I guess you guys famously don't do the like coming tomorrow or like the cryptic, you know, tweets before model launches that everyone else seems to do.

• 41:34Speaker 00

  Well, awesome. This has been fascinating. Thanks so much for the time.

• 41:37Speaker 01

  Thank you so much. It was great chatting.