Pod Insights Decoder The good, the bad, and the future of AI agents About this episode This is Hayden Field, senior AI reporter at The Verge and your Thursday episode guest host. Today, I’m talking with David Hershey, who leads the applied AI team at Anthropic. I wanted to have David on because earlier this week, Anthropic released a brand-new AI model called Claude Sonnet 4.5 that’s been making waves.
So I wanted to sit down with David, who spends a lot of time testing out what modes like Claude Sonnet 4.5 can and can’t do, to ask him where we are on this promise of AI agents, and also what the path forward looks like as agentic technology progresses.
Links:
Anthropic releases Claude Sonnet 4.5 in latest bid for AI agents | The Verge
ChatGPT’s built-in Buy Now button has arrived | The Verge
OpenAI really wants you to start your day with ChatGPT Pulse | The Verge
Anthropic’s Claude AI is playing Pokémon | The Verge
AI agents are science fiction not yet ready for primetime | The Verge
Agents are the future AI companies promise and need | The Verge
Amazon is betting on agents to win the AI race | Decoder
Credits:
Decoder is a production of The Verge and part of the Vox Media Podcast Network.
Our producers are Kate Cox and Nick Statt. Our editor is Ursa Wright.
The Decoder music is by Breakmaster Cylinder.
Learn more about your ad choices. Visit podcastchoices.com/adchoices
Transcript 9,507 words · 11 speakers
0:00 Speaker 00 Companies shouldn't have to choose between speed and control. Amex Corporate connects cards with flexible spending controls and real-time visibility to help teams adapt faster. Built to simplify, designed to scale. This is Amex Corporate. Get started at go.amex slash Amex Corporate. Terms apply.
0:28 Speaker 01 Hey, Bose here. When your next track starts, listen for the decisions that the artist makes. The vocal pushed right up front. The bass that waits half a beat. That sample from a 70s underground gem. Every choice deliberate. With Bose, you'll hear it all. Your music deserves Bose.
0:54 Speaker 02 New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess Collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want online only at homedepot.com.
1:26 Speaker 03 Hey there and welcome to Decoder. I'm Hayden Field, senior AI reporter at The Verge and your Thursday episode guest host. I'll be subbing in for Neelai for a couple more episodes, and I'm excited to keep diving into the good, the bad, and the questionable in the AI industry. Today, I'm talking with David Hershey, who leads the applied AI team at Anthropic. David works with startups to help them figure out how to best apply Anthropic's tech. Plus, he tests new AI models to understand their limits. I wanted to have David on because earlier this week, Anthropic released a brand-new AI model called Claude Sonnet 4.5 that's been making waves. For reference, Claude is to Anthropic what ChatGPT is to OpenAI. This new model, Sonnet 4.5,
2:10 Speaker 03 is being billed as a big breakthrough in autonomous, agentic AI, especially for coding purposes, which is a big battleground in the AI market right now. All these companies want to get a slice. These types of AI products can, in theory, be given complex tasks and then go off and complete them over the course of many hours, or in some cases, even multiple days. And Anthropic says this particular model, Sonnet 4.5, can run for up to 30 hours straight without any human intervention, all while working on a singular task, like building a software application from scratch. For the last year or so, companies like Anthropic, Microsoft, OpenAI, and others have been promising that this agentic technology would be
2:53 Speaker 03 the next phase of AI, the next big hype-filled thing that comes after general-purpose chatbots. They say it could really unlock generative AI's potential, and it's true, they've made some strides. But as we've seen so far, agents aren't quite there yet, and they have a ways to go. Most of us are not, in fact, sending agents off on the internet to do our bidding, and we're certainly not giving them tasks that might take 12, or 24, or even 30-plus hours of autonomous work without human handholding, at least not yet. At the same time, many companies are looking at agents as the breakthrough that's supposed to unlock huge productivity gains from AI models, including the opportunity to use them to replace or augment human labor. So I wanted to sit down with David,
3:37 Speaker 03 who spends a lot of time testing out what models like Claude Sonnet 4.5 can and can't do, to ask him where we are on this promise of AI agents. I wanted to talk about what these types of products are good at from a consumer standpoint, beyond just programming purposes, and also what the path forward looks like as AI agents progress. Okay, here's Anthropic's David Hershey on the state of AI agents. Here we go.
4:17 Speaker 03 I wanted to ask you about your view of the current state of play for AI agents. Like, we hear all the time that agents are the next big thing for generative AI, but are we still in the prototype stage, the testing phase, or what? How would you characterize what AI agents do today, right now, relative to what AI companies actually want to offer in the end? Which I hear from a lot of execs is basically Jarvis from the Marvel movies.
4:43 Speaker 04 I am less confident in our ability to predict the end, but I'm happy to talk about now. I I've seen agents come a long ways in the last year working with customers and my view is there are places where we're starting to see what it looks like when agents work really well. And there are still a lot of places where they don't work really well. And I think that kind of makes it confusing. I think for some people, code is a great example. When you're writing code, and especially with Sonnet four point five, when you watch the model, spend a lot of time as an agent, developing software itself, it's incredible. They can do a ton. It's like gotten much better. Literally this week, as we release models, it's really visible and obvious how much better they're getting.
5:26 Speaker 04 Sort of if you're plugged into that at being an agent, doing really long running, complex tasks. When you look at other sections of the economy or other like jobs that people want agents to do large pieces of, sometimes there's stuff they're still not good at. In some cases, they're not good enough at deciphering what's on a computer screen, so yeah, to be able to navigate complex UIs or whatever it is. And so they fall over and stumble over themselves on on something sort of silly. And it's easy to point at that and say, what are agents all about? Like, is this kind of fluff or hype or whatever it is? And I I think the way that I see this sort of generally happening is we've like slowly been ironing out the kinks of the stuff that models fall over themselves on.
6:08 Speaker 04 And we've done that the best so far in coding. I think the industry has done that the best so far in coding where we're making really fast progress on how much an agent can accomplish when writing code. And we are, I think, starting to make that progress in a lot of other domains. For example, like I talked about clicking on UIs in a computer. This model, Sonnet four point five, is way better at that. I don't know if it's like exactly the point we're going to tip into people like trusting it to automate a whole bunch of stuff they do when they're clicking on a browser yet, or getting there. And so I think my general view is we're making really fast progress. It's just not necessarily visible in every part of the economy yet. And then every job and every person and every individual.
6:51 Speaker 04 I have a feeling sort of each model that comes out will get one bit closer to being something that everybody can sort of interact with and see.
6:58 Speaker 03 So it's great at developing software. It's great at coding. It kind of reminds me of the robot hand moment. You know, how robots can do things that are really, really complex and hard for humans. But actually grasping something has always been a real headache. What about consumer facing stuff? You talked a little bit about UIs, but you know, what do you think agents right now are the absolute worst at? Like, what's the simplest thing that they truly just cannot do?
7:24 Speaker 04 I honestly sometimes struggle to put my finger on this because I think it's it's surprising little funky things in a lot of different agents. So for example, if you're trying to do something related to finance, maybe there's like a bit of manipulating a spreadsheet that is really hard and it falls over. And so it's like 99% of the stuff it can do, it can do the math and it kind of understands how a finance model works, but it will stumble over a spreadsheet. This is the thing you talked about. I don't know if there's like, I think it's hard to boil down all of the jobs that we want to help people do and one tiny little thing that the models aren't there. If it was that easy, I guess we would probably be on top of it already. And I think like maybe a better model is for each of these different things that we want models to help us with.
8:09 Speaker 04 There's like a million little components it breaks up into. You need to be able to like see the right cell on a spreadsheet and know how a formula works and know the macroeconomic model. And thing by thing, you can like find any different tasks, the little like thing that's not quite there yet. And so as we think about it, like coming from Anthropic, when we think about it, it's like, we just have to sort of be able to iron out, like fix each one of the little gaps in each of these to to help everybody use it. But I honestly like, this is probably an unsatisfying answer, but I can't quite put my finger on like, there's just like this one thing. I think this is why this field is hard, where we're just sort of like constantly working on the whole universe of the stuff people do around a computer and trying to help them out. And that just means a really wide scope. And there's just a lot, it's like from my side, it's a lot, it's fun.
8:53 Speaker 04 There's a lot of hills to climb to try to make the models better at all of the things that we wish they were good at. So we're working on all of the ones.
8:59 Speaker 03 Being someone that works with a lot of different startups and a lot of different industries on how they're actually applying AI, what are the industries that have surprised you the most? Like, what are the, you know, what sectors are clients in that you just wouldn't really expect? Or maybe the ones that you've seen the most growth in in the past six months, the past year? You know, what are the trends you're seeing in terms of which sectors are really adopting agents at scale?
9:27 Speaker 04 I have two versions of this, but I'll give you the fun one first. I think like one of the domains that surprised me the most in my personal customer work is the legal domain, which is at some value, at face value, it's apparent why that can be really useful. There's this, all this information you need to know about case law and studies and it's just fusing information is something that's like pretty obvious models are good at. But actually, there's so much depth and complexity to the legal field, which I didn't appreciate. I'm not aware. I didn't appreciate it until I started working with people in the legal domain.
9:58 Speaker 03 I used to write a lot about legal AI and how the law sector was kind of the last to adopt a lot of AI tools because they were super old fashioned. I did a couple of trend pieces on that. And then when they started to adopt it, I was really surprised at how quickly it came.
10:12 Speaker 04 Yeah, that's exactly sort of like what caught me off guard was you would think like there and honestly, there's so many things that make it hard. Writing a really good legal system, you typically need a lawyer to tell you if it's good or not. And like having to have lawyers in the feedback loop of how to build is challenging. And so I've been really impressed with a lot of the companies I've worked with and their ability to sort of like work around that where they like have this interesting build of companies that have like lawyers on staff to help with product building, like, like, like big amounts of lawyers on staff to help them build products. They build agents and cool things like be able to like comb over case law and look for the right details and and pieces. And then obviously use the stuff that AI is obviously good at of synthesizing answers at the end.
10:56 Speaker 04 But how quickly that field, I think it's just like a, it's probably like the, the scale of the upside there of how much just the pure volume of work that needs to be done and how much it can help that has driven the speed that people go. But that domain has surprised me. I promise you a two part answer. My, my second part of this answer is I think the funny thing about this space is it's really hard to guess where the next agent is going to take off because of that thing we talked about before, or I talked about before, where it's like, sometimes there's just this like one little tiny thing. That's not super obvious to, to any of us that's blocking an agent from working. So like, if you wanted a model to do your taxes for you, I think that'd be very nice. I don't look forward to doing my taxes every year.
11:40 Speaker 04 And you just find out that like, actually, what it's really bad at is you upload your W4 and it can't quite see the difference between the two boxes on your W4. And that's why the whole thing doesn't work, you know? This is a toy example. I don't think that's actually where models are, but.
11:54 Speaker 03 Yeah, that would be pretty complex. I remember when I worked in two states or I had a job in one state and I lived in another. It was also really hard for me to figure that out. So not surprising that AI can't either.
12:05 Speaker 04 Yeah, it is complicated, but like I think we can get there. Like that's clearly it feels in the domain of what the models can do. I've seen them do much more complicated. I don't know. They're better than me at math by a decent amount. So I would expect them to be able to do this. But sometimes it's just like this one little thing. And so I'm kind of like constantly surprised and it mostly is around when we release new models. Like it turns out there's some set of things that the model can do now. And then you see new agents crop up that we're doing things we couldn't do before. And so there's like, like nice little micro surprises that come out. I don't spend a lot of time thinking about accounting normally in my day job, but then you run into an accounting startup that can certainly do something new and interesting. And that's pretty cool.
12:43 Speaker 03 Do you think that data annotation is going to be a big part of that? Do these models get tripped up because there's some data that they don't have enough of, especially with a niche industry? You know, is that kind of where you see some of the obstacles come into play?
12:56 Speaker 04 I think we need to learn from specialists and we need like great ways to learn from specialists. And I don't know if it's necessarily data in the classical sense, that there's some like pile of data that we're going to learn from that makes it better, or there are just better ways we need to incorporate the intelligence of accountants and lawyers and other people into our models. Yeah, I think, I think that could come from data. Some of it certainly will. I think that can come from like talking to and interfacing and working with people in those domains. I think a lot about learning directly from our customers. I think there's a future where more companies can contribute more directly to making the models do the stuff that they care about.
13:38 Speaker 04 It would be really nice if someone saw this big important thing that they'd like to achieve instead of having to sit around and wait for a lab to hopefully build a model that helps them do it. They can sort of like directly work with a lab. And I think that's probably somewhere that, well, I know that's probably somewhere that we'll, we'll head, where we can have more direct mechanisms of working with sort of experts. And yeah, it's partially a data thing, but I also think it's just like.
14:02 Speaker 04 I don't think it's surprising that one of the things that models are great at today is software engineering. When the building that I'm in, in Anthropic headquarters, is filled with software engineers who know how to make models great at software engineering because they write software all day, you know?
14:16 Speaker 03 Yeah, that's true.
14:17 Speaker 04 And, yeah, exactly. Like, it's deeply unsurprising that that's true. And I think as we grow up as a company that's only a few years old and has spent most of our time hiring software engineers and figure out how to consult and work with more people in more diverse places and doing more diverse jobs, we'll get better at building models that are great at all the other things that people wish models were great at.
14:40 Speaker 03 We need to take a quick break. We'll be right back.
14:55 Speaker 01 Hey, Bose here. When your next track starts, listen for the decisions that the artist makes. The vocal pushed right up front. The bass that waits half a beat. That sample from a 70s underground gem. Every choice deliberate. With Bose, you'll hear it all. Your music deserves Bose.
15:21 Speaker 02 New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want online only at homedepot.com.
15:53 Speaker 05 This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends, like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an Ultra. Because it's not just a superior light beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser-Busch InBev Michelob Ultra Beer, St. Louis, Missouri.
16:25 Speaker 03 We're back with Anthropic's David Hershey discussing the landscape for AI agents. Before the break, you heard David explaining some of the trends he's seeing, both from his work internally at Anthropic testing new models and from clients who are now using this tech in their industries. But now I want to ask David about Anthropic's big announcement this week, Claude Sonnet 4.5, and why it's being billed as such a step forward for AI agents.
16:52 Speaker 03 Just released Claude Sonnet 4.5. And to me, that was a big deal. I'm really in the weeds on this stuff, so I was really into the, all the specs. But for the average person, you know, that's just a word and a number with a decimal and another number. So why is this a big deal? And how does it differ from your latest models before that? You know, what's the meaningful difference and the meaningful step change here?
17:19 Speaker 04 Yeah, I'm very excited about Sonnet 4.5, too. And I'm also very cognizant that when my mom texts me asking about it, it just is a model with a decimal and a number after, until I talked to her, too. So there are a lot of things that I think are exciting. One thing that I want to say upfront, before I get into a lot of details that like have jumped off the page to me about the model, is it's sometimes hard to predict like what the big impact is going to be on everyone. These models get generally smarter, they get generally more capable. And not to keep harping on the same point, but sometimes we just like don't know the blind spots that they had until we get over them. And so when we released a model last year, Sonnet 3.5, and suddenly all of these vibe coding startups happened,
18:03 Speaker 04 I don't think we actually like knew that that was true. We didn't know that there was going to be like the moment we released that model. I don't think we could have predicted that so many companies would crop up and be able to help people start writing code with agents in this amazing way that we've seen happen. So part of what is exciting about Sonic 4.5 is the unknown to me. I'm really confident this model is the smartest model we've ever created. I'm really, I've seen, and I'll get into some of the things I've seen it do that I've never seen another model do before. And that's really cool. And part of it's concrete stuff. We'll talk about, I'll tell you, I'm excited to talk about some of the software engineering stuff I've seen it do. But some of it's stuff that we really don't know until our customers and the people we work with try to build cool new things that they couldn't make work before. And then they suddenly make it before.
18:44 Speaker 04 And so when I, if I had to guess how my mom might see Sonic 4.5 and it might make a difference to her. Is that there's some product that she couldn't use before or didn't exist before. In the way that perplexity has happened in the past for consumers or, or cursor exists now for developers. That it's going to happen, that's going to impact your life because this model is capable of a new thing that we didn't know before. And I have a hard time predicting it. It's part of the, the fun part and the hard part about being someone who works with customers now is it's really hard to predict this stuff the day we launch a model. But it tends to happen and that's cool.
19:19 Speaker 03 Have you seen any trends start, even though it's been like one day? Anyone that's starting to use it, you know, in a new way, one of your clients or even in beta testing?
19:29 Speaker 04 I, too early for customers, I think to, to see anything brand new. It typically is like, uh, it's a very fast field, but I, I have like a give it a month rule to find out what the new companies are going to be. My testing, I have certainly seen some new stuff and with the testing some of the team has done that is really exciting. One of my favorites to talk about is my team has been working on seeing how much of a software engineering task a model can take on at a time. And so I think we're all aware that models can be used to write code. Like that's sort of obvious to a lot of people, at least in this industry and probably people listening to this podcast at this point. But it's often in pair with a human really directly, like going back and forth, using Claude code or an IDE.
20:12 Speaker 04 Like write one thing at a time, check and review it. And the longer you let a model go trying to implement something big and complex, the more likely it is to like do a whole bunch of stuff you didn't want to happen. And so we were like really curious, like what is the most we could stretch that? Like if I give Claude a really huge task, like what can I accomplish? And one thing we've seen with this model is in a way that is sort of like.
20:37 Speaker 04 Really not true of models I've tested and seen in the past. There doesn't seem to, if you give like a sufficiently good overview of you, what you want a model to accomplish. I haven't really seen a ceiling on how much it can keep like coast consistently working on making something better. And so my favorite actual example of this, we released a video of this yesterday, but I like, I'm going to talk about this until people get bored with me. Is we asked Claude to recreate Claude.ai, like our, our consumer chat application from scratch. And Sonnet 4.5 just like worked overnight. We woke up and it just did it like this beautiful clone of Claude.ai that works incredibly well.
21:16 Speaker 04 And my favorite moment from it, there's a feature that people like of ours called artifacts where when you ask Claude to make a document or make a webpage, it will make it and render it. Next to the chat. So you can like play with an app that you built live or whatever it is. And I was saying to the person on my team who was working on this demo, like working on this thing. It'd be really like cool if Quad could like build artifacts itself. That would be amazing. It's like a complex feature. It's pretty hard to figure out. It's like, we should, we should try and see if that happens. And then he messaged me like two hours later and he didn't do anything. He didn't intervene. And he's just like, hey, Quad just built artifacts on its own and it's like currently testing it live. It's just like trying it out itself.
21:57 Speaker 04 And I really, it's just like, we released artifacts a year ago. It was like artisanally created by a whole bunch of like wonderful engineers around me at Anthropic. Who were doing all of the software engineering to build this really complicated things themselves. And then like we, a year later, we released a model and we asked it to build Claude.ai and like let it go for 12 hours. And just did it itself, you know, it's like, instead of this giant team of people at Anthropic that work so hard on this thing. The model is just like really capable of, of biting off really meaty, complicated tasks. Like this is something that would take me months to do if I did not have Quad. And overnight, we, we sort of looked at it and watched it happen.
22:40 Speaker 04 And that progress of just like, like going from a point where it was neat that a model could write a snippet of code from a year ago to like. Oh, it can do like a big chunk of the stuff of the complex developer work that. I need to do, it's just like, I don't know, it blows my mind. It was like a pretty big wow.
23:02 Speaker 03 So that took about 12 hours, you said?
23:05 Speaker 04 That specific one was like a 12 hour thing. We've seen like up to 30 hours of continuous dev work.
23:11 Speaker 03 Okay, this is, this is what I was gonna ask you about because something that unexpectedly went viral from my own article about Sonnet 4.5 was the 30 hour bit, the fact that Sonnet 4.5 could code autonomously for up to 30 hours with no interruption. So I heard one engineer at Anthropic used it to code a chat app that the company compared when they talked to me to Slack or Teams. Obviously, it was only 11,000 lines of code, so much smaller than Slack or Teams, but, and it seemed like just an example project. But people online are really excited about that detail and calling for Anthropic to release it and they want to know more about it. So can you give me any details? Was that someone on your team that was testing it? And how impressive was it really? Or was it just kind of rudimentary? Give us the deeds.
23:53 Speaker 04 Yeah, yeah, yeah. This was the thing that also, I don't know, like all of my excitement and the reason I'm here is because I've been working on this exact thing. So it was someone on my team. His name's Justin. Shout out Justin. You should give praise to him. He's amazing. Recently joined the team on a side note. He's pretty good. This was born out of this thing of, a lot of people have built demos or proofs of concept and there's this sort of vibe coding trope that you can, yeah, you can build it. You said to quickly mock something up, but can it really like build a real application? Justin really wanted to like test that. And so he was experimenting using our Claude agents SDK, which is just sort of like a more programmatic version of Claude code to some extent.
24:34 Speaker 04 He was testing, like, can I give a full spec of like a complex thing, like an app similar to Slack to the model and just like watch it build? And we did some take green experimentation around it to get it right. But yeah, you asked like, was it impressive? It's impressive. It has like DMs and threads and channels and a slick search functionality. And you can upload images and GIFs and render them and like multi-user authentication. And Claude, we didn't ask it to, but implemented a whole bunch of like AI users for testing. So if you like login, you can send a message and there's like Alice, the PM is in there that you can send a message to who will respond to you back, stuff about PM work. It is, it is remarkable. It is like.
25:20 Speaker 04 It is not by any means Slack, but like if you didn't spend a lot of time thinking about it, you'd like look at it and think that was a pretty reasonable productivity app that you would use to chat with your coworkers.
25:32 Speaker 03 Wow, what did you guys name it?
25:35 Speaker 04 I don't think we have a name. I need to ask Justin to give it a name. We need, I actually know, I lied. He has a work chat, very boring.
25:41 Speaker 03 Work chat.
25:43 Speaker 04 Clearly not the product folks in the org, so.
25:46 Speaker 03 Since though, I love that. Yeah, someone who worked at Slack, I think, messaged me and said, you know, our code base is a thousand times that. And I was like, yeah, it's just an example, but I mean, it happened in 30 hours, so it's a big deal. I was also going to ask you what surprised you the most during your own testing of it, of Sonnet 4.5?
26:03 Speaker 04 I'm just going to continue double clicking on this thing. The way that it built the app was surprising and really interesting. I'm not surprised that the model's getting better at building complex apps. The thing that was really interesting for the way it does this is it just has a tendency to like bite off little pieces that it can handle really well and do that continuously. So a lot of times like models in the past would get really eager and ambitious. They would say like, I'm going to build this whole thing and I have like these grand ambitions. And it would kind of just like meander everywhere trying to do this miraculous piece of work. And the thing that was really cool about Sonnet 4.5 is it's just like pragmatic, kind of like it. It's like, okay, right now I'm
26:45 Speaker 04 going to test, like, does image upload work? And then it's going to do that. It's going to spend a little while doing that, but it just like bites off one little chunk at a time. And that feels a lot more like what I want a coworker to work right there, collaborator work like. It's like, if I ask you like, hey, I need you to go build work chat. I don't want you to go off on this like crazy escapade trying to make everything magical. I want you to just like bite off a piece at a time, show it to me, like commit it to get like that kind of thing. And it's just a little bit more natural in collaborating if it feels with it feels more natural. And funny enough, like I think this is unrelated, but we've been chatting in our internal company Slack with Claude a lot lately and just like chatting with it.
27:28 Speaker 04 And it's been really natural, like it feels just a little bit more like working with a coworker does.
27:34 Speaker 03 In terms of the tone?
27:36 Speaker 04 Yeah, like the tone, how it responds, like how it acts in Slack, how it participates in a conversation, what it tries to do. It's a little less like over the top and eager. That has jumped out a little bit, which was surprising. Like that's not something that I normally expect, but it's also been funny, I guess, which is a funny thing. Like it cracks better jokes. Like it's a little bit more witty, that kind of thing.
27:57 Speaker 03 Do you think it also dialed back the sycophancy a bit? You know, I know that's been been a big word in the space lately in terms of, you know, being over eager, over validating. Did you feel like it was more real and that it was less like that a little bit?
28:11 Speaker 04 I have felt that. And I think it's certainly something we're pressing on. I it's like one of my least favorite things about every model is when there's sycophantic like that. I think it just gets in the way of doing good work. And it's been a big focus of ours. Like I think nobody at Anthropic likes that. And also just like, I think it's it's bad for the, it's just bad. So I do think we have made meaningful progress and this model seems to be a little bit more willing to push back. And that's part of that thing, like being a natural coworker is is someone who actually can tell you when you're wrong.
28:40 Speaker 03 When it comes to rebuilding Claude.ai, that's pretty big. Did that worry you at all? Because, you know, it's kind of doing, like you said, you know, organic work that you worked on for months. Does that worry you about job replacement for engineers, anything like that? What were your thoughts?
28:58 Speaker 04 We have a great team and I'm really not worried about, I'm going to frame this two ways. I'm not currently worried. Right now Claude is a collaborator. Like it works really well with me. It accelerates me. Like I think it makes our whole team better and faster writing software. I am general like.
29:17 Speaker 04 This is not a now thing. To be really honest, like watching Quad go for thirty hours, like it does trigger a little bit of like, oh, my God, like, that's like a pretty different thing. It is a meaningful step change. And I think it does like this technology Anthropic is founded on the principle that this technology is going to be hugely impactful in the world. And part of that is it's going to change how we do jobs. And something like doing a whole week of work for me, like, that's just like meaningfully going to change the industry of software engineering. And so, yeah, there's like a little bit of, and I would be like, it'd be goofy for me to say there's not any amount of like, this is like, how are we going to work next? Like, how do I incorporate this? What is it? My net-net,
29:59 Speaker 04 though, is like, I just think there's a ton of room to make us better, to make better software for users, to make the world a better place with this technology. And I'm really confident in that still. But, yeah, there's like a smidgen of like, we're gonna have to figure out some new ways that we build with Quad and that we like operate as software engineers to work with the thing. If it's really gonna like, build the whole app itself, like, we have probably a different role that we need to play here.
30:25 Speaker 03 We need to take another quick break. We'll be right back.
30:40 Speaker 01 Hey, Bose here. When your next track starts, listen for the decisions that the artist makes. The vocal pushed right up front. The bass that waits half a beat. That sample from a seventies underground gem. Every choice deliberate. With Bose, you'll hear it all. Your music deserves Bose.
31:06 Speaker 02 New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want online only at homedepot.com.
31:37 Speaker 05 This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends, like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an Ultra. Because it's not just a superior light beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.
32:10 Speaker 03 We're back with Anthropic's applied AI lead, David Hershey. Before the break, we were talking about the capabilities of Claude Sonnet 4.5 and whether he sees a future where this technology might even automate parts of his job. But now, I want to ask David where he sees the model falling short and what might be next for AI agents. Well, what are the primary limitations to Sonnet 4.5 that you wish you guys could have offered with this release that you couldn't? And what was something that you tried to make it do during testing that it couldn't do? Basically, yeah, the things you, the features or, you know, the context window or anything else that you wish you could offer that you couldn't? And also, what was dumb about it in testing?
32:49 Speaker 04 I have a pet thing that I always test on, which is I really like to make Claude play games. I'm accidentally famous for creating Claude plays Pokémon in the past.
33:01 Speaker 03 Oh, that was you! I've heard a lot about that.
33:04 Speaker 04 Yes, that is, that is my project. And I have also tried a lot of other games, like I had Claude playing chess this time. And one of my favorite comics people really wants me to make Claude play Catan effectively. And it's really bad at spatial awareness still. And this annoys me to no end. Like, it just like basically doesn't know the difference between left and right and up and down. And this is just like, it's just one of those examples that exists in the world of... It can do PhD level math and I can't and and just for it to not really understand that it can't walk straight through a building hurts my brain and models are kind of weird and funny that way. So that one, that one was the one that like probably jumped out the most is I really wanted Quad to be this like great chess player. It's so logical and interesting.
33:50 Speaker 04 And then it like doesn't know where the pieces are on the board and where other pieces are. And it's like, ah, you're almost there Quad one day. And features that I think are still out there. I have been so in on this model that I have to think about the the next level. I honestly like the real answer, I don't know. I like, I can't think of some like feature that I wish was there. There's like a lot that I want Quad to get better at. There's like so much that I want Quad to get better at. I want it to be able to beat Pokemon one day on its own and I want to see it like how it does it. I want to like, again, there's all of these fields where I'm like, know that Quad still needs to get better. Like I, I, we talked about legal, like I think Quad is becoming a better lawyer, but I don't think it's like as good as a lawyer as it is a software engineer yet. And like, I still wish we worked on that.
34:36 Speaker 04 And and we're making progress. Like I know the teams that are working on all of these little things and focusing and thinking about all these little things. And there's always this like so many hills in my mind that we have to climb still. So I could probably like, there's probably this intersection of there's a million things I wish Quad was better at. Not one that I am personally making on except for Pokemon, which is one day the research team will listen to me that it matters. They haven't quite listened to me yet.
35:00 Speaker 03 Well, when I, when I went on a reporting trip to London and visited the office, it was all I heard about. So it's making waves somewhere.
35:07 Speaker 04 That's good. Well, it is, it makes people happy here. It's a fun experiment. I don't think it has crested the peak of what we need to train Quad to be good at yet, unfortunately, one day.
35:17 Speaker 03 Let's talk a little bit about what's next. So talk to me about why you think Anthropic is pursuing such gains on the AI coding front? And how it stacks up to competition? Obviously in testing, you had to, you know, take into account what the market looks like right now and what other models can do. You know, how did you think? Sonnet 4.5 stacked up during testing and what stuck out to you there?
35:40 Speaker 04 Coding market is really important for a lot of reasons. It's really well positioned for people to build with our models. It's a place where you can have a huge impact. People have figured out great ways to integrate models into how they do a job writing code. And I think more so than essentially in other industry. And so if there's like a right now where you can make a huge difference by continuing to make models better. I think coding is probably like the best use case. And so it's a huge focus for us. We want to keep helping people who are relying on our models to write code, be able to get more out of them. And our belief, we've talked to a lot of customers, done a lot of testing. Like I'm pretty confident that this is the best coding model in the world. We have a lot of benchmarks and other things to prove that out.
36:20 Speaker 04 And it certainly has like for all of the people here in our early testing, just been a huge step function improvement. And what it feels like within Quad Code or other service areas to develop and write code. And from my perspective, this is like a really big step function. And personally, like this is just talking for myself. This is the most noticeable change and improvement I've seen since we released Sonnet 3.5 last year. Which I think is a funny coincidence. I don't think we necessarily knew that these 0.5 Sonnets were going to be such special models for us. But just mechanically, like this is something or interface of vibes and numbers and things I've seen.
37:03 Speaker 04 This feels like a really, really meaningful step change improvement. One of my favorites, just to maybe call it out. I love the team at Cognition. I spent some time working with them and they put out a post yesterday about how much their product dev improved with the model and some of the work they did to make that product better that I loved. It's a cool blog. And that that like really, I don't know, I think that just that validation from the customer, like it's a huge jump in a benchmark that I really haven't seen have a benchmark like a jump for them in a while. I just think that's really like for the right product built the right way around this model. I have a feeling this is a huge step function change and ability to write coding. And I'm quite confident it's going to be the best model in the world for for people like that.
37:48 Speaker 03 Yeah, that's really interesting because I usually don't mention benchmarks much in my own reporting because sometimes they can be subjective and sometimes they're, you know, created by the companies that are testing their own models in very specific areas with very specific sets of questions. But I think usually what I hear from engineers is that they go based on the feeling and based on what things it can do that it couldn't do before in their own anecdotal testing. So that's why it's interesting to see, you know, you have your own pet projects that you like to test on and you've seen changes in that regard instead of just in the benchmark specifically.
38:26 Speaker 03 I also wanted to ask you about anthropics chasing consumers versus enterprise versus governments. So, you know, all AI companies right now, it seems like are kind of like working with those three tiers. And I think it's probably because, you know, those are two of them maybe are more concrete areas for potential profits. So do you think with Sonnet 4.5 and just all the stuff you're working on in general? I know you work mostly with the enterprise side of things, but is there a specific slice that anthropic is chasing more right now and why?
38:58 Speaker 04 I think they're basically all really important. And I think one of the things that we have luckily seen, and I think is true for all of the labs, is that when we make our models generally smarter, they service all of those segments. They service enterprises who are building with us. They service consumers who want to use our chat app or write code or get into that. And they're used for the public sector too. Just to tie a little bit of a link here, I work with I've worked with customers and startups that have become big consumer hits, like lovable is a great example where, you know, it's like that's an enterprise customer from my perspective that I spend a bunch of time working with and helping them be successful and our team does.
39:43 Speaker 04 But then they go turn around this thing that everybody can use to build just in time apps to service every part of your life. And so I think this kind of just bends back on itself where in reality, our focus is building really great models that are safe. And we have seen time and again that that results in sort of like success in all of these places. And there's just like a lot of different ways from our perspective. This is talking a little bit from my personal perspective instead of just Anthropic's. But I think there's just a lot of ways that you can make progress building great models. And whether it's helping enterprises or helping consumers or helping the government, like these all have a way of warping back in on themselves where the thing that we really do that makes a difference is make great models.
40:28 Speaker 04 And there are a lot of different ways you can impact different segments with that.
40:31 Speaker 03 And when it comes to consumer use cases, OpenAI seems to be pushing pretty hard into that right now, at least this week. They launched Pulse recently last week and then yesterday they debuted their instant buy button. You know, I wanted to ask where you think Anthropic is planning to meet consumers? Do you think it's more likely you'll be reaching customers with Claude directly in the future? Or is it more likely customers will use Claude through something like Cursor? You know, and that can also apply to your startup clients too, right? Where are you seeing people really find Claude?
41:04 Speaker 04 I definitely would think we are growing. It's just like the presence of the applications we build to interface with consumers. And Claude Code is a great example of this. I know it's not like the traditional consumer, but there's like a very prosumer-y thing that we've captured with Claude Code. And I think we've demonstrated that we do have some of the muscle to capture like for some sets of clients, a thing that they love that like sparks the excitement of a consumer product that people love using. And I guess like obviously we would love to do that 50 times over. We would love to be able to invent a bajillion beautiful uses of Claude that interact or
41:45 Speaker 04 that that consumers love and love to build and we're investing in trying to find and build great experiences that help people do important things with Claude.
41:57 Speaker 04 We have a sort of constant tinkering. We we had this Imagine demo that came out. It's just a limited time thing. I don't think that's the product we're going to launch that is consumer facing, but like we just like it's part of a portfolio. I guess of like how we think about, we need to keep building and trying and innovating and inventing products. Seeing if there's something there and we have a good opinion of what models are capable of that helps us have an interesting lens on what we build. So it's a big focus, like we we need to get there. Like it's important for us to build direct relationships and help people have awesome ways of interfacing in Quad. That said, like, I don't know, I'm biased as a customer guy a little bit, but I also just think like, I wouldn't ever.
42:39 Speaker 04 And I don't think, and I don't think we should, and I don't think Anthropic is betting against the ecosystem of people who are trying to build amazing products. And I think it'd be really silly to to claim that that's all our market to grab. I just don't think it is like, if we build great models, then there's an incredible ecosystem here in Silicon Valley and abroad, building amazing products that I think is like probably bigger upside than our our first party applications will ever be. So I'm like, I think that it will always be a huge focus for us.
43:10 Speaker 03 Awesome. And then last question for you, back to the AI coding market and how important it is and how much most AI labs are chasing that right now. We talked a little bit about vibe coding earlier and here at The Verge, a lot of us have tried it and to no avail. We had some success with really simple stuff, but not a lot of success with building large scale applications. A couple of us did, but not as much as we actually expected. You know, did you see a big change in terms of testing out like vibe coding with Sonnet 4.5 or was it the same as before?
43:46 Speaker 04 I have noticed like my own personal coding, like every model we release, I am trained as a software engineer. So it's just like cheating. I'm like not the best guinea pig for vibe coding, but I still do it on the side sometimes. And you notice meaningful improvements on like what, how much it can churn out before it goes off the rails and how much you can trust it. I actually, I think this is actually an interface problem and I think there's something funny about AI, which is that it has this tendency to outgrow. Interfaces really fast. So if you look at the history of coding with AI, there was probably like Copilot with Ghost Text. So Copilot would automatically complete your code. For a while, there was, you would go to us or ChatGPT or wherever it was, and you would ask it to write some code for you in a browser window.
44:31 Speaker 04 If you were a developer and then you'd copy and paste that into your editor. Cursor figured out how to sort of like bridge those two things where it could be side by side. A lot of people have started building this sort of agent that was aside alongside your ID that lets you build things. I don't think any of that is quite the thing that we need for everybody to build production applications, though. Like, there's some interface where I actually do think Sonnet 4.5 is the first model that could be that thing where anybody could build a sort of like production ready application. I've seen enough evidence of when left to its own devices, how it can build complex applications. I've seen it be able to deploy a complex application to AWS like fully autonomously and do like a security audit on it.
45:12 Speaker 04 Those are like both really incredible things that like make me think this is the model that could sort of like cross that chasm and get to the point where like anybody can make something production ready. I have a feeling, though, we need like one more interface that isn't Claude Code and isn't Cursor, but like the next step past that I think needs to happen to get so like it's more obvious to everybody instead of having to like try to figure out if you're on the right path by coding.
45:38 Speaker 03 Awesome. Well, thanks so much. I'm so glad we got to talk and, um, you know, appreciate you, uh, coming on with so last minute. I really appreciate it.
45:46 Speaker 04 Yeah, it was fun. I appreciate it. Of course. It was really nice to, uh, nice to be on.
45:52 Speaker 03 I'd like to thank David for taking the time to speak with me and thank you for tuning in. I hope you enjoyed this episode. If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email us at decoder@theverge. We really do read every email or hit me up directly on X, Blue Sky or Threads. I'm at Hayden Field on all platforms. Decoder also has a TikTok and an Instagram and now also a YouTube channel. Check those out at DecoderPod. They're a blast. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts. Decoder is a production of The Verge and is part of the Vox Media Podcast Network. Our producers are Kate Cox and Nick Statt. Our editor is Ursa Wright.
46:34 Speaker 03 The Decoder music is by Breakmaster Cylinder. See you next time.
46:45 Speaker 02 New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want, online only, at homedepot.com.
47:15 Speaker 06 At Zocdoc, we know you care about your health. You wouldn't do that early morning cardio if you didn't. So the fact that your annual physical is overdue just shows how hard it can be to get seen by a doctor.
47:25 Speaker 07 Sorry, that doctor's not in network.
47:28 Speaker 08 Press 1 for more options. You've reached us outside normal business hours.
47:32 Speaker 07 The next availability is in three months.
47:34 Speaker 06 But Zocdoc makes finding the right doctor easy. Download the app and search by specialty, insurance, and availability. Read real patient reviews, then book instantly. Zocdoc, ready to be seen.
47:45 Speaker 09 Mr. Monopoly here. Miss me? Monopoly is back at McDonald's with millions of prizes. A 2027 Jeep Cherokee, a Carnival Cruise vacation, gift cards from DoorDash, and Best Buy. You'll then play in the McDonald's app. Get your bag.
48:03 Speaker 10 No purchase necessary. 50 U.S. and D.C. age 18 plus. See playatmcd.com for rules and playatmcd.com/amoe for info and playing without purchase by mail by November 2nd. Chance card game ends November 8th. Game ends November 22nd, 2026. See rules for prize details. Monopoly is a registered trademark of Hasbro.
Previous Ford CEO Jim Farley on China, tariffs, and the quest for a $30,000 EV Next Rivian CEO on CarPlay, Lidar, and affordable EVs Page updated October 26, 2026 · transcript and summary are machine-generated
