Latent Space: The AI Engineer Podcast
Hosted by Alessio Fanelli
Watch the full show on YouTube: https://www.youtube.com/watch?v=aJxnel2_O7Q
5,835 words · 3 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
Hey everyone, welcome to the Late in Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Squicks, editor of Late in Space.
Today is one of our coolest intros, or at least I can't believe I get to make. We're excited to welcome Mike Krieger to the podcast. Welcome.
Thanks so much for having me.
It's so surreal to have you here because obviously everyone knows Instagram and obviously now you're a CPO at Anthropic. And congrats on releasing 4.5. I mean, it's such a huge update. Claude has been crushing it ever since you joined. I feel like you joined at this inflection point and it just went like this immediately. But how do you feel about the launch? What's it like?
I feel great. There's this funny moment that happens in the last few days before releasing a model where you're doing a lot of internal bashing. You're trying it out. It's been running in Claude code and our internal surfaces for several weeks as we've been rolling out. Every snapshot gets better and better and better. And then there's the point where like, okay, this is getting ready to release. And we do early access. We definitely have people try it out, but there's nothing quite like the force of the internet using it on day one. And actually, within the first day, we have more traffic on Sonnet 4.5 than we had on Sonnet 4. So basically, it's already eclipsed Sonnet 4. So people definitely try it out and switch on day one, which is exciting, but also means you get this wave of people trying out. But it's gone great. It's been a great launch day.
As a CPO, what's the handoff between the model and the product? So you get the new model checkpoint, I'm assuming, at some point in the post-training thing. And then you basically look at Claude.ai, Claude code. What's the process of taking the model and putting that in the final products?
I think I had, when I think about when I joined, my idea of how it would work is that research trains the models, then it gets handed off, and then we make sure it... Still meet some evals of our product services as we roll it out. And I think that sort of was the case kind of when I was starting because our product team was very, very small. What was interesting about this model in particular is that it was really the first one where product was upstream of research and downstream of research. So we've been doing a bunch of work kind of beyond coding and things like financial services. And a lot of the sort of use cases and evals that we were looking at, because we saw it in the field, were ones where we then helped inform some of the research direction on. That was like a cool reversal there as well.
Of course, a lot of the hard research work was still done by the research team, but we're finding a lot more chances to collaborate on things like what are the top customer problems? How should the model be better? Even within Claude Code, and we listen to a lot of user feedback on, if Sonnet 3.7 was too eager, maybe four was lazy in some places and really drove down laziness. Or when the model's like, great, I did two of the three things you asked me for. And you're like, just do the third one, too. So that product research symbiosis, I think, was the strongest on this model than it's ever been. And it's been really fun to see. But to your question, yeah, like when we're getting close to model releases, we definitely put it in all our surfaces. And then, you know, just like all of our customers do, when we drop an early access model,
our customers then adapt it to their harness and make sure it works well. For our first-party services, we have the same journey. So how are we going to adapt the Claude AI system prompt? How are we going to adapt the Claude Code harness to make the most of the model as it's developing?
What's your vibe eval? So as soon as you get a new model, what's like the first two, three things that you do with it?
It's such a good question. I had three, and the team got really tired of me by the end of this Sonnet 4.5 process. Because like every, literally every snapshot, you know, I would drop everything I was doing and run these. So one is the virtual boy. I had a, you know, I actually never owned a virtual boy. But the local video game store in Brazil where I was growing up had one. And then, if you remember, this is like a doomed Nintendo console that had these like stereoscopic wireframe, red and black 3D graphics. This is a very early product, it totally failed. But I always, I like to have. In Claude AI, like, create me a, like, virtual boy-style 3D shooter game. And it's really funny to see the, all the checkpoints from, like, early Sonnet 4.5. I was like, I don't know, guys. This is like, it's pretty early.
I know there's a lot of RL left, but it's not looking that good. To, you know, about a week ago, I was like, OK, great. This is, like, officially good. It's, like, better than Opus at this. It's, like, it generated this, like, great split-screen stereoscopic thing, three-dimensional, like, thing. So, anyway, you could really see it evolve. So, that's, that's one I always shoot for. Within Claude Code, there's a particular, sort of, change to our codebase that I, I, like, once did in Claude Code. I was like, oh, this is a useful kind of approach, because it involves synthesizing across, kind of, both our web repo and our backend. It just involves, like, a lot of, of searching for files. So, that's my, like, Claude Code, sort of, like, running, running eval. And I've been doing more and more on, like, once we added file creation capabilities to Claude,
so it can, you know, create PowerPoints and create Excels, et cetera, is I have a very specific thing, which is research Nintendo. I don't know why I have Nintendo on my mind, I guess. Like, research Nintendo's last few quarters and create a presentation to their board about what they should work on next. And, again, you know, you get to see the evolution of really basic-looking Arial 32-point font with bullets to now, especially with, with some of the later training, like, it did it. And then Nintendo Red, it had charts. It was like, it was really dialed in. So, those are my three, every single checkpoint, run through them.
I can see your, sort of, design side getting passionate there. Like, you like to see the, coming into focus, almost, like, the AGI expression of, like, you know, like, no, no, like, you know, this is, this is really what it should be like. You tweeted out an exercise of asking Claude to recreate Claude AI. And I thought, yeah, I thought that was also, like, really interesting to see the progression over time. And I think it's, like, that, that's, like, the publicly visible version of what you just said, which is, like, you, you have that vibe check of, like, what, what, what is this new version able to do?
Exactly. Yeah, and I think there's, you know, of course, the capabilities and the evolution there, but I've been really pushing, this is actually one of the three top priorities of our product team right now is, let's make sure the outputs are great. So if you're gonna make an Excel file, it's not just correct, which is obviously really important, but it also, you know, matches the style that was requested. If you're gonna make a PowerPoint deck, make it something where, you know, it's not gonna be 100% right off the bat, just like code wasn't 100% right off the bat, you know, a year and a half ago. But it should be close enough where you're not like, oh, it's gonna be more work for me to start from scratch. And similarly, on web dev, you know, can you start somewhere reasonable and then iterate from there? And it might not be customized to your company's style just yet unless you're, you know, creating something in something like Figma Make, but,
you know, is it a good foundation that you can build upon?
Yeah. I think one of the interesting things that like sort of plagued previous models would be, for example, like there will be like a yellow tint on, you know, some other image generation models, but also for specifically for Claude, he would really prefer purple websites, or it's like this like purple, blue tint websites. Did you get rid of that? Was there like any of this internal discussion going on?
There was definitely in this training process of this one, like a little bit more, you know, can we, can we try to teach Claude more about, you know, UI taste? I don't think we're all the way there. I think there's more work to be done there. But I agree, there was definitely the like, white background, purple tint on the top of rounded rectangle is something like Claude Force on it, or Sonnet for loves to do. I've seen that so many times. But yeah, I think it's definitely improved. And I've been seeing demos on X this morning, people were experimenting with it. I'm like, ooh, regular dark background, but like very little purple, great.
In some ways, I think, you know, I was figuring out with the Imagine, I guess, Playground or OS, so I don't know what you call it, experience where you would actually generate a lot of websites on the fly. And it's gone. You know, I mean, it really looks like, they sort of started a de-biasing motion there. And I think like, what's weird is kind of like your role as a product person kind of evolves, it's still like, you're sort of like a, you're distilling product sense into the model, rather than being responsible for one product, responsible for all the products that come out of the model. And that's like a very strange. In direction that's happening right now.
It's really true. I mean, when you think about how much UI is going to be dynamically generated in the future, you know, right now, we have imagined with Claude as kind of a research preview kind of demo. But even internally, we have found that having Claude generate UI on demand for internal dashboards is useful, not just like an interesting demo. If you believe that's the case, and this is maybe tied to a conversation we can have also around like design tools at Figma, but you want to instill in the model, good UI, you know, techniques. I was a human computer interaction major, so I like spent a lot of time thinking about usability. And then also whatever sort of design guidelines that make each company's products kind of unique and feel, you know, theirs. But it is true, like you think about the software that is going to be created in the future by these models.
They have some sort of upstream, almost like responsibility for good usability and design.
Yeah, we're releasing our episode with Dylan Field in a couple of days. One thing we were talking about is when you have a code diff, you can give it to the model and the model reads the code and understands what's being changed. It's kind of hard for the model to understand the aesthetic change in a way. Like, how do you think about explaining to the model these things? Like, have you come up with like a good model of like taste, semantics to put it in the latent space?
I actually think Figma did a really good job with make in that they, you can tell that there's much more of a bridge between the sort of underlying design model and what the LLMs are doing in kind of coordination between Sonnet. And that's, I think that's part of it, which is giving the model a sense of sort of the design building blocks is important rather than just the finished product. I think you can reason about that as well. But the second part, and I think we need to make some strides here too going forward is the models don't see as well as they could. They see, okay, you know, you ask them to analyze a complex photo and they're able to do it. But I want them to be as persnickety as a like really good visual designer, like, no, that looks, the baseline looks a little off, you know, or this needs to be good.
And I think that's going to come from sort of additional vision capabilities that, you know, we'll work on. But I think that's going to be a really key piece of closing that loop so that, you know, model, and I've seen people do this with Claude Code and like take an MCP with Playwright, for example, in a browser and basically do the loop of, right, you generated the UI and now look at what you did and is it right? And can you iterate on that? And I think the is it right still needs some, some visual help before it's, it's fully done.
Yeah, and there's the pixel level thing of like, is the padding right? The margins right? And then there's kind of like the broader design UX UI level. For example, the Claude AI example that you tweeted, the final one with Sonnet 4.5, it has the chat history right on the side versus on the actual Claude AI, you have to click through to go to the history. Like, do you use these models to like, think about all the different permutations of like, how to build these UIs and products? Or do you feel like the models are still, you know, pretty median results so far?
One of the better projects that we did here was get a lot of our kind of products into artifacts, or at least into a harness that we could iterate on them really quickly. So one really fun use, Nate Parrott is one of the designers on our team. He got Claude to be able to prototype Claude code UIs, even though it's a terminal UI, Claude can sort of imagine what that's like. And it's very valuable to say like, all right, well, what it would look like if we revamp settings in this way and not have to go and necessarily code the end. So I think they can be very useful in sort of exploring that space of what, how could this product evolve? What do it mean to do this differently? And they can also actually just implement the changes as well. But even sort of from a prototype phase, it's valuable to have it sort of iterate quickly over ideas for non-engineers on the team.
Yeah, I would also shout out the Claude plays Pokemon effort with David, which we've had him on the pod before. And then, you know, I think, I think that's like a kind of a trivial benchmark, but also not that trivial in a sense of like, there's a lot of you software UI that needs to be fed into Claude that it still isn't perceived that well, despite, like, he has to do a lot of work to like augment just the vision side, but he should be, he should be able to only drive by vision, you know?
The thing I think about a lot where, you know, I get the question sometimes, like, is the future all MCP or is it browser use or computer use? And I've come to believe that the, I've swung, you know, at first I was like MCP, everything, like, you want things that operate at the speed of computers, then at the speed of, you know, human interfaces. And then, you know, we do a bunch of work with a bunch of verticals, one of them being like legal or sales. And both of those often will have to go to some compliance website that has a 50-part radio button. Form and that thing's never gonna have an MCP around it like it's just like who knows if the company created is even around much less like ready to sort of expose their kind of underlying constructs as API. So I think you will need to be able to do both.
And if that's the case, it's not just the case that Claude needs to be able to generate good UIs, it also needs to be able to be good at figuring out bad UI too, or legacy UI and and work through all those different pieces as well. And I think that, you know, training on like, well, you should be able to solve the UI or work on this UI, but don't believe that that is good UI. Like, there's a better way of doing this as well.
I was gonna move on to a little bit of the sort of eval process and like, you know, any other sort of coding features. I almost call it, it's like, yeah, it's like, yeah, yeah. At this point, we're like sort of desensitized to like every model being like better than the previous model and like being the best coding model in the world. But like, I think like you've done a lot of work on like just the Claude like sort of as a developer platform, as a, as a sort of code agent model, and even like the, the API features where you have like sort of context aware
concatenation or compression. What was your sort of overall design principles on the, on sort of like the Claude for developer side?
Yeah, for sure. And I think, you know, it's a thing that we launched our API and it was, you know, you think about it like I described, it was like tokens in tokens out. There was a messages API. It had some opinionation or opinions on, you know, the way that blocks should be formatted, but it was, you know, pretty, pretty lean. But over time, you realize like people aren't hiring you, you're hiring your model to do just that. They're hiring it to solve some like downstream user problem, right? And that's going to involve like helping them manage context, helping them manage memory and ideally do that in a way that's really aligned with how the model likes to do that too, right? Like because there is such a tight loop between, you know, the tool use. And of course, the models can generalize, but if you can provide it the same tool as it does in
in training, why not? You know, one of the first times we did that was actually when we opened up the computer use API. It was the first time we actually took our tools and actually made them available as sort of first class citizens in, in the platform. And I really think like over the next year, what you'll see the platform do is I think of it as helping people express The problem they're trying to solve for their downstream user and seeing if we can help as much as possible, right? So if you're, if you know, if you're trying to provide a coding agent that's having to manage a lot of context, like if we can do that and take that off your plate and make it, you know, lead to a better outcome for the user, great, we should be able to do that, right? We added a code interpreter and we've evolved it a couple of times similarly.
Like there's some companies that will want to own that whole stack themselves and that's great. We make it all very pluggable. But if you're, you know, we work with like a very large legal analysis company and they're like, we don't have the in front or the know how to like manage containers. If you can do that for us, great, like, please do. And so you'll see us add these sort of composable, pluggable building blocks in the platform. And again, all in the service of we like, you're hiring the model to solve a problem, like help us, we want to help you solve that problem.
How do you decide which of these spaces are worth doubling down on? So if I look at the Sonnet 4.5 benchmarks, you have retail, airline, and telecom at the three agentic tool use, tab bench categories that you mentioned. How much of alignment is there between that and what you think is important? Or like for example, financial services, you know, there's like the financial analysis agent at the bottom that, you know, that's another category that popped up, like is that usually directionally correct, what you benchmark against or like spaces you're going into?
It's interesting and like I'm a click removed from this, but what I've observed, like working with our research team is that the, the benchmarks are, and evals in general are a helpful parameter of how the models are doing relative to the industry. But it's important to really ground sort of in your really hard customer problems almost more so than that. I think that's the case in, even within coding, like there's been deltas, you know, maybe two weeks before the final snapshot of the final snapshot where yes, it improved on, on Sweetbench, but even more so, it went from, ah, it's mostly reliable to, yeah, it's great and I'm using it every day in Claude Code. And I can't tell you that there was a crossing line between like 76 and 76.5 in Sweetbench. And actually it was interesting is even when it was already outperforming Opus, for example,
on Sweetbench, people still didn't feel it was better, but then it. Continue to train and it was like now better than Opus and people don't want to switch back. So there is this, Jared calls it a genesta qua, right? Like there's, there's something that can happen in the models where they become more useful for, for particular pieces. So in terms of verticals that we look at, you know, we have a couple of customers that really love pushing the model in different ways, right? Either they have like a super complex, you know, agentic coding harness like cognition does, or they have a really tricky legal set of problems, or they're using it for medical analysis. All these different pieces and that's why the early access feedback ends up being very valuable. Where my favorite early access channel is one where basically the, the folks that are working with these external companies come back and report back what was said.
And they'll say, great, they ran it on their hardest legal eval and this model is like 12 points better. And you know, they're not at the point necessarily where they're going to give us that eval. So we still have to triangulate a little bit, but that's the kind of feedback that I think shows us whether we're on the right track or not.
Yeah, cognition kind of did extra work on this. So usually that's what, that's all that cognition does. When a, when a new model drops, the, you know, the cognition sort of runs the evals and then, you know, gives that feedback back. But 4.5 was so good that like they got all the engineers excited and, you know, it's basically fast forwarded a rewrite that was already sort of, kind of in the works. And I think specifically planning was like the key main, the keys to the trigger. Cognition has always been like very interested in like sort of the interactive planning side of things. And so like, you know, I think you have the quote on the website that it like increased planning performance at 18%. And I also don't even know how you post-train for planning. Like, like it's just better, but I don't know how it's better. It's just better.
Well, it's interesting that like you, we can see it even within Claude code for more of the interactive coding side of things. Like having it list out its to-dos and make the plan. And then what's interesting is as we think about evolving Claude AI, like you can see this with our file creation pieces, kind of the first step in this direction. But models are going to be doing work over a longer time horizons for domains even far beyond coding. How do you, how do you build trust between the user and the, you know, the model or the product in those situations? I think it's the Model should express a plan. Eventually, maybe that plan becomes something you just trust the model is going to do well. But in the meantime, you know, imagine it's like, oh, I'm going to research these three companies. I'm going to do this and then I'm going to prepare this. You're like, no, no, no, no, you missed one thing there. And you want to be able to pride that feedback.
Yeah.
Ahead of time.
And so that I think is going to be like the next step for those. And I, I think of knowledge work as being on a similar sort of exponential as code has been on, but just time shifted. Right. And so it's only this year, it's really only probably only 4.5, honestly, where you're starting to see the glimmers of, yes, this could actually do work for me beyond code.
Yeah, I would say, you know, I, I think there's one, one part of that, which I sort of really wanted to double click on, which is kind of interactive planning, you know, I think a lot of people make a lot of a big deal about, for example, there's like 30 hours of autonomy in this model versus like the previous one was like, I don't know, two hours. And like, I mean, that's great, but don't you want like a back and forth, you know, like it's not about just like fire and forget. Like you actually want to go back and forth. And I feel like we're incentivizing people on the wrong things. So I'll give you an example, right? Like the, the cognition block was, was mostly like, oh, like we rewrote Devi, but like because of the better planning, it's faster now. And so you have less hours of autonomy, but that's, that's good.
Yeah, I think of the level, it's, it's a useful sort of barometer, like, yeah, so this, you know, we had a customer and internally, we also got like a 30 hour plus kind of execution versus I think Opus Four was seven hours.
Was that Claude code by the way? Those, those 30 hours.
It was, I think, it was like a harness built on top of Claude code, at least the internal one was. And the external one was a different coding startup. So I think it was just their own, their own harness. And I think it's useful to know what the like theoretical maximum is primarily to see how long the model can stay coherent and how long it can manage its own context in memory. I think that's like a useful thing. But for most use cases, I agree, there's much more of a, let's have some interactive back and forth. Let me build confidence in what you're going to do and then go do it. And then maybe while you're doing it, you know, the engineer switches to doing something else. So they fully delegated it and then look back. So I think that back and forth remains important. And it's like the, the theoretical maximum comes into play on things like, hey, I just need you to convert this entire repo
from this Neo version of Java to the next, which we get all the time from our enterprise customers. And those are the ones where it's like, you don't actually want to intervene that often. Once you've built trust that the plan is generally correct.
I know we only got a few minutes left. Another kind of big sneaky thing is like the SDK being renamed from Claude Code SDK to Claude Agent SDK. Yeah, what do you see as like the future of like the Anthropic platform? You have Claude AI, you have Claude Code, you have this agent SDK now. Are those the main pillars in your mind?
Yeah, I think of those as the main, the main pillar. So when I think of what people are trying to do on top of the models increasingly outside of the code domain, they're building complex agentic, you know, products, often having to reinvent that harness themselves. And I started getting texts from founder friends of mine who they were like, hey, I think I've just discovered a secret. Claude Code is actually like a very good general, like general purpose agentic harness. I'm like, I know, we're like, we've been like seeing it inside too. And but then people, you know, we'd go talk to companies about it. They're like, hey, I do like document processing for, for like a legal use case. Like, oh, you should try the Claude Code SDK. They're like, no, no, we don't need it to write code, which is, that's not what we're doing. And eventually I was like, okay, we're just pitching this thing and it's framed wrong.
It's actually a general purpose agent SDK. I think that there's going to be some interesting overlap. So here's the like universe of Claude, I think, going forward. There's Claude AI, and I think you'll see us start bringing that harness into Claude AI more and more for things like document creation, advanced research. And so it'll power a lot more of the agentic workflows in there. There's Claude Code, which is directly built on top of the Claude Agent SDK. And then there's external companies building on top of the Claude Agent SDK. And then I think we'll also offer ways in which if you want to run an agent off of that SDK, but have a lot of the computation done on our side, we can offer that as well. But I think what's really nice is that they all sort of compose out of the same building blocks. When I got the company together, I guess it was about maybe May or June,
around sort of rethinking our platform strategy, even internally, and sort of composable building blocks that then reform into whatever shape makes sense. And I'm starting to see it like kind of come to fruition. And it's very exciting because that same harness could be used in training. It can be used in Claude Code, it can be used in Claude AI, it can be used by our customers. We can host it and it all, all those benefits all sort of accrue.
Okay, awesome. Well, you know, I wanted to leave the door open to you. One of my favorite questions to ask people is like, what do you wish you were asked more of? Or the other, the other version of this is what do you want to hear from engineers? Right? Like you've put out all this amazing stuff. We have all the engineers on this podcast listening. What do you want to know from us?
I think a big one for us is, you know, especially given that evals only capture part of the story is how do we, we have a really good user research team at sort of diving into, you know, interesting ways in which AI is being used by everyday folks. But the thing I really want is a really good pulse on how the models do at really hard challenges over time. And so it's a thing we're trying to figure out. We're actually, we're just scheduling some Zoom calls with some folks in the community that like reached out and like, hey Claude, like it solves this problem well, it doesn't solve this well. Can we like, you know, we were like, hey, can we just zoom with you? It's something I used to do at Instagram. Our engineers sometimes get stuck on a problem that they couldn't repro. And I was like, have you just tried going on Twitter and asking somebody who was like experiencing that crash, whether you can like get some logs from
them? And then they would and they'd be like, oh, you know, it turns out it was like this particular Samsung phone and we just didn't see it in the logs. And so I think my call to you all is like, both on what Claude is doing well, but especially on what it could be doing better because that helps us improve. Like if you're all open to like us spending time with you, it's like great user research on our side. So yeah, I think that that's great. You can feel free to, you know, submit feedback via Claude code or via Claude AI, for example. But you know, if you express like, hey, I'd totally be down to show this to you, we'd like, we're all ears.
That's amazing how eager it is. Yeah. I also like the drop that you used Twitter for support at Instagram.
It was great. It was, it was, it was very effective. I'd like, you know, we had a, one of our core principles is do the simple thing first. And it's really funny. Like I knew Anthropic was going to be my second home. When I got here and I was like, oh, what are your like core philosophies? They're like, oh, do the simple thing that works. I'm like, well, that's really close to do the simple thing first. Like, let's go with like, uh, so there's, I just felt very aligned with how, how folks think about things here.
Awesome. Awesome, Mike. Thank you so much for taking the time today. And, you know, I'm sure we'll do a longer episode in the future.
Sounds great. Thanks for having me, guys.
Page updated October 24, 2026 · transcript and summary are machine-generated