How Anthropic Builds And How Engineering Will Change Soon | Thariq Shihipar

Thariq Shihipar is an engineer on Anthropic’s Claude Code team I asked him how Anthropic makes the most out of the models for engineering and how the industry will change soon. • My ergonomic keyboard project I mentioned, you can follow along here: https://read.compose.llc/ • The Kickstarter page for it: https://www.kickstarter.com/projects/ryanlpeterman/compose-simple-ergonomics-beautifully-done Podcast links: • YouTube: https://youtu.be/2Kch3tWMnw8 • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/how-anthropic-builds-and-how-engineering Thank you to this episode's sponsor for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at https://workos.com/ Timestamps: (00:00) Intro (00:29) Onboarding at Anthropic (02:53) Internal capabilities vs external perception (06:16) Model vs Harness (08:55) What percent of Anthropics changes are fully autonomous (14:51) Computer use (17:42) How to make the most out of your compute (20:45) Loop engineering (22:47) Where the industry will go soon (26:02) Which model do Anthropic engineers use (27:38) Is learning a particular model worth it (30:56) Prompting tips for todays models (35:04) How to get the models to do tasteful work (39:00) How much of writing is done by AI at Anthropic (45:36) Code ownership and maintenance at Anthropic (52:04) How Anthropic prevents breakages (55:24) Visibility and sharing your work (58:42) Luck surface area example (01:00:57) Should people still learn to code (01:07:42) Advice for his younger self (01:09:58) Outro Where to find Thariq: • X/Twitter: https://x.com/trq212 • LinkedIn: https://www.linkedin.com/in/thariqshihipar/ • Personal Website: https://www.thariq.io/ Where to find Ryan: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman Referenced in this episode: • Anthropic's post on removing 80% of Claude Code's system prompt: https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Most software we wrote before LLMs is not very good.

• 0:03Ryan Peterman?

  This is Dharik. He's an engineer at Anthropic on the Claude code team. And I asked him all about how their engineering team makes the most out of the models today.

• 0:12Speaker 00

  Can you move up like a higher abstraction level, you know, like, can you build the system that builds a system?

• 0:17Ryan Peterman?

  Is there something that you feel is kind of the next big shift that is gonna diffuse into the industry?

• 0:24Speaker 00

  Yeah, I think there are a few different ones.

• 0:28Ryan Peterman?

  Here's the full episode.

• 0:34Ryan Peterman?

  My goal with this conversation is to ask you as much as I can about how can you get the most out of the models, specifically for software engineering, so that people in the industry can kind of learn from the best practices where Anthropic's having success. And so to start the conversation off, I'd like to ask you, let's say I was coming from a company that's maybe less AI-pilled, and I was onboarding onto your team. What are the most impactful things that you'd tell me to start doing to successfully onboard?

• 1:08Speaker 00

  I think increasingly, we're like, just talk to Claude about it. I think the number one tip we have for, you know, both people inside and outside Anthropic is that like, if you treat Claude like a thought partner and give it like the context that you need, then you can usually figure out the next steps, you know? And so I think that like, starting with, okay, can Claude do it? If not, why not, you know? And using that as the starting place, I think is really important. And I think just like, always thinking about like, okay, what's, you know, reflecting on, can you move up like a higher abstraction level, you know? Like, can you, you know, set up a system? Can you build the system that builds a system, right?

• 1:50Speaker 00

  Instead of just like, building, you know, the, the product itself.

• 1:55Ryan Peterman?

  I remember when we used to onboard people, you would get assigned an onboarding buddy. So someone who kind of really knows the code base and you can ask all of your trivial setup questions to. It sounds like Claude fills in a lot of those gaps. And in practice, is it 100% you don't need an onboarding buddy and you can just ask the model for everything?

• 2:17Speaker 00

  You do need an onboarding buddy, but more from like a, kind of like social and cultural perspective than a technical perspective. You know what I mean? I think like, from a technical perspective, you can basically just like, work with Claude. And, you know, if you're like, good at it, like, you can get what you need to onboard. But I think just like, having the context of, you know, how to work with the team and also just like, how to like, you know, get buy-in on what you're building or, you know, understand like how the team works together. Or just even like have a friend to work with, you know, I think is really important. So, yeah, we still have onboarding buddies, but not, it's not like nearly as much technical lift as it used to be.

• 2:57Ryan Peterman?

  There's a big difference between the external perception of the AI capabilities and the internal Anthropic usage. Because I'll talk to friends and say, yeah, what Boris is saying is the reality. And so my question is, why is there such a big difference between that, that internal perception and that external perception?

• 3:18Speaker 00

  I think we have a good track record, kind of, of when we make these like claims. You know, you're like, oh, you're, you're like, it turns out to be true on a long, large enough time scale. You know, I think a lot of engineers are not really into IDE anymore, you know, like, yeah. The code being written is like, I, I talked to larger enterprise customers where like, you know, they haven't like typed a line of code themselves in six months, right? So we think of it part of our jobs, you know, to figure out how to work at a higher abstraction level. And like, if I spend like a day trying to figure out how to get Claude to like, you know, do this autonomously and I fail, that's like, fine, you know, and maybe even good, you know, because now I can be like, oh, hey, like Claude is not good at this. How do we make it better, you know?

• 4:03Speaker 00

  But I think if you're working like an average job, you know, your job is to produce the output. And like, I think automation always has a cost because you're like, it's an investment. And if it works, then great, you'll make, you know, a return on your investment over time. But if it doesn't, now you've like wasted all this time. And so I think the nice thing about the models is that, you know, generally the chance of the investment working out is higher and higher. Because like, the models are getting smarter and smarter. But you still have to sort of take that leap as an individual. Your boss is not going to be super happy with you if you're like, you know, if you've been working on your harness setup the entire time and like not shipping code.

• 4:43Speaker 00

  But yeah, I think it's mostly just like culture and thinking of it as our job to sort of live in the future. And then we try and like build it into the product and our harness as well, so that you don't have to do as much manual step, yeah.

• 4:57Ryan Peterman?

  Are there examples that you come to mind when you think of things you don't see people doing on Twitter? That are really having, making a big difference at Anthropic that you'd kind of recommend?

• 5:08Speaker 00

  Like, using Claude code for knowledge work, I think is really valuable. The way a technical person does knowledge work, I think is very different than the way a non-technical person does knowledge work. These days, because you can get the models to do it. And so I think like, you know, increasingly, if you can figure out like, hey, this is a task. The task is made up of code-like things, you know? And like, how do I tell the model the code steps to take, you know? In order to do it, right? You can like, you know, I think you can do a lot more, right? So I think I do like a lot of accounting with Claude code using, on the personal side, using like scripts and Python instead of Excel, you know?

• 5:53Speaker 00

  Or like, I do video editing using FFmpeg and these libraries to render visuals and things like that. And so I think most of knowledge work is like, reducible to code and coding agents, if you like, think about it well, you know? And I think that that's like one, I think like, big difference that I see like, other people doing less of, I think.

• 6:21Ryan Peterman?

  We talked a little bit about the, the model and the harness. And I want to ask you, what's the relationship between the model and the harness in terms of getting the best end results? Obviously, the model's important. So how important is the harness? And are there any examples you could kind of share?

• 6:39Speaker 00

  Yeah, I mean, the harness is super important. I think you, I think that sometimes there's this idea that the harness doesn't matter because the models will get better and better. And if the model just does everything perfectly, then why do you need a harness at all, right? And I think that in practice, what we see is like, the models get better and better. And so the harness needs to become more and more complicated to allow the model to do more things, right? And so an example of this is auto mode. And so auto mode, you know, is a classifier that runs after every, you know, task that you normally have to ask a permission prompt for, for Claude.

• 7:20Speaker 00

  And back when we were like, you know, Opus 4 or even Opus 4.5 maybe, like, it was not so bad to hit enter on the permission prompts because like you were, you know, the turns would only last a few minutes anyways, right? And now Claude is running, you know, can run for hours. And so you really need that ability for it to, you know, like, do work safely, stick to your instructions. And auto mode is really complicated software, you know, sandboxing is really complicated software. And then like, you know, you also think about things like, okay, like Claude can do so much more work now.

• 7:55Speaker 00

  How does it represent the work that it's done? It's worked for like eight hours and you want to know what's done in eight hours, right? So this is what we use artifacts for, right? And artifacts themselves are a form of prompting, because like, how does Claude represent that in a useful way, right? Like, there's a lot of different ways that it could represent that information. And so I think roughly, what we see is like, the harness, harness engineering is definitely like, this like mix of science and art. I think it's very unintuitive in a lot of different ways.

• 8:27Speaker 00

  But I think it has like just big abilities to like, unlock new parts of, you know, like model behavior. And yeah, I just see the harnesses get more and more complicated. And it's kind of harder and harder to like, actually vibe chord your own, which is like a little bit unintuitive to me because of like how good the models have gotten. But yeah, things like auto mode and workflows and things like that, that which are like, actually quite complicated pieces of software, become like really load-bearing, I think.

• 8:59Ryan Peterman?

  As LLMs have advanced, more and more of the human part can be taken out of the loop. You know, at Anthropic, what percent of your or your team's changes are fully autonomously made versus actually pairing with a model like people were doing more of like around a year ago?

• 9:21Speaker 00

  Um, I think it's very dependent on, you know, what you consider autonomously made or um, what uh, what team or what function you're working on as well. You know, for example, a designer might give you a Figma file and then you pass the Figma file to Claude code. Uh, Claude code is great at using the Figma MCP, but like, you know, the designers put a lot of work in there. Um, I think that like roughly, um, our goal is to sort of get Claude being able to do all of the like, uh, the glue work that like ties everything together, right? So, okay, you've got like a Figma file. Do I really need to like rewrite that in React or something like probably not, right?

• 10:03Speaker 00

  Like, I think like that work has been done once, right? And so the way I think about it's like, what's the unique work that I need to do every day, right? And uh, the more I'm like doing that unique work, the better. And, and like, I think there's just a ton of demand for, you know, unique work and thinking. And the more I'm doing something I've done before, I'm like, okay, like, can, can Claude do this? Or like, if I'm just translating what someone else has done, you know, I'm like, oh, like, can Claude do this as well? It's really blurring the lines of like, what, what does it mean? Even, even if Claude has fully generated this PR, you've probably done a lot of, uh, made a lot of decisions and, and add a lot of context along the way, so.

• 10:45Ryan Peterman?

  How close are we to a world where someone just says, hey Claude, here's the ticket, just don't tell me till you're done?

• 10:54Speaker 00

  Yeah, so I mean, it depends on how good the ticket is. If someone has like perfectly written out essentially like a software spec for this ticket, then yeah, actually Claude can probably do that, right? But I think that like, um, to start, you know, like, let's say that we get a GitHub issue. That, you know, is this worth fixing? Is this a feature, like oftentimes issues are like also feature request or something, where there might be like multiple things happening that maybe you combine together in a different way. You know what I mean? And so I think that there is a, a lot of that work is like, okay, what's the vision for the product? Where, where do we want to go? How do we like, you know, um, make sure what we were building is cohesive?

• 11:35Speaker 00

  And so I, I think for a specific spec, like a specific enough spec Claude can do it. In practice, people don't have, haven't figured out what they really want, you know? And haven't figured out the unknowns or the shape of the problem and things like that. And maybe the way you would have done it before is like, you start writing the code and you're like, oh, like, what am I supposed to do now? Like, or like, you know, like you, you figured out that way. Uh, now I think you need new ways of figuring out what you don't know yet, right? And I think you can still chat to Claude really, um, but I think, uh, that's the hard part. And I, I think definitely a failure mode is like, you, you tag Claude tag and you're like, hey, please do this.

• 12:17Speaker 00

  You know, one sentence or less description, no previous context or memory. Um, and then it does it and you're like, oh no, I don't like it, you know? And then you're like, you know, just iterating forever on that, right? Versus like figuring out what you actually want, like really quickly.

• 12:34Ryan Peterman?

  When I was working at Meta, like the ambiguity of the task really ranges oftentimes proportional to people's. I mean, levels aren't everything in, in software engineering, but it's roughly senior engineers kind of take that ambiguous uh, business need and convert it into something that's a lot more concrete. And then people who are maybe newer in their careers to kind of do the implementation work. Um, and like intern projects were kind of almost line by line specified. If I was still uh, working at Meta and I had interns, like, just kind of give the, give Claude my interns back and it would kind of do it.

• 13:15Ryan Peterman?

  What kind of work does an intern do then at Anthropic if Claude can kind of handle it?

• 13:21Speaker 00

  What we see in practice is that there are just so many new types of work that no one has ever done before, right? And I think that like, we all have to figure that out. And so, for example, how do you do an eval, you know what I mean? Against like, uh, these new coding behaviors, right? Like, how do you make sure like, uh, that like, you know, how do you measure Claude's performance across like millions and millions of users? Uh, mid, across doing all sorts of tasks. Sometimes these tasks might have trade offs, you know? So I think there is a lot of new work to be done, especially like in the research side of things. And um, I think that like, it's harder and harder to write those specs,

• 14:04Speaker 00

  but there's also less and less experience that is relevant, you know what I mean? And so like, I think that, yes, having some experience means that you can sort of, you know, y- y- you know how to get work done and you know how to like learn. But I think you also get those skills out of college and, and you don't now, I think as an intern, trying to figure out like, okay, what are the things that people have not done before? And, and doing that, I think is really exciting. And I think that there is, um, yeah, I think internships are less write the React code, you know what I mean? And there's so many new problems to solve. We need people to solve them. It's helpful if you have a fresh set of eyes on it, you know?

• 14:44Speaker 00

  Um, but it is definitely be more proactive and like opportunistic maybe than uh, than before, where if you're, it's maybe more of like a pipeline.

• 14:54Ryan Peterman?

  We talked a little bit about knowledge work and that makes me think of computer use and browser use. And I wanted to ask you, you know, how far away is that from being widely adopted in the industry and impactful? Maybe you can talk about how Anthropic uses it since it feels like you're always far ahead.

• 15:14Speaker 00

  I, I think computer and browser use, the models have gone a lot better. Opus five, I think is like a really good computer use model. But there are like these weird edge cases where like, for example, it can't type a password on my behalf because my passwords aren't one password or something. You know, it can't access that. And so like it gets stuck. And um, I think there's still like those edge cases which are kind of UXE edge cases. Um, and then I think also obviously computer use is also, uh, you know, the more and more people turn APIs and MCPs into like new ways to use Claude. I think that can also take a lot of the use cases that you're using computers for, right? And so going back to like, oh yeah, knowledge work, everything is code.

• 15:58Speaker 00

  I think like, you know, there are some things where there is no API, there is no way to execute in code other than just like opening up your browser. But I think increasingly there are more and more ways. And I think, yeah, Claude tag is a great example of, we've just sort of like tried to roll up all these things into APIs that it can use. And so, um, it feels like it can do a lot of work on your behalf, even though, even though it's not literally running like a virtual computer and clicking things, you know?

• 16:27Ryan Peterman?

  I have noticed that whenever I use uh, any kind of computer use tooling, it feels painfully slow. I look at the cursor and it's sitting there for 10 seconds, moves over, you know, goes there for 10 seconds. What, where is all that latency coming from? Do you have a sense?

• 16:44Speaker 00

  I think that like, it's hard to make a small model that's really, really good at it. I think just because there's a lot of knowledge that you need to have about this task and how these things work together and stuff. And so, um, you want like a smart model and smart models, you know, take a little bit longer time. It, it's like, I think a lot of people thought we'd get here sooner, but I think it's just been a harder task than we expected. Um, I think like one thing I've, someone's told me about computers before is that like, it's a state machine where you don't control the entire state. If you are on the DoorDash website or something, you want to add something to the cart and you add it incorrectly, now you have this new flow to like undo it, you know? Now you need to go click and now you need to go delete it. And you can make a mistake along that side as well, right?

• 17:29Speaker 00

  Whereas like in code, you can sort of undo git, you know, whatever, you control all of the state. Um, but for computer use, like each action is, if not irreversible, it's like, uh, you know, much harder to review, reverse than, than others.

• 17:45Ryan Peterman?

  Anthropic, um, I get the sense that employees have a lot of budget in terms of the compute to kind of speed up whatever it is they need to do. And so to, if we were to spur the imagination of people who use the models to make them more productive, assuming they had infinite compute, like what, what type of workflows would you start telling someone to do if they had infinite compute?

• 18:13Speaker 00

  I, I think this difference is slightly more exaggerated than you think, I think, you know what I mean? I think that like, for example, I use my Mac sub on the weekends and I have almost never hit a five hour limit. I think the models are really smart. And I think that like, a lot of times when we're spending a lot of compute, we're just trying to find sort of capabilities or we're trying a bunch of different things. And, and it's more about like us figuring out model possibilities, you know what I mean? Than getting a lot of work done. When we're testing for math, for example, we're trying to understand how smart is the prob, the, the model? And that is useful to us, that's useful output to work.

• 18:55Speaker 00

  And it can also sometimes solve like the Riemann hypothesis or something, or like not make progress, you know what I mean? People can replicate what we do at home just by thinking at a higher level of abstraction. For example, I'm trying to like get Claude to draft feedback for you. And so I want the funnel for like, Claude has drafted some feedback to the user, has submitted feedback to be really good, right? And so I monitor that funnel and then I ask Claude, I, I had some ideas, but then I was also like, oh, what if I ask Claude to try and improve the funnel? You know, and it'd be like, here are some ideas, can you come up with some as well? Let's figure it out. All of those things, you can kind of do yourself right now, you know?

• 19:37Speaker 00

  If you're like, okay, let me, when I'm making a feature, let me like, uh, annotate it with events, you know? Let me make sure that Claude has access to those events. Uh, let me run a loop in the morning every day to check, you know, what events have fired and what changes were made. Maybe even like, let me proactively suggest some ideas, right? This can all happen, I think, within a fairly reasonable amount of compute. I, actually what I see more often is people running into limits where they've actually done kind of the opposite. They've started with a small mid scope task, you know, it's like, oh, hey, like, uh, you know, refactor this function in this way. And then Claude does it and, and maybe it's like, has some following effects.

• 20:19Speaker 00

  Cause like refactoring this has, means you have to do some other work too. And that's not exactly correct or like, you know, you will, you're iterating there. And it's like, you know, you've just spent a lot of time, whereas, uh, if you had sort of like stepped up a level, told Claude your goals, then figure out, okay, what are the details? You know, do some exploration, uh, maybe write out the schema or like, you know, and, and then work with it, then let it run. You could probably get the same.

• 20:47Speaker 00

  Output.

• 20:48Ryan Peterman?

  I remember it was going kind of viral, this idea of creating loops. And I mean, that that does feel like a pretty expensive sort of thing to set up if you're just asking it to kind of keep hammering away. Maybe first, could you define this loop engineering thing? And then I'm curious, how often do you use it? And, you know, would you hit limits if you were on a max plan doing that kind of stuff?

• 21:13Speaker 00

  Yeah, so I, okay, so loop engineering roughly, it's like instead of prompting Claude directly, you're setting up a system that prompts Claude, you know? And you don't have to think of it. If you use Claude tag, a lot of this comes naturally. You just ask it like, hey, every day do this thing, you know? And that will, that's a loop. You can definitely do that sort of work right now. But I think if you're trying to set up like, let's say like 10 loops or something, you know, that are like

• 21:48Speaker 00

  triaging your feedback and implementing and things like that. We do that sort of work, but we also spend a lot of time making sure our skills and things like that are useful, you know, and like are good at like triaging, that we can have, we're good at verification and so that we can like make sure that the changes land, you know? And then I think at that level, you know, what it means is like now we have someone monitoring issues that we just could never have kept on top of before, right? And it just like increases our software development velocity and it's worth a lot of value to us. And so, yeah, I think that like, if you're kind of at the scale of like, okay, you want it to essentially be autonomously running, you know,

• 22:32Speaker 00

  doing a software engineering job for a certain cases, I think it can do that if you set up the verification well, if you set up your skills well, if you give it the right data sources. But it is a lot of work, right? And I think that like, you have to make sure that that work is valuable to you in the same way that like, if you hire someone, you know, you have to make sure that work is valuable.

• 22:50Ryan Peterman?

  We've talked a little bit about Anthropic being kind of just the head because that's the job of the company on adopting AI. Is there something that you feel is kind of the next big shift that maybe Anthropic's already felt? That you feel is gonna diffuse into the industry? And if so, what might that be?

• 23:09Speaker 00

  Yeah, I think there are a few different ones. Obviously Claude tag, I think is our, you know, the way we do a lot of our work right now, right? And I think the way I think about it is that Claude code is really good at the implementation of code and Claude tag is for the rest of the software development lifecycle, you know? So like, so getting feedback, you know, doing code review, like babysitting, building, C I C D incidents, like, you know, all of these other things Claude tag is really good at. At a high level, turning every part of your software development lifecycle into kind of like a routine or a loop using Claude tag or something like that, I think is probably where things are headed, you know?

• 23:54Speaker 00

  I think probably generative interfaces are still to come. And I think that like artifacts and, you know, yeah, basically artifacts, I think are going to be like an increasingly large way of how you like interact and read with Claude. And, and so I think that, you know, I use artifacts for almost everything. And, and so I think that like, we're going to see that also become big. And I think that's also ties into Claude tag, right? So like, let's say that you are on your phone and Claude tag has done a bunch of work for you. It creates a report, the report is readable, you know, on your phone because it's made the like, the artifact look really great.

• 24:32Ryan Peterman?

  Could you explain artifacts a little bit or kind of what are they and how do people use them?

• 24:37Speaker 00

  Artifacts are basically,

• 24:41Speaker 00

  Claude can upload essentially a web app for you to use, you know? And I think that

• 24:47Speaker 00

  you can use it in a really wide range of capabilities. They can now, for example, call your MCP. So you can make an artifact, for example, to read your inbox, you know, and like display it or sort it or tag it, right? As like a one way, you can also use artifact to show you a plan for coding, right? And then that might show like diagrams and file snippets and code and schemas. And so it shows you like sort of, it's like an interactive display for the tat, for the job you're doing right now, you know? And as Claude does more and more, you know, like jobs, we're realizing that like just text in, text out is probably not useful for everything, right?

• 25:31Speaker 00

  And Claude is doing, is better and better at creating essentially the exact interface for you at the right time. I think that there is like a lot more to do there. And I think it's another one of those things where you have to think about like, oh, you know, could I be interacting with Claude in a different way, right? Like, could I interact with it through an artifact? Could I, you know, use it to like learn more or like to understand it better or stay in the loop? Or

• 25:57Speaker 00

  improve some of my like own knowledge work or something? So, yeah, I think it's pretty exciting. We're still kind of early to it.

• 26:05Ryan Peterman?

  Does the daily driver model vary among engineers? Or do people typically just pick the most intelligent one that's available at Anthropic?

• 26:14Speaker 00

  I do think, you know, similarly with models, if you use the smart models and you use them well and you give them tasks where they can, you know, are well shaped and you've spent some time setting up a good verification harness and things like that, you can get a lot out of them. I don't think it's like choosing which model at the right time, you know? I think it's like, oh, how do you get the most out of the frontier models? Because I think technology works the way it does, right? Like everything gets more abundant, more available. And so I think we're in this current weird spot where, you know, we don't quite have enough compute for everyone to have Fable at 100% of the rate limits. But I don't think this is like a durable skill to build. Figuring out like, oh, when do you use Fable and when do you use Sonnet, right?

• 26:59Speaker 00

  So I think it's like unintuitive there. But I think in practice, if you're today, what I would do is like, I use Fable for planning, for brainstorming, for finding unknowns, for coming up with a detailed spec, and I'd use Opus Five to implement it.

• 27:17Speaker 00

  And I would probably use Opus Five with workflows, using like a verification aid, like schema or like harness that's built with Fable, right? So I'd use Fable for those high leverage tasks.

• 27:29Speaker 00

  And yeah, Opus Five for the execution and implementation. But yeah, I think increasingly, probably next year, I, I think you're just not going to be thinking that much about like which model.

• 27:41Ryan Peterman?

  You mentioned that skill of, I guess, prompting. And I've heard some people, they say it's not too durable of a skill because it's kind of, it's really specific to a model. Like these models, they almost have their own unique, I guess, spiky intelligence. So if you, if you knew everything that was very specific to, let's say today's Fable and you're a master of today's Fable. I mean, maybe you don't need to know any of that stuff like a year from now and Fable three or four or five or whatever comes out. Let's say you were talking to software engineer who's looking for career advice.

• 28:21Ryan Peterman?

  And they're thinking, hey, should I really become a master of engineering my prompt?

• 28:27Speaker 00

  I do want to say prompting is like a little bit more than just a prompt you put in. It's also, you know, you might have done, made a skill or you might have like, you know, added some data or something. It's not just a prompt you write, but it's like everything you've done before that builds up into your context, right? So sometimes people see us write small prompts and they're like, oh, what does that mean? But we've, we just spend so much time on the, the harness and the verification and the skills. So I think it will be like really valuable to just keep better at prompting. I think to what you're saying about each model is different, you're right. Like I think each model is kind of its own kind of like almost organic digital thing, you know?

• 29:09Speaker 00

  And so there are quirks you have to learn and you do have to unlearn them. So we recently wrote about like how we removed 80% of the system prompt from Claude code, right? And one of the learnings we had was like, we needed to remove examples from the tool descriptions. And, you know, this used to be the only way you could get good output from the models was through tools, right? Or, or through examples, right? So you'd have to be like, hey, this is the right tool. Use this here, here's an example of writing a file well, and here's an example of not doing it well. And now we found that like examples are mostly negative, I think, unless you really see Claude doing something you don't like because

• 29:53Speaker 00

  it's just like quite imaginative. It's good at sticking to your intention and working with you. And so we removed a lot of examples. So in that case, yes, you do have to sort of like adapt. But the skill you're building is the skill to adapt. You learned how to use Fable. And now you know a lot about Fable, but you also know how to learn, you learned how to work with a model, right? And so Fable 5.5 comes out, you need to learn how to use it again as well. But you'll be much faster because you're better at learning. You're better at working with Fable 5, you know? And I think for me, I think the first model I worked with was G P T two. And I remember like it was, it was so hard to get a JSON output out of G P T two.

• 30:38Speaker 00

  Like if you could just get it to like choose one of the categories that you gave it, that would be like incredible, you know? And so I think that, but like building that skill, G P T two is such a different model than Fable five. But I feel like the skill I spent doing that has like helped me be better at prompting Fable five.

• 30:59Ryan Peterman?

  Is there any like tribal knowledge or quirky tips in today's models? Or you'd say someone should, should know that to get more when they prompt?

• 31:07Speaker 00

  I actually need to counter one, I think that people have been saying where it's like, oh, just believe in yourself or something. I, I know that Jared, Jared's post about the Reiman hypothesis had like, he was just like, keep going. We're just believe in yourself. I think in this case, it was mostly just Jared saying it's okay to use compute to solve this problem and I'm giving you permission to do it, you know? And I think that like, that's not exactly the same as, you know, I believe in you, you know what I mean? It's really just like letting the model use compute. So I think that that is something I would, I like to tell the models right now is like, okay, hey, I think this is a hard problem. Use subagents, you know, use workflows, like if you need it, right?

• 31:52Speaker 00

  So like, I always tell it to like use its own judgment, but I'm giving you.

• 31:59Speaker 00

  Permission to do this stuff, right? And I think that uh you have to sort of remember that the models by default, you know, do what maybe the average user wants, which is like they wanted to respond and start doing work as fast as possible. Roughly like complete the task, but not spend like a crazy amount of compute on it, you know? And so I think that like um you have to sort of, if you want the model to do it differently, you have to nudge it slightly, right? So you might have to be like, okay, hey, like, I don't want you to do any work yet. I want you to brainstorm, you know, I want you to like think with me, right? Um, and if you prompt it that way, it will start doing that. If you want it to spend a lot of compute, if you're like, hey, you know, sometimes I'll say like, yeah, hey, I think this is a hard problem.

• 32:44Speaker 00

  Uh, feel free to use workflows. If I'm running overnight, I might just be like, hey, I'm going to sleep, you know, set a slash goal or something and then uh let it, let it run. So um, yeah, I think there is a uh just like giving it permission to do the thing you want.

• 33:01Ryan Peterman?

  When you recently removed so much of the system prompt, uh how did you prove that the end result was better?

• 33:09Speaker 00

  We have a bunch of user metrics, just like how, you know, how much do people like the output of Claude? Do something, you get that survey and you see it. We run evals against, you know, our internal eval uh and external evals to see like how it performs at these different tasks. Um, but I, I think it is hard, like sometimes you don't realize that they're not, if Claude is telling the user, if it does all its work and then it's like, hey, maybe you should go to sleep. There's no eval for Claude tells you to go to sleep, you know what I mean? And now we're like, have to like catch this like new behavior. Um, so it is hard. I think we spent a lot of time basically just um, you know, like removing lines in system prompt, running evals,

• 33:52Speaker 00

  uh seeing how it works, seeing how people uh reported it internally and then like adjusting. But it was like a full-time job for several people over long periods of time. And so I don't think I necessarily recommend everyone do this. I think that's kind of why we wrote that post about what we learned from like adjusting the system prompt. And we think that's pretty general, so hopefully you don't have to like now go through this like crazy iteration process.

• 34:19Ryan Peterman?

  OpenAI, Anthropic, Cursor and Versal all use this product to make their lives better. And the problem it solves is when you're building SaaS or an AI product and you want to sell to other companies. There's all these requirements you need to meet. There's SSO, there's SCIM, there's RBAC, there's audit logs. These are all things that take time to integrate, but aren't the main focus of your app. WorkOS is an API layer that lets you meet all of these requirements in just a few lines of code. So let's say you have a new SaaS product and you want to sell to other companies. WorkOS will solve all of these critical feature gaps for you. You can check them out at workos.com to learn more and get started.

• 35:01Ryan Peterman?

  And I appreciate them for supporting my work and sponsoring this podcast. I think a lot of people, when they use Claude code, they get excellent results when it's kind of getting very objective work done. But when it's kind of prompting models to do beautiful or tasteful work, it's kind of not always, it's pretty much more hit and miss. And so, you know, how do you best instill like a very particular style you're going for or particular taste? In the models outputs when it's a much more subjective domain, maybe like front end?

• 35:38Speaker 00

  The way you do it is sort of like you stay in the loop. I think you give it references, right? And I think the more references you give it with data, the better. So like, it's better to give an HTML file than a screenshot, right? It's better to give a Figma file than like a raster image or something, right? Cause now if Claude wants to know the border radius of this thing, it's like, oh, what's the Figma component border radius? And let me just copy it over. And so I think giving it a bunch of references, ideally in code is a really good way, right? Of like getting it to stick to this. Um, I think if not, you can then ask it to, if you don't have like, uh, let's say you're not a designer.

• 36:21Speaker 00

  There's probably step one is being like, okay, I'm not a designer. There's a lot I don't know about design, you know? And like, there's a lot I don't know about iteration. I don't even know what good looks like, right? And I think this is like part of the art of working with the designers, like they will just know what good looks like. And they'll be like, this isn't good enough in this way, right? And they like prompt it. And that, I think is also a skill that will keep getting valuable and even more valuable over time. It's just like, what is good output? What is worth doing, right? Like, I think, uh, for example, with the uh the Reiman hypothesis, Jared prompted it, but he had no idea if it was correct. Until like Lev, who's like, you know, one of the world's best mathematicians.

• 37:04Speaker 00

  So it was like, you know, okay, like, how do I, is this correct, right? And he worked with it and he asked it like tons of follow-up questions. And we could not follow that at all. We had no idea what he was saying, right? But he was like really intrigued. And, and so I think more and more being that like high taste user and like knowing a lot about a problem in a domain space is how you get good outputs, right? That's how you solve these problems. Uh, otherwise, like maybe Claude did solve like physics or something, but you just wouldn't know it, right? Like you're like, uh, you, you don't know enough about it. And so I think when you're talking about design, the first thing you do is how do you become more tasteful with design, right? And so you can ask Claude that as well, right? Like you can be like, hey, I'm not a designer.

• 37:48Speaker 00

  I want to be better at design. I don't even have the language. First, maybe let's find some reference sites and, and then maybe you pull some reference sites. And then you're like, this is what I like, or this is what I don't like, right? And then you sort of build up that, you know, those references and you give Claude it. And, and now maybe you tell it, hey, let's do some exploration. This is sort of my taste. And then it'll do like a few different mockups, right? And I like to do these mockups all in HTML because it's all self-contained, easy to edit. And then, once you have that, you know, that reference, now it's a reference. So now you can be like, you can make a new session and be like, hey, this is a mockup of a design I want. Uh, start implementing this and you know, it will have all of it in code and you can start getting there.

• 38:32Speaker 00

  Um, but I think that like, the really hard part is just knowing like, oh, when is something good enough? You know, versus like, when can you, you know, push harder, right? And you see this with a lot of the math proofs too, right? And when like Terrence Tao is like, okay, enough with the half complete theorem, just do the full theorem, you know? And, and so I think like, it sounds simple, but it takes a lot of domain knowledge and expertise to get to the point to be like, okay, like, it seems like you're smart enough to have done that, now do this.

• 39:03Ryan Peterman?

  When I was a software engineer, there was a lot of um, I guess it was like glue writing work kind of where you, you finish some work and you write a launch post or you are part of some work stream and you gotta post an update every two to four weeks. Or maybe you have a direction doc or a design doc and all this like writing around the software that you actually write. And cue your thoughts on if that's changed at all at Anthropic? How much of that is written by LLMs and how much of that is, you know, human written still?

• 39:44Speaker 00

  My rule of thumb is like, if I would be happy to show someone the prompt, I would send them the output, right? And so a lot of times the prompt is just collecting context is the most common one, right? So for example, uh, before every one-on-one with my manager, I asked Claude to, you know, read every Slack message and GitHub message and, you know, or read PR and compile like, you know, a report of what I did, right? And uh, that's context that my manager doesn't have because, you know, they haven't been literally reading every PR or something. And so I don't feel bad if like, you know, like my manager could also run this command, but it doesn't have my context.

• 40:26Speaker 00

  And I don't feel bad with them knowing that this is the prompt I used to generate this command, right? So um, I think likewise, like if you're sharing updates or something like that, you know, like uh, you might want to prove that you've read it, you know? I think this is important and, and sometimes like little edits are ways of proving that you have also understood this work, right? And so if it's just like a data readout, you know, maybe you're sanity checking like the numbers make sense. And this like, these are all the numbers you intended to include. And as part of that, maybe you format it differently or you add like a little sentence on your behalf, right? And, but I think ultimately it's a data readout.

• 41:06Speaker 00

  Everyone knows the prompt you wrote is like, hey, like generate a readout on this feature based on this and, and you're fine, right? But like, I think if you're pitching like a new concept or a new idea and your prompt was like, help me come up with a new concept for, you know, this product, right? Like you probably at least people want you to know that, want to know that you've like believed in it, right? And so like, even if you think Claude's idea is incredible and just verbatim, you wouldn't change anything. What I would say is like, I'd be like, hey, Claude generated this, um, but I think it's great. You know, like I, or I've did like a hundred different generations and I think this was really good. And here is like something, you know, that I want to send you, right?

• 41:48Speaker 00

  Um, and so I think that like, it's good to be upfront about it, I think, right? Because I do think what people don't like is when they feel like they've been like misled a little bit like, oh, like you, we thought you were doing this work. But you know, um, it, it's really Claude, right? And I think that like, um, writing sometimes can have a lot of, like, there are some parts of writing where individual words matter. You know, like funnily, like tweets are I think an example of this where like the individual tweet matters, right? So um, you like more and more, you can't use Claude to do that because it's like, like every word has some thought that you've put into it and some intention that you've put into it. So like, you know, yeah, a pitch or an essay or something like that.

• 42:32Speaker 00

  We do a lot of like internal essays at Anthropic being like, hey, this is why I think we should do this. And that's generally like all human written. It's very like looked down upon, I think, to have like Claude, like your essay written by Claude, you know what I mean? Uh, because like every word is something that you like are intentional about.

• 42:52Ryan Peterman?

  So it sounds like the proportion of the writing that's all that boring route writing, like the data readouts, the the one-on-one updates, the work stream updates, that's increasingly becoming AI, but always reviewed by a human. And then the novel thoughts, novel direction is still very human written. And it feels like it should remain that way, even if the models were a little bit better, too.

• 43:19Speaker 00

  Yeah, I think it's like if you're, you know, so like writing is also a way of thinking, right? And so like, maybe if you need to think about the data stream or data readout more, then maybe you need to like summarize it, you know, and so. But yeah, I think like especially gathering context is one of those things where no one will ever like hold it against you kind of, right? Like, oh, like, you know, you did a bunch of research and, you know, Claude did this. But yeah, like, I don't want to ask my agent to do the same research. Like, you know, it's a shortcut, but like just acknowledging it is good.

• 43:51Ryan Peterman?

  Someone else I was talking to, they had this thought of it would be valuable to have almost like a get blame, but it's like, like a prompt blame of, because it would be nice to reverse look up what was the prompt that generated this change to kind of debug things. Do you have any sort of meta version control on the prompts that generated the software? Or is it still very vanilla, you know, Git history?

• 44:20Speaker 00

  Yeah, that is a little bit tough because again, like, you know, what goes into a prompt is not just the prompt, but also the context and skills and things like that. So like, maybe, you know, a prompt might seem basic, but isn't. I think it's kind of hard to judge, but I do, you know, like going back to like, when I'm giving a PR, I don't think a PR at this point is any different than an artifact or something. Like Claude is doing basically all the code writing, right? So if I send someone a PR, I usually also attach an artifact of every prompt I sent to Claude, including failed like approaches and things like that, you know? So that they can see like, okay, I've considered a lot of other things, you know?

• 45:04Speaker 00

  And if I haven't, if this is just a one-shot, I just tell people, I'm like, hey, this is the one-shot example. This is the prompt I used, right? And I think that's like usually impressive and interesting to them as well. Because they're like, oh, like it's cool that Claude could one-shot this. But the worst is when you get like, you know, you just, I'm just trying to avoid cases where I send in like this 10,000 line PR and they're like, did you like, how much have you like read this or like, you know, like, how much do you work with Claude on it? And if I'm doing like a large PR, I am going to like show my work as much as possible.

• 45:42Ryan Peterman?

  On maintaining code, because AI can generate such high volumes of code at this point. Do you have any tips on what's worked well at Anthropic for code ownership and maintenance?

• 45:55Speaker 00

  I do think you have to revisit, like, what is important with code maintenance, right? And so I think that like, there are some things where like naming used to be really important, right? Because like, it was like how you as a team thought about this, like abstraction and feature. But I think naming is becoming less and less important. A lot of stylistic things in code are becoming less and less important, you know? And I think that like, this is not the same to me as maintenance, right? So I think that like, if you, you might want to sit down and be like, okay, like what things really matter now, what don't, what opinions do we have that don't matter? That's one. And then I think on the second, on the maintenance side is sort of like having good scaffolding, right?

• 46:40Speaker 00

  So I think that it's like, having a good verification harness, having a good like, sort of skills. We use the simplify skill a lot. I think that like, you know, sometimes even from a model perspective, like it might do a lot of work. And then like even just giving it permission to be like, hey, I think this is the right idea. Let's simplify it, gives it that permission to do it, right? But like, you actually don't want a model to by default do work and then simplify, right? Because maybe it's not correct, right? Like you don't want it to simplify work that's not correct. It's like, you're wasting tokens, right? And so I think this is also how humans think, right? They're like, you know, you think generatively, you try and approach and then maybe you like simplify and abstract a little.

• 47:26Speaker 00

  So I think there's like some work inside your own codebase or like setting up those skills and that for those practices where you're like, you know, you're not just submitting like the first take, but you've like simplified it and tried it.

• 47:40Speaker 00

  And then like, yeah, having a really good verification harness where you feel like you're catching, you know, like you have a good belief that Claude is like testing every part of it. Like, you know, when you submit a PR to Claude code, you get a recording back of it using the feature and testing it as an example, right? And you can just get really, really creative with different ways to like test stuff. Like, I think you should basically have on the order of, I'd say more like 100 times more testing code than you've ever had before, you know what I mean? So like, you should have fixtures for everything. You can just pull a production code and create fixtures and mockups on the fly for databases. You can have storybooks for front end and things like that.

• 48:25Speaker 00

  And you can have all these different ways of testing and verifying your code. And that I think is really valuable for maintainability, right? It's like, just having like all these ways of verifying it. And then, yeah, of course, like there's just like the human element of like, where do you want your codebase to go? If you know that's the case, probably you start thinking about like, you know, how you'd replay and undo and redo becomes really important, right? Like, just like that sort of stuff. And so the, but if it's single player only, that actually matters a little bit less. You know, like you, like the average user is not going to undo 100 times or something, right? And so like, you know, many like single player applications undo stop working

• 49:10Speaker 00

  pretty quickly, like, you know, after like four or five times, right? Just because it's not that important, but in multiplayer, the ability to compose different things together is really important. Or compose different operations together is really important. And so that's just an example where like, if you know the direction of your codebase, there are things you care about and you want the models to know. And, you know, you can include that in your skills. You can also just like, you know, like, that's sort of what maintainability means to me is like having a vision for what your codebase is good at, where you're, where you're going, you know, keeping it in line. I do think the models are getting better and better and better. And like, almost every codebase probably I think will have this moment where you're like, do you just ask the model to rewrite all of it, you know?

• 49:53Speaker 00

  And like, because now you're like, oh, like I can do it in the most performant language. I can mix and match, like, you know, like there's not a reason that every software in the world shouldn't run in WebAssembly in the browser. You know what I mean? There's not a reason why, I don't know, like your Xbox game can't run in your browser, actually, right? Like the Xbox, like the hardware, like the consoles are much weaker than your average like MacBook right now. You know what I mean? But it's just like the codebase, right? But we could like, and so I think probably over the next year or two, like everyone's going to need to think about that, right? And so I don't think you want to spend too much time

• 50:35Speaker 00

  like worrying about maintainability in this like way that might not matter anymore. I'm not saying maintainability doesn't matter. You just have to update your mental model of like, what does it mean for like code to be maintainable, you know?

• 50:47Ryan Peterman?

  I was talking to this friend who was saying that he leaves tons and tons of tech debt leaving around because the next, you know, the next why, why waste time fixing that now in the next iteration of Fable is going to one-shot all this tech debt and refactor this for me? So kind of funny.

• 51:08Speaker 00

  Yeah, I don't think that's completely incorrect. It depends on the circumstance, right? Like, I think this is also just a classic thing in startups, right? Where you're like, you know, even in normal human engineering, you always have this problem of like, hey, do I refactor, do I do tech debt, or do I deliver more customer value? Would refactoring help me deliver more customer value right now? I do think that like, just generally, I think you should think on your projects on shorter time scales. So you're like, okay, how do I deliver value over the next month or two? And if the project you're talking about is delivering value over six months or 12 months, you know? Then maybe, yeah, maybe like wait for the next model a little bit, you know? And then like that, like.

• 51:51Speaker 00

  Just deliver value on like the short time scale, make sure it's like, you know, really good. And if the models are not quite good enough, like they might get there soon. It depends very much on the like specific case, right? But and I'm not saying this is true of everything, but like just something you should keep in mind.

• 52:08Ryan Peterman?

  I've talked to some friends who work at big tech companies like Google, Facebook, those types of places. And then they, as they've become more and more AI-pilled, one thing that people have noticed is there's a lot more incidents or SEVs in their usage. And it's natural in those organizations because they they read the code less and there's more code flying out. You know, what countermeasures have worked really well for Anthropic to prevent breakages, given that the code velocity is so much higher?

• 52:41Speaker 00

  Yeah, I think this is something that like is a byproduct of moving faster sometimes and we have to figure it out. Like, I think that, uh, you know, I don't think our uptime is exactly where we want it to be either. But also, as a company, we're a little, like almost six years old, I think, around, right? So it's like, no company has grown this fast before. And a lot of that is because we've been able to like create more products faster than ever before, right? And so you can use Claude to make your uptime better, right? And I think that like the way we think about this is just like really good. What's the dream testing environment, the dream like, you know, deployment environment? Like, can you like, you know, take requests and replay them across like, you know, mock databases and fixtures across everything?

• 53:26Speaker 00

  Can you chaos monkey everything, you know?

• 53:30Ryan Peterman?

  In my personal workflows, I have so many more custom, random tools and scripts that just make everything faster. And just curious, you know, on your team at Anthropic, for instance, does everyone have a set of miscellaneous tools that help them, you know, get random things done?

• 53:52Speaker 00

  Yeah, everyone does for sure. Like, I think part of this is the job. Like, I think that sometimes, uh, what we try and do is we play around with harnesses. Like, sometimes some people on the team build their own harnesses for a little bit to figure out like, oh, is this like useful or not? You know, and then they they figure out if it works and if not, they like integrate it, right? So I think there is like, that's part of the job, uh, in some ways. Um, I think like other examples of like, sort of like Misk stuff, people will do. But a lot of people have like, uh, unique Claude tag setups. I think Claude tag is one of these things where like, you know, you can have it like, I have it scheduled my calendar advice, right?

• 54:32Speaker 00

  And so like, if someone wants to like schedule something with me, that I'm just like, hey, you can just like tag Claude here in this channel and I'll accept whatever it like puts on my calendar, you know what I mean? So, um, I think there's like some stuff like that. I know like a lot of people use it for like email. I think that, um, I've seen like some people do like interesting like multi-clawding, sort of like TMUxing like setups, right? Like, what's the ideal scenario for you to like, display like 50 different Claude codes? And, you know, what's the best way for you to figure out what's going on at any one time? So, um, yeah, I think there's a lot of different ways and we're trying to make Claude code more hackable as well.

• 55:14Speaker 00

  So that like, you know, more people can, uh, sort of like even, uh, make their own version of Claude code, like more different. And everyone might have their own little twist on it.

• 55:27Ryan Peterman?

  Your role at Anthropic is really interesting because of the external visibility that you have. And, um, I think a lot of people, when they give career advice, visibility is a good thing, but they don't have this level of external visibility. And so do you recommend to software engineers, like they should be posting on Twitter and X? And, you know, so what what advice would you give in that sense?

• 55:52Speaker 00

  Generally, the thing I say to people is that you should share your work externally as much as you can, especially, um, I think within certain companies, like you might not be able to, but like maybe you have side projects or something like that. I think just, um, before I joined Anthropic, what I did was like, I spent a bunch of time working with different companies, building stuff and writing about it and talking about it. Um, and, uh, like, this was really valuable because it like increased my surface area of luck, you know? And so I think that like, uh, it's really, like, the bar is much lower than you think. Like, basically, whenever someone asks me for advice, I'm like, okay, I'll sit them down.

• 56:36Speaker 00

  I'll be like, I know statistically, I tell this advice to a lot of people and almost no one does it. Um, and everyone who's done it is like either in a job that they're pretty excited about or running their own company. Um, and there are reasons why you're not going to want to do it. But like, this is, I just can tell you, one, like, I don't care about your resume, like, like, you know, don't do that. Um, you have to choose like an interesting project to work on. You have to work really hard on it. You have to like lock in and then you have to ship it and write about it. Like, you have to do it. You can't like, there's going to be so many reasons why you don't want to. It's like not good enough yet or like you haven't like, you think the write-up is not very interesting or something, but you have to do it. And you can't get discouraged, you know, like maybe the first one might not work, but you have to do it again.

• 57:22Speaker 00

  And maybe Twitter is not the right place. Maybe it's Reddit, uh, maybe there's a specific Reddit or Hacker News or something like that. Part of what you're trying to do is you're just trying to find people who like, like what you're doing, right? But I think that like, people in general want really, like, you know, we consume more content than ever, right? And you know this, right? Like, I think that like, um, people want really high quality content. High quality content, as you know, is a lot of work. And so I think going back to like what we said before about like, would you show someone the prompt you did, right? Like, I think a lot of times people are like, oh, what's the shortcut? Like, oh, like, should I just ask Claude to manage my Twitter account? And is that how I like grow? And I'm like, no, like, like, you should not do that, right?

• 58:06Speaker 00

  Like, you have to sort of engage authentically. Um, you have to like post, like, you know, do good work and talk about it. And I think like build up networks and things like that. But I think that as long as that's one of your goals and you try hard at it, I haven't seen anyone not succeed at it. But it is really hard. It's kind of like saying like, oh, I want to like go to the gym every day or I want to like lose weight or something. You know, like there are simple things that you can do that take a lot of discipline and are easy to get discouraged with and, um, but like worth doing if you do it. And I think like posting or more specifically, like writing about your work is like, I think, really, really valuable.

• 58:45Ryan Peterman?

  You mentioned luck surface area. Do you have an example that kind of illustrates the value of expanding your luck surface area?

• 58:55Speaker 00

  Yeah, I mean, okay, how did I get my job at Anthropic? I did, uh, a fellowship with a company called Goodfire, where I did like some applied research. And so they're an interpretability AI company. And I was trying to figure out like, how do I use interpretability in like to make better products? And so I learned about their research, um, and I, uh, like felt, I want to work on a project. And it was really important for me to like share it. And so like I, when I agreed to work with them, I was like, that's my goal. I want to like share what I'm building. I think that's good for you as well because Goodfire is a startup and, you know, people, I want, they want people to learn about them. And so this is something I'm going to do.

• 59:38Speaker 00

  So I built this like interpretability visualization and I shared about it. This was my first like big post on Twitter, but it was only 500 likes or something like, like now, like, you know, that's like not very much to me. But like, it's like back then it was like a huge post and, um, several people saw it and like some of them DM'd me about it. And then, uh, like I then got an intro into, you know, like, uh, into a role here. And so, um, I think that like, I spent like a month on that project, I think in particular. So it wasn't like, you know, crazy. Um, and, uh, yeah, it just like showed people that I could like do interesting work.

• 1:00:21Speaker 00

  And, um, and I got paid for it too, so it wasn't like I was doing it for free.

• 1:00:26Speaker 00

  And there's plenty of examples of like, I think people are happy to do that sort of thing for you. You know, like if you like sort of show that kind of initiative. But yeah, you have to like, it is like you really have to like do interesting and novel work and work, work hard to be proud of your work as well. You know, there's not like a shortcut to that, I think. But if you do, I think everyone is always looking for interesting work and wants to support you and, you know, wants to hire you or yeah, give you money, like lots of good things.

• 1:01:01Ryan Peterman?

  I see this tagline going out a lot with some of the stuff that Anthropic's been. Maybe like, I think it was like Boris went on some podcasts and the tagline was coding is largely solved. Should people still learn to code if coding is largely solved?

• 1:01:18Speaker 00

  I think being technical is really, really important, knowing how do computers work? How does like, yeah, how does, how do computer programs work? How do languages work? Like, what are the hard things? And like, what's a backend service? Like, what's a cache? Like, what's, what like, like, you know, what, like, what is memory allocation? Like, all of these things are actually kind of really important to learn. I do think it's hard to motivate yourself sometimes to do, in the same way that like, you know, doing math by hand was not that motivating to me, you know, but some people just love math and did it. I, I think that like, that's probably something that people have to figure out. I think it is really worth it, like being technical is really, really important.

• 1:02:01Speaker 00

  Like we talked about at the start of, or like earlier, we're like, oh, the only way you can tell you solved, you know, the Riemann hypothesis or like made progress or whatever is like, or the Jacobian conjecture is like, if you're a great mathematician, right? And in the same way, the only way you can tell if you're like built great software is like if you're a great software engineer. I think that like, how do you do that is hard, but like we've talked about some of this stuff before, just like staying in the loop. Like, you know, like put like, like putting, like reflecting on your process and, and getting better. And you can use Claude to learn as well and, and sort of explain things to you. So like treating it like a thought partner.

• 1:02:43Speaker 00

  But learning like truly, I, I think like, you know, Karpety says like, learning should feel like effort, you know? And I think that's like, one of the hard things is like, a lot of times, even if you ask Claude to explain something to you, you might just like nod along and you're like, oh yeah, like I learned it, but you didn't really, because you didn't put any effort in, right? So I think that like, it is really technical, you should learn, it's hard to learn. And then, and sometimes like what school forces you to do is to learn that. I don't know truly, like I'm not learning programming from scratch, so I don't know exactly what, how to do it now. I think if short of better ways, I would still like type out and build programs and run them and learn them. You know what I mean? There might be better ways that are more Claude informed as well, but I think it's really important.

• 1:03:29Speaker 00

  I think when Boris says coding is solved, I think it just means like,

• 1:03:35Speaker 00

  you know, we don't get stuck in the same ways that we used to before. Like I think coding used to be this very high variability thing where you're like, oh, like, could this bug take a day or could it take two weeks? You have no idea sometimes, you know, and I think like,

• 1:03:49Speaker 00

  on the whole, coding used to be like in real terms, like something that was very rare for something to go well. Do you know what I mean? Like very few people in the entire world could write software and they were very, very rare. And even if you got them all together, there were so many other reasons why it wouldn't work, right? And coding was this like, one of the rarest things in the world where like the chance of software going pro- project going well was like, on absolute terms, very low, you know? And if you were like hiring someone to like make software for you for something that's not like, a huge product, like if you're hiring someone to make software for your car dealership, you were almost certainly not going to get the software you wanted.

• 1:04:34Speaker 00

  You're going to get like essentially scammed, you know what I mean? Not because anyone was trying to scam you, just like software is really, really hard. And you know, it, it could only be spent on like the most important scalable things in the world. And now that coding is solved, I think what we mean is that like you can use coding to do all these other things that we've not done before. But that's not to say that like, that's not a lot of work still. It's just like, it's not this like incredibly rare, difficult thing that mostly like just doesn't work. And you have to spend like eight hours a day locked in to do.

• 1:05:10Ryan Peterman?

  Well, it's really insane, that shows the difference in expectation is when I used to write software, I would be shocked if it worked on the first try. I was like, whoa, wait, why is this working? And you, you expect to kind of bash your head against the wall a little bit and then it, it works. Even if it's just like, you missed the semicolon or something like that. And now I almost have the flip expectation where when it doesn't work, I'm like, wait, what, why, why did Claude? What, what happened here? Usually I expect it to work almost the opposite, like on the first try, it's crazy.

• 1:05:45Speaker 00

  Immediately, yeah, yeah. I think humans got really used to abundance, right? Like there was this like article someone shared recently about like going through a modern apartment and talking about all the like wonderful things that we have now that people could not have imagined before. Like something that can play music on demand that's suited for your mood. You know, like before you'd have to like hire a musician, you know, to like go write, like, you know, incredible music, right?

• 1:06:13Speaker 00

  But I'd say on the whole, probably more people are getting paid to make music now than ever before, right? Like, and I think that like music is reaching more people than ever before. And I think probably the same thing will happen with software, will happen with math, but will happen with all of these things. Where, you know, when you get abundance, people are like, great, like I want more abundance. I want my software and everything, you know, like I want my music everywhere, you know, so, yeah.

• 1:06:39Ryan Peterman?

  Yeah, one potential other data point from this conversation, saying that maybe people should still be technical or learn how to code, is I think earlier in the conversation, we talked about automating knowledge work. And you mentioned that people who are technical had kind of a leg up because they could kind of understood how to coordinate Claude to do a variety thing. Like you're calling FFmpeg to automate some video editing and like, I don't think someone who is not technical would have that thought. So it does seem like even in this case where models are doing a lot, there's still so much value in being technical.

• 1:07:16Speaker 00

  Knowing how computers work, you know, in the same way that like, like probably, like the most technical CEOs are like, the best CEOs are technical, right? Like Zuckerberg, Elon, right? But like, you probably haven't written like a line of code truly in a long time. But you know, they understand how systems work, they understand, you know, constraints and things like that. And that, and that's really, really important. And so, you know, even if all that work is changed, I think like being technical is really, really important.

• 1:07:45Ryan Peterman?

  And then last question for you is if you could go back to when you just entered the industry and give yourself some advice, knowing what you know now, what would you say?

• 1:07:55Speaker 00

  They're like, kind of like two wolves, sort of. I, I think like you have to believe in yourself, you know, I think like this is really important. And I think that like, at least when I joined in the industry, it was rarer to believe in people when they were kind of younger. Like I think that was like the whole point of Y Combinator or something was like believing in young people early on to do really great things. And so I think that like, you know, even going back to like the intern discussion we had earlier, right? Like I think that like, thinking of yourself not as like someone who's like being trained to do something, but someone who can do like something incredible right away, I think is really important, you know?

• 1:08:32Speaker 00

  And I think I wish I had done bigger, bolder things that I'd written and shared about. I think there were lots of ideas where I was like, wow, like I think, I think I had like, I thought I had done original work that I wish, maybe I'd even shared in some form, but maybe not like form that like survived the internet, you know? And it wasn't like a, like goal of mine. And so I wish I'd like sort of been bolder and like, you know, done and shared some of this work. But at the same time, you also, there is a lot to learn from people, you know what I mean? And like, I think that you, like, I think when you're young, you're like, you know,

• 1:09:15Speaker 00

  you tend to fold, fall, either you're not bold enough or you're, you're not like,

• 1:09:22Speaker 00

  you don't like learn enough, you know, or you're like, you're not like, you don't like, you're too bold and you don't like figure out what people have done before and figure out why it's not working, right? And there's this balance and I think you, everyone has different failure modes. But I think like, I think I probably didn't take enough advantage of like, you know, mentors and people who had learned a lot. And I think I probably had to like relearn a bunch of things as a result. So there's probably not a universal good advice, but just things to think about as you're, you know, as you're entering.

• 1:09:56Ryan Peterman?

  Awesome. Well, thanks so much for your time, Derek. I really appreciate it.

• 1:09:59Speaker 00

  Yeah, of course. Thanks, Ryan. It's fun.

• 1:10:01Ryan Peterman?

  Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskov, Mike Stonebreaker, Mark Booker, these were all people that I brought on because someone left a comment. On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype, it's a split keyboard, so there's two sides. This is in the case, but yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of the early units. We're now working on the long journey of building the tooling now.

• 1:10:45Ryan Peterman?

  And so if you still want to pick one up, I've left the late pledges open on Kickstarter. So you can grab one there, I'll put a link in the description. Thank you again for watching the podcast and I'll see you in the next episode.