944: Is Coinbase Really Writing Half Their Code With AI?

Wes and Scott talk with Kyle Cesmat about how Coinbase is writing nearly half its code with AI—while keeping quality and security front and center. They dig into tools like Cursor and Claude Code, agent-driven workflows, code review challenges, and how AI is reshaping developer productivity without replacing developers. Show Notes 00:00 Welcome to Syntax! 03:29 Defining and measuring “quality” at a large company 05:05 How AI-generated code is reviewed and shipped at Coinbase 07:35 A developer’s workflow using AI—from ticket to production 10:30 Standard vs. team-specific tools for AI development 12:55 Using MCP servers to power internal AI workflows 14:42 MCP vs. custom agent integrations 17:16 What kinds of code AI is writing at Coinbase 19:48 Which languages and tasks does AI handle best? 21:21 Getting AI to follow existing code conventions greptile 24:36 Brought to you by Sentry.io 25:01 How AI affects hiring, velocity, and developer focus 28:16 AI’s role in speeding up code reviews 31:28 The most valuable AI tool in Coinbase’s stack 33:48 Managing AI costs and model choices 35:10 Security considerations for using external AI models 37:04 How often Coinbase tunes and adjusts its AI systems 39:26 Using AI to write code vs. embedding AI in customer-facing features 42:29 Kyle’s big-picture take on AI as a tool—not a magic bullet Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity 44:33 The future of hiring engineers with their own “backpack” of agents 45:53 Sick Picks + Shameless Plugs Sick Picks Kyle: UltraShelf Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Wes Bos?

  Welcome to Syntax. We got a really interesting one for you today. A couple of weeks ago Coinbase came out and said that they are writing 40% of their code with AI. And I thought that was really interesting because you hear these stories of like, yeah, you have these like brain dead vibe coders pooping out an app in an afternoon. And then you hear stories like, like Google says like 80% of their code or something. And I talked to people at Google and they say, yeah, it's, we're just converting Angular apps to like a more modern version or something like that, you know? So it's, I am always so curious when companies come out and say, "We are writing X% of our code with AI." I'm so curious. I just want to be like, "But what are you using? Like what tools are you using? What are you using it for? What kind of code are you writing?" You know? I want to get into the nitty gritty.

• 0:45Wes Bos?

  So Coinbase, not a small company, rather a large company, you know? Lots at stake, lots of financial, has come out and said that they're writing 40%, hoping to get to 50% of their code with AI. And I immediately was like, "We gotta get somebody on." And it turns out, I, do you remember Kyle? We went out for beers many years ago in Toronto?

• 1:07Speaker 01

  I think it was Toronto JS. It was quite a while ago. Yeah.

• 1:11Wes Bos?

  Yeah. So I went to message you and I was like, "Oh, I think I know this guy already." So welcome Kyle. Thanks so much for coming on.

• 1:19Speaker 01

  Oh, absolutely. Yeah. Thanks for having me. And yeah, I, you know, remember that time fondly. And then ever since then, I think I follow you on Instagram for lately home improvement tips and, you know, general DIY inspiration. So yeah, definitely have, you know, moved through the sphere of influence in my personal feed.

• 1:41Wes Bos?

  Oh, that's great. But was it you that told me the story about Walmart and toilet paper?

• 1:47Speaker 01

  It was. Yeah.

• 1:48Wes Bos?

  Yeah. So this guy is responsible for the spooky story episodes that we do every single October, which is, we tell stories, horror stories and whatnot. And, and he told me this story. I'm not sure if we can get into it, but I appreciate you telling us that because it jogged my memory. And I was like, "We gotta do Syntax on this. It's really good."

• 2:12Speaker 01

  Yeah. Yeah. A good episode about testing in production would be a good theme there.

• 2:20Scott Tolinski?

  Cool. Well, who are you and what do you do at Coinbase?

• 2:24Speaker 01

  So my name is Kyle. I've been at Coinbase for about four or five years now. And currently I'm leading a developer experience team that is so, I guess, task focused on this sphere of AI enablement. So how do we enable engineers across Coinbase to use AI, to leverage AI for their coding tasks, for their developer productivity tasks, kind of all along the sphere of the work before the work, the work during the work, and then ensuring quality all throughout. So the time at Coinbase has been filled with different experiences as the company has changed

• 3:05Speaker 01

  in size and in shape. And currently, quality is one of our leading initiatives that we've been so focused on. So as we're exploring AI, we're doing so with very strong guardrails for quality. And so, yeah, I'm really excited to talk about, I think, how we're approaching AI enablement, as well as safety, and how that ultimately impacts the developer productivity down the road.

• 3:29Scott Tolinski?

  Yeah. When you say quality, could you give the audience an idea of like, what is that? Like, how do you even quantify quality?

• 3:35Speaker 01

  Yeah. So at a large company, quality has many different shapes and sizes. We are always laser focused on our customers, so trying to measure from the customer perspective, what quality looks like. And obviously, as the company grows, you can feel further away from the customer, you can have a lot of noise throughout the system. So there's obviously some high level metrics around incident severity and kind of outer loop impacts. But then on the inside impacts, or excuse me, the inner loop impacts, we're looking at how developers are, I guess, able to test, able to kind of go through their checklists of what is the bar that we're holding. And we have service scorecards.

• 4:15Speaker 01

  We have all sorts of, I guess, internal measures that we get to developers to kind of, you know, stay at a certain high bar. But what quality really boils down to is can a developer use a tool friction? Painlessly, like, we all love our tools and ultimately, I think that's where the quality starts is if you're using a quality tool, you're going to have quality output. That's kind of my my thesis. And so as as we're kind of, you know, enabling AI and enabling other kind of developer productivity wins along the way. We've done a lot of work around, you know, our internal build systems and how we are building apps at scale. We want, we want the tools to be really high quality. And then ultimately, we want really fast iteration speeds.

• 4:57Speaker 01

  We want quick deployments, all the observability inputs that that we kind of need in order to gauge the success of our systems.

• 5:05Wes Bos?

  Okay, so let's, probably people are dying to hear, let's dig into this is like, you say 40% of code is is written AI. So your, are your developers just boop, boop, boop in the box, pooping out a whole bunch of code? And then ship it, you know, like it's probably not, given that like the the stakes of what Coinbase is, right?

• 5:28Speaker 01

  Yeah, when we're talking about lines of code, specifically in that metric that I think everyone's pretty aware of. We're talking about lines added or deleted. So one of the first, you know, pieces of critique that that we kind of get on this is they shouldn't just be adding code to a code base. As you add more code, complexity increases, quality usually diminishes. These are all pretty correlated metrics. So, you know, lines added, deleted. As developers are, you know, using their AI tools. Look, like I kind of want to, you know, move back and talk about the tenants of AI for developers. The key tenants that I kind of have in my mind is AI should always be attributed.

• 6:08Speaker 01

  So we are, we are essentially asking these systems to produce code on developers' behalf. So developers are still taking responsibility for the code that is produced. So strong attribution is kind of a key tenant. And, and, you know, along those lines, AI should never be making decisions. AI should never be kind of the final piece between a feature and the customer. So everything that I guess these lines of code are producing is merely a suggestion. And this is, you know, it falls into the review steps that we all know as software engineers, we're able to interact with every day. So code is produced, it goes into review.

• 6:51Speaker 01

  And this is actually where I think one of the key problems around AI and velocity is going to emerge down the future is code review. You need really strong code review these days when it comes to flagging potential risks, having good quality rules, you know, through your kind of like lint type, all, you know, the, the kind of quality build system.

• 7:12Wes Bos?

  Yeah.

• 7:12Speaker 01

  And then ultimately, having, having the right approvers to say, yes, this is the right code, we're going to ship it to production and then, you know, lean on all the strong parts of, of the deployment flow. So I, I wouldn't say that the code that we're writing is going straight to production. We have a lot of really high quality guardrails intact that, you know, ensure that the right feature is landing in front of customers.

• 7:35Wes Bos?

  What does that actually look like? So let's say a dev in your company gets a ticket to add a feature to, to your dashboard. What does the like, like start to end look like for a developer that is writing the code, maybe, maybe even the PM before that of figuring out what to do, all the way through to like, who's code reviewing it, testing it, security, et cetera?

• 8:00Speaker 01

  The different ways that AI is used, I think, indicates how comfortable developers are with AI. So, so we kind of introduced tools along the, along the way. So you kind of talk about from, you know, ticket to, to code. We in some cases have product managers that are, you know, creating really robust tickets and perhaps they're leveraging AI to do that. So they're, you know, culminating a bunch of different docs together, perhaps using some sort of inputs to create a really high quality ticket. And in some cases, we have workflows where that ticket can be worked on by Claude, can be worked on by an agent, essentially picked up right away and that agent will work in the background. Push a branch to GitHub, at which point it lands in kind of the traditional review

• 8:43Speaker 01

  flow. I'd say that's the future that we're aiming for is this kind of, it'd be great if 60% of our work ticket wise could be kind of just done by a background agent. And where the attribution piece comes in here is, you know, there's an engineer that's kind of responsible for maintaining these background agents for their PMs. There's PMs who, you know, feel compelled enough to just author code. And this has always been allowed in systems is if you're a product manager and you want GitHub access, you want to end up in the review flow and kind of triage that all the way through, we let that happen. So this is where the lines are blurry, but they've also always existed. And so

• 9:24Speaker 01

  that's probably the case that we're aiming for. I'd say the day to day piece where developers are using Cursor, using Claude Code, using any of our kind of other agents that we allow within Coinbase, they might be using it in tab completion mode, which is probably the most fine grained control of the code that you're writing. And then there's agent mode, obviously, that is kind of fully agentic. And there's different levels of enablement there. We can either, you know, turn on Yolo mode and let the agent kind of develop however it wants. But yeah, I'm going to kind of, you know, go back to my original point. All

• 10:06Speaker 01

  of this code gets to this review step. And at that point, it's kind of the same step throughout of strong developer review, strong, strong testing, quality bars. So how the code gets there, I think is, you know, enabled in different ways. But ultimately, there's the traditional steps that the code has to fall in before it gets to production.

• 10:30Scott Tolinski?

  Yeah. I'm curious, I mean, you mentioned a lot of different types of tools here. Is everybody on the same set of tools? And are those mandated from up high in terms of like, you're using Cursor, you're using VS Code, you're using Claude Code, etc.?

• 10:49Speaker 01

  Yeah, so Cursor is a really high-quality tool. We were early to use Cursor, middle of last year. And it's just so powerful because what I talked about earlier, where it's blurring the lines for what is an engineer. We have product managers, we have data science. We have all sorts of users throughout the company who are going to Cursor as kind of their first place to say, "Hey, I have some code in a GitHub repository, and I want to chat with it. I don't really know exactly what I'm doing, but I just want to do some data visualization, get some understanding. I just want to learn a little bit more." So Cursor is kind of the all-integrated solution that I think just about 100% of our engineering

• 11:31Speaker 01

  and plus a good handful of our PMs and other job functions are using. Claude Code, we have some custom tooling that enables Claude Code. We have other tools that we're kind of exploring internally, such as Goose, Codex, other kind of open-source alternatives. And really, what we don't want to happen is for us to go too deep on a tool. And then, we all know how the AI story is unraveling. There's a new tool tomorrow.

• 12:02Wes Bos?

  Right, yeah.

• 12:02Speaker 01

  So how do we stay on top of that? We kind of have these good primitives that let us pull in the models with the right sorts of context. All developers are encouraged to use these. I wouldn't say that there's strong mandatory enablement, but yeah, it's definitely, I think everyone wants to use them. And our MCP culture kind of backs it up, too. We have so many MCPs that teams have kind of created these custom agents with. So they might be running an agent inside of a Docker environment that calls different MCPs. And those are now passed around from team to team, saying, "Oh, I automated this really unique on-call situation."

• 12:46Speaker 01

  And now that's just kind of exploding as this kind of sourdough starter bespoke configuration.

• 12:55Wes Bos?

  That's really interesting to me because I'm always curious about like, how do large companies who have probably a lot of domain-specific stuff inside of their company that is only for their company? I'm like, how do you make AI work for that? So you're saying like, you're making all these MCP servers that help? Do you have any examples of stuff you could share?

• 13:18Speaker 01

  Yeah, so we have probably 30 or 40 developer-facing MCPs today from all the tools that we use.

• 13:26Scott Tolinski?

  Cool.

• 13:27Speaker 01

  You know, things like Datadog, PagerDuty, all of the kind of internal suite of tools that we use. And it's wild, just the amount of creativity that we're seeing. And our kind of approach is enablement. It's not prescriptive, you have to use this tool in a specific way. So we have all these, you know, agents that I was mentioning, but then we also have like LangChain, LangGraph internally, that teams are starting to really write super robust, fully agentic flows. You know, this is, this is kind of the catalyst effect that's happening. As we, every time we add a new, a new MCP integration, someone runs to that and says, I've been waiting for this. It unlocks this, this bespoke flow that I've been doing.

• 14:09Speaker 01

  And yeah, the, you know, excitement around these like tools is really, really new. I don't think we've ever had so many developers clamoring for, you know, Python in production. In production in the sense of producing a production internal instance. But yeah, we've never had so many devs that are just running towards new domains because they're saying, "Cool, I can work with LangChain, LangGraph because I use Cursor, because I can now kind of, you know, iterate in a new language that I wasn't fully comfortable with before."

• 14:42Scott Tolinski?

  Yeah, that's amazing. I, I'm curious on your take on this. I've noticed that when I'm switching tools, they interact with agents in a different way, and I'm having to translate it across tools. Do you find that MCP is a more solid surface to be able to keep those tools in, rather than using and adapting custom agents for whatever tool you're using at any given day?

• 15:08Speaker 01

  MCP is not a perfect, it's not a perfect abstraction. I think it's a, and it's such an interesting world where I think you've seen even so many vendors saying, we have an MCP available now. And it's kind of this overused term. It's like, it's just a protocol. It's just a way that you're interacting with a service remotely in a very generalized way. But then, yeah, you see agents like Claude Code that natively call GitHub, for example. And that is a much tighter way for agents to work is just to call an API directly rather than proxy through an MCP that it has to kind of figure out what to do with. So I think as developers are kind of transferring through tools, this is where there might be

• 15:51Speaker 01

  very specialized use cases where it makes sense to go deep on a tool integration and use the native API instead of an MCP. But broadly, I think a lot of our tasks are generic enough to where MCPs are kind of, letting us prototype really fast, letting us use something instantly in a really easy format. But if something gains more adoption, if say we produce an agent that does really unique bug triaging, for example, we might move away from an MCP and move towards a direct API call just to get that even finer level of control. It's kind of the one-two automate philosophy that we've used at Coinbase.

• 16:34Speaker 01

  So we're going to prototype everywhere. Cool, someone else likes it. And now we're going to make it into a production thing. And that's, that's in general, I think how DevX, my team kind of works is we see what this team is doing in one part of the company that might be a more ambitious part of the company or less regulated part of the company. Man, it's just exploding over there. It's doing great. All these hundreds of developers are now enabled. How can we bring that to other parts of the company to fit their kind of compliance profile in a very safe way? And so, yeah, trying to up-level the, I guess, innovation to other parts of the company is kind of a key part of working at this company at scale.

• 17:16Wes Bos?

  And what kind of code are you primarily writing with AI? Like, is this React components and CSS? Or is this like backend code? Like, I don't even know. What is the stack of Coinbase?

• 17:28Speaker 01

  Yeah, so we're primarily, you know, React TypeScript in the front end and very heavily Go on the backend. Traditionally, we were kind of a Ruby shop and have grown into a Go company over time. There's been a huge push early on to grow test adoption. And this is, this is probably where a giant bulk of our AI code was, was written early on was, I want to use Cursor to go and bring in the test coverages that I need. So we started not mandating, but you know, putting out bars in places saying, okay, if you want to use Cursor, if you want to use all these AI tools, you're going to have to meet these certain criteria of, you know, test, test coverages, deployment frequency, scorecards, et cetera.

• 18:13Speaker 01

  And so teams early on would say, okay, we're below the bar. We're going to write a bunch of tests. And it's one thing to have high test coverage, but it's another thing to, you know, have them be very accurate tests. And we've, we've created certain tools for this with, you know, providing the right sorts of context, mocks, et cetera, to create really good tests. So, so yeah, early on, a lot of test coverage. I think where we're seeing the bulk of the company write AI-generated code today is probably 60% TypeScript. I think that, you know, it's just a much more fungible place where we're able to iterate on pretty quickly.

• 18:54Speaker 01

  And yeah, traditionally isn't quite as sensitive, secure as, you know, some, some of the more backend systems.

• 18:59Wes Bos?

  Yeah. And is that like internal tooling or is that like customer? Like is the, is the Coinbase?

• 19:06Speaker 01

  Definitely both.

• 19:07Wes Bos?

  Dot com. Yeah. Okay.

• 19:09Speaker 01

  Yeah, definitely both. Coinbase.com, it's a pretty highly trafficked repo for AI today.

• 19:13Wes Bos?

  And does it, how much Go does it write? Or you say it's, it's mostly TypeScript, but does it, it still dips into Go as well?

• 19:20Speaker 01

  Oh, yeah, yeah. A lot of our backend engineers are using Cursor. You know, this is, it's a static language and it's actually a pretty simple language. So it's a great use case for AI. Look, again, this, these are core systems and in order for these teams to be using AI, we need to keep the quality bar really high. And, and yeah, there's so many different ways to do that, but we're, we're not just letting people write code and push to production with backend systems by any means.

• 19:48Scott Tolinski?

  Do you find that it, the, your AI tools handle some aspects of the codebase better than others? I know you've mentioned languages specifically, but even as far as languages go, yeah?

• 20:00Speaker 01

  There's different tools that, that have done well. So we've, we've heard solidity devs, some of our kind of like blockchain systems engineers, they were not having great luck with Cursor and kind of however, you know, Cursor is kind of enhanced on their context. So they, they started using Claude Code and whatever Claude Code's training data or sampling is from, it's just a lot higher of a bar for them. It's trained on more solidity data. So there's, there's different tools that I think maps to different codebases pretty well. I think the, you know, very generic parts of developers' jobs of setting up a new, a new route, setting up a new screen, the copy and paste of, of what we were already doing

• 20:45Speaker 01

  from, you know, I, I want to, you know, create a new route essentially. Very good at that. It's very good at pattern matching and pattern learning. More specialized and like new architecture is not necessarily the place that I would leverage AI. I think pushing it into a place where there's not existing parts of the codebase to look at, maybe the, the best practice that it's proposing is not the best practice that, you know, you as an engineer agree with. So letting it write net new is, is not necessarily the best use case for it.

• 21:21Wes Bos?

  Interesting, because we get that question all the time and maybe you could answer this, is someone says, we have an absolutely massive codebase. We have strict ways that things need to be done. If someone comes to you and says like, hey, we have this. Codebase, that's 15 years old. There's ways that we do it, right? We have, we do have some tooling around it. But how do you get AI to write code that, that looks and acts like the code that we have been written? Like, what, what would you tell someone to do? Would they fire up Cursor? What tools do you need to, in order to support that?

• 21:56Speaker 01

  Onboarding new tools to existing systems and kind of downloading all the cultural context that you've created is, is, is an ongoing task. We've had developers have good success with, you know, the Claude MD files or the Cursor rules. Internally, we've created code conventions, kind of AI code conventions that are applied generically to different tools. And, you know, historically, this was, this was something that we've always done. Two, three years ago, we'd have weekly, monthly meetings of, you know, senior staff engineers getting together, saying, what is the best way to do tests? What is the best way to, you know, use state management?

• 22:40Speaker 01

  And we'd kind of put it on, put it on paper, document it, set it in stone and preach that throughout the rest of the company. And now we're able to codify that quite well with, with, with AI. So it takes time and there's some tools out there that are doing it quite well in terms of learning about your code conventions over time. Reptile is one review tool that we've started experimenting with at Coinbase where it actually looks at review comments and it starts to build an internal registry of things that it's reviewed on or things that other developers have called out and reviewed. And so that's where I think, you know, the, the long tail problem is, is it's going to be around capturing context in a really nice way. I think the Claude MD file, I've seen a lot of engineers start with that.

• 23:25Speaker 01

  They say, hey, I got Claude in a codebase, it committed a file. Maybe it gets updated in the future. Doc writing is something that different teams have always had different levels of success with. How can we? Automate that with AI to then turn those into AI rules is a conversation and tooling that we're building. But yeah, for for teams that are just getting started, it's different levels of success with with different AI tools. I think Cursor has its own way of kind of indexing and creating, creating a map of your codebase, whereas Claude Code just uses LS everywhere and it, you know, it kind of manually crawls your your codebase.

• 24:05Speaker 01

  So experiment with the right tool and, you know, for the size and shape of your codebase is my my suggestion.

• 24:13Wes Bos?

  I'm really excited about the new GoBase TypeScript compiler, because I think that, given how fast it is, it's going to make the prompts much easier to to like, kind of spider through the pieces of the codebase that it needs.

• 24:27Scott Tolinski?

  Yeah.

• 24:27Wes Bos?

  So I'm curious to see how that.

• 24:29Speaker 01

  Yeah, integrating that with an agent would be really sweet.

• 24:33Scott Tolinski?

  Yeah.

• 24:33Wes Bos?

  Yeah, I think it's just a matter of time with that.

• 24:37Scott Tolinski?

  And if you want to see all of the errors in your application, you'll want to check out Sentry at sentry.io/syntax. You don't want a production application out there that, well, you have no visibility into in case something is blowing up and you might not even know it. So head on over to sentry.io/syntax. Again, we've been using this tool for a long time and it totally rules. All right.

• 25:01Wes Bos?

  So now that CoinVoice is writing 40, 50% of its code with AI, do you do you just fire half of your staff?

• 25:10Speaker 01

  Definitely not. So.

• 25:13Scott Tolinski?

  Yeah.

• 25:14Speaker 01

  So let's talk about velocity. There's always a, there's always a demand for more velocity. This is, everyone says, we want to, we want to create features faster. We want to iterate faster. We want to be hyper competitive. What we hope to see, and this is, this is where the metrics are very hard today. And yeah, we can talk a little bit about metrics here, where code coverage is probably one of the only finite output metrics that we really have nailed down today. We can say AI code, you know, generated, generated lines of code is going up as AI's usage is going up. On the other side, we have a lot of input metrics. So we have input metrics around what number of AI tokens are engineers

• 25:58Speaker 01

  using. And, you know, we see that. Growing at an adoption rate right now, it'll probably get to a resting rate until some sort of tool comes along that consumes twice the amount of tokens. What we really hope to see is developer velocity increase in a very long tail way. And right now, there's a lot of, there's a lot of issues in that kind of developer velocity story. There's issues around, I as a developer, I'm able to produce three PRs a day. Awesome, but reviewing those three PRs a day is still just as painful as ever. Or testing those is just as painful as ever. Or, you know, I'm staggering out the deploys because my, my system, the way that we, we promote code

• 26:42Speaker 01

  hasn't really been modernized in a while. So there's all these kind of existing pipeline problems that as you, you know, put more code into the wood chipper, you're going to kind of create this backlog downstream. You're going to see these systems not scale well. And, and that's, I think, what we're really hitting internally right now. The other side of it is engineers are able to focus on more long-term work. They're able to, you know, focus on, on things that they weren't able to before. Because I had to, you know, produce a feature by the end of the week, but I also had to go and ensure that, you know, our SLO reporting was right, that our customer complaints were addressed, that kind of, you know, triage of our, of our bug tickets happened correctly. So there's all these kind of like work around the work tasks that

• 27:27Wes Bos?

  Yeah.

• 27:28Speaker 01

  People are now able to spend more and more time doing, which hypothetically should increase the quality over 10.

• 27:35Wes Bos?

  Yeah, yeah. If you're always putting out fires like that and there's no time to, to do the deep work and then the product will suffer.

• 27:43Speaker 01

  Exactly. Yeah, yeah. So yeah, long story short, more code written by AI doesn't necessarily translate to less engineers in the long run. You know, perhaps it does scale it at some point. Perhaps, you know, teams won't be constantly hiring forever. Because I, you know, obviously, organization building, there reaches a certain sweet spot where, yeah, more people just, they don't solve more problems better.

• 28:11Wes Bos?

  Yeah, yeah. And you mentioned some stuff around code review as well. Like, that's a bottleneck right now. And I see it in so many of these open source projects, especially some of the open source projects where it has a very specific skill set. You know, like, I was looking at the OBS Web Socket implementation in C the other day. You know, like, there's one guy that understands how that works. And it's really frustrating, because you see all these pull requests of people being like, "I don't know, C, but I just typed the problem into a box, and maybe this works. Let me know," you know? And now there's this huge backlog of like, we've got one guy that knows what he's doing. So is AI helpful in that code review space as well?

• 28:55Speaker 01

  Absolutely. So, yeah, the example of the one engineer who is responsible for an entire open source library, this is everywhere.

• 29:05Wes Bos?

  Yeah.

• 29:06Speaker 01

  We adopted Expo early on at Coinbase, and I remember there was like, one engineer who maintained a V8 fork for Android. So shout out to Kudo, if you are listening, because, yeah, he's keeping so many codebases alive. But, yeah, the kind of, the code review step is probably more important than ever. This is where there's more demand, and there's probably so much opportunity here, too. And what code review has traditionally been, especially in open source and kind of very async teams, is, okay, on Monday, someone submitted a PR. Awesome. I left some comments. I reviewed it on Wednesday, left a couple more comments.

• 29:47Speaker 01

  We were going to get my version on Thursday, but then CI broke, so we're still triaging it. And that's pain. And so what if we could automate 30 to 40% of that? What if we could say, okay, that first round of comments, the low-hanging fruit, just, you know, have AI kind of run through that with these code conventions that we've built, with this knowledge, you know, graph that we've developed? And, you know, how do we kind of like, take out the initial round of review? Cool. What if we get to a point where CI and integration is a little flaky? Something's breaking in the PR? Why not just have a Claude code and GitHub Actions bot triage on that?

• 30:26Scott Tolinski?

  Mm hmm.

• 30:27Speaker 01

  Say, hey, it looks like a test was changed on master. Here's a, here's a piece of code to fix that. And now your branch is back up to date. So kind of the, the branch health. And, and so if we can, if we can fix some of these just pain problems around code review, as well as scale the idea that everyone's going to receive a really high quality review by default. That's probably one of the biggest opportunities as more code is written and quality bars don't want to be diminished.

• 30:56Scott Tolinski?

  Yeah, man. That it, it's so fascinating to hear you, you talk about these, these systems at a, at such a scale, considering like, it feels like a, just a giant version of the same things that I'm, I'm doing on my own. So like to hear it's so affirming in a way to know that like, these are the systems that work at this type of scale. And it's also fascinating how you guys are essentially reducing latency there. You're reducing that latency to, to iterate, to fix all that stuff. And this is kind of like a, a broad question, but what's the one tool in your arsenal right now that gets like the most bang for buck?

• 31:37Speaker 01

  I mean, so yeah, Claude code is just such an amazing tool. And I'm going to go back to that because it's, it's a great primitive. And, and the way that we've kind of brought it to Coinbase is we have our own internal, you know, list of, of models that we've kind of allowed and, and they're provided through a light LLM proxy. And we're able to, you know, configure that with our tools. And so Claude code is, is, is a, it's, it's very portable. We're able to use it in GitHub Actions, like I mentioned, we're able to use it in containerized environments and scripted environments. And, you know, you're able to provide so many different pieces of context to it. And it's, it's this kind of puzzle piece that just fits in so many different systems.

• 32:19Speaker 01

  You know, we want to build more advanced triaging bots for, hey, our, our sentry instance picked up an error and Claude code picked it up, tried to work on it. And here's a PR for it. I think, I think, you know, those types of systems and interactions have a great future in front of us and reduce the latency like we're talking about. So a new bug is detected. Instantly, we're able to kind of put string together these automations and the Claude code is probably one of the best well-maintained, featureful pieces of software that are out right now. I haven't had the same experience with Codex

• 33:00Speaker 01

  or I guess other offerings on the market. But that said, you know, we kind of need tools that are specialized to models. I think Claude fits that really nicely. And I think perhaps the entire developer ecosystem right now is just banking on Anthropic being a long-term competitive partner here. We hope that, hey, just, just, you know, keep creating a great tool that lets us, you know, write code. But what if we get to a day where that support runs out and we don't have a tool that's built quite as ergonomically around a model? A little bit of a risk. I, you know, we kind of felt that with GPT-5 coming out that, okay, what's going to be the right tool here? Cursor is working on this problem as well. You know, they want to be more than an IDE. They're offering a full suite of

• 33:46Speaker 01

  different integrations with their cursor agent.

• 33:49Wes Bos?

  Man, and what about cost for all of this stuff as well? Like, are you sensitive to how much this costs? Or are you all just on the like, what, 29 bucks a user per month or whatever? Or are you like, straight up API request, spending thousands of dollars on a feature?

• 34:07Speaker 01

  Yeah, so costs are an interesting one. You know, a broad statement right now is relative to the cost of an engineer's time. Model costs are just, it doesn't, it's not quite there. So, so we want to let engineers use the model that they think is the right tool for the job. This is, I think, still a gap in the tooling is why am I as an engineer having to think about not only the problem I'm solving, but what, like, what, what model do I send it to? What tool do I use? You know, what, what context window? These are all little problems I think are going to be ironed out over time. I think there's amp code in the space that's doing a really nice job here. Having an oracle that kind of decides what is the best model to use for a task. And yeah, the cost optimization is not something that I'm necessarily concerned too deeply about today.

• 34:52Speaker 01

  I think planting down on OPUS over usage in certain environments is definitely something that to keep an eye on. But with Cursor, we've been able to work out some pretty good cost controls, some good kind of model management features.

• 35:10Wes Bos?

  That's good. And what about security as well? Is it fine just to be sending Coinbase's code to these random models? And you guys feel comfortable with that?

• 35:22Speaker 01

  Yeah, I mean, so I think every org should be comfortable with zero data retention model. Every model provider is doing this. And I think where it gets a little fuzzy is different integrations having indexing and having some sort of like database persisted of, I guess, context that is provided to them.

• 35:46Wes Bos?

  Yeah.

• 35:46Speaker 01

  That's the messy part that we have to go through our due diligence with every time we work with a different tool.

• 35:50Wes Bos?

  Yeah, I didn't think that. You didn't think about security, but people are, I'm sure anytime you talk about, especially when you talk about something that has money, essentially a bank, people are.

• 36:00Speaker 01

  Yeah. I mean, just to like, you know, double click on that point a little bit more. I've been so impressed by the level of diligence with our security teams. And I think the willingness to move at a speed that lets us keep up with this ecosystem. This is a very fast moving ecosystem. We've worked out a framework of where the risks are. I think a year or two ago, AI was very spooky. It was, we don't want to trade on our data. We don't know where the data is going. We need data controls to make sure the right types of data is going to AI. And I think now we have a really mature framework that limits any sort of exposure.

• 36:43Speaker 01

  Yeah, the teams just ultimately need to get comfortable with certain model providers. I think there's, we have a whitelist internally of essentially what models we've deemed appropriate based on some kind of analysis. And we don't use any models that are excluded from that list.

• 37:05Scott Tolinski?

  So I would imagine you guys are tuning these systems constantly. Like, how often are the knobs being tweaked on these systems that you're implementing?

• 37:16Speaker 01

  I think it really depends on the system and the technology. So we don't do much fine tuning. I don't think we're, you know, developer productivity is a place that necessarily warrants fine tuning. I think there's other parts of the business that it makes a lot more sense. Some of our compliance controls and ML teams, the work that they're doing. On the kind of, you know, fine tuning, tweaking, turning the knobs, I think managing context windows is one of our daily tasks of...

• 37:44Scott Tolinski?

  Yeah.

• 37:44Speaker 01

  Are you just providing, you know, a single query that you want really finite results for, or are you providing just a giant, you know, piece of context for the model to iterate through? So that's one challenge. And then some of our more bespoke systems, so it's more link chain, link graph agents. You know, you can mess with temperature controls and kind of different interpretations. And yeah, you're kind of creating a persona at that point. And that's where a lot of the tuning comes in. Have you guys, have you guys created any agents with, I guess, link chain or MASTRA? Any of the kind of like agentic frameworks?

• 38:23Scott Tolinski?

  No, I haven't. I've been mostly just like augmenting mine into death as I go. Yeah. Just my Claude, my Claude agents or my open code agents, I'm just tweaking them constantly. But I haven't been using any outside tools other than those.

• 38:38Speaker 01

  I was just gonna say, it's kind of a fun prototype space to be in though, right? With Claude code where...

• 38:43Scott Tolinski?

  Oh, yeah.

• 38:44Speaker 01

  It's this, it's this thing that you can just like manipulate in different ways. You can add bash scripts around it.

• 38:49Scott Tolinski?

  It's endless and it's actually become a problem for me because I'm just constantly messing with it instead of building with it. So, yeah.

• 38:58Wes Bos?

  Yeah, it's a hard part. And kind of on that point, I wanted to talk about using AI as as part of the application. So this whole thing has been talking about using AI to write code. And I feel like we don't necessarily talk about like putting AI in the application itself. And you said one of your tenants was AI should never make a decision, which means that like, yeah, you probably shouldn't put that as as part of some of your flows. But like at Coinbase, what does it look like for people who say, hmm, instead of having to build some sort of algorithm for this or a function to work, can we just throw it to an AI and come back with the result?

• 39:40Speaker 01

  Yeah, that's it's outside of my scope, primarily. What broadly the suggestion has been is, and you know, we have a lot of controls around this, where if you plan on, you know, there being some sort of AI touching any sort of the customer journey, there's a very long process to kind of go through to to enable that, because we want it to be, you know, highly accurate for it to be essentially scaled in the right way. So that's a very different problem set. I actually think one really interesting area that, you know, I haven't I haven't seen us leverage yet. You can do the on-device SDK. So you can use the Apple Intelligence to through an expo SDK now.

• 40:24Speaker 01

  And I don't have any good use cases quite yet. And I don't know how you do things like observability and, you know, I have so many questions about it. I'd love to try to use it in a personal project or something.

• 40:36Wes Bos?

  Man, the one example I always give people, I like, I'm I'm big on like the on-device or the smaller model that is good for one thing. But just like one simple example is we should be done with date pickers. We should be able to just type in the last three months. And like, anytime I talk about this, people are like, well, there's a library that's pretty good at regexing it. Like, no, those things are hot garbage. I want to, I built one. And I want to say the last game that Michael Jordan did until three weeks before somebody was, someone specific was born, you know, I want to, I want to be able to type in literally anything I want.

• 41:12Speaker 01

  That's the data.

• 41:14Wes Bos?

  Grab the information, figure that out and bring those into Unix timestamps for me.

• 41:18Scott Tolinski?

  And for me, it's spelling, Wes. For me, like, Google is still the best spell checker because whatever machine learning they're doing in spelling, right? Like, Apple's spelling, you could be one letter off and it's like, I don't know, those suggestions.

• 41:31Wes Bos?

  Yeah, you gotta copy paste it into Google to find the real spelling.

• 41:34Scott Tolinski?

  Right, yeah.

• 41:35Speaker 01

  So I think all problems boil down to forms again. It's like, if we're creating forms online, are you saying we should just, all these forms should just be AI, AI blitz now?

• 41:45Scott Tolinski?

  The whole thing about the future of UI in general and like, like, okay, we have all these bespoke UIs. I love bespoke UIs, but like, you always have to learn them. You have to figure them out. When ultimately you're trying to accomplish a task, like, what does that look like, right? You know, I don't know. There's, there's too many head exploding things there when you get into too much of everything becomes a chatbot, you know?

• 42:07Speaker 01

  I think there's an opportunity here to create an open source library that's just, boom, a single form that says, hey, tell me, tell me about yourself.

• 42:15Wes Bos?

  Yes.

• 42:16Speaker 01

  How many days after the Super Bowl were you born?

• 42:18Wes Bos?

  Yeah.

• 42:19Speaker 01

  And that's your validation. Send it off to a, to a backend server.

• 42:24Wes Bos?

  Oh, that'd be great. Anything we haven't touched upon yet that you, you'd like to talk about?

• 42:29Speaker 01

  You know, the developer journey is, is really the, the biggest piece of what we're focused on. You know, lines of code is exciting. Like I mentioned, it's, it's, it's an output metric today. All the input metrics that kind of, you know, go into how, how someone does their job day to day. We're in this crazy time where we don't want to put too much of a microscope on that or, you know, try to massage that story too much. And I think AI is simply a tool. At the end of the day, it's not going to replace what, you know, developers are making decisions on. It's, it's not a magic bullet that's going to solve every problem, you know, universally with the right.

• 43:12Speaker 01

  Ergonomic or success output. There's been some cool studies online, like the, what was the meter study? I don't know if you guys saw that one, but it was, they sampled 15 or so open source developers on how AI impacted them. And it ultimately produced some negative results, saying, hey, it didn't make us faster. In fact, it made it slower. And digging into that study a little bit was really interesting because you learn, okay, there's some, you know, senior staffish level engineers who are just mandated to use AI for every task and that actually slowed them down. So it's not something that I think we can dictate to every engineer across our company, across any company and say, you have to use AI 100% of the time.

• 43:58Speaker 01

  It's just not, it's not going to be right. And ultimately, what I hope for in the next three, four, or five years, especially as we're hiring more entry level engineers, is that we have AI natives. So I want to see folks who, you know, come to Coinbase saying, this is the model that, you know, I prefer to use. This is the tools and kind of the workflow that I really like to, you know, work with. How can Coinbase, you know, fit that for me? Yeah, like I think we're going to get to a place where there's more AI natives, I guess, than there are people who are just poking at it in different ways.

• 44:32Wes Bos?

  Yeah. And I saw something the other day where someone said, if you want to get hired, like you're going to come to a job interview with your backpack full of these 20 agents that you built and they're going to want to hire you because you're bringing the guns. Do you think we would get to a spot where we hire somebody because they got the sauce?

• 44:53Speaker 01

  I, I mean, yeah, haven't we always kind of done that in some ways of, oh, you know, show me your GitHub profile. Show me, you know, we want to obviously bring on people who have, who have the experience to solve the problem, however they need to, in a really high quality way. Yeah, I mean, on that backpack analogy, I've, I think most jobs I've gone to, I've, you know, transported around my bash profile or, you know, all the kind of like little N files that you come with. And those are my productivity hacks. I think now, I wouldn't be surprised if we start hiring engineers who say, hey, I have this agent that I use, can we onboard them too?

• 45:31Scott Tolinski?

  Yeah, it's so funny, Wes, because I, that's such a good. Like way to say that. Can I onboard this agent? But I remember when I got my job at Ford, one of the reasons why like that interview went so well is when they came to me, I was like, I have this whole SaaS CSS toolkit and they're like, I want that, you know?

• 45:49Wes Bos?

  Yeah.

• 45:51Scott Tolinski?

  Same thing.

• 45:52Wes Bos?

  Man.

• 45:53Scott Tolinski?

  All right. So now's part of the show where we get into sick picks and shameless plugs. Sick pick being something that you're just enjoying in life right now. Shameless plug, anything you want to plug and just, you know? Kyle, do you have a sick pick for us today?

• 46:07Speaker 01

  I guess things, things that I'm enjoying in life right now. So I'm currently on my, it's my four by four sabbatical and I'm just doing a ton of woodworking around the house.

• 46:16Scott Tolinski?

  Nice.

• 46:17Wes Bos?

  Oh, right on.

• 46:18Speaker 01

  And, uh, yeah, it's, it's, it's going great. So I don't really have any sick picks here. Maybe the ultra shelf people are just creating these amazing floating shelf brackets. I don't know. But, but yeah, I think if you're a engineer, someone who develops in front of a screen all day, find some sort of thing to do with your hands is kind of my like pitch to people. I've been doing pottery woodworking, uh, trying to do things very away from screens in my free time, which is just such a dichotomy coming back on. And now we're just moving at the speed of light with agents. But it really helps to, I think, solve problems in a different way. So, uh, would recommend that. And then, uh, excuse me, what was the second, the second piece here?

• 47:00Scott Tolinski?

  Just anything you'd like to plug. Yeah.

• 47:02Speaker 01

  Ah, shameless plug. Um, you know, I don't really have any plugs right now. I'm, uh, I'm currently, you know, really loving the developer ecosystem, seeing kind of like, you know, what, what cursor and, and, and those teams are doing. I think the plug I have is, is probably just, uh, to follow along, I think what Coinbase is doing with our agentic systems. We actually, you know, we're doing a lot in kind of the cryptocurrency, uh, agentic protocol sphere right now. And there's some really cool work being done around the X402 protocol with Cloudflare. I think this is going to unlock kind of a future of agentic payments and, you know, how do agents kind of transact online? So, yeah. I don't want to necessarily shill it to everybody,

• 47:47Speaker 01

  but it seems like a really interesting space to start thinking about. As you're building perhaps e-commerce or any sort of checkout steps, what if an agent were to interact with that? And how do they do that? How do you transact value between two individuals? Yeah, that's probably my plug for the day.

• 48:06Scott Tolinski?

  Tech, love it.

• 48:07Wes Bos?

  Awesome. Well, thank you so much for coming on. I appreciate, like, this is such a fantastic interview. You're incredibly well spoken. Such a, I feel like we have been missing this context and like this inside scoop in the industry. So I appreciate that you're able to come on here and like props to Coinbase as well. Because like every time we talk to these companies about like, hey, can you come on and talk about it? They're like, sorry, legal killed it.

• 48:33Wes Bos?

  So appreciate that.

• 48:34Speaker 01

  Yeah, love to share what we're building in the open. I've chatted with other eng leaders from different companies. And you know, it's such an interesting time right now to be building. So glad I could share and kind of give a peek of what we're doing.

• 48:48Scott Tolinski?

  Yeah, thank you so much, Kyle. Massively, massively interesting and valuable here.

• 48:53Speaker 01

  Absolutely. Thanks, guys.