Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling

James Cowling is the CTO at Convex and was previously the most senior engineer at Dropbox. We discussed technical details of his past projects, simplicity vs complexity, and career advice given where AI is today. • My ergonomic keyboard project I mentioned, you can follow along here: https://read.compose.llc/ Podcast links: • YouTube: https://youtu.be/3XkmNSuHFmY • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/dropboxs-former-most-senior-eng-building Thank you to this episode's sponsor for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at https://workos.com/ Timestamps: 00:00:00 Intro 00:00:53 Systems work during his PhD 00:13:05 Dropbox technical deep dive 00:21:57 Why Dropbox migrated from AWS 00:36:40 How to do massive migrations 00:44:31 Simplicity vs complexity in promos 00:49:23 What technical teams should be focused on 01:00:25 Doing the right thing vs promo hypothetical 01:08:13 Why he dipped into management sometimes 01:11:36 Why you should not lead by example 01:23:23 How to mentor Senior Staff engineers 01:27:30 Career advice for the AI era 01:37:21 Why he started his own company 01:46:05 The most technically challenging work of his career 01:48:10 How he got involved in Silicon Valley 01:52:16 Career regrets 01:55:54 Top technical book recommendation 01:56:36 Younger self and permanent underclass advice Where to find James: • LinkedIn: https://www.linkedin.com/in/jcowling/ • Twitter/X: https://x.com/jamesacowling • His company: https://www.convex.dev/ Where to find Ryan: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman Referenced in this episode: • His PhD Thesis: https://www.usenix.org/system/files/conference/atc12/atc12-final118.pdf • Masters paper: https://www.cs.princeton.edu/courses/archive/fall19/cos418/papers/vr-revisited.pdf • Papercuts writing he mentioned: https://medium.com/@jamesacowling/embracing-papercuts-e6390055dfc4 • "Don't lead by example": https://medium.com/@jamesacowling/dont-lead-by-example-4f86b1174e64 • His writing about orienting teams around missions: https://medium.com/@jamesacowling/your-system-is-not-a-sports-team-e17f9eb16b94

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The best engineering comes from a deep understanding of why.

• 0:04Ryan Peterman?

  This is James Cowling, formerly the most senior engineer at Dropbox and now CTO at Convex, and we dived into all the technical work of his career.

• 0:12Speaker 00

  It's not about getting a system to work, it's what do you do when it doesn't work? Simple systems are way harder to design than complex systems.

• 0:22Ryan Peterman?

  He also had some interesting takes on technical leadership.

• 0:25Speaker 00

  Really, a team should be oriented around what problem do they solve? They should not care about the system that survives.

• 0:31Ryan Peterman?

  I've had many friends whose promotions were rejected because their work wasn't complex enough.

• 0:37Speaker 00

  Yeah, I mean, it almost angers me. I just like it so much.

• 0:42Ryan Peterman?

  Is there any career advice that majorly changed in the last five years?

• 0:47Speaker 00

  Someone will argue back, no, what you can do is get really good at using Claude. Well, guess what? It's not very hard to use Claude.

• 0:53Ryan Peterman?

  Here's the full episode.

• 0:59Ryan Peterman?

  Well, first off, your PhD thesis was huge. It was, it's 156 pages. I didn't know, like thesis, I thought papers, you know, maybe 10 pages or something like that. It's its own book.

• 1:12Speaker 00

  It's very similar to Spanner ultimately. And it was, what's funny is I did that work. It came out a little bit before Spanner. So I got a citation in the Spanner paper. But then Spanner came out and then everyone kind of, everyone forgot about my paper.

• 1:27Ryan Peterman?

  You know, about the research, like if you could kind of describe the the problem that granola was solving and, you know, how it solved it, those types of things, can we talk about that?

• 1:38Speaker 00

  Yeah, absolutely. So, I mean, my, my interest in my career has been two, two parallel threads. One has been just abstractions in general, the idea about how to build simple models for complex problems. And, and I find that a very, very difficult and very intellectually stimulating, like, design exercise, how to design APIs, basically. And the other has been large scale transactional systems. Big fan of, of transactions and transaction, you know, is do a bunch of things at once. And, and again, I think transactions are just one of the most incredible abstractions we've invented because it allows us to, to, to manage the, probably the most difficult problem in computer science, which is, which is concurrency.

• 2:22Speaker 00

  And so granola is a, you know, that was a long time ago in my life now, but it was, it was a, it was an algorithm for how to do distributed transaction coordination. And in particular, transactions that I described at the time as one-shot transactions. I'm not actually sure if that was a standard term at the time or whether I made it up. But a one-shot transaction, meaning you kind of send some code to the server and say, run this function, all these reads, all these writes on two, three, whatever nodes at once. And how to make sure this commits atomically across multiple shards in a distributed system. But yeah, that was my, my PhD thesis and my master's thesis was on a, on a, um,

• 3:05Speaker 00

  on Byzantine fault tolerance. So a consensus protocol in the presence of malicious nodes. So how to, how to achieve agreement in state across multiple parties when there's malicious entities. And I guess the most well-known work I did as part of grad school was work on this paper called Viewstamped Replication Revisited. And that was, uh, that was a paper on redefining a protocol called Viewstamped Replication, which was a predated Paxos a little bit, very similar algorithm. Paxos, Raft, VR, they're all basically, virtual synchrony, all basically the same thing. And that was a, that was a paper I wrote at the time, which ended up being, I guess,

• 3:48Speaker 00

  influential to a, to a few really great companies like Tiger Beetle.

• 3:53Ryan Peterman?

  You mentioned transactions and I saw in the paper, there's this idea of an independent transaction. And if I'm, you know, understanding correctly, a lot of the efficiency in a, in a distributed system is lost by needing to get consensus and to vote for consensus. And I saw that in your research, you found out a way to avoid needing to vote to reach consensus and, you know, how did you do that?

• 4:20Speaker 00

  Yes, I mean, a lot of people, I think, think about performance maybe from the wrong angle because they think of performance as a factor of just, of this raw horsepower. You know, how fast are your disks? How fast is your network, et cetera? And, and no one has particularly faster disks or memory than anybody else. What really matters to performance in a large-scale system is eliminating points of coordination. So it's how to allow systems to progress. Without having without having contention between parties, you know, where like you're basically reducing parallel throughput to serial throughput across large numbers of transactions. And so in Granola, we had this idea called an independent transaction.

• 5:03Speaker 00

  Again, that was the terminology at the time. Where there were, you know, two entities are basically processing pure functions. So they have independent state and they will both come to the same conclusion as a result. And all they need to do then is serialize those transactions. So they have to decide if it was to happen atomically across multiple nodes, what timestamp should it get? And in Granola, the paper was mostly about how to exchange these timestamps very efficiently. So how to how to have multiple parties each propose a timestamp? And then choose, you know, the maximum of these, basically. So you could safely serialize the transaction. And there's a lot more complexity that goes into this.

• 5:44Speaker 00

  But really, the focus there was about how to maximize throughput in a large distributor system without resulting in the alternative, which is two-phase commit and two-phase locking. And two-phase commit with two-phase locking is basically the standard approach for when you want to have multiple nodes agreeing on the same thing, where they both agree to lock their state, not process any other data, and then and then and then commit a transaction. Two-phase commit can be quite, one, it can be low performance because you're blocking basically the systems for the duration of the transaction. And it can be high risk, too, because you are taking a dependency on another node. You basically blocked waiting for another node to return. And so that was what Granola was now.

• 6:30Speaker 00

  I've kind of, you know, it's funny you asked about Granola because I forgot about it. You know, that's so far in my in my history, I kind of forgot about that work. And I even forgot the phrase independent transactions, to be honest. So it's it's great to hear you bring it up again. But I guess all this stuff does feed into all the work you do later on in interesting ways.

• 6:48Ryan Peterman?

  When I think about distributed systems, I think about you're at a big company and you've got a bunch of machines. But when you were at MIT building this thing, how did you build and test it? Did you, were there spare machines that the college had? Or was this in the cloud?

• 7:05Speaker 00

  Yeah, this was a long time ago now. I'm showing my age. This was pretty early in the days of AWS. And so we had a rack of servers in our office that we could use. And I was fortunate enough to be at MIT where we could afford a rack. And there was also a service called Planet Lab. And Planet Lab was like a big communal set of nodes that academics could use to run tests on. But Planet Lab was a communal system. And so it was continually having problems. I mean, it's just a free for all. And so for the longest time, I would just sleep next to my desk and wake up whenever Planet Lab was free. Because, you know, you need to run some benchmarks. These days, you just spin something up on AWS.

• 7:49Speaker 00

  But then I would, yeah, I would sleep next to my desk and I would wake up in the middle of the night or random times and check Planet Lab status and then kick off a benchmarking job. Now, this was right at the cusp when I think in some respects, Google ruined systems research. And I say that with affection and respect to Google. Because before then, most of the research was coming out of academia. And a lot of distributed systems research was done on smaller scales. And a lot of the value prop was the ideas. So like, hey, I wrote a paper. Here's an interesting new idea. And yeah, it's all in theory. And so the value that came out of it was the idea. Around about this time, you started seeing papers

• 8:33Speaker 00

  out of Google, Amazon, and these various companies, where it wasn't necessarily a paper about an idea. It was a paper about a system. Hey, I've built this giant system. It has a whole bunch of features, some of which are interesting, some of which are not. And by the way, it powers Gmail or whatever.

• 8:52Speaker 00

  And that that kicked off a pretty interesting transition because at that point then, you know, program committees reviewing papers started to expect to see realistic benchmarks that frankly grad students are not able to, at least at the time, produce. I mean, we, I wasn't running Gmail on my system. And I think it did, in some respects, obscure intellectual ideas, right? Because I'm all for like papers about systems, but I think there's also value in a paper about an idea. You know, it's not, we built this big thing and it works and up to you to figure out what's interesting about it. But instead, here's a new, you know, there's a paper, you know, an old paper just called leases,

• 9:34Speaker 00

  right? Where someone invented the idea of a time-based lock and it's just a paper called leases. And that was a cool era of systems research. I think a lot of it now has shifted towards industrial research, which, you know, people are building stuff with, you know, for practical purposes. Whereas I think it's hard for academia to compete on pragmatism. And, you know, academia is really a great place to do impractical work.

• 9:59Ryan Peterman?

  When you look back on, you know, you getting the PhD in academia, being enabled to completely explore an idea versus going into industry, but maybe doing, you know, maybe you worked on spanner at Google or something like that, you know, when you look back, which path do you think would be better and why?

• 10:18Speaker 00

  I think that's a bit of a misconception a lot of folks have about PhD programs. I think a lot of students are like high achieving students in college and they think that the PhD program is like college plus plus. It's like, hey, I like learning things and so I'm gonna go do a PhD. But it's not. A PhD is training to be a researcher. And for most people, they shouldn't do that. If someone wants to be a professional software engineer, they probably shouldn't spend their time training to be a researcher. But with a really important caveat, I think there's a really interesting and challenging developmental experience you go through in a PhD program at a top university at least. Which is you get to a certain point in time when you have a problem that you're facing

• 11:02Speaker 00

  that no one else in the world knows the answer to. You have a problem that you are the world expert on. And you can chat about it with folks, but you can't ask your advisor because your advisor doesn't know either, right? So you're faced with these difficult challenges that there's, you can't read a book, you can't, you know, you can't ask ChatGPT, right? And so you're forced to go through a quite difficult and frankly emotionally challenging experience of being unsure, being uncertain and learning to think for yourself. I think that is really valuable for all engineers. I think that was really, really valuable in my career. I do see a lot of folks early in their career

• 11:44Speaker 00

  think that all knowledge comes from reading or, you know, absorbing it from someone else, or that it's a right way to do things. But I think being in a PhD program

• 11:55Speaker 00

  or being faced with really demanding open questions does train your mind to be comfortable with that discomfort. I mean, look, I feel, I feel a little bit lucky that I went through grad school without L L Ms existing. Because I had to deal with this uncertainty. I, there was, there was no, there was no crutch to help me out, which I think was very valuable. So, um.

• 12:22Speaker 00

  I mean, a lot of, a lot of, you know, if you want to, if you want to be a researcher, go, go do a PhD. If you want to build stuff, go build stuff, you know? And, and, you know, since leaving academia, I've done a lot of work that I, I guess one could call innovative. Like it, it advanced the industry in certain ways and did novel stuff, but our goal was never research. Our goal was just to solve problems. And I think that's the difference. I mean, in academia, your goal is to advance knowledge. In, in industry, your goal is to solve problems. And I gravitate more towards just solving problems. And I find that a more comfortable, comfortable environment within which to work.

• 13:05Ryan Peterman?

  Going to your time in industry, you know, at Dropbox, you became the most senior engineer at the company. And looking through all the projects that you did, I had a series of just technical curiosities. I saw this idea early in your career about multi-homing, and I wasn't familiar with that concept. What is multi-homing? What's the problem it solves?

• 13:28Speaker 00

  Yeah, I mean, multi-homing is the ability to have data in two locations, two homes. And multi-homing can be valuable for a variety of reasons. One is called, you know, primary, secondary, or, you know, lazily replicated multi-homing, whereby all writes commit authoritatively in one region and get replicated to a secondary region. And so this is normally used for, you know, business continuity. You know, one thing we did at Dropbox, we made sure that if the entire West Coast blew up, you know, which hopefully wouldn't happen, Dropbox would keep running because there would be enough, the data would be replicated in other regions, but with a window of vulnerability, with a window of time where there may be some data

• 14:11Speaker 00

  lost. So that is, you know, that's kind of primary, secondary replication or multi-homing. There's something else called active, active multi-homing, where there is truly an authoritative copy of the data in multiple locations. So basically, when you, for example, write to a system, you don't externalize that write as having succeeded until it has landed in all the regions. And so, for example, in the storage system at Dropbox, the block storage system, we truly had a multi-region replicator system where we could take down an entire region, say, take down the, a region in Ashburn, Virginia, where everyone's data centers are. And there'd be zero downtime for the company, and the data was still safe

• 14:53Speaker 00

  in multiple regions. I think multi-homing is something a lot of engineers aspire to work on, because it seems like

• 15:03Speaker 00

  The right thing to do. But I think the reality is for most companies it is not. For most companies not because there's very high costs. There's very high latency costs for this. Very high, um, because the speed of light is just not getting faster. The speed of light is fixed. And if you have to, you know, synchronously write data across multiple regions in the United States, you're going to have, say, sixty milliseconds in the commit path of your protocol, which for most applications is not tenable. Uh, so there is a lot of desire. Um, you know, a lot of engineering teams reach out for kind of advice on how to adopt active, active multi-homing for their company. I would normally say, don't do it. I would normally say, frankly, if US East is down,

• 15:46Speaker 00

  if Amazon is down that day, that's okay. If Amazon's down that, that day, your, your company will be down. That's a shame, right? But by avoiding that complexity, you're going to be able to move much faster and build a much better product. And so in my mind, systems is all about trade-offs and making the right ones. I would generally recommend most people do not make the trade-off to have partition tolerance or, or, or multi-region availability. Even though for a company like Dropbox, yes, that does make sense. When you, when you, when your, when your job is storing data and you have several exabytes of it and, and hundreds of millions of customers, I think that's, I mean, it's when it starts to make sense.

• 16:26Ryan Peterman?

  So it's, it's just another term for, I guess, data replication and having it available in other regions. Okay. I imagine also, I mean, the cost of storage is a concern. How many replicas would you keep for something? Like, let's just say I stored something in Dropbox. It's my document. Is that on, on the order of one or two or is there multiple?

• 16:47Speaker 00

  No, it's on the order of many, many. And so, um, so if you were to store a file in Dropbox. I modeled the storage to be, as we advertise, at least 12 nines of durability. Internally, the models look around 24 nines of durability. And so that means the data is secure with 99.99999% where there's 24 nines, right? Which means, at least according to the model, you know, the universe will be extinct before any data is lost. And, and the way that is done is by a combination of, of what's called erasure coding. So erasure coding is how you take several blocks of data and combine them together in, with an encoding scheme and spread them around in, in different locations.

• 17:33Speaker 00

  At that scale, at Dropbox scale, you're taking things into consideration like putting data in different racks, different rows of a data center because they're on different power feeds. You're taking into consideration different eras of hard drives and different manufacturers of drives because they could have correlated failure patterns. And then replication across regions. So, you know, you, you could be looking at 27 fragments, for example. That doesn't mean you're storing 27 times the data. It's, it's kind of encoded in, in many regions. But it's really, the replication schemes get really quite sophisticated at that point. And we actually had our own custom encoding matrix we had developed. It's called a Van de Monde matrix, where you kind of take a bunch of data and you combine it together to produce outputs.

• 18:18Speaker 00

  And we would plug in some variables like, how much do disks cost? How much does network bandwidth cost? Because there's a trade off. You can either store, if you don't want to lose your data, you can store more copies on more disks. Or you can store fewer copies. And anytime a disk fails, you re-replicate it really fast. And re-replicating really fast costs disk, costs network bandwidth. So these kind of variables go into this equation. It ends up being an extremely complex field of endeavor. But it's kind of abstracted away into a part of the system that doesn't, doesn't leak into anywhere else.

• 18:53Ryan Peterman?

  So with erasure encoding, my, my document in Dropbox is fragmented into a bunch of different chunks of data and loaded potentially from many different machines?

• 19:03Speaker 00

  Yes, absolutely.

• 19:06Ryan Peterman?

  I mean, the, the first thought I have is now there's, I might be waiting there and one of the 27 machines is slow and I can't look at the whole doc. So how do you prevent against that?

• 19:17Speaker 00

  It's actually faster than not replicated. Because if you imagine, I'll, I'll pick a, a A simplified example. Imagine, this is not the encoding scheme Dropbox uses, but imagine to reconstruct a file, you have to read six out of nine fragments. So if you read six out of nine fragments, you can just ask all nine and reconstruct and return the data as soon as you've heard from the first six. It's actually faster and you can construct these encoding matrices such that it's actually faster to have a ratio coded data than none.

• 19:53Ryan Peterman?

  I see. Okay. So you can like oversubscribe, overrequest, and then you complete on a portion of them being received.

• 20:01Speaker 00

  Yes. Now, in practice, it was a bit more complex in that we'd often have a copy in a single disk to provide fast access. We'd often try to make sure that you could serve your data out of a region close to your home region. So we'd make sure that your data was mostly served with low latency. But if that region had failed, you could reconstruct it from the remaining regions. So there's a lot of, a lot of people, you know, there's a lot of talk about building your own infrastructure and you can save money by moving off the cloud. Almost definitely you can't unless you, unless you, either you have very small requirements or very fixed requirements or very, very, very heavy investment. Because if you want to compete with Amazon, if you want to build

• 20:45Speaker 00

  a more efficient storage system than Amazon, you have to have a supply chain team that's working with, you know, Western Digital and Seagate constantly and negotiating on prices of disks and buying shipments at certain time and capacity teams and data center teams. There's a lot of work that goes into optimizing this because ultimately, our desire was to use the disks to 90, 95% of the disk size to maximize storage efficiency. To put it this way, I mean, that was, you know, I guess it's a ballpark figure, a billion dollar project, right? And it was, you know, and I think at the time, as far as I know, was the largest ever data migration in history, I think at the time.

• 21:28Speaker 00

  And so extremely large engineering project with very high technical investment. So yes, if you have that scale and you have the engineering team to do it and you're willing to keep innovating, if you're willing to keep Optimizing and investing effort in it, then yes, you can do it.

• 21:48Speaker 00

  But I think the real, I mean, the cloud has been an incredible innovation, right? Most people are not experts at this and most people should not be experts at this. You know, most people should focus on their applications.

• 21:57Ryan Peterman?

  When you say that most people shouldn't, my immediate thought was, but one of the big projects you worked on at Dropbox was migrating away from S3.

• 22:06Speaker 00

  Yes, yes.

• 22:07Ryan Peterman?

  So, you know, why, why did Dropbox migrate away from S3?

• 22:12Speaker 00

  Yeah, that was a desire at the company for a long time. You know, from even before I was there, so I started Dropbox in 2012. And I spoke to Drew, the Dropbox founder, I think in 2010 about this project. And so I think there was a desire to control the destiny of the company from a strategic perspective. I mean, at the time, this is before Dropbox kind of reshaped itself as being more about collaboration. You know, at the time, it was a file sync and share category. That was, that was, that was the, that was the market sector, you know? And so, and owning the file system was really valuable to the company. Ultimately, we saved a huge amount of money. I mean, we, we, and this is before the public, company went public. We really drove massive cost efficiencies through the project.

• 22:57Speaker 00

  But it was hard, you know, in a way that I think it'd be very difficult to emulate without a huge investment. And I do think that there is, there is a benefit to an organization from having hard problems to solve. Because if you have a company with extremely hard technical challenges, you can attract engineers who like working on those hard technical problems. And when they've solved those problems, they cycle off and work on different parts of the system. So, you know, after we all worked, we had such a great team. It was a very, very small engineering team. And after we kind of shipped the, the storage system reliably, we all went off and, you know, Jamie went and, and redesigned the sync protocol, the desktop client.

• 23:40Speaker 00

  And I worked on the, on the file system and the distributed databases. And so, yeah, there's, there's value to a business to have that level of technical investment. But it is a, you know, it's like, it's like having a baby and then you have to, you've got to raise the baby. You can't just build a system like this and be, that's it, we're done. You own it and you have to keep investing in it.

• 24:03Ryan Peterman?

  Did S3 do any counter negotiation before you set out to leave them? And say like, oh, you know, we'll cut you a deal.

• 24:12Speaker 00

  I guess I'm allowed to talk about this now. It was a long time ago.

• 24:17Speaker 00

  Yeah, for the longest time, I don't think they were particularly aware that this was happening. But you know, the data center folks talk. And certainly, it was noticed that Dropbox was buying up a lot of data center space. So yeah, we had, we had, obviously, at the scales that we were at, I mean, we were negotiating very good rates with Amazon. You know, we weren't paying sticker price. We're paying very, very, very good discounted rates. But yeah, at a certain point, they weren't able to meet our cost efficiency. Because when we launched the system, it really was more efficient than S3. And that's for a variety of reasons. One was that we were using kind of new experimental disks called shingled magnetic recording. We were the first ones, I think, to use these disks at scale.

• 25:02Speaker 00

  And two, we had a very tight understanding of our workload. So we were able to design the system specifically optimized for our workloads. Whereas S3 has to design the system for everybody. So you know, it got to the point where, you know, Amazon would not have been able to offer us a more competitive deal because we had a more efficient system.

• 25:21Speaker 00

  I wouldn't recommend another company do this right now. But I think at the time, it certainly made sense for us as a company.

• 25:28Ryan Peterman?

  Can you give an example of a tight understanding of your workload leading to like something you could do that S3 couldn't?

• 25:36Speaker 00

  Yeah, absolutely. So for example, I know that, well, I'll try not to leak any confidential data, right? But when you, when you upload a file to Dropbox, there is a pattern of access, right? So typically, people access the file very quickly shortly afterwards because you're sharing it with someone or maybe Dropbox is processing that file to generate an image preview. And then it decays at a certain rate. And so we understand in general the average block size. And we understand also the access pattern. So we could do things like At a certain point, we had these two clusters. One was designed for kind of temporary storage.

• 26:21Speaker 00

  That was kind of storage inefficient, but access efficient. So it was very cheap to read and write to, but it was inefficient to store. And data would get written to there first. And in the background, it would get moved in bulk to this colder storage system. And this colder storage system was far more static. And so it was able to have kind of more efficient algorithms and be written to in bulk. And if it went down for writes, that was no problems because it wasn't in the live path. And so we're able to trade off, again, like systems is all about trade offs. So we're able to trade off the live data write path from the long term read path. That was one of many examples where knowing the size of your data, where it's accessed from, how frequently it gets accessed,

• 27:06Speaker 00

  how long it takes to delete that data, you can really tune a system to your workload. You know, stuff like looking at even things down to knowing how much power to put in a rack. You know, you have a rack of hardware, there's a power distribution unit, a PDU at the top of that rack. It has a circuit breaker which can handle a certain number of amps. We would have to figure out, you know, how many amps are required for that rack based on access patterns. And there were times where we got it slightly wrong. There was times where we got maybe a bad batch of hardware and disks were failing too frequently. And as a result, we were re-replicating the data more regularly. And I was getting, you know, messages from the data center team saying, hey, we're running the racks really hot right now.

• 27:51Speaker 00

  You know, so that's a level of optimizations you can make when you get to that scale. But again, that's multi-exabyte scale, you know, million hard drive scale.

• 28:03Ryan Peterman?

  Yeah, when I was reading about this migration, I saw somewhere in the migration, you initially started with Go and then the racks or something that you're running on the hardware itself was oming too much? Or is requesting too much memory? And then you migrated to Rust.

• 28:20Speaker 00

  So initially, the prototype was in Python, if you can believe that. And to be fair, Python is actually pretty efficient for I O. I think people give Python a bad rap for I O bound workloads. It's pretty good at I O, but obviously not great for concurrency, not great for memory management and very hard to refactor. And it was critical the system was correct. And so we migrated everything to Go and we built most of the storage system in Go. This is before Go was in general availability, I think it's before Go was GA. And we built the system in Go. Go is a great language for concurrency, a great language for proxies. You know, it's a really well-designed for like servers that moved out of one place to another place.

• 29:03Speaker 00

  At a certain point, though, we would have, you know, let's just pick a number, let's say a million. Let's say we have a million nodes in the system. And every node has some amount of memory, some amount of disk, some amount of sheet metal in the chassis, right? And we would itemize all these things. You'd have a pie chart of how much money is spent on all the things. And it would have stuff like, yeah, sheet metal and screws and stuff. And so you're trying to optimize the storage. And a big problem for us was the amount of memory these nodes were using. Not just the memory they were using, but the unpredictability of it.

• 29:41Speaker 00

  With, you know, with Go having a runtime and, and, you know, in a storage system, an out-of-memory error is pretty bad. Because if a node runs out of memory and restarts, that looks like a disk failure. So that looks a lot like a disk has failed and has to be re-replicated. And so a batch of nodes ooming can lead to cascading failures throughout the system. Because maybe, I remember a time, it was a band, it might have been De La Soul, I can't remember. There was a band released an album on Dropbox. So there was a big spike in load to a few files. So it was very, very high bandwidth and it caused the machines to oom. So those machines oomed, those disks oomed.

• 30:27Speaker 00

  And so as a result, no problems. The system went to try to recover that data from a whole bunch of other replicas, right? Now, all of a sudden, you've taken one amount, one fire hose worth of load coming in. And you've turned this into seven fire hoses worth of load coming in. Because now you have to do a more expensive reconstruction operation, right? So now you have seven x the load. And these are the kind of cyclical kind of behaviors that can lead to something called congestion collapse. Congestion collapse is when workload to a system crosses a threshold where it all kind of collapses. So designing against congestion collapse is really the hardest part, or one of the hardest parts of Magic Pocket, which is the name of the storage system.

• 31:08Speaker 00

  So ultimately, we switched to Rust for the storage nodes themselves. And this is again, this is before Rust was in GA, so that was a bit of a risky move. But we rewrote the switch to Rust coincided with getting rid of the file system entirely on the disks and directly addressing the disk heads. So there was a, there's an instruction set, I think it's called ZBC, Zone Based Block Control, something like that. There's an instruction set for accessing disks that we were using. The disk manufacturers gave us the draft specs of these new disks and we were operating off the draft specs and directly controlling the disks. And so all that product was tied up together into a product called Disco Tech.

• 31:52Speaker 00

  It was the Disk Technology Project. And ultimately, if you look at that pie chart, one congestion collapse stopped and reliability improved. But if you looked at the pie chart of where all the money was going, it really shifted to be almost all disks. And what we wanted to do is get that pie chart to be almost all money spent on disks and as little money spent on RAM, on compute, on network, on power, on sheet metal, et cetera.

• 32:17Ryan Peterman?

  The cascading failure you mentioned, was there, so that, that happened and then there was a postmortem?

• 32:23Speaker 00

  I don't think we needed a postmortem, I think we, we knew as it was happening.

• 32:28Speaker 00

  I think when I was on the game page in the middle of the night on these things, you know, you'd be pretty, pretty evident. And so, look, this is, again, this is an argument in favor of the cloud. And imagine you've spent hundreds of millions of dollars on a, on a storage system and it's out in production and it's on physical hardware. You can't just go and put new memory chips in every one of them. I mean, you can, but you have to pay people to come in and swap them out, you know? And so that's a tricky place. And so as a, and this is, this is what I love about industry, you know, because in, like, some of the more challenging moments on Magic Pocket were stuff like, in one week, this is a weird coincidence, two trucks crashed that were delivering service.

• 33:13Speaker 00

  So there's two trucks showing up to deliver racks and they both crashed. I don't know, you know, the drivers were okay. So we lost capacity for two weeks. So we lost capacity for, for more than two weeks, for, for probably six weeks. Um, how, what do you do? What happens, you know, it's the equivalent of your disk filling up on your laptop, except it's, you know, it's a million disks. Uh, and you can't tell the customers to go away. You can't delete their files. So a lot of, a lot of tricky, a lot of tricky capacity work. And ultimately, what that, what that led to was trying to build in

• 33:51Speaker 00

  all these protections against the unknowns. You know, making sure we had the right amount of buffer planned out for anything bad that could happen. Making sure we, we designed the system so there couldn't be congestion collapse, so there wouldn't be memory spikes. And, you know, at one point, there's a, there's a process called, I think it's called FMEA, which is like a threat modeling process where you had a big spreadsheet and you kind of write down every bad thing that could possibly happen. And then, you know, all the, how bad it would be if it happened. Existential risk. Does someone die? You know, if there's a fire in the data center, all these kinds of things get put into this spreadsheet. And you kind of do a bit of a, almost pre-mortem kind of work to figure out all the, all the potential failure modes

• 34:35Speaker 00

  and then design around them. And I mean, I, that's, I love that work. I, I really, um, I don't know. I do like the firefight. I like the, I don't, I can't say I like getting paged because I've spent my whole life on call. But I do like that rubber hits the road stuff. I like that. Wow, there's congestion collapse and, and there's no one that can help you. And so you've got to think through this problem.

• 35:01Ryan Peterman?

  When I worked on infrastructure Instagram, we had this concept of DEFCON knobs, which are basically these configs that you could flip that would gracefully degrade your system where you can still operate it, but you know, maybe in the case of Dropbox, you store less replicas. So you take on a temporary increase of risk in losing data because you need to. Did you have something like that? And did you flip it in that type of case?

• 35:29Speaker 00

  Never when it came to durability. So, I mean, we just had a just absolute zero non-negotiable, like there was no room for negotiation on, on users durability. And so, um, We had those knobs for background processes for CPU and memory, for example. So if there was like a spike in load, you could turn off background processes and turn off, you know, the test load, for example, on the system. Eventually, we built this system called Trampoline. And what Trampoline did was, which actually was, it saves us a ton of money, right? When if we ever got too close to the threshold, we would just start writing data to S3. Because this is right there, right?

• 36:11Speaker 00

  So it's, you can run your capacity way closer to the edge. If you're willing, under worst case scenarios, just to dump 30 petabytes on S3. And then move it back when it's done. And now, that didn't happen very often. We would do it to test it. We would do that just to make sure the system worked. But yeah, being able to have a escape hatch for worst case scenario was really nice.

• 36:35Ryan Peterman?

  I see, so S3 is kind of, it's like elastic storage.

• 36:39Speaker 00

  Exactly.

• 36:40Ryan Peterman?

  When I looked at this project, it's such a massive migration. And my first thought is, how do you coordinate this whole project without breaking the system as it's running? Yeah, what are your thoughts on doing such a large migration without breaking things?

• 36:58Speaker 00

  Yeah, now, I mean, in terms of the engineers that like built the initial version of the system, it was a handful. Three, four, five, six engineers. It wasn't a team of a thousand, you know? It was a very, very small team. And so how do you build such a large system with a small number of people? And then how do you do a high risk migration? One of these, one thing you do is you keep things simple. You try so, so, so hard to build very cleanly, abstracted, simple systems that their failure modes are very understandable. And so they, and they're decoupled so they don't have congestion, collapse, et cetera. So focusing on simplicity, it gets tricky, by the way, with people doing agent development.

• 37:43Speaker 00

  They're not the best at building simple systems. Simplicity is still the domain of human beings for now. But a big focus on simplicity. The other was. You know, this very thick layer of validation checks during this migration. And in fact, we had, when we did the migration off of S3, we had this, something called the Dark Launch, where we would be moving data off of S3, but we were keeping it in both locations. And we had to demonstrate to the, we had this kind of contract with the Dropbox founders. We'd have to kind of keep the system running with no incidents, no downtime, no data loss, whatever, for six months before we would delete any of the data from S3.

• 38:27Speaker 00

  So we had double write it. And there was one point in time halfway through this process where there was a bug got through to production. And it didn't, nothing bad happened with the bug, but it was, it was like, it was, it was like a bug slipped through our multiple layers of, you know, release process, et cetera. And so I went to the VP and I said, hey, a bug made it through to production and we're going to reset the launch clock. And as a result, it's going to launch later and it's going to cost us some amount of money, let's say double-digit millions, you know? And they were like, great, thank you, that's good, I trust you. And that was, that was the cool thing.

• 39:10Speaker 00

  I mean, Dropbox had a lot of incredible cultural values, but like, that was like, okay, cool. If you're prioritizing user safety, that's the right thing to do. And no one was mad about that. It was almost like they were proud almost, you know, it was, it was just like, it was like, yes, you're, you're operating in accordance to the principles of this company. Yeah.

• 39:32Ryan Peterman?

  Okay, so it was double writing both systems for a while and then?

• 39:35Speaker 00

  Some subset of data, yeah.

• 39:36Ryan Peterman?

  Okay, and then you switched over reads?

• 39:38Speaker 00

  Once we were sure the system was durable and we had all our validators running in production 24/7, they were just migrating as fast as we could. I think at some point we got to 700 gigabits per second, 764 gigabits per second of pairing bandwidth between Amazon servers and ours. Certainly someone on the network team over there noticed that there was that much data moving out. At one point I got a slightly nasty email from someone saying, It was super weird. I hate the phrase "super weird." It's like, it's super weird that you're doing so many reads and not that many writes.

• 40:14Ryan Peterman?

  Hmm.

• 40:15Speaker 00

  And I didn't respond to that email. But to be fair, like, Amazon was a great, we're a great partner for Dropbox. So Dropbox still uses AWS. They always, um, there was no concern that they would do the wrong thing by us as a company. I mean, like, we only had excellent experience with AWS. But yeah, it was a, it was a moment for us, you know? It was a, it probably strained the relationship somewhat.

• 40:41Ryan Peterman?

  You, you mentioned simplicity and intuition wise, it makes sense. Do you have a concrete example, though?

• 40:48Speaker 00

  Yeah, I've got a concrete example for you. And maybe it shows the difference between academia and industry.

• 40:55Speaker 00

  So, um, the storage system is a giant distributed system with a file stored in various locations. And so you need a, a mapping from, from the file to where it lives on these disks. And so all we did was had a cluster of a thousand MySQL nodes. Alright, big giant database. And it was indexed by the block ID and it said, "This block is on these disks." And that's a pretty, like, simple, it's not sophisticated, you know? And it, and every time we'd hire someone out of academia or maybe from other companies, they'd say, "Oh, this is not very sophisticated because like, you could use a Patricia try or you could use a distributed hash table."

• 41:40Speaker 00

  And that would map, um, a block to a set of, um, of, of locations. And I think that's optimizing for the wrong thing. Because the really nice thing about dumping a list of files and the locations in a giant database, it is written in one location. If I want to validate what happened, if I want to check all the data is where it's meant to be, I just walk over the table and check. And we did, we had services constantly walking over the table and checking. Whereas if it was a distributed hash table or some giant complex data structure, it's very hard to, to validate. So, I mean, designing for validation is very important, designing for understanding is very important. It's not about getting a system to work, it's what do you do when it doesn't work, right?

• 42:25Speaker 00

  And so having a very simple boundary, that's, that's a, it's a very basic example, you know? There's, there's more, there's more sophisticated examples that take more time to explain. But like, something like that is... A lot of engineers will, you know, a lot of engineers will want to do interesting work, will want to advance in their career. They, they want to be seen as a, as a intellectual problem solver. And so the tendency can be to design complex systems. And my argument is always that like, simple systems are way harder to design than complex systems. Like simplicity is so hard. And I think too, like maybe the untrained eye, a simple system can seem like obvious.

• 43:08Speaker 00

  And the, the, I think the best compliment you could ever get about anything you design is people say like, oh, isn't that the, isn't that the obvious way of doing it? It's, it's like same as convex, you know? People say, oh, isn't that, what's that? That's just like the obvious way of structuring. I'm like, great, because it wasn't obvious when we did it. No one else was doing it, right? Everyone thought we were idiots. If after the fact, people think it's obvious, then you, then you really nailed it. I think, um, but I think that's a, um, it requires a, an understanding that's, that simplicity is, is the hardest thing in systems. And cause simplicity is, is scalable. And I don't, yes, simplicity is scalable in terms of numbers of queries per second, right?

• 43:48Speaker 00

  But what I really mean about scalability is you can take a simple system and, and have it run for five years and have people work on it for five years and have all sorts of features added to it and have requirements changed because the company realized the product didn't work the way it wanted to work and it wants to change things. And it still stands the test of time. Whereas a complex, over optimized system will not. And that's the tough thing about distributed systems design, especially, um, L L M augmented distributed systems design, is just because something works, doesn't mean it's maintainable over a long period of time, doesn't mean it's understandable, doesn't mean it's cleanly up, architected and abstracted. That stuff's really very hard.

• 44:31Ryan Peterman?

  Absolutely. And I agree with you. I think it's the long-term beneficial thing to do. One unusual thing though, in the industry that I, I've seen is The incentive system for engineers is actually, I mean, you mentioned the desire for an engineer to want to be seen that they can do something difficult. There's that, but there's also the incentive system of promotions. And I've had many friends whose promotions were rejected because their work wasn't complex enough. And so that kind of forces, it's a forces complexity, which is kind of unusual. I wanted to know what you thought about that.

• 45:08Speaker 00

  Yeah, I mean, it almost angers me. I just like it so much. Partly why I started my own company, you know. I think the ideal for anyone is to be doing work where the, you're being appreciated for solving the problem. Like, you know, and this is, if we get philosophical, this is what it was like going back to the farming days, right? There was no incentive to make it really complicated to milk a cow because the goal is to like milk the cow and then the reward is you got milk, right? And I think at a, it sounds so silly, but at a startup, that's the same thing. A startup, the goal is to build the system, have it work, have the users like it, have

• 45:51Speaker 00

  it grow, and everyone gets rewarded and celebrated for solving the problem. It gets hard to scale that. So at large companies, you end up with so many layers of organization that people end up building alternative incentive structures, right? It's like, I'm so far away from whatever the hell we're trying to do over here, that my goal now is to get all green checkmarks on my OKR plan. But who cares about your OKR plan unless it solves the problem? And so the, you know, the thing that really drives me, um, it really drives me insane is when people try to chase, um, artificial goals, right?

• 46:33Speaker 00

  And I, and I understand that if you're in a company with, um, like this, that you may have no choice in the matter. But what I want to tell people, there is a better way. And you know, and that that better way may not be available to you. You may not have job opportunities near where you are, for example. But if you do have the ability to go work at a company where like you are being appreciated for problem solving, that will make you so much better as an engineer. Like, and I see this when I, when I interview people, you know, um, and I, if I, you know, do a, um, I mean, everyone knows Google has tremendous engineering and tremendous engineers.

• 47:13Speaker 00

  Um, a lot of folks there, though, I'm not that interested in hiring because, you know, if I'll do a deep dive with them and they'll say they built a system and I'll say, well, why did you build it? And they're like, I don't know, the VP told me to. And like, oh, how's the system used? And they're like, I, I don't really, I think ads uses it, I'm not sure. And this is, this is a caricature, but I think it's very, very hard to do good engineering in that environment. You can do competent engineering, but the best engineering comes from a deep understanding of why. And this is something we just drill into, uh, you know, the, the team here at Convex or, you know, the team embodies so strongly at Convex is like, everything exists for the why. Do like, don't build a fancy low balancer unless it's not needed.

• 47:57Speaker 00

  Turns out we do need a fancy low balancer, we're building it right now. But, but you know, you should always start with why are we doing this? What's the point? And I feel for people stuck in environments that are not like this, but you know what? Like, I don't know. Try to fight the system a little bit. I think I, I, I do see a lot of maybe nihilism, a lot of defeatedness sometimes amongst junior engineers, a lot of this, um, cynicism. Like, you know, what does it matter? Like, who cares? It's just a big organization and nothing matters and, but I think it does matter. Like, I've just, if I think of like the happiest times in my life, it's been like just dedicating

• 48:38Speaker 00

  myself to a cause and, and, and trying really hard and, and trying to do the right thing. And, and I felt good when I went home and, and not trying to get promoted, just trying to do the right thing. And then, and then assuming I'm going to get promoted, if not, go somewhere else. I know it does sound quaint when I'm saying this, but I think it's possible to do this and especially possible if you surround yourself with people like this. And if someone is in a big company and they're feeling frustrated by politics, look around and see if there's a team of folks who just seem to. Want to do the right thing, just you know, want to do good stuff. I don't think that's selling out. I think that's being true to yourself, like just that's what real engineering is. Not trying to make a complicated fancy thing to get promoted, just build the coolest thing that solves the problem.

• 49:23Ryan Peterman?

  This really reminds me of something you had written. I thought it was really good writing. And in the writing, there was this idea of system bias. And you you have this quote you're you're writing, it says, here's some examples, it says, "The team is spending six months to improve performance by 10% when it was completely fine to begin with." Or, you know, "The team is trying desperately to force their tooling on clients who don't need it." Or, you know, "The team is riding their outdated system to the grave, like the captain going down on the Titanic." I I've definitely seen examples of all those types of things in industry.

• 50:03Ryan Peterman?

  And so, yeah, I I think it was it was in the the it was in the context of your writing about what you should orient your your team around, not systems, but actually missions. And maybe that's a way to fight system bias.

• 50:20Speaker 00

  Yeah, I mean, one of one of my jobs at Dropbox wasn't the most fun job, but it might have been one of the most impactful jobs is was shutting down projects, you know, looking around and being like, huh, that thing over there that has had 60 people working on it for two years, doesn't seem to make a lot of sense to me. And then I, you know, it wasn't like a hostile thing, but I'd go and chat with the team and I'd say, hey, what are you all doing? Do you believe in what you're doing? Does this make sense? And the team in private would say, oh, I don't really know, I don't really. But inertia is so strong, you know, this whole desire to not get in trouble, you know, to just keep doing what you were previously doing is so strong. And and talented people can end up doing things that don't make a lot of sense.

• 51:03Speaker 00

  One of the things I said in that article is it's kind of a cheesy story, but when we started building the storage system at Dropbox, the team was called the Magic Pocket Team, because that was a silly code name for the for the for the system we built. And so the team's oriented around building that system. But as soon as we shipped it, I renamed the team to the storage team. And that actually took a bit of work because you had to like rename all the email addresses and the channels and the repos and the things. And so it seems like a waste of time. And um, but my argument to the team was that the responsibility of the storage team is not to advocate for Magic Pocket, the storage system.

• 51:46Speaker 00

  It's to solve the needs of storage for the organization. Right. Because who else in the company knows more about storage than the storage team? Right. And if there was a point in time where S three was a better idea, it would make sense to move back. Or maybe there was a different kind of storage system that we went to use. Right. It's the job of the storage team to advocate for moving back. Right. And so I've seen this before. You have like a team called the the puppet team, you know, for people don't really use puppet that much anymore. But like for, you know, the the puppet, the the job manager or puppet versus chef. And then they'd be kind of advocating for their team's thing. When really a team should be oriented around what problem do they solve?

• 52:28Speaker 00

  They should not care about the system that survives. Because if if if you are on the Magic Pocket team and someone says we should move back to S three. That's pretty threatening to your identity and to your career. Right. But are you, if you're on the storage team and it turns out it makes sense to move back, I don't think that's the case. But um, then that's an exciting new project for you to own. And so I think it's, it seems like such silly management philosophy. But I think it's really, really important to orient a team and an identity around solving a problem and not owning and defending a system. Because you just see this in in big companies, it's just inertia is so strong. Yeah. And you see people doing things they don't believe in because that's just what they do.

• 53:14Ryan Peterman?

  If inertia is so strong, how did you fight it and close down all those projects?

• 53:20Speaker 00

  I think I got lucky insofar as I was there pretty early on. And I worked hard enough, you know, I was putting in probably sixteen hour days at the start. I'm not advocating for that, but I was like, I was dedicating my life to to the company. And I think it became pretty obvious to people that I cared, you know. It this is this, here's a guy over here that really wants to do the right thing and cares about the company. At that point, you build up enough. Confidence, capital, you know, that you feel comfortable saying things. You know, I wasn't, at that point, I wasn't afraid for my career. I just, I was afraid for the wrong decisions getting made. So I, so, so I felt, you know, psychologically comfortable making these observations.

• 54:01Speaker 00

  And then at a certain point, you, um, you know, there's another, you know.

• 54:08Speaker 00

  A lot of engineers, so at one point I would mentor, um, you know, all the, a lot of the staff plus engineers at the company. And, and people would sometimes get grumpy, you know, because they were like, oh, we're not doing the right thing over here, or this is inefficient. And every, every engineer listening to this has a story like this. They're annoyed about some inefficiency at the company. And my response was generally like, do you think we should solve this problem right now? Because if we should, let me know the team to take some engineers off and the product to shut down and I can redirect resources and we can solve this problem right now. And they'd be like, oh, well, we shouldn't shut down any other stuff. I'm like, cool, well, we just have this many engineers right now.

• 54:48Speaker 00

  And, and so if there's anything higher, lower priority, let's stop doing the low priority thing and do the higher priority thing, this thing. And oftentimes the answer was, oh, no, nothing else is lower priority. And then the answer is we just have to accept, just have to accept, right? There's no point in being angry or upset that we're not doing the right thing all the time. So I think there's a dimension to you break it, you bought it. I don't think, like when I said part of my job was shutting products down, it wasn't going around just causing problems, right? It's ideally like solving problems, like, oh, this product is not going in the right direction. Let's redirect it and do this alternative thing. And so I think the thing that helped me, I guess, was, was having a sense of ownership

• 55:33Speaker 00

  that like, instead of complaining, I just wanted to go fix problems. And that was, but that was part of the culture. I mean, there was, when I started a Dropbox and, um, I remember being a Dropbox and Infra was, I don't know, seven, eight, nine people. And, you know, we'd have someone join from Google, for example, and they'd say, well, someone should go build. I can't do anything without this logging framework, couldn't possibly do anything without this logging framework. I'm like, well, we're going okay without it. So, and like, someone needs to build this thing. And everyone would be like, well, who is someone, right? Because it's just us, it's just us. We, we build it or we don't build it. And they'd pretty quickly come to understand, oh, wait, it's just us.

• 56:16Speaker 00

  It's just we, there's, there's no other idiots out there. We're the idiots, right? So that was, I don't know, I love, I loved that time because it was just a, a time of accountability. But life gets easier and harder when you realize that everyone else is not an idiot. When you realize that everyone else is just dealing with their own stuff, right? And so I think, um, I, I do not think someone will have good luck going around complaining about stuff and, and just saying this is a dumb idea and being negative. I think people will have, um, everyone wants problems solved though. So if you're someone in an organization who was, who's willing to put their head up and say, you know what?

• 57:00Speaker 00

  I think this thing over here is a bad idea, but here's a different idea and I'm willing to own it and put the effort behind it. I think that's a, that's a recipe for success.

• 57:11Ryan Peterman?

  From that article, you had a great quote, or I guess a great question to think through this. It said, you went to everyone or a bunch of people and you would say, if we could be spending these resources working on any project at the company right now, would this still be the best use of time? And I feel like it frames exactly what you just described really cleanly.

• 57:32Speaker 00

  Yeah, I mean, I guess this is like maybe a, a trick for being a tech leader or a manager is almost nothing's a yes or no question. It's a like, it's a prioritization question. Yeah, it's not like, should we redesign the database? I don't know, maybe, I guess. Is it the most important thing to do right now? No, cool, let's not do it, right? I think it's much easier to have those conversations than to think about it as a yes or no. You know, I often hear like, oh, my VP won't let me do blah. Okay, well, it could be that your VP is dumb. That is probably not, right? It could be your VP has a different set of priorities. Maybe their VP knows that you need to ship these features. And if you don't, then the company is going to be struggling. I don't know, but to really frame it around prioritization.

• 58:15Speaker 00

  And the way I think about the career ladder for engineers is, is I think some people, maybe, maybe it's not common, but think that like becoming a senior engineer means you get better at programming. But like, I don't know, I think my programmability went down from like level four onwards. You know, I think once they got to like level four, that was peak programmer for me. And then I probably went downhill from there and I got wiser, whatever. But I think the real thing that happened is the scope that I cared about increased. So at a certain point, you're just thinking about, cool, what matters most at the company for the next five years? Um, and I don't think a junior engineer on their first year of the job should try to do this.

• 58:59Speaker 00

  Cause you probably don't yet have the wisdom, insight, um, knowledge to be able to make a good assessment. And I think it probably would be a mistake to be like, to try to come up with like redirecting company strategy. But as you grow, I think the real key part about growing is the, the, the IC six, seven, eight engineers are not necessarily the best programmers. It's they're just getting better at having broad perspective in decisions within a company.

• 59:29Ryan Peterman?

  OpenAI, Anthropic, Cursor and Versal all use this product to make their lives better. And the problem it solves is when you're building SaaS or an AI product and you want to sell to other companies. There's all these requirements you need to meet. There's SSO, there's SCIM, there's RBAC, there's audit logs. These are all things that take time to integrate, but aren't the main focus of your app. WorkOS is an API layer that lets you meet all of these requirements in just a few lines of code. So let's say you have a new SaaS product and you want to sell to other companies. WorkOS will solve all of these critical feature gaps for you. You can check them out at workos.com to learn more and get started.

• 1:00:11Ryan Peterman?

  And I appreciate them for supporting my work and sponsoring this podcast. You mentioned this idea of do the right thing and then, you know, the byproduct, you also get promoted. And I think that's, that's the dream, you know, do the right thing, get promoted. But in reality, oftentimes people would have to make the trade off. Imagine a two by two matrix of doing the right thing, doing the wrong thing, getting promoted, not getting promoted. Obviously, the, you know, do the right thing, get promoted, great. Do the wrong thing, don't get promoted, obviously bad. But I'm curious about the other two quadrants. Which one would you have picked when you were earlier in your career? Let's say I came to you, I said, hey, you could do the wrong thing, but you're going to get promoted.

• 1:00:55Ryan Peterman?

  Or you can do the right thing and I guarantee you, you're not going to get promoted.

• 1:00:59Speaker 00

  Absolutely, second one.

• 1:01:00Ryan Peterman?

  Absolutely.

• 1:01:00Speaker 00

  The second one. So I'm going to say a really tacky thing, right?

• 1:01:05Speaker 00

  There are a set of engineers who at a certain point just make infinity money. Like the amount of money they can make, you know, going and whatever, going to work at Anthropic is, is a huge amount of money. Right? And so there's a certain point where it just doesn't matter anymore. Like the money is whatever, right? But so that, so if you really want to, if you really want to maximize long-term income, I don't think you should try to do that. But if you wanted to maximize long-term income, if you get to a certain level of experience and skill and seniority, you've made it, that's it, you're done, the money's fine, right? And so I think there's this desire amongst junior engineers early in their career to be kind of over optimizing

• 1:01:49Speaker 00

  for promotion and, and income and salary, et cetera, as opposed to investing in themselves. And I guess it's, it's, they probably don't want to hear me say this because it's easier for me as an old, old guy to say this, right? But, but I think, you know, for the longest time at Dropbox, there was a time at Dropbox where like, we just didn't have our level system right. I was tech leading the team and I was making the least amount of money of the team. I'm because I was at some point as tech lead manager, I saw everyone's salaries. I was making the least money, but whatever, right? It all worked out in the end, right? And that was a happy story, but I do really think. I benefited so much from working with the best people. Like, if you have two choices, like making twenty percent more money now,

• 1:02:34Speaker 00

  or working with the best people in the world, right? Just be around the best people because like you will be,

• 1:02:43Speaker 00

  that's gonna set you on like the, on a ship to success, right? And now, not everyone wants to do that, not everyone wants to go on that. Like, it, there's no shame in just wanting to be, have a regular job and just be chilling and just be getting paid, and fine, there's, there's nothing wrong with that. But if you do wanna maximize, if you wanna maximize your career growth,

• 1:03:07Speaker 00

  then the way to maximize that is to maximize your skills, right? Like, if, and, and hopefully you're not so cynical to think that there is no correlation between talent and compensation.

• 1:03:19Speaker 00

  I don't, if you think that's the case, okay, I don't know what to say. But, but, but there is a correlation between talent and compensation and growth. And so, my advice to people early in their career is land at the best company with the best people doing the most important problems. And your life will be great, because we are so lucky as engineers. Like, what, what, what else, where else can you work solving problems for a job? Like, solving puzzles for a job and getting paid so well? Like, engineers get paid so well compared to most jobs.

• 1:03:53Speaker 00

  The privilege, the, the real privilege that we get is to work on something cool. And that's what I advocate for.

• 1:04:02Ryan Peterman?

  I love the idea you mentioned earlier, that's kind of counter to the common opinion I, I often hear. Like, the common opinion I hear, they, someone working at a big company, they're in this machine, it's kind of defeatist, they're gonna, ah, I shift this thing, but I hate this, or, I don't believe in this. And the, I, I really liked your perspective on, you know, doing the right thing and like, basically giving a damn that, you know, someone outside of you is doing the right thing too, and expanding your sense of ownership. And I want to know, what is your motivation for that? Because there's so many other people who are faced with the same inputs, and they get to a very different conclusion.

• 1:04:44Ryan Peterman?

  So, what motivates you to actually do the right thing when the machine doesn't necessarily incentivize you to do so?

• 1:04:51Speaker 00

  Yeah. Well, I think the machine does incentivize you to do so, but I don't think it's visible immediately. Like, I think people, for example, who do less job hopping really do grow the most. Because I would, I would say very strongly, if you're only in a job for three years, you're not gonna see whether your decisions were good. Like, you, you can get more money as a junior engineer, but you cannot become a senior, a very talented senior engineer without being around for long enough to own the consequences of your decisions. It, you know, it's like, it's like playing a basketball game and leaving before the game's over. Like, you're just not learning, right? So, so I do think there's an actual structural incentive towards staying in a job. Now, if you're in a bad job, leave the bad job, right?

• 1:05:36Speaker 00

  But there's a structural incentive to be able to stay long enough to have an impact.

• 1:05:41Speaker 00

  But also, I don't know, I, like, again, I guess it's easy for me to say. And also, I joined the tech industry when it wasn't like a very lucrative field. Like, you know, it, it wasn't like it is now, you know? But I just like it. The reason I left academia was because I wasn't confident I was making the right. So what would happen, how I would, how I was in academia? I'm not saying everyone was like this. But you, so firstly, you, you don't know what to do. So you have to make up a problem to solve. And you make up a problem. And then you make up a solution, you know, and hopefully a good one. And then you write a paper where you try to convince everyone that was a really good idea.

• 1:06:24Speaker 00

  And I hated it, because I didn't want to convince people. I just wanted to build it and see if it was good. Like, I just wanted to build it and ship it and see. I didn't want to play pretends and, and argue about it was good. Um, that's what makes me feel good. I don't know, and like, obviously.

• 1:06:44Speaker 00

  If you're an engineer who's living paycheck to paycheck, you should maybe ignore what I'm saying. Like, I don't want to come across as unempathetic to anyone who's really struggling financially. But if you are not struggling financially, the best thing you can do for your quality of life is, you know, in be enjoying what you do every day. You know, like, I don't know, you could have a fancy car,

• 1:07:12Speaker 00

  or you could enjoy what you do every day. And I love cars, like, I'm a, I'm a motorhead. But I promise that enjoying what you do every day is gonna have a much bigger impact on your life. And personally, I don't enjoy, like, going to work and, and working on projects I don't believe in. Like, I can only operate, and so Jamie, my co-founder, is the same way. Like, we, we really get along really well in this respect. We can only, um, we can only put up with doing things we actually care about and, and believe in. And I think that's, that's the luxury. That's more luxury than taking a first-class flight. It's more luxury than going to a three-million-starred restaurant. The luxury is you don't get up and have to do a terrible job, you know?

• 1:07:57Speaker 00

  You get up and go to a place that you like your coworkers, and you solve cool problems, and you go home and feel proud of yourself. If you can, if you can construct your career that way, and it's not easy, you have to, it requires very active effort, then you're gonna have a good life.

• 1:08:13Ryan Peterman?

  I, I saw in your career journey, it said that you're an occasional manager. And, you know, you seem like someone who really enjoys the, the technical aspects of things. So how'd you decide to occasionally dip into management?

• 1:08:27Speaker 00

  I feel like most people should not want to be managers. And I sometimes think it's a bit of a red flag if someone wants to be a manager too much, right? Of course, um, being a manager is a hard job. And, uh, so I think most people should go into management because they need to, because it's necessary. And so what happened every time I went into management, I'm in the management role now as well, because you have to, right? But, um, someone needed to manage the team. And, and so I do genuinely enjoy accountability. I like responsibility. I like, I like, um, having weight on my shoulders, I suppose. And so I went into management.

• 1:09:07Speaker 00

  Several times, but as soon as I had the opportunity to get out, as soon as someone else would come in and manage the team, I would, I would kind of bounce out of that role back into engineering.

• 1:09:17Speaker 00

  Now, going to management is tremendously educational.

• 1:09:21Speaker 00

  If, being a manager, really, unless you see the world in a different way, you realize companies are more complicated than you thought. You realize the engineers are more complicated than you thought. You realize everyone on the team is going through something. And you're like, oh, wow, now I understand why that thing happened. So being a manager is very educational. One thing I would say, I would strongly caution people against, is going into management too early in their career. And I see this happen, I see this happen a lot with well-intentioned people who want to push people into management as a means of career advancement, right? I think it's doing people a disservice. I think you should let folks take their time in an organization, in a, in a career journey. I don't think you should be going into management

• 1:10:06Speaker 00

  under most circumstances in the first three years of your career. I think you should get to, ideally get to staff engineer before you do that. That's not going to happen for everybody, but ideally you take the time because if you go into management before you're an excellent technician, it will limit your ability to influence strategy later in your career.

• 1:10:24Ryan Peterman?

  How does that play out?

• 1:10:26Speaker 00

  It plays out with people who, who stuck or have roles as people managers where, you know, they, they see their job as making, maybe making a team happy or maybe coordinating a team. Or, you know, dealing with all the day-to-day challenges of like people on a team, but they're not organizational leaders. They're not like pushing the team towards excellence. They're not like, hey, how can we reframe what this team is about? How can we, you know, help influence technical strategy? And.

• 1:11:01Speaker 00

  Again, people management is a fine job, but I think the best companies are ones where all the managers are very technical and so they're able to make sure the company's doing. At a certain point, if you're not a particularly technical manager, you're not going to be able to evaluate the work of your team. You're not going to know whether your team is even doing well. And I guess maybe you might contribute towards some of the stuff you were saying about, cynical organizational attitudes where like my manager doesn't understand me. Maybe your manager doesn't understand you if they haven't spent enough time developing technical skills and struggling.

• 1:11:36Ryan Peterman?

  Yeah, there's another piece that you wrote, I thought was really good, about leading by example. And you actually say it's bad to lead by example, but there's a quote in there. It says, "Modern tech workers can be an anti-authoritarian bunch at the best of times." Let's say you're a tech lead or tech lead manager or manager. How do you strike that balance so you don't lose credibility as a tech lead?

• 1:12:03Speaker 00

  Yeah, there are command and control companies that the manager just tells people to do things and they just do them. Those typically are not excellent companies. And not every company needs to be excellent, but if you want to have a company where your team's really innovating, or the engineers feel very personally responsible for making really high-quality decisions, you have to have them believe in what you're doing. And so, in a context, I'm the founder and CTO. I can't really go to someone's desk and say, "Do this thing." Now, they might do it just because they like me. There's a good chance they would do it, but not because I'm the boss. They'd be like, "Well, why?" And that's

• 1:12:45Speaker 00

  because that's our culture. I mean, we don't just do things because we're told. If I want someone to do something, I'll spend time with the team talking about, "Why are we doing it? Where are the markets trending? Where's the gap in our product?" And then once we're really well-articulated...

• 1:13:01Speaker 00

  Why something matters, people are just gonna do it anyway. Now, there's a, sometimes you have to have hard conversations. But I think, the way I think about it, in terms, certainly in terms of conflict in an organization, there's like this kind of hierarchy of the values you have. And then why, what, and how. And engineers are very often debating the how. They're very often debating like, what algorithm should we use for this? And like, you know, should we use this container service or that container service? And these are kind of like the implementation details. But most times when I see organizational conflict, it's because well-intentioned people are debating as best they can about how to do something, but they don't agree on why we're doing it.

• 1:13:43Speaker 00

  So if one team thinks the most important thing we can do right now is get more features out to expand our customer base. And the other team thinks the most important thing we can do is increase reliability because there's a risk to the business, right? They're both very valid perspectives, but it's gonna lead to them doing very different things. And within an organization, I'm a strong believer that everyone needs to have 100% why alignment. So the stuff we argue about, or we debate, or we talk about ad nauseam is why. And then largely, I just trust the team to do the right thing. And I think that's the case for in any tech lead. If you want to have credibility on a team, it's not, if you think that you can just tell someone to do something,

• 1:14:25Speaker 00

  and they'll listen to me because I'm the senior guy, and no, they won't. They won't. If they listen to me, it's because I've come in and explained it in a way that resonates with them. And again, one of my jobs at Dropbox was to resolve the situations. You know, someone would say, oh, that team's an idiot, they won't do blah. And then I would go talk to that team, that team was not an idiot, right? And we talked through it, and they would decide to do the project. And it wasn't because they were scared of me, I hope they weren't at least. It was because, you know, I take the time to figure out what their motivations are. And that's something you really have to learn. If you're a tech lead, you don't really have that much authority over people. And really, you have to kind of encourage them and get them to believe in what you're doing.

• 1:15:08Speaker 00

  And if you are leading through authority, you're not going to have a culture where good ideas arise from within the org. People are just going to do what they're told.

• 1:15:20Ryan Peterman?

  Influence without authority is huge. And I think there's, I mean, even big companies like Meta, they technically don't have titles. Everyone's just a software engineer. Is that also how Convex is run?

• 1:15:33Speaker 00

  I mean, we're a very flat organization. There are people in tech lead roles. I don't completely buy the everyone's a software engineer thing. It's like, you know, at, I guess, is, you know, anthropically, everyone's a member of technical staff. Because it's kind of like a wink wink thing. You kind of know, it's like, no one's mentioned the title, but you kind of know. If like the most senior person at the company comes to your desk, you're probably going to notice, you know? And, and so I think there's no point in, in, in, you know, playing pretends. People, people know, you know? But I will say that you should have an organizational culture where you don't do something just because a senior person says something. Now, I do think that reputation matters.

• 1:16:15Speaker 00

  Like, I certainly think that, you know, I don't know, if, if, if a, if a very experienced person at a, at a company comes and tries to explain something to you, you probably should listen to them because they probably have some wisdom. There might be something to be learned there. You should be open to it. But ultimately, um, yeah, even, even the most senior person, I don't think,

• 1:16:36Speaker 00

  should be leading through authority. They, they, they, they should use the benefit of their experience to be able to, to win the hearts and minds. And, and that's how you, engineering culture is so important. And if you want a culture of, of ownership and innovation and, and drive and enthusiasm and people trying to do the right thing, not get promoted, right? You have to have a culture where everyone believes in what they're doing. And no one's going to believe in what they're doing if the senior principal engineer comes to the desk and says, I'm not gonna tell you why, but you have to delete this database and do this other thing. No, that's not an empowering statement. What the empowering statement is, hey, let's spend some time together to talk about where this product's trending and how this is probably not gonna work out and how there might be a different way we can solve this problem.

• 1:17:20Ryan Peterman?

  On the topic of tech leadership, I mean, in that article I mentioned, you know, the title was "Don't Lead By Example", and I think that kinda might be confusing for people. Can you explain why you think you shouldn't lead by example?

• 1:17:33Speaker 00

  Yeah, leadership by example is a very passive thing to do. And engineers are passive people at the best of times, you know? I think that's stereotypically a little bit part of that personality. So, I mean, the very concrete example is, you know, when I first started becoming an engineer leader, I was trying to lead by example. So, I wanted to kind of demonstrate the behaviors I want everyone else to have. And very specifically, like with regards to on-call and people getting paged, I wanted people to have high ownership. I want people to jump on issues as soon as it happened. So, I would do it. I'd be always the first one to respond to a page. I would always be like, you know, writing up the reports.

• 1:18:14Speaker 00

  I would always be jumping on all the bugs and stuff and like really falling over myself to kind of show how I want people to do, to be. But from their perspective, all they see is that the lead's just doing all these jobs and they don't know, they're like, oh, maybe that's James's job. Or maybe James knows how to do it and I don't know how to do it. Turned out, I didn't know, I was just kinda figuring it out. Or maybe he likes doing those things. And I think at a certain point, being a leader is about understanding human psychology. And I don't think you can just act a certain way in front of people and wait for them to copy you, right? I think you should, obviously, you should act with integrity and values and you should, you know,

• 1:18:59Speaker 00

  you shouldn't, you should own the values of the team. But you have, someone's have to explain stuff. Someone's have to tell people, hey, I mean, very, very, very specifically.

• 1:19:08Speaker 00

  There's a trajectory almost every leader goes through. Every high achieving leader where they become a tech lead and they care so much that they become a micromanager. Like they review every line of code, they're involved in every decision. And at a certain point, they are the bottleneck for the team. So this person is so overwhelmed, you might have gone through this yourself, you know? They're so overwhelmed and they're like, wait, my team doesn't even seem busy right now. And I'm so busy, I'm reviewing all this code, I'm doing the strategy, what's going on? And then a manager will come along and say, hey, like you're micromanaging, you gotta let your team have more ownership. And so the tech lead says, okay, sure, whatever, I'll let them own stuff.

• 1:19:51Speaker 00

  And they'll just take their hands off the wheel and then the team falls apart. Because you can't just stop doing the things you're doing, right? You have to go and have a conversation with people. And so I see it, I see leadership as this kind of slider between oversight and accountability. So when, when someone's new to the team, when they're very junior, they're in a mode of oversight, like you're checking their work, right? But at a certain point, you have to dial down the oversight and very importantly, dial up the accountability. So instead of saying, I'm not gonna look at what you're doing anymore, you say, okay, cool, you've got this project. Let me know when it's gonna get done, next Thursday, cool, all right, what's the plan?

• 1:20:37Speaker 00

  This is gonna happen, okay, how are you gonna know it's correct? Great, great, great, it's on you, I expect you to do that, great, let me, let me know if there's any issues, right? And then the, the ownership relationship is explicitly on them. Because what you wanna do is basically encourage ownership within teams. You can't go from owning something to yourself to not owning something yourself and expect that to develop. You have to go have those conversations, you have to go and give people accountability. Now, what I have found, even though that can feel like an awkward conversation, most people genuinely like accountability. Most people like to own their work, most people like to say, hey, this is on you. We're all going down with the ship.

• 1:21:22Speaker 00

  I'm not gonna leave you high and dry. Like, you know, I'm the tech lead. I still take responsibility to. You're accountable to this project and go for it. And that's how you can develop people in your team. You know, a tech lead shouldn't be about you as the boss and the team as like the people who, you know, do the work or something, right? It's about this kind of flow where you're the more experienced person typically, maybe the more organized person, the more strategic person, and you're working on developing your team members so they can take your job. And then you can do something else.

• 1:21:55Ryan Peterman?

  That slider you mentioned, what if you give someone accountability and they blow it? Do they go back down the slider to, you know, where you start micromanaging?

• 1:22:06Speaker 00

  They have to recognize it, right? So this is growth. I mean, this is growth, right? It doesn't always work. And so I think I had another article about paper cuts or something. So, you know, we'd call these paper cuts, you know? So some decisions don't matter that much. You know, you get them wrong and maybe it sets you back a week. You get them wrong and maybe something's a bit suboptimal. And these are great candidates to give people accountability for because they do it and if it doesn't work, they get to experience it. And it will feel bad. And I guess that's good. It's good to feel bad. You know, you shouldn't be demoralized, but it's good to try something and it doesn't work. And you're like, oh, wow, that didn't work.

• 1:22:49Speaker 00

  And then you feed that back into your little, you know, LLM in your brain and you get better next time, right? That's a growth experience. I think what you don't want to let people do is like lose their arm. You know, like, like, paper cuts are fine, but I will not let someone, a junior person especially, design the replication system at Convex or whatever. Like, you know, you have to have safeguards. And one of the arts as a CTO or leader, a tech lead, is figuring out the right level of altitude. For how to know whether you can trust someone to take on ownership for something.

• 1:23:23Ryan Peterman?

  You mentioned, because you're the most senior engineer at Dropbox at some point, that you had to mentor other very senior engineers. And, you know, when I think about mentoring a junior engineer, it's relatively straightforward patterns. But when I think about, let's say, I need to mentor a senior staff engineer, or maybe even a principal engineer. How do you mentor someone like that, that's already so polished? They already know how to take ownership.

• 1:23:51Ryan Peterman?

  Yeah, how do you mentor someone so senior?

• 1:23:53Speaker 00

  Yeah, I mean, there's in two ways. One is there is still just a whole bunch of commonality. Everyone goes through the same problems. They, they don't know how to deliver harsh feedback. They, they don't know, you know, there is a bunch of just standard stuff people are working on. But I do think that, um, that at a certain point, everyone needs to become the best version of themselves. Sounds so cheesy, right? But there is not an archetype for what, according to me, there's not an archetype for a senior principal engineer. There, you've got to be your own brand of engineer, right? So you might be, for me, I'm like the strategic kind of collaboration, simplicity, abstraction engineer. And my coding just fell off a cliff, you know? Some engineers are like the, the deep, uh, science fiction,

• 1:24:38Speaker 00

  you know, the, the super hardcore science, you know, problem solving engineer. So everyone's going to find their own brand. So oftentimes what I'd be working on them is, is a combination of two things. One, finding their, their strengths and helping them be more spiky, helping them to like really, um, excel in the area that makes them special. At the same time, almost always working on the personal side of the engineering, like the, the organizational, personal, understanding people, understanding the why. It's just, I don't know, we're, we're so mathy, you know, as an industry. We think that somehow like even silly things like, like for example, I, I have to tell so many senior engineers. That you can't change someone's mind in a meeting.

• 1:25:23Speaker 00

  You just can't do it like you can, you can, you can disagree with someone and they're gonna be mad or whatever, right? But like, but to change someone's mind involves going through a complex series of like, of neurological processes, right? That like don't happen live in front of ten people in a meeting, right? And so if you force someone to agree with you about them being wrong, they're just gonna go along with it. And and their ego is gonna get bruised and whatever. So so one thing, you know, firstly, meetings aren't generally aren't for decision making. Most of the time when you identify a disagreement, I would point out the disagreement and why,

• 1:26:04Speaker 00

  and why I I don't agree or, you know, where the problems are, provide enough information. And then let us sit and let them go and reflect on it and like come back and have a conversation a week later after they've, after they've gone back and thought about the why, right? And so this is kind of a very psychological, but it's it's it makes no sense to be like, well, they should be able to change their mind in a meeting, well, they won't, right? That team should do this, but they didn't, right? You know, people should like my API, well, they don't, right? And I think it's like that whole like, it's almost um, it's almost like a capital, a capitalist attitude towards like interpersonal behaviors, right? It's like doesn't like, you know, capitalism rewards success, you know, or or or or impact.

• 1:26:49Speaker 00

  It doesn't matter how well intentioned you were if you didn't manage to convince them. You could be the smartest person in the world, but if you can't change someone's mind, then you are kind of useless, right? And so I think it is a lot of senior needs still struggle with this. It doesn't matter how smart you are, it doesn't matter how right you are. Are you effective? And a lot of being effective at that level is about, you know, uncertainty and project management and simplicity, blah, blah, blah, blah, blah. But also kind of interpersonal dynamics, because most hard problems happen with a team. Not always, but generally when you're at that level, you're gonna have to have ten, twenty people working with you to get something done. And that that's a whole different ballgame.

• 1:27:30Ryan Peterman?

  On the topic of career advice, the the industry's changed a lot in the last five years because of all these agentic tools. And I wanted to know if you thought, is there any career advice that majorly changed in the last five years? Something that you used to say five years ago, you don't say anymore, or vice versa?

• 1:27:52Speaker 00

  No, I I don't know if I've changed my perspective much, but I think the industry has changed its perspective dramatically. I think, I mean, let's just let's just be honest about it. There is incredible demand in Silicon Valley for senior engineers, and it is getting harder for junior engineers to succeed and grow for a variety of reasons. Why is there demand for senior engineers? Well, because L L M's can't do everything. But, you know, the architecture and simplicity and design, they're still the domain of human beings, despite what you might hear on Twitter, right? And every company, including the labs, are desperately hiring senior engineers, right? But junior tasks are getting a little

• 1:28:32Speaker 00

  bit commoditized, and that worries me. Because I do think that that learning, it's very, you know, wisdom is kind of facts put into practice and then synthesized, you know? And so a lot of people would argue that, oh, well, you know, it's easy to learn now because of chat G P T, because you can just go ask it a question about how does two-phase commit work, or what's the difference between snapshot isolation and serializability? And it will give you probably a pretty good answer. But I think growth as an engineer does require wisdom, and wisdom only really happens when you synthesize it, in my my

• 1:29:12Speaker 00

  opinion. And so I think I'm still very bullish on young people. We're hiring junior engineers at Convex. I'm very excited about, I mean, I love working with junior engineers who are really hungry to grow. But what I would say is train your mind, like do not listen to anyone who tells you that there is an advantage to having less knowledge. Yeah, I'm not sure if you've seen people say these ludicrous things that like, oh, maybe in the future, like not knowing engineering will be an advantage because you won't have biases and you'll just use Claude. I think these are like ludicrous statements, like, like software engineering is an intellectual

• 1:29:54Speaker 00

  discipline that helps you think like, like the software engineering, the best software engineering is not about knowing syntax and it's not about knowing an algorithm. It's being really good at conceptualizing problems and be able to break them down to building blocks and coming up with clean solutions to them. And that requires experience, that requires like, it's like doing, doing weights with your mind. And just like if you went to the gym and you just like, it never hurt. Like if you went to the gym and just picked up really light weights, um, you're not growing. And also if you went to the gym and picked up a heavy weight and like, okay, cool, I think I can do it. And you let the robot pick up the weight for you, you're also not really growing, right? You have to do the reps.

• 1:30:38Speaker 00

  And so I would say, and it's tough, but find a way to stress your brain every day. Find a way to, to avoid. Now, obviously agentic coating is here, right? Obviously that's, you know, I could never tell someone to like, never use a coating agent because that'd be silly, you know? But I can say that it is easy to fall into a passivity trap, like where you're just being passive about your learning. And I do think you need to spend some time in the intellectual wilderness of not being able to solve a problem and struggling. If you, I, I would say if you're running into a, a new, um, a new problem, try to

• 1:31:22Speaker 00

  think of a solution yourself and then go check with an L L M.

• 1:31:27Speaker 00

  It's gonna be hard for you. It's almost like, for example, I'm not very good at reading anymore. I gotta be honest, like I would find it hard, I mean, I can read obviously, but I find it hard to sit down and read a book because my brain has been fried by the stimulation economy, right? Like, like I, like I find it, I go home and I'm tired and I watch a YouTube video. I don't tend to go home and read a novel. I should be better at that. They take discipline to do that. I would say similarly, it's getting harder to solve a difficult problem without reaching for the help. It's getting harder and harder every day to be faced with a very difficult intellectual problem and not being like, well, I'll just, I'll just do a Google search or I'll just ask Claude, you know, whatever.

• 1:32:12Speaker 00

  I'm still doing the work. And I don't know. I, I would really encourage people to practice using your brain every day.

• 1:32:20Ryan Peterman?

  If I was just devil's advocate or just thinking from the junior engineer perspective, I might think, well, the proof that Claude can do this work today means that I don't need to know it today or tomorrow or in the future. So why, why even build that skill in the first place? Like, and also these agentic tools, they have positive trajectories too, so.

• 1:32:44Speaker 00

  Yes, so firstly, um, the agents are not particularly good at a lot of parts of engineering currently. Um, and so probably everyone should agree there's, you know, Claude is not good at designing distributed systems protocols right now, for example, you know, or, or managing a, a three million line code base, you know. Um, and so there are parts of engineering where engineers are valuable. And like I said, you know this because the labs are all hiring engineers. No matter what they say, they're still hiring engineers, right? Desperately hiring engineers, um, really, really aggressively hiring engineers.

• 1:33:18Speaker 00

  So engineers still have value. And maybe they won't in the future, but I doubt it. I really do think there's a role for human ingenuity in engineering. And so here's the, here's the trade off I would pose to people. I mean, there's, there's kind of three paths. One, get out, get out of engineering, go mow lawns and do whatever if you want. That's the, that's the most nihilistic attitude. I, I don't believe in that. I believe, I really love engineering. I believe that there's promising roles for human beings in engineering. So the second is to realize that, um, there's value right now for humans in engineering. And maybe one day AGI will be here. Let's say in a year's time AGI will be here.

• 1:34:02Speaker 00

  And there's two people. One person just gave up on problem solving right now and they're just like feeding the machine and they're running seventeen coding agents in parallel and they're just accepting everything Claude says and they've given up. Right? And one person said, you know what? I still want to have an active role in my learning. I want to understand what it's doing. I want to think hard about problem solving. Right? Play it out one year, AGI arrives. Who is going to be better placed for the future? Right? The person who's been training their mind. Right? Like there was, there's you don't, engineering is not a means to an end. It's a mechanism for, for improving your mental processes. It's like, like I still do whiteboard and coding interviews. And by the way, Anthropic still does whiteboard coding interviews.

• 1:34:47Speaker 00

  Just in case you were wondering, they don't say like, just use Claude. Right? So I still do whiteboard coding interviews with candidates. Candidates are getting worse at coding. That's absolutely true. But I don't know any better vehicle for evaluating someone's intellectual capacity for problem solving than seeing them solve an engineering problem. Right? It's a, it's a great mechanism for doing so. It's like, you know, um, if you were doing a math degree, right? A big part of doing advanced mathematics is proving theorems. And by the way, the theorems are already all proven. Right? When you're like, part of the, you know, the exams and is proving a theorem that has already been proven. And so you might say, well, what's the point of proving that theorem?

• 1:35:31Speaker 00

  It's already been done. Well, the point is that the act of proving that theorem improves your mind. Right? And then you can go and do more innovative stuff. So I would say sure, you may not, I still do think that there is a very, very promising role for human beings in engineering. Maybe not in coding, but coding and engineering are very different things. Right? Um, but even if you don't believe me, you can either give up now, but all you could just keep trying to improve your brain and you'll be better off anyway. Imagine if you're wrong. Imagine if you take the defeat of you. Imagine if you say, well, AGI is coming next week. Why bother doing anything? And imagine you're wrong. Oh my God, you just got off the, you just got off the, off the ride. You go off the ride.

• 1:36:16Speaker 00

  There's so much cool stuff. I mean, this is a cool time for engineering. Right? This is a really cool time. There's so much cool stuff happening. And I don't, and I, I hate this talk about like, AI means humans don't have a role anymore. AI means no one's gonna have a job. AI means we're gonna do nothing else new and there's no one's gonna. I, what I like is, hey, look at this, all this new cool stuff we can build. Like, look at how these ways we can make people's lives better. And to be honest, I really wish my peers and, and, and my cohort would stop it with the,

• 1:36:49Speaker 00

  the real doomer, like, um, human elimination kind of narrative. And because I think there is a little bit of a, like, um, you know, a psychological anchoring, you know? I, and as part of Convex, wanna make the, you know, we didn't start Convex to eliminate jobs. We started Convex, we wanna make it easy for people to build cool stuff. And, and I think the more we as an industry get behind, let's do everything we can to make it possible for people to do more cool things. I think it's a really exciting future we have ahead of us.

• 1:37:21Ryan Peterman?

  I wanted to talk about what you're working on now, or Convex, and why did you quit Dropbox to build Convex?

• 1:37:27Speaker 00

  Dropbox is a great place. At a certain point, I, I was there eight years. Whether I, I don't know if I say I outgrew the company, but I'd been the most senior engineer for a while there.

• 1:37:39Speaker 00

  You know, at a certain point, I, I wanted to grow and do my own stuff, you know? And so, so I left the company very unannably. And started Convex. Now, why Convex? Convex started pre-agentic era. Because my observation was that, that, um, the real differentiator in the success of a project, especially a large project, is the quality of the abstractions, the quality of the design, the quality of the architecture. Right? Systems that thrive over time and are extensible are systems that are, uh, architects well. And in my mind, the most difficult challenge in engineering, according to me, is distributed state management. How do we store state reliably and reason about it, you know,

• 1:38:23Speaker 00

  modifying it concurrently with other users? And so, so we designed, um, a platform for application building based on our experiences building large-scale systems. So Convex is a, is a transactional database where the transactions are written in TypeScript. They wanna store procedures, TypeScript store procedures. They're serializable. There's, you know, automatic reactivity. So what the client sees is a consistent view of what's on the, on the server. And I would say Convex is a very designed platform because Convex is designed to be very composable and fit together well. And so we designed Convex for developers to use. In particular, we wanted to make it so that application developers were able to build complex full-stack applications. That was the goal of Convex. Now, all of a sudden,

• 1:39:08Speaker 00

  agenty development came along. And that's been really interesting for us because it turns out that what humans find hard is also what agents find hard. You know, the coding agents are not particularly good with large code bases. They're not particularly good with reasoning about action at a distance, you know, race conditions across services. They're not particularly good at simple architectures over time. And these are the things that Convex gives you as a developer. So the idea now, and almost everyone using Convex is using Convex because they have their coding agent doing their front end, but they need a backend abstraction, a higher level abstraction, there's something like

• 1:39:50Speaker 00

  AWS or something like hosted Postgres, which makes their problems go away. And that's the company.

• 1:39:55Ryan Peterman?

  That was one thing I wanted to ask because immediately when I think, oh, I just need a backend or something like that, just go to AWS or, you know, just host something like that. So this is a layer of abstraction on, kind of on top of those types of primitives?

• 1:40:08Speaker 00

  Yes.

• 1:40:08Ryan Peterman?

  That makes it easier for an application developer.

• 1:40:11Speaker 00

  It ties into a lot of the stuff I said earlier about making problems go away. And frankly, I mean, I watched the interview you did with Barbara Lyskov, and Barbara was my advisor in grad school, and we worked a lot together and a lot on abstraction and, you know, the value in clean designs that minimize complexity. And so AWS is a fine tool. Postgres is a fine tool, although none of the mainstream databases are that great, frankly. But they're fine tools, but they're not, they don't make problems go away, right? And so the idea of Convex is a higher level set of abstractions that you can use and not reason about state management, not reason about concurrency, not reason about scheduling, not reason about transactions, not reason about polling and data sync and type safety

• 1:40:56Speaker 00

  and all those things. So Convex is a, if you think about, you know, the history of engineering, over time, the abstraction floor raises. You know, when Barbara was first starting, She was using punch cards, you know. I don't know if she mentioned it to you, but when she started as a programmer, she never heard the word programmer before. You know, that was the first time she heard the word, right? And then, you know, you went from punch cards to like, you know, having proper operating systems and, and, and, you know, then, you know, languages like C and then high level languages and, and then you had cloud computing. And over time, the abstraction floor raises and you largely forget about what's going on beneath the surfaces. Most people don't think about how S3 is implemented. I do, but that's what I used to work on.

• 1:41:41Speaker 00

  But like, most people just use it and it just stores your data and it gives it back and that's great. That's a successful abstraction. But I do strongly believe that the world is and has been overdue for a new abstraction. One level up the stack. And especially now that people are doing agentic development, largely they don't want to own a Postgres instance. Largely, they don't want to think about Kafka versus Rabbit MQ. They don't want to think about, you know, what set of tools to use. They want it just to work so they can focus on building their application.

• 1:42:11Ryan Peterman?

  When you talk about, you know, the abstraction, there's obviously a lot of stuff going on behind the scenes in Convex and the, the technical side. And, you know, what, what is it that Convex is building behind the scenes that you're most excited about and why?

• 1:42:27Speaker 00

  Basically, Convex is a new operating system in some respects, right? So we have the primitives queries, mutations, actions, subscriptions. What I think is kind of cool is how we built this, you know? We have our own database that we built. We have our own distributed database that, you know, does, you know, tracks read ranges and write ranges and does very efficient subscriptions over web sockets, et cetera. So that's, that's the current operating system set of primitives. But Convex is getting much larger workloads now and much more interesting workloads and more high performance workloads. And so we're in the process of developing a slightly lower level API

• 1:43:09Speaker 00

  for doing very efficient kind of background processes, singletons, APIs like fork, like, you know, like operating system primitives. And I'm pretty excited about launching these and how much is faster is going to make various Convex components, like the workflow system.

• 1:43:28Speaker 00

  And to be honest, the thing I find exciting every day, challenging every day, I still find Convex very hard. Like, to be honest, like, I struggle every day. I don't find my job easy. I mean, I feel confident at my job, but it's not easy. Like, designing the new API for this is super hard. I can't just go ask ChatGPT, it's not gonna give a good answer, right? And because it's innovation, it's new ideas. I really enjoy it. I find it stressful sometimes. I find it challenging and tiring, but I also find it exciting. And I would, you know, I would encourage engineers to try to find this kind of stuff to work on.

• 1:44:09Speaker 00

  Where it's like, you're on that edge of like, I'm really liking this, but also, it's a bit tricky, you know, it's a bit tough.

• 1:44:16Ryan Peterman?

  You mentioned fork and in operating systems, I'm familiar, you know, you just take the existing process and kind of split it. What's the idea of fork in a distributed system?

• 1:44:27Speaker 00

  So Convex almost never has scale issues with regards to live traffic. You know, live traffic is like typically bound by user-facing interactions, people clicking on stuff, running website, you know, accessing a website. Every now and then, someone will come to Convex and wanna kick off a million background jobs to do something, background processing. It's a big workload. You programmatically, you can trigger huge workloads, right? And so one of the things we have to scale is kind of these background workloads, and a lot of them involve things like scheduling. And there are a lot of workloads in Convex that would be very efficient if you had a background singleton process to perform things like aggregates, you know.

• 1:45:09Speaker 00

  I'll give a very silly example, right? If you have a, let's say you're building an election on Convex, a voting system, and every vote is a new row in the table, and you wanna show a tally of the votes. One way of doing this is having a bunch of background processes or Crohn's adding these things up. One way is doing a table scan, which is, you know, the obvious way to use Postgres, which doesn't scale. The other is to have a background job, which if there's new votes, it adds them all up, keeps a tally. If there's no new votes, it goes to sleep and waits on like a condition variable to wake up, to wake up again when there was a new job to perform. And so these are the kind of primitives they were working on right now. Most people won't even know they exist, but allow us to build this very high performance

• 1:45:53Speaker 00

  primitive for scheduling aggregates, you know, background aggregations, et cetera. And I'm pretty excited about like the, the next generation of workloads we can support as a result.

• 1:46:06Ryan Peterman?

  When you reflect on your career, and it sounds like you've done a lot of gnarly technical work across your PhD, Dropbox seemed like pretty intense systems work and Convex is also doing a lot of cool stuff. When you look back on your career, what was the most technically stimulating work you've ever done? And, you know, why was it hard and what'd you learn from it?

• 1:46:29Speaker 00

  There were certainly times in grad school where we were like formally modeling consensus protocols and stuff. And, you know, I'd be on the phone with Barbara on weekends and talking through, trying to reason about this in our heads. That was pretty intellectually stimulating and fun. But I think the stuff that I found most stimulating was stuff like working on very large storage system with a team where, you know, things are going wrong, you know, where, where the rubber hits the road. That's where I find, and this is every day at Convex, you know, the rubber hits the road, like, you know, oh, hey, we have a compaction process that runs in the background, but it's running into issues. We might have to redesign it using partitioning, et cetera.

• 1:47:09Speaker 00

  I, I feel most intellectually stimulated where, where there's a really clear constraint in front of me. And that to me is engineering. Like if, I don't actually know what the definition of engineering is, but I'm just gonna make it up in my mind. Engineering is science with constraints. It's like, it's like how to, how do you solve problems in the presence of resource constraints? I'm not particularly interested in constraint free environments. That's art. I like craft and engineering and the, the more visceral and difficult the constraints, the more fun that is for me. And I've been lucky enough to, whether it's luck or intention, I don't know.

• 1:47:54Speaker 00

  But I've always placed myself in those environments, you know, like, let's go get on the hardest team and own the hardest problem and then, and then, you know, put the effort in to survive.

• 1:48:10Ryan Peterman?

  This question might be a little bit off topic, but I know you were a consultant for the show, the TV show Silicon Valley. I love that show and I gotta hear, how'd you get involved with that?

• 1:48:21Speaker 00

  Yeah, that was a lot of fun. So a lot of folks might not know this.

• 1:48:28Speaker 00

  I had nothing to do with season one. So a lot of TV shows, they don't know whether they're gonna survive as a TV show. So, so Mike Judge, who started, who wrote Silicon Valley, also was a beaver's butt head in Office Space fame. He started his career as a software engineer at, I think Lockheed or something. So he actually was a software engineer that a lot of people don't realize. And so Silicon Valley was like a throwback to the kind of work it did. And if anyone's seen the movie Office Space, you would, you would get this. That's really a dystopian cubicle era tech industry film.

• 1:49:02Speaker 00

  So they did season one of Silicon Valley and then it was very popular and they got picked up. And so they had to figure out what to do for season two, but they didn't know what to do because they designed, they've written a storyline that gets to the point where there's a compression algorithm. And, and then what happens? And so they needed to find an expert on compression. And I guess ostensibly that was me. And I don't know whether I was an expert on compression. I guess I was an expert on storage at least. And so, and so they came to the office and, and, and we just chatted and it was so much fun, you know? And so I was involved in, yeah, pretty heavily involved in the show.

• 1:49:42Speaker 00

  A lot of it was, you know, storyline. And so first like, yeah, so what would you do with the compression algorithm? What would, could you design a storage system? And so coming up with story ideas that are technically accurate. They were really, you'd be surprised to know how much they care about accuracy. A lot of people I know can't watch that show because it's just so creepily accurate. They find it so cringy. And partly why Silicon Valley can, this TV show can be so cringy is because. It's it's real like those stories are almost almost. I mean, everyone, so many of the stories I was looking about are just real stories. They just they went to a bunch of companies, just farmed everyone for like stories of crazy

• 1:50:26Speaker 00

  things that happened in the tech industry and they wove them into the series. And all the characters are based on real people and and real archetypes. But they also cared very much about technical accuracy. So I would also do technical consulting and then they'd say stuff like, oh, we're building a data center in our house. What should the rack look like? And, you know, what should the diagram on the wall look like? And and partly, you know, I wanted to be like, oh, well, it doesn't really matter. No one's gonna care. And they're like, no, no, no, it matters. They really cared to get it right.

• 1:50:56Speaker 00

  So, yeah, I love the show, but it it can be hard to watch just because of, oh, my God, how how real it can feel.

• 1:51:04Ryan Peterman?

  Yeah. Are there any Easter eggs where you look at it and you go, that's unusually accurate? Or, you know, that system diagram actually is like very spot on.

• 1:51:15Speaker 00

  I can't, I don't think I can even say them because there were stories of like early Dropbox. There were stories of having ideas ripped off by other companies and being tricked into having meetings with folks on the to, you know, it may be the the competitor's team there to to steal the information. A lot, a lot of those stories are real. And so there's people who who watch Silicon Valley and like, oh, wow, that was that was something I went through and probably it was because it was about that situation.

• 1:51:48Ryan Peterman?

  That's that's such a cool experience. Did you get paid for that or was just?

• 1:51:52Speaker 00

  Yeah, that's a complicated question.

• 1:51:56Speaker 00

  I got paid because I had to get paid because it was like a Hollywood Union thing. I didn't want to get paid because it made my visa more complicated. So I've got a green card now, I'm all good. But, yeah, but that there was something where they had to pay me $400. Anyway, I made a grand sum of $400 off that show.

• 1:52:16Ryan Peterman?

  Looking back on your career, is there any regret that comes to mind that maybe other people can learn from?

• 1:52:23Speaker 00

  Yeah, I mean, ooh.

• 1:52:26Speaker 00

  I think I under invested in my personal life, to be honest. I hope people don't probably say that much, you know, because I think you can be all about like growth. And you're sure I could have grown more, I could have dropped out of grad school, say, three years in or four years in, I probably would have learned just as much. I could have taken a job at Dropbox back, you know, two, three years earlier and made a lot more money.

• 1:52:48Speaker 00

  Everyone who's been in the industry long enough has been offered to co-found several billion dollar companies. So like, everyone has a story about the times they could have been a billionaire several times over. I don't really regret those. I think, um.

• 1:53:03Speaker 00

  Yes, it's, it's been a lot of sacrifice, to be, to be blunt. I've been on call my whole career. You know, I've been, I've carried a laptop almost every day. Um, there's many dinners and parties and events I've had to skip. And there's people in my personal life who have suffered as a result. And I really appreciate those people and they, you know, they love and care about me and they know that I have a passion for this. And so, you know, they, they accept me for who I am. I think it's, um. It's, it, this is like, this is, you know, um, this is old person talk. But yeah, I think everyone has to decide like how much they want to really drive their career. Cause there's a trade off, like absolutely, like I made a tremendous amount of sacrifices in my career.

• 1:53:47Speaker 00

  And I really prioritized building as probably the number one, I mean, values first and then building. But if, yeah, we went back in time, yeah, I would have had, I had a good life, but I probably would have done more vacations and, you know, I just. I had a bit more of a balanced life. I, I really do think, I mean, I see that with all the nine nine six stuff and, and this kind of performative like. You know, photos of being in a bar and a laptop and stuff. And I'm like, that's not, that's not real. Like, that's not, I mean, sure, I was working more than nine nine six back then, but like, I probably still do work more than nine nine six. But that's, I, but I don't do it like, um, as a checkbox. You know, I do it because I just really want to be doing stuff. And, and so I think just, um.

• 1:54:31Speaker 00

  I would caution people against that hustle culture. I would, that's not, firstly, like, you know, you're only young once. You should, you should have fun. Um, you know, I've got quite a few grays in here, you know. Um, but, um, but also that's, that's, that's acting. I mean, focus on solving problems, work hard, be passionate. Yeah.

• 1:54:55Ryan Peterman?

  Yeah, when I studied your career, I mean, there's, there's mentions of you working sixteen hours a week in various. But, um, you know, you, you like on-call, you like fires. Um, you like ownership, like all those things are a recipe for working obscene hours.

• 1:55:13Speaker 00

  Yeah, and you know, I go home and I'm tired and I wind down by building stuff now. Like I, I'm lucky enough to have a little workshop at home and so I go home and I, you know, make things with my hands. Uh, that's just, as you know, that's just, you become an infor person. Um, you become an, you become an engineer like in all aspects of your life. Um, but yeah, I would just, I would just say like, the cool thing is doing cool stuff that humans use. The cool thing's not working long hours, the cool thing's not like showing off about, about that you were running a coding agent all night. Who cares? The cool stuff's, the cool thing's enjoying doing important things.

• 1:55:54Ryan Peterman?

  Do you have a best technical book recommendation for people?

• 1:56:00Speaker 00

  Um, I mean, I have to be honest, I, um, I haven't read almost any technical book.

• 1:56:08Speaker 00

  I mean, obviously I was in academia for a long time, so read a lot of papers, read a lot of papers. You know, learning is awesome, reading is great, but balance it, right? Read something and then go put into practice. And most of my career, I have, I mean, sure, I did a PhD, so I guess that, that is like the academic side. But after that, most of my learning's been by doing. Cause there's nothing like being faced with a real problem in your face to like, to really, to really develop as an engineer.

• 1:56:36Ryan Peterman?

  Yeah, I think if I answer the question, I'd probably say the same. I think learning by doing is, that's, that's really where, where it matters most. Um, and then last question for you is if you could go back to the beginning of your career and give yourself some advice, what would you say?

• 1:56:51Speaker 00

  I'd say, you know what, it'll, it'll be okay. Like, don't sweat the small stuff as much. It's hard, cause like my. My whole like engineering brand is about caring about details and like I I love like Dieter Rams and like design, you know, and and and so so being obsessive is a little bit part of my DNA. But I think I would go back and say careers are long. I mean, there's just any story anyone reads about twenty two year old billionaire, blah, blah, blah, just ignore that story. That's not that's not real. That's not repeatable. That's not

• 1:57:29Speaker 00

  normal. And it's not that healthy. And it's not that good for the people either. Right? You probably won't ideally won't max out your growth for twenty plus years as an engineer. I'm still learning all the time. I've been an engineer for several decades. So my advice would be don't sweat it. There's time to grow. And I do think that again, it's a little bit of a modern phenomenon, but there's a feeling right now, oh, my God, AGI has come and better max out my growth in the next three months. Well, guess what? You ain't gonna do it. It's not gonna happen. Right? You can't do it. You can't max your growth out in the next three months. It won't happen. Don't have to be running seventeen agents

• 1:58:14Speaker 00

  at the same time. All you gotta do is orient your career around learning every day and getting better at what you're doing and trying to solve things in the most simple ways.

• 1:58:24Ryan Peterman?

  Yeah, I mean, on Twitter, I I see this take all the time of this permanent underclass idea where if you if you don't make it in time for AGI, then you know, you're gonna be part of this permanent underclass.

• 1:58:38Speaker 00

  Yeah, I mean, look, it's a bit challenge economic times for a lot of folks. And I don't want to be I don't want to be unsympathetic to people who are, you know, having financial difficulties. At the same time, it's just not an instructive attitude. There's not much you can do with that information other than feel bad about yourself. And and and I don't and I and someone will tell someone will argue back, no, what you can do is get really good at using Claude. Well, guess what? It's not very hard to use Claude. I was like, grab some. Sometimes people tell me like, Oh my god, I'm I'm finding it hard to keep up with all the new models, the new model drops. I don't even know what the new models are like. I I use them, but I forget the latest model because it doesn't doesn't matter, right?

• 1:59:20Speaker 00

  Like, like, there was this thing called Ralph, right? I guess it's still Ralph, it's like a loop thing. I don't really know what Ralph is, and I haven't heard anyone mention Ralph in the past few weeks. But it was the biggest thing in the in Twitter for like a month. And it just doesn't, this is noise. Somehow, sometimes I feel like it's like, um, it's like tech tabloids. It's like people think that they're learning by somehow knowing, like, listening to what Jensen said today. Or like, oh, my god, Boris said that Claude writes itself. I don't think people, you're, that's like, it's just like reading about Beyonce. But you're a nerd, and so you're reading about Jensen, right?

• 2:00:03Speaker 00

  But like, it doesn't matter, you're not growing, you don't have to know. It's okay, the new coding agent could come out and you could miss it. And then next year, if if it turns out it's the big one, you'll just use it. Like, it's not hard. But I haven't seen any skill so far. I guess there's some skill, but it's not a hard skill. If you're good at engineering, you can figure out how to use Claude code, right? Or open code. So just watch out for tech tabloidism. It doesn't matter, just just be building stuff, just just just do real crap.

• 2:00:36Ryan Peterman?

  Oh, I I love I love that mindset and, um, yeah, well, yeah, thank you for your time.

• 2:00:43Speaker 00

  I'm on Twitter too, like, I'm, you know, I'm part of the thing, you know, but like, but just ignore me too. Like.

• 2:00:51Ryan Peterman?

  Oh, god, all right, well, thank you so much for your time, James, I really appreciate it, this is a lot of fun.

• 2:00:56Speaker 00

  Brian, it was great, thank you.

• 2:00:57Ryan Peterman?

  Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskoff, Mike Stonebreaker, Mark Booker, these were all people that I brought on because someone left a comment. On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype. It's a split keyboard, so there's two sides. This is in the case. But yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of the early units.

• 2:01:38Ryan Peterman?

  We're now working on the long journey of building the tooling now. And so if you still want to pick one up, I've left the late pledges open on Kickstarter, so you can grab one there. I'll put a link in the description. Thank you again for watching the podcast and I'll see you in the next episode.