Most Valuable Skill of 2026: Managing AI Agents

I welcome Ryan Carson back to the show to turn anyone into a world-class agent operator. Ryan spent 25 years founding companies, scaled Treehouse to around 110 employees and a million learners, and now runs Untangle, an AI divorce agent for family law firms, as a team of one while his revenue tracks toward 4x this month. He walks me through his full stack: running cloud agents in parallel, building automations that watch production and improve themselves, and shipping 22 to 40 PRs a day, often from his phone. We also get into keeping token costs sane through model routing and building a durable reputation by sharing your work on X. By the end, listeners hold a clear playbook to run agents, automate the busywork, and ship faster. Timestamps 00:00 – Intro 01:43 – The New Agent Paradigm 04:24 – Inside Ryan’s eight-screen desk setup 07:23 – Why Devin and cloud agents 10:00 – Everyone Is Now a Manager of Agents 13:15 – Local vs Virtual Development 14:50 – Agent Management System 24:22 – 3 Automations to build 32:54 – Self-improvement loop with Grace 34:38 – Token costs and model routing 40:33 – Building reputation on X 44:03 – Closing thoughts Key Points Ryan frames every knowledge worker as a manager of agents, and mastering that role is the new edge. Cloud VMs let Ryan run five to ten agents at once and ship 22 to 40 PRs a day. Roughly half of Ryan's work happens on his phone, which keeps his agents moving in real time. Automations like a production watchdog and a daily self-improvement loop hand Ryan a running summary of what matters. Model routing keeps costs in check: budget around $5k a month per employee and lean on cheaper fine-tuned models. Sharing your work publicly on X compounds into relationships and opportunities over time. Numbered Section Summaries Meet Ryan and Untangle Ryan Carson returns to walk me through his stack. He spent 25 years founding companies, scaled Treehouse to about 110 employees and a million learners, and now runs Untangle, an AI divorce agent for family law firms, solo, with revenue on track to 4x this month. The World-Class Agent Manager Ryan's core premise: everyone is now a manager of agents, and becoming the best in the world at it is the goal. He argues that managing agents makes you more technical over time, much like a skilled engineering manager. The Desk Setup and Key Security Ryan lays out eight screens, a 52-inch monitor, a UG Monk paper to-do system, a vertical mouse, and Whisperflow for voice. He stores all prod write keys in 1Password and hands them to agents only when a task truly calls for it, keeping production safe. Cloud Agents Over Local Machines Ryan explains cloud VMs: click a button, spin up a fresh environment, and run many agents in parallel while the code stays cleanly separated. He credits this shift for his 50x jump in output and urges builders to move work into the cloud as fast as they can. Staying Sane at 50x With 22 to 40 PRs a day, Ryan treats his job as making 10 to 20 high stakes decisions before lunch. He pins his most important threads, checks them on a roughly 25-minute cadence, and works from his phone so his agents keep moving. Automations That Run the Business Ryan shares three automations: an end-to-end signup test that browser-tests three times a week for about $60 in tokens, a 9am production watchdog that summarizes customer activity and links to real UI, and a daily self-improvement loop where an agent named Grace grades chats on a rubric and ships roughly three fixes a day. Token Costs and Model Routing After a $20k token month, Ryan settled on roughly $5k per employee plus heavy model routing, using cheaper fine-tuned models like SWE 1.7 for loop work. He favors independent agent labs such as Devin, AMP, Factory, and Cursor for affordable long-term engineering, while enjoying the subsidized Codex Mac app for everyday tasks. The #1 tool to find startup ideas/trends - https://www.ideabrowser.com LCA helps Fortune 500s and fast-growing startups build their future - from Warner Music to Fortnite to Dropbox. We turn 'what if' into reality with AI, apps, and next-gen products https://latecheckout.agency/ The Vibe Marketer - Resources for people into vibe marketing/marketing with AI: https://www.thevibemarketer.com/ FIND ME ON SOCIAL X/Twitter: https://twitter.com/gregisenberg Instagram: https://instagram.com/gregisenberg/ LinkedIn: https://www.linkedin.com/in/gisenberg/ FIND RYAN ON SOCIAL Ryan’s Website: https://www.ryancarson.com X/Twitter: https://x.com/ryancarson Untangle: https://untangle.us

• 0:00Speaker 00

  I don't think it's crazy to say that the people who know how to run teams of AI agents are going to be able to outperform everyone else in this AI age we're in. So how do you actually run teams of AI agents? What does this mean? What's the step-by-step way to actually do this? And what are the tools you need to know, like Devon AI, to actually go and do this? Well, on today's episode, I brought on Ryan Carson, and he clearly explains all of this for you. He even gives you a tour of his insane setup for how he manages his AI agents. So by the end of this episode, you're going to understand what cloud agents are, how you can set up automations, how you can create a software factory that creates software 24/7.

• 0:43Speaker 00

  And you're going to be able to understand it just really clearly because he does such a good job at explaining all of this. So enjoy the episode. I'll see you at the end and have a creative day.

• 1:00Speaker 00

  Ryan Carson, back on the pod. By the end of this episode, Ryan, what are people going to get out of this?

• 1:06Speaker 01

  So you're going to be able to do three things really well. You're going to be able to run cloud agents. You're going to be able to do automations and you're going to be able to ship faster. So whether you're a founder, a solopreneur, a stay-at-home parent, you're just going to become an agent pro by the time we're done.

• 1:27Speaker 00

  Okay, because I think a lot of people listening to this might have, they've played with agents, cloud agents, but you're going to help them just get the confidence, basically, to be world-class. I think that's what I've been thinking a lot about. It's like, okay, there's a bunch of people who have played with agents, but how do we become world-class agent operators?

• 1:50Speaker 01

  Yeah, essentially, you are a manager of agents now. So no matter what you used to do, whether it was a people manager and IC, you are going to become a manager of agents now. And you need to be the best in the world. And it doesn't matter whether you're a VC, a founder, an IC, a mom, a dad, like a college student. You should manage agents and be the best at it. And we're going to show you some tips for that.

• 2:15Speaker 00

  Okay, so where do we start?

• 2:17Speaker 01

  Okay, so let me give some context, right? So people understand where is this coming from? So I've been a founder CEO for like 25 years. I'm working on my fourth company now. At my last company, Treehouse, which was acquired, we had, I think, up to like 110 full-time employees, taught a million people how to code. So I've done the whole build organization thing and manage organization thing. And you learn a lot about managing people on that. And now this startup, which I'm running, I am one employee right now. So we've raised a seed round. We're scaling really fast. Our revenue probably is going to 4X this month. And I'm probably going to be hiring my first employees.

• 3:01Speaker 01

  We are essentially an AI divorce agent for divorce firms. So we do the dirty topic that no one seems to want to tackle, which is family law. So I've learned a lot about how to scale me, right? And so I'm going to help folks understand how do you manage a lot of agents well? How do you check the work? How do you automate things? How do you work from your phone? I do almost 50%, probably more of my work from my phone. And it's pretty wild. So that's where we're going to take you. And I will say as well, like, this is going to change every three to six months. So we're going to share a lot with you. Get your hands dirty, try it.

• 3:46Speaker 01

  And the best way to learn is to do.

• 3:50Speaker 00

  Cool, yeah, I'm actually, I'm particularly interested in the phone stuff because I actually do 0% of my stuff on the phone.

• 3:56Speaker 01

  What?

• 3:57Speaker 00

  Yeah.

• 3:57Speaker 01

  Okay, let's talk. So let me start by just showing you, well, when I'm at my desk, what is the world like, right? So right now, I'm in my home office. I work from Connecticut and I do all my desk work here. And I've got this big 52-inch Dell monitor. I used to have like a couple of monitors and I was like, screw it, I just want a huge monitor. So let me share my screen. And all right, so what I do is I set up eight screens on my monitor. And I also enjoy food and snacks as I code.

• 4:37Speaker 01

  But the key is to have eight things up at once because I have to multitask a lot of agents at once. I also use a paper system to keep track of some of my to-dos, which is a little bit weird. I use the Eugene Monk system, which is kind of fun. And I've got a killer Razer mouse. I was getting a lot of wrist pain, actually. So I decided to go vertical mouse on that. I also have a button, which actually is connected to Whisperflow. So I almost use Whisperflow for all of my stuff. So I've got Slack in the top left. That's where I get notifications and things about what my agents are doing or what is happening.

• 5:17Speaker 01

  That's the primary thing I use Slack for. And then what I do is I lay out various other screens. So I've got one password always set up. It's very important to keep your keys safe, secure, and separated from the agent, right? So I've got all my prod write keys in one password. I do not give my agents prod write keys. So production writing is dangerous, right? And your agents will do something bad. And so you need to set up a system where they ask you for the key when they need to write to prod. And then you go into one password, you copy and paste it into the session, and you're both very clear about what you're doing. So that's one tip. Keep your keys secure. And then I use Devon.

• 6:03Speaker 01

  So Devon is created by Cognition. I think it's one of the best software factories in the industry. It's not cheap, but it's good. The things I want to talk about today are not Devon-specific, but they do work very well in Devon. And one of the keys is cloud agents. Then I usually have two screens set up where I'm testing my app. It's just a lot of testing. And then, of course, I've got X in the top left and top right. And then on the right, I've got Codex, which we'll talk a little bit more, how I do most of my work in cloud in Devon. And a lot of that work, like we talked about, 50% is probably on my phone.

• 6:44Speaker 01

  And then whenever I do desktop or whenever I'm doing things locally, I'll do it in Codex. Usually that's in the bottom right. So that's kind of my, when I'm at my desk, my setup. And what I do when I'm here, but the truth is, I do a lot of work from this, my iPhone. This is my beautiful wife.

• 7:07Speaker 01

  It's just I get a lot of stuff done. I'll be in the shower and I'm thinking, okay, I need to check on that PR. Or maybe I want to land that PR, right? You know, grab it, talk to Devon, which is just done in a browser on my mobile. And I get a lot of stuff done that way. So that's my setup.

• 7:24Speaker 00

  So I keep hearing this, that Devon is almost, it's the most powerful coding agent. And can you just, for folks who haven't heard of Devon or who haven't played with it, like, why is that?

• 7:38Speaker 01

  What is going on? So about two years ago, maybe two and a half years ago, Devon came out. And they pitched it as this idea of like, it's a software engineer, right? And honestly, it didn't work very well, right? The models were just not good enough. And no matter how good the harness was, the models just weren't there. And then fast forward to 2026, the models are definitely good enough. And when you have a good harness that is cloud-based, it's really good. So think of it as you've got your choices, right? So if you're using agents to build things, you're probably using Codex from OpenAI or using Claude Code from Anthropic.

• 8:19Speaker 01

  Or you're using one of the indie, I call them the indie agent harnesses. You've got AMP, you've got Cursor, which is not independent anymore, but sorta, owned by Elon. And then you've got Cognition, which makes Devon. You've got Factory as another one as well. So you've got these choices, right? And so I thought, okay, I want to scale myself horizontally. Like, I want to be able to run as many agents as I possibly can. And I don't want to be working on my laptop and trying to figure out which code is colliding, which work tree am I on? All a lot of technical stuff that really, it keeps you from shipping.

• 9:05Speaker 01

  And I know there's a lot of different folks that listen to this show. You might have solopreneurs who are newer to coding. They don't have computer science degrees and not super technical, but they're building things. Then you may have software engineers who are multiplying themselves 10 to 100X. They have deep understanding of software and architecture. And then you have founders who are coding but are abstracting themselves. And you have VCs who are coding, you know.

• 9:32Speaker 01

  But everybody's going to learn a couple of skills. And what's interesting is I'll sort of zoom out for a second. I think we all thought that engineering and being technical was going away. But actually, what's happening is the more you become a better agent manager, actually, the more technical you become. And I think a good analogy is that you are basically an engineering manager, right? And so to be a good manager of many agents, you do need to become technical, right? There's this reality of you have to understand what Postgres is. You have to understand the production environment versus a dev environment. You have to understand migrations.

• 10:15Speaker 01

  But the truth is, you'll learn all that by using agents, right? And so I am way more technical than I have ever been in my whole life. And so I think the idea is become more and more technical, therefore. Okay, so it's backup. Why Devon? What is it? It's a really good agent. They have their cloud environment really nailed down. So that's the first takeaway I want to talk about today is cloud. Okay.

• 10:42Speaker 01

  How can you run more agents, get more done? And the answer is you work in the cloud, not on your local machine. Now,

• 10:52Speaker 01

  For the folks that aren't super technical, sort of explain, what does that mean? So when you're writing code or you're building things, you are editing code, right? That code has to be edited and then hosted somewhere, right? And, and usually what we used to do is we used to all work on our Macs. And that's called local development. And you have to set up your whole local development environment. You have to have your database, you have to have your authentication, you have to have everything local. You have to have your dev server set up.

• 11:20Speaker 01

  And then what happens if you want to do two things at once? You have to either use Git work trees, which is pretty technical and hard to understand. Or you have to literally make another copy of your code in another directory. And then, so then to work on two things at once, you have to have two copies of your code, which is hard. And then what about three or four? What if you want to do five or 10 things? It just doesn't work. So what Devin does, and I think a lot of good independent agent harnesses like Cursor is doing this now. I think Codex is starting to do this okay, is they create what's called a VM or a virtual machine in the cloud. And it's, it's basically a little server. And it is your development environment.

• 12:04Speaker 01

  And you click a button and it spins up and all of a sudden you can code in there. And the beautiful thing is if you can, you can create infinite numbers of these things. So I often have, you know, at least five cloud agents working at once, often 10. And there is no risk that the code is going to collide. So what it does is it takes away all of the mental overhead of trying to coordinate these things. Like, I want to work on this part of the code, but what if it overlaps this part of the code? Or what if, you know, I want to do that? Have I synchronized and pulled the latest? Like, I don't know. And when I work in the cloud, I literally just open up my browser and I click new session.

• 12:47Speaker 01

  And a new, and a new VM spins up and I never, ever have to think about it. So I think if you are working locally, I honestly think you are a caveman. Like, I, I think you're, you are holding yourself back and you are shipping 10x less than you could be. And it is not smart. No matter if you think the cool people work locally and that all the smart ends, it's not true anymore. You work in the cloud.

• 13:15Speaker 00

  Yeah, I mean, there are a lot of people like on X who are sort of like local Maxies, who look at people who spin up virtual machines and cloud agents as sort of like amateurish. What do you say to people like that?

• 13:33Speaker 01

  They are not, they are not doing real work.

• 13:38Speaker 01

  Here's the deal, right? So as soon as we hit PMF with Untangle and we, and we started acquiring real customers and having to ship real fixes, real features, like I had to probably 50x my output, right? And you just can't do it if you're working locally. So I think it's, it's nice to work locally. And sometimes you have to if you're doing heavy front end stuff. So say, say you're shipping a brand new feature with a lot of UI, like a lot of new UI, right? Then yes, of course, you're probably going to spin up a local agent and you're going to do some light front end, you know, probably wire framing. But as quickly as you can, you want to move that into the cloud and then let a cloud agent take over.

• 14:23Speaker 01

  So I think this is happening to a lot of people. I think a lot of the traditional

• 14:30Speaker 01

  knowledge of how to work is, is, is out of date.

• 14:36Speaker 01

  That's where we're at.

• 14:37Speaker 00

  Okay, so if you do start using cloud agents and you do 50x outputs, you know, in some ways that is a little stressful because you have so much stuff going on, right? And now all of a sudden you have to manage more. So how do you, how do you structure a system so you don't go absolutely crazy?

• 14:56Speaker 01

  So it's interesting. I think first of all, you have to shift your expectation of what your work is. And what your work is now is making high stakes decisions almost all day. So in the past, like we all had the luxury of making probably, you know, at most maybe two to three high stakes decisions per day, right? You'd have an important meeting with your key team or with a customer. And you'd make a couple of important decisions and you would communicate those decisions. And then they would get done over, you know, two weeks. You know, I think you have to sort of mentally pace yourself to say, probably by lunch,

• 15:37Speaker 01

  you're going to have made, you know, 10 to 20 high stakes decisions.

• 15:45Speaker 01

  And so you have to learn how to do that. So which is your question, how do you do that? And I think there's a couple of simple things, right? So when I'm working with my agents, I actually have to organize my threads by, and I pin the ones that are the most important work we're getting done today, right? Cause there's a lot of quick bug fixes or, you know, small things you're shipping. And you kind of learn to separate into, into two buckets, buckets. What are the big, big important things I'm getting done today?

• 16:17Speaker 01

  Pin those. And, and then the other things, just let them rip and get back to them when you can. And then set yourself this kind of almost timer, right? Where you're like, okay, I'm going to go crazy if I just click through my threads like this all day. You, you just get completely wiped out, right? And so it's almost this discipline of saying, I'm actually, you know, the, the phrasing smooth is, slow is smooth and, and smooth is fast.

• 16:46Speaker 00

  Yep.

• 16:48Speaker 01

  I, I think there's a little bit of, of that where you realize I can only check on my agents and make high stake decisions probably every 25 minutes. And so what you might do is settle into this cadence where you check on your high stakes threads, like every 25 minutes, just so you can like literally mentally rest.

• 17:09Speaker 01

  And then the other side of that is just understand that this is the world we live in and it is kind of exhausting. And, and that is kind of where we're at. And in, in order to survive in this new world, you're, you're going to work a lot more, not a lot less. And on that note, I'll share my screen because there is essentially a world we live in now where you're going to ship probably, you know, so this is my PRs, right? And we're looking, these are days, right? And you can see what, you know, the average here is, is sort of this 22 to 25 PRs a day, right? And, and sometimes, you know, 40 a day.

• 17:54Speaker 01

  Most of these are being, you know, merged, right? A couple are being closed. And what's funny is what happened this day, I went hiking with my son.

• 18:03Speaker 00

  Which day is that? Can't see your cursor.

• 18:05Speaker 01

  Oh, it's the, it was the 18th. And so we went to climb Mount Washington and I literally couldn't access my, my phone. But notice immediately the morning of, right? You know, while he was sleeping, I, I shipped, what is that? You know, eight PRs.

• 18:22Speaker 01

  So this is kind of, you know, the world we live in now.

• 18:27Speaker 01

  And we could talk about that. But let's zoom out to the, the lessons here. You've got to go cloud, right? You have to understand in order to compete in this new world, you're going to be working half the time from your phone. And it is tiring, but, but, but it's like going to the gym. Okay, I have this new muscle like I never had to exercise before. And it turns out this weird back muscle, it turns out like that's the most important muscle in the world. And if you don't do it, you're not going to win. So the way to do that is then set some sort of cadence for yourself. Pin your most important threads, write it down. So this, you know, what I'm going to sort of flash up on the screen here is my written to do list.

• 19:10Speaker 01

  Every day, you know, I use this hilarious analog system to like, to remind myself, what are the most important things we're going to ship today? Because there's going to be 10 probably PRs that we ship that are fire, you know, putting out fires.

• 19:26Speaker 01

  And, and you can get distracted and exhausted. So, so cloud and then exercise this new weird back muscle, which is making high stakes decisions all day.

• 19:36Speaker 00

  Including on your phone, right? We were basically saying is like, you're gonna, because of the velocity of,

• 19:45Speaker 00

  of, you know,

• 19:46Speaker 01

  Knowledge work.

• 19:47Speaker 00

  Knowledge work and you're, you're, you just, you'll have.

• 19:52Speaker 00

  Stuff going on and you will need to be there to say yes, no, actually this.

• 19:58Speaker 01

  Yeah.

• 19:58Speaker 00

  Giving feedback in real time and you know, why, your point is why wait until you're in front of your Mac to give feedback?

• 20:08Speaker 01

  It makes no sense.

• 20:10Speaker 00

  If you're trying to hit product market fit and if you're trying to Forex revenue in, you know, in a month or whatever it is, you know, why?

• 20:18Speaker 01

  You gotta be available.

• 20:19Speaker 00

  Gotta be available.

• 20:20Speaker 01

  And you know, I think it just, it's easy to think of these things as humans, right? Like you have a team now that is infinitely scalable. They're going to do work very fast and they're going to be blocked by you making important decisions, right?

• 20:35Speaker 01

  And you can't delegate the decisions yet. I think we're still very much, even if you use Fable, like this is a trick is, is use a Fable thread as kind of the manager and then ask it to spin up children. And I do this a lot. So, you know, I'll spin up a Fable thread and I'll say, okay, you know, we want to accomplish these five things, you know, spin up five, you know, child Devon sessions. Don't use Fable, I use Fusion for that, which is a cheaper model. Or it's cool. It's like their, their new model where they've got, Fable is kind of the parent and they've got a sidekick. Point is you're not going to, it's not viable to run Fable or any super premium model all the time, but have a sort of a parent, you know, very intelligent model run children.

• 21:18Speaker 01

  You can do these things, but in the end you're gonna have to pay attention. And it's just like if you ran a big team and you were out to lunch on your boat all the time, you gotta get in the office and the office is your phone. And so get it done. So Greg, this is my homework for you. Next time we talk, I want you to be doing more than 50% of your work on your phone.

• 21:42Speaker 00

  Yeah, I mean, that's why, that's why you're here, right? Like you're, a lot of people ask me like, how do you choose guests and why do you explore topics, certain topics over other topics? And it's, it's really, it's for everyone. Like I love arming people with the information, the tactics, the systems, but I'm also everyone. Like I'm a part of that, right? Like a part of that is like, I know that you're, you're doing this and I want to understand why you're doing it. And I also want to understand like, you know, one of the takeaways I have is your system for dealing with tasks is not, is not too dissimilar to a very organized person pre-AI.

• 22:21Speaker 01

  Right.

• 22:21Speaker 00

  Like the idea of like pinning, you know, your most important things, like it's, on one hand, it's pretty obvious, but on the other hand, if no one tells you like, hey, you know, the way to, to do this so you stay sane is via that system, you're, it's overwhelming.

• 22:39Speaker 01

  Yeah, it is funny. It's like back to the basics, right?

• 22:42Speaker 00

  Yeah.

• 22:42Speaker 01

  It's just in an age and age. And it turns out like being a very, very good engineering manager is, is probably one of the most valuable skills you can, you can teach yourself. And you know, I'm, I have the honor of sort of mentoring my nephew right now and he's coming to me and asking me, how do I win? You know, he's just come out of college soon. How do I win in this new world, right? And you know, I basically teaching him what we're talking about on the show. It's like, okay, your job is going to be managing as many agents as you possibly can. Get good at that, right? Which, which means, you know, you need to start to adopt these, these, these behaviors, right? Work in the cloud, prioritize what you're doing, learn how to make high stakes decisions quickly, get technical.

• 23:28Speaker 01

  You know, it is a farce that engineering is going away. Like if anybody tells you engineers are going away or becoming less technical, they're just not doing the work, right? And it's like being a carpenter, like say you had robot carpenters around, if you were a real carpenter, you'd be a much better manager of those robot carpenters because you understand how to make the cut, you know, where to put the nail. And so, and so it's important. So to back up, and if you're listening to the show, like the thing I want you to walk away is, is cloud agents. Get your reps in, figure out how that works. Try it a couple of times.

• 24:10Speaker 01

  Do it from your phone and get comfortable, you know, looking at your small phone interface and, and, and learning how to work that way. So I think that's, that's thing one, it's very important. The, the second thing I want to dive into is automations. Okay, so say you're working in a cloud, well done. You know, you're starting to separate yourself from the crowd. The second thing is automations.

• 24:34Speaker 01

  You need to sort of think about things that you would have normally had a meeting with, you know, once a week with your team, where you check on something or daily you check on something and build an automation. What does that mean?

• 24:49Speaker 01

  So the first thing to do is go back to your agent, start a session and say, okay, I want you to automate this task every X amount of days. And so one of them that I do, it's really simple, is that I want to automate the test of signing up for Untangle, creating a case, onboarding a client, having that client go through discovery, you know, basically our user experience, but I want that to be automated in a browser. Like this is not rocket science. This is basic user testing, right?

• 25:23Speaker 01

  But it turns out that good agents can do this. And so the big unlock is set up your agent so that they can make themselves better instead of you trying to build these systems. And so I set that up and then in Devon, it's really cool. It's called a playbook, but the idea is it's, it's, it's sort of like, here's how to do this thing. It's different than a skill. It's more of a list of things to do, how to do them correctly. And I have a, it's called end to end signup test and that runs three times a week. Cause it is expensive.

• 26:03Speaker 01

  It's, it's probably 60 bucks in tokens because it's doing a lot of, it's doing a lot of browser testing. And I think that's sort of a side point I want to explain here is it's, it's now vital that your agent can, can properly browser test. And it really should work out of the box, right? This is part of the reason I use Devon is because they've been using cloud agents that can do browser testing for two years now. It's so good. It records a video, it annotates the video, it then looks at its own video and it fixes, fixes bugs that it sees, right?

• 26:45Speaker 01

  So you have this agentic loop that is specifically around browser testing that then is automated three times a week, right? And then what, what, what do you do with that? And then you need to trigger a triage based off of that. So say that it's like, oh, wow, during this, the, the third test failed in this automation, I'm going to spin up a Devon session or an agent session to then fix it.

• 27:11Speaker 01

  And then you have to figure out where do you know that that failed? How do you know? Did the agent just kind of find, you know, silently fail and then fix itself? Or what about the PR that was created? So you have to start thinking about how do you see these things when they happen, right? As the manager of all these agents, how do you know what's failing and what's not? And, and that's again, we should talk to your agent and say, okay, we've got this end to end test, you know, I understand it runs Monday, Wednesday, Friday. I understand it, you know, it, it triggers a child triage session if anything goes wrong. How do I know about it? And it, and then your agent might say, well, you know, why don't I post in Slack? You know, and you could say, well, okay, how are you going to do that?

• 27:54Speaker 01

  Well, we need to connect the MCP, you know, how am I going to know, look in that channel? There's just all this machinery that you need to, to automate and figure out and, and work through to build that system of automation so that you are doing less of the work.

• 28:10Speaker 00

  So this specific automation, is this more of a like bug testing QA automation? Or is it more of a, you know, UX QA perspective?

• 28:25Speaker 01

  This one is more about finding bugs that have been introduced to prod that somehow we missed in our, in our user tests. In our automated test suite. So that one in particular is like, hey, we just can't have signups go down and not know about it. But you know, sometimes your, your automated test suite just, it, it doesn't catch everything. But a good old fashioned clicking through the site in a browser catches that. It is not really about the UX. I think that there's, there's no, one thing I will say is there's no substitute and we

• 29:06Speaker 01

  probably won't have a substitute for a while of you using your own app. There just isn't a good substitute right now. So right now. Every so often, I literally go through our entire app myself. And then I obviously quickly spin up, you know, sessions to fix bugs, I find, but we're just so far from agents still. The smartest models in the world still lack this sense of obvious, you know, intelligence where you're like, why did you think that was okay?

• 29:42Speaker 00

  Well, yeah, I think the bigger question is, you know, if there's a division of labor between human beings and agents, what are the tasks that you as the human being are going to be doing? Or are the sets of tasks and what are the sets of tasks that the agents are going to be doing? That you're okay with them doing and keeping you in the loop? So my question for you around automations is beyond QA, what are some other automations that, you know, a founder could be implementing?

• 30:10Speaker 01

  So let me pull up my list for you. So the kind of automations that I have are clearly QA bug fix, fine. The other is what I call my production watchdog. So we've got a lot of real law firms using Untangle. We've got a lot of real clients, you know, going through discovery. And we've got just a huge amount of activity and honestly, I can't keep track of it, you know, mentally by just looking through logs. And so what I do is every day at nine AM, I have what's called a production watchdog automation, which what it does is it goes through all of the events in the database

• 30:53Speaker 01

  that happened for our paid customers and it summarizes that in a JSON file. And then that JSON file is in our admin. And then I, you know, know every morning I go in and I click on the production watchdog and I read through the summary of what the customers did.

• 31:14Speaker 01

  And it's so valuable. So it's sort of like you can imagine, you know, your chief of staff showing up and saying, you know, all right, Greg, here's the important stuff that happened yesterday for our customers. Right? And I'm going to roll it up and explain, you know, the things that went well and a couple of bugs that we saw. So that's an automation. Essentially what happened yesterday, that was important. So that one.

• 31:38Speaker 00

  That's a big one.

• 31:39Speaker 01

  Yeah.

• 31:39Speaker 00

  It's honestly a big one. Like it's like one of those ones that sounds small, but is a big one because you're going, you think you know what's going on, but you don't.

• 31:48Speaker 01

  You don't. I mean, and it's, it's so shocking to me like, yeah, the amount of times I'm like, oh, and this is the other trick, man. So use a production watchdog automation, pick your most important customers and then make sure that in that report, it's like the customer did X, here's a link to view that, right? And so I actually have a production instance where you click it and you actually see what the customer was doing. And then I'll be like, whoa, that's weird. Like, and we have this amazing ability that, you know, over, you know, you know, millions of years of evolution in very quickly picking up things that are off. And so there's a couple of times I'm like, that's weird.

• 32:32Speaker 01

  And I click into it and it turns out, oh, the user experience that the agent built on that wasn't, was, wasn't quite what I thought it was. And so, yeah, production watchdog automation that links out to real UI is just, man, that's a game changer. So that's a big automation. This is a really important self-improvement loop. How does this work? Okay, everyone's talking about self-improvement, right? So in Untangle, we have an agent called Grace and she is essentially a paralegal.

• 33:07Speaker 01

  And she has a lot of chats with both our customers who are attorneys, our customers who are paralegals and our customers and their customers who are clients that are getting divorced.

• 33:19Speaker 01

  So how do I actually grade those chats and then improve them without managing the fine details? So what I do is I, every day I have an automation that looks at these chats and then grades them on a rubric. And so you, and again, just talk to your agent about this. You pick the most important part of your app that you want to self-improve and say, here's how you agent judge whether this thing that happens is good or bad. And then every day I want this automation to kick off, look at those things for us, their conversations that Grace is having. And then I want you to grade them. And then if anything is bad or below the score, I want you to kick off a child session and fix it.

• 34:03Speaker 01

  And it is shocking to me, the number of small, you know, fine UX details that have been picked up by this loop and have been fixed. And honestly, there are things that I just wouldn't bother doing. Either I didn't know about them or I'd be like, meh, you know, it's a paper cut. Like, are we really gonna, you know, ship a fix to that? But because the agent already identified it, spun up a PR and it's ready to ship, I just go ship it. And, you know, we probably ship, I would say probably three of these a day.

• 34:35Speaker 00

  Wow.

• 34:36Speaker 01

  Yeah.

• 34:37Speaker 00

  That's, that's huge. And, you know, a lot of, a lot of the backlash on, on loops is, is just that it's expensive. How, how, how have you, with, with Devon specifically, has it been like absurdly expensive? Like, what are we talking?

• 34:52Speaker 01

  No. So, well, so here's the thing. So last month I spent probably 20 grand in tokens, which is just too much. Like it's not viable. I think all of us are in a place where we were getting to the spot where it's realizing, okay, for real engineering work per employee, you're looking at probably five grand a month. Like, that's probably where we're going to shake out here. And anything more than that, you really need to figure out model routing. And so thankfully cognition knows this and everyone's figuring this out. Like, yeah, whether it's cursor cognition, I would say factory AMP, all of the independent agent labs understand like we, our customers have a limit. Like they're not going to pay infinite token budgets.

• 35:38Speaker 01

  And then you have the, this is me ranting a bit, but it's like you have Claude Code and Codex who are just severely, you know, fun, they're, they're funding tokens and making it seem cheap, but this is not going to work long-term. And so what I do is basically I have learned how to use the right model for the right task. And so these reinforcement loops, they use what's called SUI 1.7. It's a fine tuned model that cognition has specifically built for coding. It's super cheap compared to, you know, Opus 48, you know, GPT 5.6. And so a lot of these loops happen with this, you know, cheaper, you know, finely tuned model. And we're just going to see more of that.

• 36:20Speaker 01

  So you're talking, I don't know, it, you know, actually I could probably, I could probably look and dig it out, but you're probably talking like five bucks a session on that.

• 36:27Speaker 00

  Okay, that's not bad.

• 36:28Speaker 01

  I mean, and if you're not willing to pay 15 bucks a day to improve one of the core, you know, feature sets of your, like, what are you doing? Like either it's not a real company or you don't care.

• 36:38Speaker 00

  Yeah.

• 36:40Speaker 01

  So spend the money like, and, and then, you know, figure out the, the system to reduce your token costs. I will say, this is why I don't understand why anybody would build their engineering on a frontier lab stack. Like, I'm just going to call it out. Like if you are, if your whole engineering, you know, motion is happening inside of Claude Code or inside of Codex, what are you doing? Like, because they are not incentivized to make it reasonable for you long-term. Right? They're going to lock you into their models, to their process. Whereas if you use an independent agent lab, you know, like an AMP, like a Devon, like

• 37:22Speaker 01

  a factory, like sort of a cursor, but they're kind of weird now. Those systems are going to optimize for affordable engineering long-term.

• 37:33Speaker 01

  And I don't think you want to be locked in to Anthropic or, or OpenAI, you know, solely.

• 37:41Speaker 00

  I mean, to be clear, you can still use their models within some of these products, right? So you're not abandoning ship.

• 37:48Speaker 01

  No, it's actually, so it's, it's yes and, right? So the beautiful thing is if you pay, you know, an independent agent lab, like a factory, like a cursor, like a cognition, like an AMP, you know, et cetera, they're incentivized to figure out, how do I give you the best results for the lowest price? So they'll, they'll say, okay, we're going to use GPT 5.6 soul on extra high for this type of task. And then we're going to route, you know, to a, you know, to a SUI 1.7, you know? And then we're going to have it check with a Fable. Like they're going to figure out that hard stuff.

• 38:21Speaker 00

  Yeah.

• 38:21Speaker 01

  And the other thing I will say is if you're trying to build a software factory for yourself, stop. Like it, you know, the reason why you see, you know, Ramp Launch Inspect, which is their custom in-house agent, is because they got to a size where they had to build their, their software factory in-house. And if you get to a certain size, you will do that. But now if you're a one-person shop and you're just building something, it's just you, you can probably get away with this $200 a month, you know, angle on a pick your frontier lab. But as soon as you graduate into, okay, we're building a real product, we have product market fit, we're going to start hiring people, you got to build a software factory. And at that point, you want to be on an independent agent lab.

• 39:05Speaker 01

  Like, otherwise you have to build it yourself, which is stupid, or you're going to pay and be locked into, you know, a walled garden. And you can't do that.

• 39:17Speaker 00

  The, the analogy I use for it is, you know, working with an independent lab is sort of like a.

• 39:24Speaker 00

  Like working with like a mortgage broker or travel agent, like they're going to make a lot of calls to make sure to get you the best price and stuff like that.

• 39:34Speaker 01

  That's a great analogy.

• 39:35Speaker 00

  Yeah, to me, I use both, you know, I I'm in Claude code, but I'm also, I use Cursor personally. Although Devin, I'm interested in, I might, if people are interested, I might just do like a whole breakdown on Devin if people, if people want to, want to do that.

• 39:51Speaker 01

  Devin's so good now, you know, I just, it's so good. It's amazing. But I will say, I do use Codex, right? The, the, the Mac app is so beautiful, like, and there's, and they are subsidizing the tokens so much that if you want an agent on your machine to just to do really cool, helpful stuff, the open browser tabs and create Google Docs and check your email and, and do all this information work. And, and basically get it for free and you're paying 200 bucks and you get almost infinite tokens, like awesome. But don't build your company software factory on that, like it, I just think it's bonkers to do that.

• 40:33Speaker 00

  We've covered a lot today. Is there anything else you wanted to cover?

• 40:38Speaker 01

  I feel like I'm just vomiting, you know, all, all the stuff in my head.

• 40:42Speaker 00

  Well, no, I, it's important that, like, this is what I wanted. I wanted you just to vomit everywhere because like, it's sort of like advice that you, you, you would give to your, to your nephew, I think, or 22 year old. You know, a 22 year old fresh out of college, it's like, what are the minds I need to know in terms of managing agents? So is there anything else that people need to know? Cloud agents, automations?

• 41:11Speaker 01

  Yeah, cloud, automations, like those are the two that I want people to walk away from. I guess the third is, is really, you will be using a software factory, like, and what I mean is the agents are going to be writing a hundred percent of your code, reviewing a hundred percent of your code, shipping a hundred percent of your code, like, that's where we're going. And so the, the more you can get there, the, the better. Thing one. Thing two, I think, you know, thing three is, it, it is important to build

• 41:44Speaker 01

  a credible reputation for yourself, right? And I will say, you know, the work I've put in over, you know, the last 20 years on X, like, it really pays dividends. And what I mean by that is, if you, if you share what you're learning publicly and you generally give, like, write very helpful articles on X. And obviously, the algo really does reward articles still, so you should be writing articles on X. You should absolutely be on X, no matter how you feel about Elon.

• 42:17Speaker 01

  Because sharing your knowledge really does build up credibility, which will, will pay dividends. And I don't mean just like getting paid to post, I mean, the, the, the network and the relationships. I mean, the reason why I'm on the show is because we built a relationship over X. And so I would encourage people, take the time to share what you're learning. It doesn't have to be super polished, but get out there and be useful to people and it will open up doors. So you know more than you think, just get out there and share it. And, and you'll reap the rewards, not immediately, but eventually.

• 42:51Speaker 00

  I mean, even if you don't know, you can just say, I don't know, and that's your, and that's your strategy, right? And that's interesting, too.

• 42:58Speaker 01

  Amen. And if you don't know, then gas an agent to help you.

• 43:01Speaker 00

  Yeah, so. I mean, my original co-host of this show was Sahil Bloom. Sahil Bloom worked in private equity. And during COVID, he just started writing about topics he was interested in, like, he would be, like, just finance topics or, you know, a particular individual who was interesting. And it like, it, it, it read as like Wikipedia articles, but really focused on Twitter specifically, like optimized for Twitter. And, and that was his thing. And he wasn't saying, I know everything. He was just saying, like, I'm learning this thing. Here's the thing I'm learning. He grew in his audience, over a million Twitter followers, wrote a bestselling, New York Times bestselling book, started companies on top of it, raised a fund on top of it.

• 43:46Speaker 00

  So anything is possible for sure on, on X.

• 43:52Speaker 01

  Yep, just, just get out there and do, do, do, do, and learn.

• 43:57Speaker 01

  You know, there, there's no try, there's only do, as Yoda says, right? So.

• 44:01Speaker 00

  Totally. Ryan Carson, I feel smarter, and that's, that was my goal.

• 44:10Speaker 00

  And speaking of X, you're a wonderful follow. I'll include the links where you can follow Ryan across the internet, including X. He's a must-follow, so please do that. Thank you for coming on, sharing your generous thoughts, incredible setup. And is there anything you want to leave people with?

• 44:32Speaker 01

  No, thanks, Greg. Thanks for having me on the show. I appreciate you putting out the constant knowledge, constant knowledge and sharing. It really makes the internet a better place. So happy to be here and thanks for having me.

• 44:42Speaker 00

  All right. I'll see you next time. Take care.