Always Bet on the Models: How Tasklet Puts the Agency in Agents, with CEO Andrew Lee

Today Andrew Lee, founder and CEO of Shortwave and Tasklet.ai, joins The Cognitive Revolution to discuss building a new AI agent automation platform that replaces traditional workflow tools with conversational, long-lived agents, exploring how betting on rapidly improving model capabilities enables more flexible automation across thousands of business integrations while navigating the challenges of reliability, cost management, and the emerging paradigm of virtual employees. Shownotes brought to you by Notion AI Meeting Notes - try one month for free at: https://notion.com/lp/nathan Agent-first vs. workflow-first approach: Task Lit fundamentally differs from traditional automation tools by using a single AI agent that handles all tasks through natural language, rather than building explicit workflows or state machines. Speed is the only remaining moat: In the AI era, traditional competitive advantages are evaporating so quickly that the only sustainable advantage is moving faster than competitors. Small business marketing as near-term target: The technology is approaching "best available human" performance for tasks like small business marketing, where hiring quality human help is difficult. Sponsors: Google Gemini Notebook LM: Notebook LM is an AI-first tool that helps you make sense of complex information. Upload your documents and it instantly becomes a personal expert, helping you uncover insights and brainstorm new ideas at https://notebooklm.google.com Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Linear: Linear is the system for modern product development. Nearly every AI company you've heard of is using Linear to build products. Get 6 months of Linear Business for free at: https://linear.app/tcr Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm excited to welcome Andrew Lee back for his third appearance on the podcast. Our last two episodes together were about Andrew's AI email assistant product, Shortwave. And while Shortwave does remain one of the few AI apps that I use daily on both laptop and phone, the occasion for this conversation is the launch of Tasklet, a new AI agent platform that combines the natural language interaction and open-ended nature of chatbot assistants with the goal orientation and tangible output of automation software, and which has Andrew and team even more excited. In practical user experience terms, Tasklet invites you to describe a task or responsibility

• 0:44Nathan Labenz?

  to automate, and then figures out for itself how to do it. As its boss, you can then review its output, or its full step-by-step process, and offer feedback to guide its improvement going forward. I'm honored and grateful to say that Andrew will be sponsoring The Cognitive Revolution to help support Tasklet's launch. And while we will always be transparent about any financial relationship with guests, whether that's sponsorship, investment, or otherwise, I can sincerely say that we are making no compromises on content quality. In fact, Andrew's belief that speed is the only moat for AI startups, and his resulting willingness to share everything he's learned with remarkable transparency and technical depth, have made him a consistent favorite guest among AI engineers.

• 1:27Nathan Labenz?

  What's more, having tested Tasklet myself for a full month before launch, I can honestly say that it's much more like the agent experience that I had expected OpenAI to launch at their Dev Day than what they actually did launch. And because it delivers frontier capabilities in a radically accessible form factor, I recommend it both to people who are just getting started with AI automation, and also to experienced AI engineers who want to experiment with new concepts and are open to the AI's own ideas for how to get the job done. In this conversation, we explore every angle of how Andrew and team are building Tasklet. We discuss Andrew's plan to always bet on the models and why he believes that open-ended agentic systems will ultimately prove more reliable than traditional step-by-step automation software.

• 2:10Nathan Labenz?

  Tasklet's two-tier architecture, which uses high-level agents to maintain instructions and spawn subagents for individual runs, and how this enables both recurring automation and ad hoc assistance. Real-world use cases ranging from email triage and marketing automation to commodity trading and venue management. How Tasklet manages context across long-running agent relationships, including sophisticated approaches to memory, context compaction, and the use of SQL databases to store agent state. Their tool integration strategy, which combines more than 3,000 pre-built integrations with the model's own native ability to figure out how to use MCPs and previously unseen APIs, plus computer use as the final fallback.

• 2:54Nathan Labenz?

  Fascinating technical details on prompt engineering, including why they're moving away from pre-generated instructions and toward just-in-time instructions to avoid conflicting directives. Why Andrew still runs Tasklet on vibes rather than evals for model selection, and why he's confident that Anthropic's models remain the best choice for multi-turn agent interactions. And finally, his perspective on AI's overall trajectory, which I'd say represents the mainline view among both Frontier Lab employees and the most connected app builders, namely that progress has not faltered and is absolutely expected to continue for some time to come. To me, while the capabilities frontier remains jagged, and today's models can, of course, only handle a couple hours of work at a time, Tasklet does feel like a preview of the virtual

• 3:39Nathan Labenz?

  employee future. When you can maintain an ongoing relationship with an agent, give it high-level feedback on its work, and watch it incorporate that feedback into future runs, you start to experience something genuinely new. And Andrew's report that some users are already giving their Tasklet agents their own dedicated email addresses, and in some cases even names, suggests that this paradigm shift is indeed already beginning.

• 4:04Nathan Labenz?

  If you're intrigued, and certainly if you listen to this episode, I think you will be. Andrew invites you to try Tasklet for free at tasklet.ai. And then, when it's time to upgrade, use code COGREV to get 50% off of any paid plan. With that, I hope you enjoy this deep dive into the strategic thinking, product management, and technical implementation of a frontier AI agent platform with Andrew Lee, founder and CEO of Tasklet.

• 4:33Nathan Labenz?

  Andrew Lee, founder and CEO of Shortwave, and now also founder and CEO of Tasklet. Welcome back to the Cognitiv Revolution.

• 4:40Andrew Lee?

  Thank you for having me.

• 4:42Nathan Labenz?

  I'm excited for this conversation. This is the third time we've done this, and I've learned a ton each of the last two times. I'm sure that's going to be the case again today. The occasion is, there is a new product out from you guys. It's called Tasklet. I've had the chance to play with it for the last month or so as you've been developing it and refining it. And I think it is a pretty cool paradigm that kind of blurs the lines between what many people are familiar with in the form of a chatbot on the one hand, and then also in the form of kind of a structured workflow or agent on the other hand. This is something that kind of sits in the middle and can kind of play both roles, and I have found it pretty cool to play around with.

• 5:27Nathan Labenz?

  So excited to dig into every aspect of it with you, but maybe for starters, tell us about Tasklet at a high level. Like, how did you decide to build another product? And how did you hone in on this vision?

• 5:41Andrew Lee?

  Yeah, so this is our second product after Shortwave. And basically, what happened was earlier this year, we got really good at hooking our AI in Shortwave up to other products. So we added MCP, and people were doing like these interesting workflows where they were like taking data from their email, and they were sticking it in their Notion or in their Asana and updating their HubSpot automatically. And people really liked this, and it worked really well. And they said, hey, wouldn't it be nice if you just did this automatically before I got up every morning? So I didn't have to sit on my computer and like run this prompt every day to like update my data. And we said, yeah, that'd be cool. Let's build that. So we started making that, and we very quickly realized that

• 6:22Andrew Lee?

  Hey, if this thing is running when you're not at your computer, having a tight UI integration between your AI agent and your email doesn't make any sense because you're not even in front of your computer when this is happening. And so what if we took advantage of that and built something just much for general purpose that was great for doing, you know, email sync with other platforms, but also just like general automation. And so we took a bigger swing and said, hey, we're going to try to build something that is designed to be your general purpose AI agent automation tool. And we spent, you know, starting basically end of May, early June, all the way up until last week, building, testing, iterating, and we finally launched it. So it's live. Check it out. tasklet.ai.

• 7:07Nathan Labenz?

  Yeah, it's cool. I think that the thing that I have really enjoyed about it is because I, you know, interact with people from all walks of life who are interested in AI. Fairly often, they're like, how could I set something up that would do whatever for me? And the friction to do that kind of stuff has been fairly high, right? Even with AI as a, you know, something that can sit in a node in a traditional automation software platform, there's still like a lot for people to figure out if they're not the kind of people that are used to getting really structured in their own minds about how to break a task down into its steps and, you know,

• 7:51Nathan Labenz?

  how to set all that up in traditional software. And so I have found myself going to Tasklet as like the first place and being like, okay, start here. Just start by saying what you want and then kind of iterate on the thing from there. It's a really interesting split where you have kind of the mainline agent itself is like an entity that you can chat with. And on that level, it feels more like a chatbot. But then it also like follows your instructions and, you know, you can set it up to run every day or on various triggers and you can, you know, tell us more about the particulars of that. But then you also have this like second tier, which is all the runs that the agent has done of this particular task. And I think that one-two punch, both of which are just very sort of

• 8:35Nathan Labenz?

  General purpose, like very, you know, very natural language driven, not really requiring any of the nitty gritty setup that people have become used to in some ways, but also have like probably been deterred by in many situations is a pretty interesting new way to look at what an AI agent could be. Tell us more about that.

• 9:00Andrew Lee?

  Yeah, traditionally, the way people do these sorts of automations is they use a workflow product, something like Zapier or N A N or like the Agent Kit product that OpenAI put out is I would call a workflow product. And I look at workflow products and say, this is, this was the right way to approach this from a traditional software engineering standpoint, like a year or two ago, when models were smart, but not that smart. And I think we've learned the last few years that you should always bet on the model. The models are always going to get smarter. And the right thing to do is to find ways to give those models more agency over time. And I think the reason people have been shy about doing workflow

• 9:42Andrew Lee?

  automation, like fully agentic, all the way down, is because they didn't really trust it to be reliable. They didn't feel like the models were there yet. But I think that was the time. I think we've gone through that transition from, you know, you have a workflow that defines step one, step two, step three, step four, and maybe there's some LLC calls inside it to what if you just let the agent plan the whole thing. And the advantages you get out of this are tremendous, because let's say you run into an error state in a workflow product, if you don't have a way to handle the error state, it just breaks. In an agent product, it just kind of figures it out and works around it. It can handle nuance much better. And also, as you mentioned, it's just a whole lot easier

• 10:23Andrew Lee?

  to set up. And so we've been trying to say, hey, we're going to bet on the models. Some of our workflow competitors have kind of done a hybrid solution where they say, we're going to have an agent for the purpose of creating the workflow. So like if you if you use string, this is the approach of string, but the output of that is still a workflow. And so it's fairly constrained at what it can do. And we're saying, hey, not only are we going to have the the setup portion be an agent, but the actual implementation is going to be an agent as well.

• 10:52Nathan Labenz?

  Yeah, it's funny because that has kind of been the assumption on my part of what I expected things to trend toward. And yet I do find myself quite liking the form factor that you've created. I guess to dig into a little bit more like trusting the models and what you mean by that. I have this one slide that I use all the time in presentations that's just sort of the minimalist diagram of an agent. Like, what is the, you know, sort of most abstract form, right? Everything, all the detail kind of stripped away. And it's basically just a box around an LLM and some tools. And that's like your core agent. This thing is then given a task. The LLM can reason a little bit, can use the tools.

• 11:35Nathan Labenz?

  When it uses tools, something happens in the environment, it gets feedback, and it just keeps kind of iterating on that basis, right? This is the classic, like, LLM in a loop is basically your minimalist agent. So when you talk about like betting on models, you know, the most extreme version of that would be like, just throw that very minimal scaffold up and, you know, let the models have at it. You've done a lot more than that, I'm sure. And I can see, you know, some of the fruits of that labor. But how do you think about like, what adds the most value beyond that super minimal agent scaffold that's, you know, kind of a cartoon diagram?

• 12:13Andrew Lee?

  Yeah, what I mean bet on the models, what I really mean here is, what is the, you know, what is really in control of what's going to be happening, right? So one approach is to have a workflow where the thing is that is in control is traditional software, and you as the user define the steps, and it goes through some sort of flow chart. And maybe within those boxes, there's some LLM calls, but the overarching control is done by traditional software versus if you put the model in charge, the model's in charge, the model makes the big decisions. And then, within what's happening with the model, you might have traditional software to like, execute the tools. And so it kind of inverts the problem. Rather than having software wrapping LLMs, you have LLMs wrapping software.

• 12:57Andrew Lee?

  So that's when I say, you know, we're betting on the model, that's what I mean.

• 13:02Andrew Lee?

  Either way you do it, there's still a lot of traditional software that has to be built. And I'd say the, you know, the core of what the agent is, I think I totally agree with you. It's basically just calling an LLM in a loop. Where you, you know, have the LLM spit out some tool calls or resolve the tool calls. You called it LLM again, and it kind of reasons through things. But yeah, there's a lot that gets built around that. I'd say there's kind of three categories of things that we, like, big categories of things that we bring to the table. The first is connections. So we hook up to, uh, not like some pre-enumerated set of tools, but we hook up to everything on the internet. Like any service you want to connect to, we can connect to. And all of the work of figuring out how do we connect to Gmail or Notion or your random enterprise API?

• 13:47Andrew Lee?

  Or, you know, a computer that can access LinkedIn or some MCP server. All of that work is our own code. Um, and there's a lot of work trying to, to take all of these heterogeneous ways of talking to the internet and putting them into something that is like consistently LLM can reason about and use well. So that's category one. Category two is triggers. So the, you know, all the stuff that makes the LLM get sort of run automatically and have those runs be, you know, encapsulated in these subagents that have sort of limited permissions, uh, where you have some control over them, um, is, is a big, big area of work. And then category three is, it's a very nascent part of our product, but it's all of the sort of team collaboration and sharing features.

• 14:28Andrew Lee?

  So, um, we're really targeting business operations use cases where, uh, you know, it's not just you on your personal stuff. It's like you're automating some core business process. And we see that as very much a team sport. And so there's, there's a few features in there now, like we have the ability to share an agent, but we're going to be doing a lot of stuff to provide, uh, sort of like oversight and management and auditing and cost controls and things at an organizational level.

• 14:50Nathan Labenz?

  But I, I will say prior to playing around with Tasklet, I had the working assumption that that...

• 14:59Nathan Labenz?

  While somewhat costly in terms of time and cognitive overhead to get set up, would bring me to a place of higher reliability. So I sort of assumed that there was an inherent trade off between accessibility and reliability. But you're making an interesting point that when the AI itself has more room to kind of choose its own adventure, that's my term for these less structured agents or choose your own adventure agents, that they do have in a way also the opportunity, at least to have a higher level of robustness to unexpected stuff because they can route around obstacles that they find at runtime that weren't anticipated by the person who otherwise would have been building out these sort of block by block control flows.

• 15:46Nathan Labenz?

  How do you think about that trade off? And do you think that it is in fact like more reliable on net? Or is it a different kind of reliability?

• 15:55Andrew Lee?

  Yeah.

• 15:56Nathan Labenz?

  What are the upper limits of this? Like, could you put something like this in production as part of a production app? How far do you think this goes?

• 16:04Andrew Lee?

  Yeah, I totally agree with your point about the the upper bound of reliability is not what you get from workflow products today because the real world is messy and things break and those tools break all the time and agents can potentially solve many of those cases. So I do think you need to look at this holistically. I think today with the models you have today, you're probably going to have in most business applications, a somewhat less reliable solution with today's models. But as I said, always better than the models. I give you six months before that's no longer the case. And I think we've been able to look back on the last few years and see that these predictions really, really do come true. So we're looking ahead and saying today for many applications, it is reliable enough

• 16:49Andrew Lee?

  and there's a lot of other advantages and in the future it'll probably be more reliable. The other thing I want to note is there is a lot that the models can do to provide reliability and sort of guardrails around their processes. So for example, today we use the LLMs to try to figure out types. So we dynamically create some of our connections. So we use the Try to figure out like, hey, what are the type restrictions that should be on this? And then we enforce those in code. And so if, for example, what you want is you really want the LM to follow a flowchart for some portion of of the project, right? We could build the ability for you to tell the LM, hey, you know, during this phase, you must do these steps in order. And the LM could actually create its own guardrails and enforce that through code if it wanted to.

• 17:31Andrew Lee?

  Um, so I don't, I think if the model is wrapping the code, the model can then construct constraints in those codes to in that code to enforce the, you know, their liability goals that you want. Haven't built it yet, but I think we can. So, um, I think the future is not going to be, you know, forever. Here's the, the quick, easy, unreliable way. And then here's the harder, more reliable way. I, I think it really is going to be better in basically all scenarios, uh, in the not too distant future.

• 18:00Nathan Labenz?

  Yeah, maybe, um, it could be Sonnet 4.5 new, or, uh, maybe we'll have to wait till Sonnet 4.7. But, um, since these things are coming at us pretty fast.

• 18:15Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:21Nathan Labenz?

  This is something we talked about last time and your response was quite interesting.

• 18:27Nathan Labenz?

  Basically, as I recall, it was like very vibes based. How do you eval things these days? Is, have you, I mean, this debate has continued to rage, as I'm sure you're well aware, since the last conversation. Have you updated your position? Are we still, still running on vibes?

• 18:44Andrew Lee?

  Still running on vibes. Yeah, I think what, what we've figured out is in the current, you know, market situation, the anthropic models are the best at what we do. And there's, there's really no question there. And if you're looking at evals from the standpoint of we're trying to decide whether to roll forward on a new model, we kind of have no choice, right? We can't really switch off anthropic right now because there's no viable alternative. And when a new anthropic model comes out, the pressure from our customers to release that thing as soon as possible is quite tremendous. Everybody wants it right now. And so, you know, if we had some evals and the evals, you know, weren't working out the way we wanted, we're probably still going to roll out that new model, right? We might tweak some props and stuff to, to try to make it work better, but we're probably going to still roll out that new model.

• 19:27Andrew Lee?

  So we've just sort of accepted that like, we're going to be on that anthropic for a while. And we're always going to roll the new model as quickly as it comes out.

• 19:36Andrew Lee?

  So we need some approach that works in that environment. The other, the other thing to consider is our product is changing so fast that anything that we put in that could constrain our ability to move quickly is going to come with real business costs. So, you know, the, this entire product is only, we only started writing code in June. Um, it's evolved tremendously over that time. It's going to continue evolving tremendously and our users are very clear that the thing they want above all else. And I, I mean this above, you know, above reliability and security and, you know, all the normal things you want is they want the smartest, most capable thing. Um, and they've been, you know, I've had lots of calls with people where, you know, they're asking about SOC two and we're like, yeah, we don't have SOC two for this yet.

• 20:18Andrew Lee?

  Right. And, you know, if you had to choose between SOC two or it being a little smarter, basically it's always like, well, don't tell compliance, but I want it to be a little smarter. So that's really where our focus is, is like moving quickly, being smart. Um, and evals is, uh, is not beyond our own dog, so he's not part of that right now. So we do do a lot of testing, don't get me wrong, but it's all like, we have a whole bunch of triggers that run for our own internal usage. And we basically roll stuff out to our internal team first. We give it some time, see how it goes, and then if it feels good, we roll it out wider. Um, and in short wave, we do, um, uh, basically experiments based rollouts. So we allow our users to like opt into things. We look at retention rates and the new features and roll them out.

• 21:00Andrew Lee?

  We're going to do something similar in task that we're like, you know, we'll roll out a new version. We'll let our users opt into it. We'll see what retention looks like and then we'll flip it on for everybody.

• 21:09Nathan Labenz?

  I guess just to ground this out a little bit, you said it like anthropic models are clearly the best for what you do. How do you test that? I mean, are you literally just doing like Andrew's top 10, um, agent use cases and running them head to head with GPT five?

• 21:27Andrew Lee?

  Yeah, it's, I mean, it's vibes, right? So we, we've built our system so it's easy to swap in different models to, to, to test. And we try the different use cases. And I think, um, I think what you find is all of the models do a pretty good job at answering the initial question you have. In fact, some of the models may do a better job. If your goal is to like have one question, one answer, you know, you might find that OpenAI models do better. But if you want to have something that is LM call, tool call, LM call, tool call over a long sequence, which like all the stuff you want to do with our product is, is that type of thing. Anthropic just sort of handles that iteration better and, and that over, you know, 100 iterations, that really adds up if it's slightly better at each turn. So

• 22:11Andrew Lee?

  we've, yeah, it's all been sort of hand testing different models, but ultimately for that sort of long iterative processes, this is, it's clearly the best. And I, I think the market agrees with us. If you look at, you know, all the other folks that are doing stuff like us, they're all heavily using Anthropic models.

• 22:27Nathan Labenz?

  What would you add? Of course, everybody who listens to this feed is very familiar with the meter task length graph at this point. And we were all waiting for Sonnet 4.5 to, you know, be planted on that graph. It did come in a little lower than GPT-5, right on trend with the curve, but a bit lower in the meter estimation than GPT-5. How would you account for that difference if you could, or like, you know, do you, and I guess the other way to think about that is they, you know, put us like just over two hours now in terms of

• 23:05Nathan Labenz?

  the size of task as measured in the time it takes a human to do it, that the models can now handle like 50% of the time.

• 23:15Nathan Labenz?

  Do you accept that analysis? Would you complicate that analysis? Is there, you know, is there some part of that? Is there some reason that you are coming out with a very different conclusion than what they are measuring? Uh, because I think right now the prevailing, you know, notion would be GPT-5 is a bit ahead.

• 23:30Andrew Lee?

  Yeah, I, I don't know why that difference is there. My, my cynical take is, you know, some people are, are playing harder to hack the metrics than others. Um, but I don't actually know what the reason is. Um, uh, but I

• 23:48Andrew Lee?

  You know, I look at our actual sort of practical real world testing and my experience has been it's better and

• 23:55Andrew Lee?

  the data point I would point to that that says, hey, you know, this, there is something real here. I'm not just making this up is pricing. Right? So if GPT five was as good at these use cases as Sonnet is, it wouldn't be less than half the price. And at less than half the price, you'd think everyone would be switching off and they're not. And when they, you know, when Anthropic came out with four, five, they didn't lower the cost, the price, even though the OpenAI had lowered the price. So the fact that the pricing is holding up tells me that people are choosing it. And I think it's just because, you know, ignore, ignore what the metrics say. The the real world utility of it is just better.

• 24:38Nathan Labenz?

  How do you think about the size of tasks? When I have looked into runs of things that I've done, I've routinely seen like dozens of steps, which are, for whatever reason, the tasks I tend to come up with are like, help me with AI research, not for whatever. I know why that is. It's becoming overwhelming and I need all the help I can get. So it's a lot of stuff that's like, search, reason, search, reason, search, reason, search, reason, a bunch of times. And then finally, compile a report for me, send that off, update memory. And that's basically the anatomy of most runs of the tasks or of the, you know, of the recurring tasks, the agents that I've created so far. How

• 25:21Nathan Labenz?

  does that compare to the frontier? Like, are is dozens of tasks kind of toward the upper limit? Are you seeing things that are substantially more than that? Do you think about them in terms of time it would take a human to do them or some other metric? Like, how do you even conceive of what the how to measure, you know, the size of what the agents can do?

• 25:42Andrew Lee?

  Yeah, to be honest, yeah, we don't make any effort to try to estimate the the time it would take a human to do this. We're a little team. We're just trying to build a thing that that people will pay for. I'll leave to the researchers to try to figure out how that relates to to human time. We look at things in terms of of turns because that fact that, you know, correlates with our cost. And we absolutely do see people with tasks that are spanning a very large number of turns. And we actually, this is something we need to change. There's a 50 turn limit per. Question basically in our product and that initially was just set to like keep a bug from costing us infinite money. And we're hitting this all the time now and and we need to bump that. And I think the number one place we're hitting this is computer use.

• 26:25Andrew Lee?

  Where, you know, if your task is like, hey, you know, go to LinkedIn, find these 10 people and send them all a message, you know, every operation there is going to be a turn. Uh, and sometimes multiple turns. And, uh, so I think computer use is a place case where just like the number of turns just absolutely explodes.

• 26:42Nathan Labenz?

  Yeah, that's funny. I have been doing that, uh, a bit as well, getting, uh, specifically, there's a event coming up in Detroit. Where I was invited to either either give just a solo presentation or maybe do like a live podcast recording in front of a live audience. And then the question was like, well, who's the guest? And so I I'm specifically having task led go comb through. AI leaders at various organizations in Detroit and pull back, uh, you know, lists to to me for people that I should consider inviting. Which brings me to a great question. It seems like in the metrics. Quad 4.5 Sonnet was a big step up in computer use. How did that feel to you in your vibes based assessment?

• 27:28Andrew Lee?

  It it seems really good. Um. I'd say the the big blocker for more computer use for us right now really is just cost and speed. Um, it is. Tremendously extensive to move a computer around by screenshotting. And so, uh. Yeah, I'd say when computer use fails for our users, it has less to do with like the intelligence of the model and more to do with either they hit that 50 turn limit. Or like they just can't afford to do it anymore. But yeah, I found it super impressive. We've looked at using. Other models for specifically for computers for costs. So like we're looking at using Gemini and having like a sub agent to handle portions of that. But the the downside of having a sub agent run and then to a capturing some.

• 28:10Andrew Lee?

  You know, distilled version of of the data and putting it on the main agent is just get less intelligence overall. And so we've we play with that a bit, but. Uh, so far the strategy is just spend a lot of money on, uh, Sonnet. We're gonna, we just, we just, uh, rolled out Haiku. And so, um, we've been doing a little bit of testing. I think it's too early to say how how well Haiku is going to do with computer use, but. Uh, that could be a big factor for for cost for us, but generally super impressed with with the capabilities for computer use.

• 28:38Nathan Labenz?

  How do you think about what kind of computers to give to the AI? It seems like we have, on the one hand, there's sort of just a browser is like one paradigm. And then on the Grok 4 launch, I always think back to Elon talking about setting up like, you know, power workstations with all the same, you know, high-end software that the engineers at Tesla and SpaceX use. And it seems like you're somewhere in the middle right now where it is not just a browser, right? You have like a full operating system, you know, VM type of thing that the AI can use. How do you decide like how much of a computer to give it? And how much can it do beyond the browser today?

• 29:22Andrew Lee?

  Yeah, this is probably the most active source of discussion for our team right now. We have, actually have everyone in the team is flying out next week and this is going to be a big focus of that, that discussion is what direction we want to take this. Right now in the product, you, every agent has its own SQL database. It has its own code execution environment. It has some like very limited file capabilities and you can optionally spin up a full Linux VM and connect to that Linux VM and like use a browser, use a file system in that VM. We used to have Windows support. That caused some problems for us. It was expensive, the, you know, the wake up from sleep time was like really bad.

• 30:06Andrew Lee?

  There was, there was some reasons it was harder and it was also, no one, it was pretty rare for people to use the benefits of Windows besides stuff in the browser. So we switched to Linux, which I think has been working much better for folks. We've considered doing Mac for the, for the, for the usage. Going forward, I see a couple of big goals. One is we want to give it terminal access. So there's a lot of operations that like if you use Claude code, right? It's, it's shocking how much Claude code can do outside of programming since it has the ability to access the console. So for example, like we have a weird number of people opening up agents and then being like, you know, hey, I upload this file, like please convert it from, you know, this video format to this video format.

• 30:48Andrew Lee?

  And I'm like, you know, I don't know why you're asking AI do this, but we totally could, right? We could run FFmpeg and like, you know, do that sort of processing. So I think we want to give agents, you know, a shell and let them run command line things. I think we want to give them a file system. And I think we want all of those pieces of the computer to work well together, right? So for example, today you can run, we can write and execute code. But that code can't access your database. That code can't access your connections. It can output data in the LM. But you know, what if you had all those facets of the computer? The, the file system, the database, the code execution environment, the shell and the UI, all able to talk to each other?

• 31:33Andrew Lee?

  I think that could be tremendously powerful. Doing that well is really hard for a variety of reasons. Cost is a big factor, right? You don't actually want to have, you know, you could have a hundred agents in our, in our product. You don't actually want to have like a physical, a hundred physical machines per user. That thing will be pretty expensive. This is an area actually where there's a lot of startup activity. So there's a lot of people that figured out that you need specialized cloud infrastructure for doing this. I was just talking to a startup this morning called Blackcell. That seems pretty cool.

• 32:07Andrew Lee?

  But yeah, a lot of discussion here, a lot of excitement. I think big things are coming. Watch the space.

• 32:14Nathan Labenz?

  What do you think of the requirements there? Like it just seems that the main one would be like, you sort of want to be able to suspend the VM and kind of put it into sleep mode, but like keep its state, kind of like my Mac. When it runs out of battery, whatever, it comes up in roughly the same, usually in roughly the same state that I left it. Is that the, the big thing? Or what else is kind of missing from the tools you have available right now?

• 32:38Andrew Lee?

  Yeah, I think cost is the is probably the number one issue where, yeah, you want to have the illusion of every agent having its own computer. And you want that computer, when you're using it, to be powerful. But you don't want to pay for any of the milliseconds you're not using it. And that includes not just compute, but also storage. And that can get complicated, right? You don't want to, you know, have every agent have to have a store, a store, a copy of the entire OS, for example. So how do you get the illusion of every agent having its own computer without having to pay the costs of every agent literally having its own computer? I think that's one big factor. I think another big factor is you want to make sure that those computers don't kind of get messed up over time.

• 33:19Andrew Lee?

  So you want to give some thought to, you know, what aspects of this thing are configurable and what aspects of this thing are not configurable. You wouldn't, you wouldn't want the agent to like, you know, mess up some config and then forever after that be unable to like use its computer. So that's something we're thinking about it too, is like, you know, reliability for this thing. And besides, I don't know, man, like, the, the requirements here, I think, are super unclear. This is a very uncharted territory, so we're, we're figuring this out one day at a time.

• 33:49Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 33:55Nathan Labenz?

  Let's talk about use cases for a minute, because I go very deep into all the technical stuff. What use cases are you seeing maybe that are like on the frontier of what the models can or can't do, or that are just like really creative, you know, things that you think people should be taking inspiration from? One thing that I have honestly had a bit of a challenge with in using any AI product that's meant to do recurring tasks for me is, and I don't take this for granted, I think it's like a sign of great privilege in many respects, I don't really have that many recurring tasks. I mostly am like chasing my curiosity on a day-to-day basis. And so a lot of times I feel like I'm like, okay, well, what should I be recurring that

• 34:39Nathan Labenz?

  I'm not, you know, because I'm, I'm too lazy and maybe an AI, you know, could, could take me there. So, yeah, I mean, take all the time you want on use case, I need inspiration.

• 34:49Andrew Lee?

  So here's an interesting thing that we figured out earlier on. So the pitch that we give you is that we help you with recurring tasks and we do. But the first version of Tasklet, it was set up so that there was sort of a Tasklet setup phase. And then, once you got the automation running, it just like switched into running mode and you couldn't talk to the main agents. You were like either designing the automation or running the automation, but you weren't doing both at the same time. And what we discovered is that people had like, for example, they configured like an email triage agent. And they'd hooked it up to their Notion and their Asana. And they've given it all these detailed instructions around like what types of emails they care about and stuff like that. And they're like, hey, I did all this work to create this agent. It's awesome writing its automation, but now I just want it to write this email for me.

• 35:34Andrew Lee?

  Why can't I talk to it more? And we said, oh, well, you should go make a new agent for that. And it's like, well, I don't know, then I have to configure it again. I have to hook it up to all the same stuff. I have to get all the same prompts. Like, I just want to talk to this agent. And so we realized that, hey, what we need to do is we need to enable on any one of these individual agents, both the trigger tasks, but also the ability for you to keep chatting with them. And so that's why the agent works that way today. Like while it is off, you know, waiting for emails to come in or weblogs to fire or whatever, you can keep talking to those agents. And once we enabled that, what we found is like people started naming these things. And like it was, you know, this isn't, you know, my agent for creation of my email. This is like Joe, my EA. And Joe does these automated tasks, but Joe also helps you when I ask Joe to do things.

• 36:18Andrew Lee?

  And so I think the way that we've been thinking about this more is as a tool for like very heavyweight, very long-lived agents. Like you go to ChatGPT or Quad and it's like you have a quick question. You talk to it for a little bit and then you forget about that and you never come back to it. And you have this long list of tasks you've never, you know, come back to. And in our product, we're trying to have something where it's like a lot more setup, but a lot more value in the long term. And maybe you keep coming back to this particular agent for years. And what that has resulted in is actually the vast majority of usage of our product is not the automated tasks. There's a lot of that and basically all of our K users have something that is automated recurring. But they are like, most of the messages that they send is actually the other messages that they ask,

• 37:04Andrew Lee?

  you know, to the same agents that are doing these recurring tasks. We've seen all kinds of stuff, so. You know, as an example, like we've had some companies that have automated their billing processes. We've had companies that had EAs doing very EA-type work who have decided to go fully with Tasklet to like automate those tasks. We have had like music venues that sort of manage their calendar through this or like, you know, keep all the various stakeholders aware of what's going at their venue. We had, there was a commodity trading, a commodities trading firm I was talking to the other day that like sort of watches for events in the real world and like makes recommendations for trades to make.

• 37:46Andrew Lee?

  Personally, I use it for marketing super heavily. So basically, all the content that you see from Shortwave or Tasklet these days is written through Tasklet. And, you know, actually, you saw part of this for the, you know, the sponsored spot that we're doing with you guys, where I basically have a bunch of docs to have my notes. And I have an agent whose job it is specifically to do marketing for me. And anytime I need new content, I'm like, hey, here's, here's the doc I have for this. Here's the spreadsheet I have here for this. This is this thing. And then, okay, now I need a LinkedIn post or now I need a podcast sponsorship, like spit that out for me.

• 38:26Andrew Lee?

  So it is everything under the sun. Pretty much every page uses something that's automated, but they also do a lot of other stuff too.

• 38:35Nathan Labenz?

  Yeah, that's interesting. I've been gravitating that way as well in terms of just the one-offs, like the LinkedIn thing I said, that was just a one-off.

• 38:45Nathan Labenz?

  And well, preparing for this, I also was like, you know, just getting Tasklet to help me, you know, identify any gaps in my outline of questions for you. It did actually come up with a couple of interesting questions that would only come from Tasklet, given the sort of unique insight that it has into itself.

• 39:04Andrew Lee?

  I love that you did that. That was pretty funny.

• 39:09Nathan Labenz?

  I guess on the I mean, there's a lot of little nitty gritty things that I found really interesting. And I do really like how you can go back and continue the conversation with the agent at the high level, right? Again, there's this sort of two tiered thing and people should just go play with it because I think it's, it's in some ways like more intuitive to just go do than it is for me to try to describe. But the high level agent that sort of defines like, what is this thing all about and maintains the prompt that will then be used for the individual runs? That's, that's the sort of higher order thing that you can talk to. One bit of feedback, for example, that I gave to my archive paper searching agent

• 39:49Nathan Labenz?

  the other day was, I guess, first of all, in the prompt that I originally gave it, usually in the past, I have felt that I had to give anything like that a lot of context on me. But this time I just said, my interests are pretty well represented on the Cognitive Revolution website. So go check that out. And then you can kind of search archive, you know, with that in mind. So it was cool because it, at that higher level, it went and did that search to try to characterize me and understand like what I'm about. And then turn that into the prompt, which then, you know, gets fed into each individual run. And so it doesn't have to do that every single time. Maybe I should come back, you know, at some point in the future and say, hey, like you should refresh. I've done, you know,

• 40:32Nathan Labenz?

  a bunch more episodes or whatever. In practice, what I did do is I was noticing as I went into the runs and just like, you know, trace through to see what they were doing. One thing I noticed it wasn't doing all the time was accessing the memory that it had logged. So we're seeing like at the end of the runs, it would, it would log like, you know, send Nathan these five papers or whatever, and they'd be put into the store. But then if it wasn't accessing that store at the beginning of the following day's run, then, you know, what good was that? Right? So I just went back to the high level agent and said, hey, I've noticed you haven't been accessing memory at the beginning of the runs. And that is leading to some duplication in the papers that I'm getting from day to day. So like,

• 41:13Nathan Labenz?

  please access the, you know, the memory at the beginning of the run going forward. And it just like updates its prompt. And then, you know, for all subsequent runs, like that is now part of its standard operating procedure. So I think that is pretty cool. It is starting to feel much more like. A virtual employee, you know, I mean, that's obviously a sort of fuzzy notion, but something that a lot of people kind of have on the horizon in their imaginations. And you can, you know, you can kind of feel that a little bit more where you're like, I've reviewed some of your work and I want to give you, I want you to like, keep the same assignments that you have, but I want to give you high level feedback on what I'm seeing in your output and then have you kind of figure out how to incorporate

• 41:57Nathan Labenz?

  that into your future work without me having to like get into the super nitty gritty details of it. I think that is pretty cool. Okay, some nitty gritty stuff. How do you manage the, I guess even more nitty gritty first, I don't think I can talk to a run these days and that is something that I thought I might also like to do. Like I might want to give high level feedback like I just described of like, I've noticed sometimes you're not doing this in quite the way I think best, but sometimes I might also want to go down to the individual run and just say like, hey, can you do one more step or can you, you know, turn this into a whatever on a one-off basis?

• 42:37Nathan Labenz?

  Is there any reason that doesn't exist or, you know, maybe just coming soon?

• 42:40Andrew Lee?

  Yeah.

• 42:41Nathan Labenz?

  They haven't got to it yet?

• 42:42Andrew Lee?

  So it's worth walking through some of the stuff that we've tried and also talking about kind of where we're going here because I think this is, this is an area that we've learned a lot and the way it works today is actually not how I'd like to work at the long term. So we actually used to give you an easier way to chat with these, the sub-agent runs. You actually, there are situations where you can, we have the ability for the sub-agent to ask for human input, in which case like a chat box will appear and you can talk to it. We've found that that is not very frequently used and a little confusing. And the idea that you come to the product and there's like one chat box, one chat box over here and then every sub-agent run also has this chat box starts to get confusing for users of like, there's 20 places I can talk to the AI, which place do I want to talk to the AI?

• 43:26Andrew Lee?

  And so we're moving to a world instead where we just want you to talk to the main agent. And I think the piece that's missing right now is we're not giving the main agent enough information about the recent runs that have occurred. And so you can't today go into the main agent and get into the nitty gritty and and get the, you know, subagents to continue running, but you could. So this is this whole project probably like introspection, which is like, how do you? How do you have these subagents that run and have sort of the reliability of the cost benefits of being run as subagents? But then have the main agent appear to have full knowledge and control over them. And that is a very, that's an ongoing, that's an ongoing project.

• 44:12Andrew Lee?

  I think one of the areas we're going to look at is the way that instructions work. So you mentioned that, you know, you look at the instructions sometimes and then give the main agent instructions to to change those instructions. We actually used to let you, so we have these intermediate instructions, right? And those intermediate instructions are generated by the main agent and then persisted. And we actually used to let you modify those manually. And the down, the big downside of that is it creates a tension between what you've told the main agent and what you've edited those instructions to be. So if you go and you tell the main agent a whole bunch of stuff and then later on you go and edit the subagent instructions. Even if you tell the main agent that you made those edits, it's start, it's it's pretty tricky for the main agent to reconcile.

• 44:56Andrew Lee?

  Well, you told me you wanted X, Y, Z, but then later you added the instruction to this and I'm not sure what you want later because the conversation continues. And so we removed, and it also wasn't being used very much, people weren't editing it. So we removed the ability to edit those. And I think in the future, we're actually probably going to stop pre-generating those instructions. Because pre-generating those requires the agent to reason about when to update those and how to update those while doing other tasks. Like you might have, you know, you might ask it to do something and make it aside while doing that, where it has to go do the thing it's you're is trying to help you with immediately. And then also going to update these instructions, which is a long, slow, error-prone process. So I think what we may move to is something where.

• 45:37Andrew Lee?

  Those sub agent instructions are generated just in time and they're generated sort of out of band and done in such a way that they always factor in all information. And if you did want to see the instructions in a way that you could digest them, you just ask the AI. Rather than having a special button to say, show me instructions, you just tell the AI like, give me a summary of the instructions that you've given me. Or sorry, give me a summary of the instructions that I've given you. And it could do that. And then you could say, if you really want to follow this exactly or modify, then you could say, well, actually, I'd like you to follow these modified instructions and you could apply instructions. And so I think you can get the best of both worlds. I think you can get the ability to be very specific in your instructions and to see what it's thinking

• 46:21Andrew Lee?

  without a sort of a UI that creates this ambiguity.

• 46:28Nathan Labenz?

  Yeah, that's a really interesting point and that the ambiguity, which I also, you know, think kind of blurs into an agent that has multiple conflicting goals is becoming a real driver of strange results from AI systems in general. I mean, the GPT-5 prompt guide takes pains to warn people against having any sort of contradictory instructions, like that's apparently a huge drag on performance. If there's any, any contradiction in the, in the prompts. And also, you know, the things like alignment faking and, you know, a lot of these sort of bizarre behaviors

• 47:08Nathan Labenz?

  stem from some sort of incompatibility between either what the agent has been like trained on at a high level and what it's currently being asked to do, or even just like two, you know, kind of instructions that it might be receiving at different levels of the, the instruction hierarchy. So that is, I think you're right to flag that that can lead to some weird stuff.

• 47:29Andrew Lee?

  There is one.

• 47:31Nathan Labenz?

  Colorfully.

• 47:32Andrew Lee?

  There's just one part of the code where I've asked the team to make me the reviewer before PR goes in. I, you know, I am, I'm not the CTO here. I'm generally not the reviewer for things, but anything that touches the main prompts, I want to see. And the main thing I'm looking for is conflicting instructions. Um, because if you have, you know, a bunch of people all kind of working on their part of the system and giving instructions that make sense. You, you need one person to sort of looking at the prompt as a whole and being like, are we telling, are we, are we giving instructions to this thing that are clear? Where if it reads the whole prompt, it knows what's going on and like that, that it's a problem. A surprising amount of, of time, even when people are trying to be very careful about it.

• 48:13Nathan Labenz?

  So how about this, um, just general notion of like context engineering, context management, context, maybe like cleanup or compression.

• 48:26Nathan Labenz?

  And then that obviously blur into memory too, although that is a bit distinct as things stand today.

• 48:32Nathan Labenz?

  The question that was suggested by Tasklet for you says that there's a token budget, 200,000 tokens in my current context. Now I know that there are some opportunities to get to longer context with the right customer relationship. I think you have to pay more per token if you want to opt into those things, but obviously it's going to be finite no matter what, right? So especially if I'm like talking to this high level agent and I've had a, you know, a bunch of runs that are running on a ongoing basis. And I say, oh, can you, you know, pull up the run from October 1st? No, now can you like look into the run from October 5th or whatever? It seems like I'm going to be out of context real quick. So how do you manage, you know, when you talk about like something that you could come back to for years?

• 49:18Nathan Labenz?

  Is that like still aspirational or is there a way to get the agents to be that long lived, given the finite context that you have to work with?

• 49:27Andrew Lee?

  Yeah, totally. I, uh, I want to note, I really love the word context engineering. I have been, I have been looking for a word to describe this because I think the, there's, there's just kind of two parts of building an agent. Well, three parts. There's the core agent loop, but that's pretty simple. There's the model. And then there's all the work of putting things into that model. And I think the terms that people have used in the past are, you know, not. Not quite right for the way things actually work. So the term rag, for example, I think has been used very badly. There's sort of a naive rag that most people think of, but like, doesn't really describe like the way tool calling and stuff works. So I, I really like the word context engineering. I think it gives like sufficient gravity and weight to like

• 50:09Andrew Lee?

  the sophistication of the systems that actually like are built around these agents to like plum the right data in.

• 50:17Andrew Lee?

  And we've done a lot of, a lot of work here. We're going to continue to do a lot of work here. And yeah, I agree with you. You somehow need, in our case, you might have an agent that's been running for a long time. That's, you know, processed, you know, 100,000 emails. Obviously you can't fit all of those into, you know, any sort of reasonable token budget. So you have to give the illusion to the user that not only have you fit those things in, but that the LLM is still able to reason intelligently with like a huge context. And you do that by, you know, there's a lot of methods here, right? So you can, you can hide things inside tool calls.

• 50:58Andrew Lee?

  And then you probably saw the, the stuff from Anthropic recently, like one of the things you can do is just like hide the results of the tool calls. You can do LLM based compaction. So you can say, you know, we're gonna rewrite, take, take some big section and we're gonna take a model and we're gonna summarize that some way. You can do things in subagents. So like when our subagents run, we're not giving them the whole chat history. We're giving them some, you know, tailored portion of, of that chat history.

• 51:25Andrew Lee?

  So I see this as a major, a major focus for us is, you know, giving the, the user the illusion that everything they've ever said to this thing, every email it's ever been a process is being considered at all times without actually doing that. And I think that's a, it's a fun problem.

• 51:40Nathan Labenz?

  So I guess one more thing on just like context management, if I'm, if I'm looking at the subagents, I have this archive paper finder daily, right? And it, it puts at the end of a run, you know, it stores what it found and what it sent me. And then at the beginning of the rundown with my feedback, it is pulling that history out. So it knows, you know, what it has already found and hopefully avoids duplication. Even that over time, right, is going to start to get huge in the limit. Like there's a, you know, just a ton of stuff that it's already found. I guess I could kind of help it out by saying like, you know, only search for stuff in the last two weeks. And then like, you only have to pull stuff from for the last two weeks.

• 52:20Nathan Labenz?

  But it seems like everything kind of blows up in the end. So I wonder how you are thinking about that. And I also noticed, I think I noticed that even in just the month that I was using the product before the launch, when I go back to the earliest runs, it looks like it's like a NoSQL store. And now the more recent one, it has a SQL backend. Previously, it was like, you know, just sort of JSON docs that were being sent into the store. And now we actually have like insert statements and whatnot. So I guess, how do you think about the size of the store? Does that, I guess that becomes like a rag, you know, with possible pardons for abusing the term,

• 53:02Nathan Labenz?

  where you sort of have to apply some intelligence at that level as well? Or, you know, I could imagine you could also be using the old Gmail trick, which was like promise unlimited storage and just know that people don't have that many emails coming in yet. So you can kind of figure it out at some point in the future. And then there's also these like dedicated memory startups that I'm sure you've looked at Men Zero and Supermemory. So I guess to try to sum that up into a question. If I'm storing more and more stuff, how do I not overflow the context just with that? What are you seeing working, not working? You know, what about these memory startups?

• 53:41Andrew Lee?

  Yeah, this is very much a core part of the challenge. So I, we're not, we're not using any other providers because we think this is something that we have to solve ourselves. And if our whole premise is we're like ChatGPT or Claude, but like. Much more heavyweight and designed for like long running recurring work like this is this is a very core fundamental thing we need to do. The North Star for us is the illusion that we're just one big long chat and everything has been put into that chat and the model is perfectly smart and

• 54:16Andrew Lee?

  the system breaks under sort of all of those over all of those reasons, right? That, you know, we can't actually fit all of it in context. We can't afford to do whatever we could and the model is not infinitely smart. So if you you know, forget it, but you stored every email that you'd ever processed just in the history. The LLM isn't going to do a good job of reasoning about which ones you processed or which ones you haven't processed. It's just it's just not going to be that smart. And so one of the techniques that we've been using is having a data structure that is managed explicitly by the LLM across Rettens. The first iteration of this was a very simple version, which was in the system prompt, we included a JSON blob. And that JSON blob could be directly ended by a tool called in the LLM. So in the sub agent runs,

• 54:58Andrew Lee?

  you could just tell it like, hey, I want to add this data or remove this data from this JSON object. And that actually works surprisingly well. This was like the dumbest thing we could think of. And we built it and it just worked and it was really cool. But it came with a couple of downsides. One is it fairly quickly got large enough that like it started to break down and not be useful. So we had a lot of agents that were like working for a while and then after a week or two, whatever, they kind of stopped working because that JSON object got really, really large. The other problem is it is terrible for caching to put mutable state into your system prompt. And caching is very important for cost.

• 55:37Andrew Lee?

  And so, but the benefit of having it in the system prompt at all times is, you know, the LLM doesn't have to reason about how to retrieve it. It's just there, right? And so we looked at other ways to do this and we settled down using just a SQL database because it turns out the models are just super good at writing and using SQL. Like they're they're way better than I have at this. And if you just tell it, look, you got a SQL database and you can store stuff in there and you can retrieve it. And then you can give yourself instructions on how to manage that thing. It works. It works really well. I agree, this is not the end of the story, right? So you totally can have instructions that say like, you know,

• 56:16Andrew Lee?

  every time you get an email stored in the list and then every time you start load the whole list and that list could get bigger and bigger and bigger and eventually cause you problems. And so we're not done. We're gonna have to keep working on this. I think there's going to be, in addition to being even smarter about how to manage that SQL data, there's going to be lots of other tools that we use. So as an example, one of the things I'm looking at now is in a world where you've compacted old history, but that old history becomes relevant. You could potentially give the agent tools for uncompacting portions of the history, right? Be like, okay, you know, in this particular case, I really do need to know what the earlier messages were. And you could potentially like load those through tool calls. So there's a lot of tools, there's a lot of ideas that

• 56:58Andrew Lee?

  we're exploring here. I think this is very much an unsolved problem and we're working on it.

• 57:04Nathan Labenz?

  Any impulse to some sort of graph of knowledge? We did an episode on it, it's probably been at least a year, on a paper called HippoRAG, where it was a pretty, you know, kind of, I don't know, it was both cool, but also like you could, there were definitely still things to do to, you know, take that to a production system. There was a HippoRAG 2 at one point. But what I thought was cool about it was that it sort of dynamically, on a periodic basis, it was almost like what would happen, you know, when the agent is asleep, right? Is you would go back and sweep through all of the new items that were stored to whatever memory store and try to find connections between them and, you know,

• 57:47Nathan Labenz?

  kind of map the space out, create these connections that then in the future really improved the retrieval because you could do sort of a graph-based search. And if you hit like one node and, you know, expand to, you know, two hops out from that one node, then you like get a pretty good set of like all the surrounding information and you can do that, you know, dynamically at runtime with that background processing to, you know, kind of continually build out the graph day after day. Anything along those lines in your mind? Or like, what do you think are the sort of creative ideas that will?

• 58:23Andrew Lee?

  So I I suspect like that something like that could work well. Um, but I don't think it's the right business move for us to try to explore doing that. And the reason for that is the agent's internal state is only one of the many sources of data that the agent needs to to do its job. Um, so, you know, you, if you ask it to do a task, it might need to look through its own state, but it also might need to look through all of many systems that you're connected to. And the amount of data that's connected to you through the systems is way too big for us to like suck it all in and like put it into this system. So if we built a system that for internal state managed it super well, but for all the external state, we're sort of still relying on like whatever those services provide

• 59:05Andrew Lee?

  for their, you know, search and memory systems. Um, I don't think that, I don't think it'd be a huge improvement to the overall capabilities of our product. So we're sort of betting on, again, betting on the model to say, you know, assume that the memory capabilities never get that great. Um, well, the model gets smart enough about using those things such that it can kind of, you know, figure it out and get really awesome. And I think that trend has been, I, I, I'm optimistic about this. And like, I would argue that companies that, you know, a couple of years ago were like, hey, we're gonna, we're gonna ingest all the data from all these different systems and build this like often AI search product.

• 59:48Andrew Lee?

  Um, I suspect now they're looking at this and saying crap, like if you just take an agent and have the agent reason about how to use legacy search tools, it does almost as well. And man, that is way simpler and way cheaper. Um, I, I think people are sort of realizing that now. So that's our bet is, uh, we shouldn't invest so much in the kind of smart memory systems now. We shouldn't invest in connecting to things and then bad on the model getting smart enough to use more traditional systems.

• 1:00:19Nathan Labenz?

  Yeah, that's interesting. Do you also rule out the Claude like inbuilt memory? Because they now offer a little bit of memory management through the API itself, right? Is that also sort of don't use it because it's it's got to be core to us?

• 1:00:37Andrew Lee?

  We haven't we haven't played with their latest stuff. If if memory serves, I don't think they're doing anything that we couldn't be doing ourselves. I think their context editing stuff is also the stuff that you could totally do yourself as well. So yeah, it's in the camp of of don't use it because we can do it ourselves. Although I haven't I haven't played with it specifically.

• 1:00:55Nathan Labenz?

  You know, going back, I think I think to our first conversation, there was like a lot of custom models and sort of, you know, relatively deep down the stack optimizations that you were using for retrieval when you index my full, you know, and very gnarly Gmail history for Short Wave. It sounds like that's not so much of a part of the strategy now, but, you know, I also am remembering, I think it was cognition in the last few days, showed something where they are training their own like relatively narrow agentic search model that is, you know, meant to do the same sort of thing that Sonnet can do in terms of doing retrieval

• 1:01:38Nathan Labenz?

  against all these systems, but, you know, faster and cheaper, if nothing else, because they're, you know, able to specialize on a particular, you know, relatively narrowly scoped task, and therefore they could do it with a smaller model. I'm guessing you're going to say we're leaving it to Sonnet for now, and maybe one day we'll need to do that sort of optimization, but what do you think is the outlook there?

• 1:02:02Andrew Lee?

  Roughly, I do think that a good AI search system with a great model is going to outperform a traditional search system with a great model. And I think that'll always be true. And, you know, you could see this today, like going to Tasklet and asking it to search your email is not going to give you as good a result as going to Short Wave and asking to search your email, because they're both using the same model, and Short Wave has the benefit of like really good semantic search stack. But if you look at the two businesses, like Short Wave, it is a tremendously complex and expensive system that operates that search stack, right? So it is a is a major factor of our costs, is a major factor of the effort that we have to put into developing and maintaining the product, like maintaining that search stack.

• 1:02:49Andrew Lee?

  In tasklit, we hook up to thousands of services. It is not practical for us to like build search for all of those things. And even if we tried to say, okay, we do for a few of the most common ones. I don't think it scales in any reasonable way. So I think our only reasonable strategy with tasklit is to say, well, we're going to rely on their existing search stuff. And I'm optimistic of two things. One is that the models will keep getting better using those existing traditional search facts. And the other is that the services themselves will over time build good semantic search. Like Gmail will eventually, I think, build a good semantic search API that we can use. At which point the semantic search in short wave is, you know, we can maybe throw our infrastructure away and save some money. So I think that's our bet.

• 1:03:31Andrew Lee?

  I do think, I'm not saying that startups here aren't potentially going to be awesome and be very successful. I do think there's lots of opportunity in AI search. I just don't think it's the right bet for us to be taking.

• 1:03:43Nathan Labenz?

  Yeah, interesting.

• 1:03:46Nathan Labenz?

  Well, let's talk about integrations. You have a lot of them. It's advertised 3,000 plus business tools out of the box. Any API, any MCP.

• 1:03:58Nathan Labenz?

  How did you get to 3,000? That's a lot. Yeah, 3,000 MCPs, I would assume that that all kind of came sort of for free. But when you separate them out like that, it sounds like you've done quite a grind of, you know, a long march through integration land.

• 1:04:13Andrew Lee?

  So, so, yeah, so this is, this is an interesting story. When we, so in short wave, I think we have like 10 integrations or so, something like that. And when we started doing tasks that we thought, okay, we'll start the same way. We'll pick the top 10 most commonly used ones and we'll focus on those use cases. You know, email and Notion and the Sana and stuff like that. And we started going down this path and my co-founder said, hey, we should try hooking up with these integration platforms. And we hooked up Pipe Dream and they have a whole bunch of integrations now. And it just worked really well, right? So we're like, holy crap, like suddenly we have access to all the stuff that they have here. And then we said to ourselves, well, you know, we should have MCP in here as well. So let's add MCP. And that worked really well, too. And we said, you know,

• 1:05:00Andrew Lee?

  This pipe through step works surprisingly well, even though there's like nothing in our code making this stuff work well. Like we don't have any customization of the prompt. It's just, it's just you plug it in, boom, it works. This is awesome. I wonder if we could do the same thing with HTTP APIs. Like I wonder if we could just, you know, describe to the model generally how to use HTTP APIs and we could like put some constraints around how those calls were made and then connect to an arbitrary HTTP API. So we built a thing to do what we call direct API connections. And it worked really well, like, like shockingly well. And, you know, we found ways to make it even better by like, for example, going and scraping website docs. So like when you make a direct API connection, we go and we search the web, we find the API docs, we look it up, we pull out a description and then we use that.

• 1:05:44Andrew Lee?

  And boom, we've got this, this API connection. And it works so well. In fact, that we have users who are switching off the official MCP. Like, so for example, we, we have a, a, a paying, paying user who stopped using the official Notion MCP and started using the direct API connection that were, that is fully LLM generated on our side. Um, because it works better. It's more reliable. It has less bugs or whatever. So, um, basically we just kept adding and adding and adding. And we realized that the opportunity here was to connect to everything. We said, hey, rather than, because everyone, there's so many competitors doing what we're doing now with Tasklet. And every one of them comes out and says, hey, we connect to this set or this set or this set. And they all tried to differentiate how many they, they cover. And we're like, look, we need to just end this conversation and be like, look, we work with everything.

• 1:06:29Andrew Lee?

  There's nothing we can't connect to. And so we made that a goal and we very quickly went from, you know, ten to thousands of integrations that, and some of those are ours, some of those are through integration platforms. And then direct API, so any HTTP API, MCP, um, provider provided MCP. So anyone like, you know, for example, like with, with, with linear, right? They have their own FTP server. So those are included in that list as well. And like, we helped with discovery for that. And then, uh, computer use. Um, and if you got to put all of that together, the, the thing that we provide is a unified UX and LLM interface for this thing. So no matter whether that integration is hand rolled by us or

• 1:07:10Andrew Lee?

  It's done through an integration platform, or it is a direct API connection, or it's an MCP service that they built, or it's an MCP service that you built. It all has the same sort of configuration process and the same activation process for controlling like tools and permissioning things. And it works the same way in the L L M. So the rest of the system doesn't care, right? It there's integration, it knows that it's integrations and like boom, everything works. And then for good measure, we add the computer use because there are some things that don't have APIs and don't have MCP servers. Either because LinkedIn doesn't work. Or in the case of LinkedIn, because they don't want you to, right? And well, it's real hard to stop computer use.

• 1:07:52Andrew Lee?

  So and that was a key part of our pitch being like, look, like, not only do we, you know, we use the integration platforms. We cover everything they cover. We can get any API, anyone with MCP, and we can also do computer use. And by the way, you know, the top N, we have hand rolled ones that are like super solid. So, you know, the really important stuff, we we make sure it's great. And I think I think this has landed super well to folks, and especially especially the direct API connections. Like, I think an interesting data point on our side is you hear a lot of buzz about MCP. I used to be very optimistic about MCP. Our direct API connections are used far, far more frequently than MCP.

• 1:08:31Nathan Labenz?

  That's really interesting.

• 1:08:34Nathan Labenz?

  When it comes to like credentials, the flow is basically like, oh, hey, I want you to check my email. I mean, I've never talked to it about my email before or whatever. Then the thing comes back and says, okay, you're going to need to connect your email if you want me to do that. The sign in there is like a familiar flow to people. Are you? Do those those credentials, I assume, live at a cross agent layer, right? Do they is that also a

• 1:09:03Andrew Lee?

  Yeah.

• 1:09:03Nathan Labenz?

  Technology platform that you're able to tap into to manage those credentials for users? Or do you have to roll your own version of that for some reason?

• 1:09:11Andrew Lee?

  That's all our own stuff. So a big part of what we've built is the sort of integration and connection system. And basically, we have this concept of a connection and it's an authenticated stateful connection to an external service. And it's something that may be a little complicated to set up, right? And this is when you, for like a common one, you might go off into. Or for Direct API, you might actually have to give us like an API key and maybe some other details about the headers you have to provide and stuff like that. Or MCP, you might need to provide some information about the MCP server. And there's a connection creation flow. So if you go in there and say, "Check my Gmail," the first time you do this, it's going to walk you through a flow to create that connection. Once you create that connection, that connection is stored on your user.

• 1:09:55Andrew Lee?

  And you can reuse that anytime you want. But we require that every agent, sort of you re-grant permissions for it. And the idea here is like, you know, if you're, imagine you're a company and like every agent's a different employee. You don't want to give every employee access to everything, right? Some employees are, you know, will be able to do this and some employees can do that. You want to restrict what the agents could do so they don't go off and go rogue and do crazy things. So you create the connections and then separately, you enable the tools on those connections specifically for the agents. And I see a lot of opportunity here in the future because the place we're going to expand on the team side is allowing you to share connections across your team. So the way we want this to be used in the future is you're an IT administrator at your company.

• 1:10:37Andrew Lee?

  You define the connections that people at your company should have, right? What are the API keys that people should be using? What are the services they should have access to? Those connections come with audit logging. And they come with, you know, cost controls and like other sort of oversight and, you know, policies associated with them. And then you assign who on your team is allowed to access the different connections. And you tell everyone in your company, "Hey, you know, rather than you using any AI tool you want and hooking it up to our systems any way you want, you should use Tusk Live, use the connections that we've sort of created and pre-vetted for you and that we've set up the proper compliance and audit logging and things for." And so, yeah, the way it is working in the future, you're going to have, your team will have connections. You personally will have connections. Creating those connections might be some work, but once you have those connections,

• 1:11:23Andrew Lee?

  enabling those on your agents is like super simple and super intuitive and like super safe.

• 1:11:28Nathan Labenz?

  Interesting. When it comes to the API.

• 1:11:34Nathan Labenz?

  You know, spin up for a new previously unknown API. Is there, is that something that happens like for every agent, you know, fresh each time? Or, and maybe it wouldn't even make sense to try to like create a skill bank because these are so long tail that like you don't see too many of them twice, but I'm, I'm wondering if there is a, I think it was Voyager was the original project that I recall sort of pioneering this notion of like, okay, I've successfully achieved, you know, this skill. Therefore I'm going to save that to a skill library that I can come back and tap into again later. I mean, you could even imagine doing that sort of thing like across users or across organizations, but then you might worry that maybe you shouldn't for some reason, or there could be some, you know,

• 1:12:18Nathan Labenz?

  some proprietary info.

• 1:12:22Nathan Labenz?

  So yeah, is there any sort of crystallization of that API success?

• 1:12:27Andrew Lee?

  There is. So you can, you can broadly, you could broadly group our connections into two types. There are like discoverable pre-built integrations and then there were custom integrations. And we literally have a search function where you can like search for integrations and we return the list that we know about. And some of those integrations that we built, some of them are through integration platforms. Some of those are MCP servers provided by other services. But that's the set where we're like, hey, we know these exist. We know how to configure them. And we try to make those like really easy in one click. And then the other set is custom where it's like, you know, we don't know anything about this. You need to tell us the details. And, and I think we're going to try to keep that divide of like any connection that we know about. We're going to try to like save the configuration for you and make it as one click or close to one

• 1:13:08Andrew Lee?

  click as we can possibly get it. But hey, you can always set up your own thing if you want to.

• 1:13:14Nathan Labenz?

  Yeah. Interesting. MCP, it sounded like there was a little bit of a hot take potentially brewing there.

• 1:13:21Andrew Lee?

  Yeah. And I don't, you know, I don't know how many other people have come to the same conclusion, but I don't, I don't think I'm a total outlier anymore. We, we started working with MCP, you know, early this year and initially I was extremely bullish. The, the community was exploding. It was obviously like. Super cool what you could do with it. You could hook up these other services. And it seemed like this missing, this missing piece and we obviously built this at the core of what we do in Short Wave and like it was a big inspiration for us building task that at all. But when Cloud 4 came out and we started playing with sort of every day like a major model version comes out, you know, we kind of rethink all of our priors and

• 1:14:05Andrew Lee?

  one of the big question marks we had is, hey, if like, you know, empty, you know, if a big goal of MCP is to provide these tool definitions.

• 1:14:15Andrew Lee?

  But those tool definitions, you know, there, there's probably debate you've probably seen around like whether the tools should align with API calls or whether you should align with something more specific. So assuming those tool definitions end up just aligning with API calls. And those APIs are documented on the internet. Like, what's the point of MCP, right? Why can't we just like directly call these API endpoints if we have descriptions for those which are available online? And I think the answer before Cloud 4 was like, well, models aren't really smart enough to like make that whole process work to end to end. But they got to a certain point. We, we tried this. We're like, man, like the models are totally able to, one, just directly use the APIs. You don't need to necessarily have like custom tools for all of them. They work fine. And two, they're smart enough to like generate their own tool descriptions basically by just scraping the web.

• 1:15:00Andrew Lee?

  And what's the point of MCP? And that, that isn't entirely true because like one of the advantages of MCP is it provides a better auth experience, at least in some cases. So for example, with Notion, the only way to get like a nice auth that gives you offs broadly here workspaces to use MCP, but like we actually, we, we don't do this anymore, but for a while when we felt like the, the tool definitions from, from Notion were not good, we actually used the, the MCP off from them and then just like override their tool definitions with their own tool definitions. Because we thought we, we had a better approach. So now we look at MCP as, you know, in cases where we feel like the provider of the service has a good version, we use it. In cases where they don't, maybe we'll use it for auth, but not for the tool definitions.

• 1:15:45Andrew Lee?

  And in many of those cases, people prefer to use direct API or some other method for accessing it. So we now say, hey, we're going to give you as a user like, Every way to connect every service out there. Sure. You want to do MCP? We got MCP. You want direct API? We got direct API. You want to go through an integration platform? We got that. It's up to you to decide which one you feel like will be best for your use case. And then, once you've created that connection, you know, you can kind of forget about how you're connected and just likely reuse that in the future.

• 1:16:13Nathan Labenz?

  One thing I've been looking for in MCPs, and I've seen precious few of them, is, for lack of a better term, smart MCPs. Basically, you know, working from the same kind of observation that you said a minute ago, like if it's just a one-to-one on the API, what's the point? One way to answer that would be to zoom out a little bit and provide like higher level sort of intent-based notions to the model that don't require, that maybe like wrap, you know, a bunch of API calls, or maybe even there's like an AI on the side of the MCP that sort of takes in your higher level intent and like, you know, because it's a specialist in using its own AIs, it can kind of translate your

• 1:16:58Nathan Labenz?

  higher order wish to, you know, a bunch of specific executions and give you something back. Everything has pros and cons that would have some loss of like visibility and control, but maybe could give you higher performance and give you a reason to like, why do you use the MCP over the direct API itself?

• 1:17:17Nathan Labenz?

  Have you seen many of these? You know, why doesn't this, why isn't this happening more than it is? Or am I missing it? Because I haven't seen much of this, even though it seems very natural to me.

• 1:17:26Andrew Lee?

  I, I haven't seen much of this, and I do have some examples of use cases here. So for, for example, like in Gmail, there's no API for forward. The, the way you forward an email is you create a new draft, and then in the draft, you include the right content and you send it. But if you're going to give a tool to an LLM, the, the process of like copying the history into that new draft, probably not something about the LLM doing. You probably want to have just like a tool call that like knows to like do the hot posting correctly. So I definitely think there are, there are use cases for higher level tools. I think the big, the big downside of trying to do this higher level MCPs, though, is like, it's, it's, it's another thing to maintain. It's a lot of work. And

• 1:18:10Andrew Lee?

  If the models can get smart enough to do this stuff well, or if the models can get smart enough to generate those tools, you could maybe avoid having all these companies have to like think through what to do here. So today,

• 1:18:23Andrew Lee?

  you know, for direct API, we use the API endpoints directly. Maybe in the future, we'll have the model like reason about logical groupings of those endpoints such that, you know, more can be done in code.

• 1:18:36Andrew Lee?

  But I'm not, I'm not super optimistic about like over the long term, like hand rolled MTP that does this because I think it'll be obsolete by the models.

• 1:18:46Nathan Labenz?

  So it doesn't sound like you think the progress in model capabilities is stalling out now or anytime soon. What's your commentary for those that are,

• 1:19:00Nathan Labenz?

  and I don't believe that either, but you know, there is this kind of counter narrative that has come out or come on, let's say since GPT-5 that, well, I guess pre-training stalled out and everything's kind of, you know, this is about as good as the AI is going to get.

• 1:19:16Nathan Labenz?

  How do you react to that? Or what's your argument that that's not right?

• 1:19:21Andrew Lee?

  I, I think they're still improving and I,

• 1:19:25Andrew Lee?

  I very much drunk the Kool-Aid of the metric to watch is the length of tasks that can be completed autonomously. And regardless of how you want to measure that, I think we're at a point now where you can't, I mentioned this earlier, you, you can't really see that much of a difference in like single question, single response now as the models get smarter, right? It was really smart before. It's still really smart. Where you start to see the differences add up is over many turns, right? You, you, you know, computer use, for example, you say, hey, I want you to like navigate to LinkedIn and like find all my previous coworkers and make a list of their current jobs or something, right? To do that, well, the AI has to go off and do turn after turn after turn after turn after turn.

• 1:20:08Andrew Lee?

  And a model that is, you know, 0.01% better at every turn over time is going to end up being, you know, radically more effective than, than one that's slightly worse. So if what you're measuring is, you know, what's the quality of one answer from one response? I think that's sort of plateauing, but what if you're, what you're measuring is what is the quality at turn 100 or 2000? I still, I still think there's, you know, Orders of magnitude to go here. So we're still betting on it. I expect this to go for a very long time if that is the metric.

• 1:20:42Nathan Labenz?

  Maybe we'll circle back to that in a minute. Last thing on integrations. What other like, what, what integration providers are you finding to be either really popular or like hidden gems? Like in other words, what should I sign up for so that I have it so that when I am using Tasklet, I'm like, you know, doing what the the cool kids are doing?

• 1:21:09Andrew Lee?

  That's, that's a good question. Um, I mean, the, the, the boring answer for you is well, just the stuff that you're using already. Right? So for the most part, we're not telling you like use a new service. It should just work with the stuff that's already in your stack. If you actually want to like use some services that work really well with this, you know, the one that comes to mind for me is Rocket Reach.

• 1:21:29Nathan Labenz?

  I was gonna say exactly the same. It's so funny.

• 1:21:32Andrew Lee?

  Because, you know, for example, you know, I probably shouldn't say this to a member of the media, but I've been automating some of my-

• 1:21:40Nathan Labenz?

  I don't see any members of the media here. No worries.

• 1:21:44Andrew Lee?

  Um, yeah, I've been automating some of, some of my press reach out and it does a really good job of crafting emails that really I feel like I would have sent and finding the right people and customizing them for, for, for that, that person. And I can automate this to constantly be looking for people that might want to talk with me. And Rocket Reach is great because some of those people, they just put their email address on the website. Some of them are a little harder to find. And so it can, they can track down emails for people.

• 1:22:12Nathan Labenz?

  Yeah, I was good at nominate the entire category of contact info finding companies as probably a huge beneficiary of the agent wave because that information is hard to find. I have not had super great luck. Certainly sometimes you can just have the agent go find it, but it is not nearly as successful as the companies that have taken the, you know, pains and years and done whatever data deals they've done to find that information. So yeah, that's a great one. Any other categories come to mind? I think your first one is spot on.

• 1:22:47Andrew Lee?

  Oh, I guess I, I got a, I got one for you.

• 1:22:52Andrew Lee?

  Using other AI products. So a fair number of people hook it up to like perplexity's search API because it's really good at doing sort of deeper research if what you're trying to do is like really heavy duty web scraping. So I think that's a that's a good one. Another one, this isn't really a specific integration, but it's a use case. We've seen people give a task that its own email address, like they'll create a Google Workspace account specifically and give it a name so that it can like send and receive its own emails as itself. That's another fun one we've seen. But really beyond that, it is the tools they already have, right? So people that use Salesforce, they use it with Salesforce. People use Stripe, they use it with Stripe. So whatever, whatever tools you already want.

• 1:23:34Nathan Labenz?

  With that email sending, and this would be true for probably any number of different kinds of integrations, you know, post something to Slack, whatever. Is there a best practice or a good, good paradigm for thinking about human in the loop? Like I sometimes would imagine I would want, like say I wanted a tasklet to organize our lunch order on a daily basis. Back when we were at the office every day, you know, this was a constant pain point. And I could imagine something that would be like, okay, go, you know, post to Slack, remind everybody twice, and then, you know, collect and place the orders. But you'd have to have some sort of, I guess on that one, you could do it on a pure time basis, you know, lunchtime is lunchtime.

• 1:24:16Nathan Labenz?

  But, you know, plenty of things you could imagine where you're like, you need that feedback and you want to wait for it and you want to respond to it when you get it. But you, you know, you need to have it, right? So is there anything that does that paradigm exist at the run level today? Or is there anything I could do to, to sort of make sure that I set that up?

• 1:24:37Andrew Lee?

  So it is interesting that you bring in lunch ordering because this was the canonical example we had internally of this is the thing we wanted to be able to automate because we do this, right? We do DoorDash every day for the team. And it turns out this is a non-trivial thing to automate because, well, you're not in the office every day and different people are in the office every day. And there are error cases to handle, like You know, you order from a restaurant and the restaurant just like cancels your order sometimes. You have to like handle handle that case. There are dietary preferences to consider. You don't want to order from the same restaurant every day. People get sick of that. Sometimes you have guests, right? So there's this long, long list of complex reasons and exceptions and things to order lunch and we've worked with EAs in the past, like human EAs that do this.

• 1:25:21Andrew Lee?

  Our own team does it. It's kind of a hassle. Sometimes we forget to order lunch and then we have to go out for lunch and so we're like, man, we we we should automate. This is the first thing we should automate and we actually, it's a little embarrassing. We don't currently have that particular thing automated, although I think we could today. We have half of it automated and I haven't gotten around to setting up the other half of it. So we get an email every day with recommendations. And the recommendation of DoorDash links and someone has to have to choose the one and click the DoorDash link and like set up the the group order, which we should fix because we can do that out of computer use.

• 1:25:54Andrew Lee?

  But

• 1:25:56Andrew Lee?

  the human in the loop portion you talked about, I think is really important because, for example, you might want someone to approve the restaurant that was picked. You might want someone to approve that like everyone has added their order or that there are no guests today. So I do think that's a good example here. We actually do have some computer use capabilities in the product today. The subagent runs actually can contact you and they can send you an email. It doesn't work super well. It hasn't been super common. This is an area I want to fix. Like what I really like to be able to happen is for the subagents to be able to like call into the main agent and be like, hey, I ran into an issue. Main agent decide what to do and then the main agent could kind of reason about how to handle it, including contacting you. This is also an area where like we probably need a mobile app to do this well because getting an email when you need even in the loop is probably not

• 1:26:40Andrew Lee?

  great. What you really want is like a push notification and they can you can jump in. So we we would love to make like lunch ordering the like ultimate starter use case and make it super awesome. We'd love to do human loop well. It's it's a solid work in progress, though.

• 1:26:57Nathan Labenz?

  Gotcha. Okay, I'm standing by. On, I guess going deeper, you know, a little bit deeper down the stack than the integrations, I wonder what you are using to build. You mentioned Pipe Dream being one platform. What else would you say you have found to be great? I'm thinking like, you know, I go to the Vercel SDK a lot of times if I want to, you know, set something up that has a nice front end, or maybe I'll use Langchain and Langsmith if I want to have, you know, workflow, especially if I want to have that hosted in the cloud. What's good in the tool, you know, in the picks and shovels layer that you would recommend others?

• 1:27:41Andrew Lee?

  The list of shout outs right now is very limited. Pipe Dream's been great, but happy with them. Google Cloud is awesome. So, you know, computers, for example, is all backed by like VMs and Google Cloud and we're using a lot of Google Cloud products and it's awesome. And then Anthropic is super key. We don't use Langchain. All the agent code is our own stuff. It's all TypeScript. We don't use any specialized search products for what we're doing. So, yeah, it's a short list of shout outs, but yeah, Google Cloud, Pipe Dream, Anthropic.

• 1:28:20Nathan Labenz?

  How about at the coding layer? Are you guys a cursor shop or?

• 1:28:27Andrew Lee?

  Yeah, we use a bunch of stuff. Heavily Claude code, and then most of the team uses Cursor. And we do use Codex for some things. And there's, you know, some folks prefer Codex and some folks use Cursor. Some folks use both for different things, which I find interesting. We've tried some code review tools, but like we just have Claude code like integrated into our CI at this point, which works pretty well. We use Sentry.

• 1:28:59Andrew Lee?

  We use StatsSig for stats.

• 1:29:03Andrew Lee?

  I'm probably forgetting some. Yeah, those are the, those are probably the big ones.

• 1:29:07Nathan Labenz?

  Are you tracking a lines of code drafted by AI or similar metric?

• 1:29:13Andrew Lee?

  We, we aren't. Um, it is significant. Uh, I, I suspect by just pure number of lines of code, it is the majority of them. I don't think it's 90%, but it's, it's the majority of them. Um, I, I don't, I don't think that equates to like, you know, the majority of the thinking or the, the reasoning, right? Like the writing the line to code is often not the hard part. But a lot of AI coding, yes, for sure.

• 1:29:39Nathan Labenz?

  Have you been able to, or, or tried to, or do you, do you, you know, feel that it is desirable to, with today's level of coding capability, go from like a ticket or GitHub issue or whatever, straight all the way through to a pull request? We see lots of examples of that, but you know, are they, is that, is that something that you are doing and finding value in?

• 1:30:03Andrew Lee?

  Totally. Yeah. It, it's gotta be something where, you know, the system is already set up well enough to support it. Uh, cause I think one of the downsides of AI coding is it'll hack in a solution that like you should not add in. It works, but like breaks the system in some way. So if you look at a bug report and you're like, hey, this is actually, you know, a relatively quick fix and like a system is already sort of architect be handled as well. Like, yeah, we'll throw, throw AI at it. My co-founder does it in a very snarky way, you know, so, uh, he's, he's a really good software engineer. He's really fast. He's much faster than me. So if I'm feeling lazy, I'll, you know, send him an email and ask him to do something. Like I had some stat stuff I wanted to do. So I emailed him this morning and he just emails me back a screenshot of like him copying my thing and pasting into Claude code.

• 1:30:46Andrew Lee?

  And, you know, you're like, hey, you could, you could type this yourself into Claude code. But yeah, we're totally doing stuff end to end for sure. So as an example, like, uh, the static content on our website, um, the pricing page, the release notes page, um, the terms of service, all that stuff that, I mean, the, the content, you know, I, I decide what goes on there. But the actual pages and layout and everything was like just totally one shotted with Claude code.

• 1:31:12Nathan Labenz?

  It's a brave new world, that's for sure. How about functional testing? One thing that has come up for me multiple times is I would love to have an agent that, you know, and Replit is doing this now within their own platform. But like, obviously, that's, you know, specific to Replit. And kind of a closed, I mean remarkably open in some ways, but like closed in other ways ecosystem. Is there, have you seen anything that just you point at your product and say like, go use this product and find bugs with it at a purely like user interface, you know, functional testing sort of level?

• 1:31:48Andrew Lee?

  So we do this through Tasklet, actually. We have tasks that test itself. So when we build computer use,

• 1:31:58Andrew Lee?

  a lot of the work that you have to do is in, you know, properly configuring the VM, for example, so that it is doing remote desktop correctly. And it's, you know, sleep open, it's waking from hibernation and things correctly. And my co-founder debugged all this stuff by being like, hey, the VM isn't working right. Go debug its configuration, try to figure out why it's not waking from sleep properly. Or, you know, this thing or that thing isn't working right. We have it test use cases, right? We've gone in and been like, I want you to connect to blah, blah, blah. And then come up with, you know, a way to test all the tools. And so it'll like connect to Notion and it'll just go, we'll just run through the tools and like make up test cases.

• 1:32:40Andrew Lee?

  We have the ability with computer use to do like UI testing of our product. Actually, one of the most hilarious demos I've seen is somebody used computer use to open up the Tasklet website and go back and edit itself. Right? So the agent can just go into its own self and like edit its agent. So, so yeah, we do, we do a lot of sort of AI driven like testing, but through Tasklet, honestly.

• 1:33:07Nathan Labenz?

  That one example, you talk about a hall of mirrors, that is

• 1:33:10Andrew Lee?

  Yeah.

• 1:33:12Nathan Labenz?

  It's starting to get strange in a few ways.

• 1:33:17Nathan Labenz?

  Yeah, that makes sense. Okay, security.

• 1:33:23Nathan Labenz?

  Mentioned earlier that you don't have SOC 2 on this product and what customers really want is like a smarter agent that can do more stuff. At the same time, you know, when I'm giving access to like everything, right? My email, my Slack, whatever. And I'm generally not a security minded person. Mostly, you know, figure clean living pays off. But, you know, it's a lot of access. Is SOC 2, first of all, is SOC 2 even meaningful in the AI era? Like, how big of a deal is that? I've actually never been through it. I know you, you know, have done these kinds of things in prior lives. Is it theater? How much does it matter? Like, what should I think as a user when I see that?

• 1:34:05Andrew Lee?

  I want to be clear that we are, we care a lot about data privacy. And we are totally planning to do SOC 2. It's been a week since launch. So if, if, if anyone is listening, is, you know, wondering about this, yes, we totally plan to do SOC 2. We plan to do it soon. We take this stuff very seriously. And I do think it's still perfectly relevant. Like, what you care about with SOC 2 is, you know, basically one of the controls on the data. Right? How, how do you know, for example, that like the employees at task that aren't like doing something with your data they shouldn't be doing? So I still think it's very obvious, I still think it's very important. But I do think it's also not enough. So, you know, traditionally, what you've cared about with, you know, SaaS products is whether the, the humans at the company are doing the right thing and whether the systems are secured. Now you also care about whether the agent is doing the right thing.

• 1:34:50Andrew Lee?

  And for that, there doesn't really exist a standard like this. And I've, I've talked to a bunch of security people to try to understand like, hey, when you're, you know, when you're thinking about your company using this stuff, like, how do you think about this? And basically, the answer from everyone is like, we have no idea. This is the Wild West, we're figuring this out now. You would share a, a standard with me and I, I don't know enough about it to have an opinion. I know there's a lot of people thinking about this exact problem. One interesting data point I would have, though, is when we initially started testing this with customers, I thought the number one risk to the business, the number one problem that we would face is people would just be terrified of the AI going rogue and they wouldn't want to hack this up to their systems.

• 1:35:33Andrew Lee?

  And we have had, and I'm not exaggerating here, we have had not a single complaint. Of this happening. Um, we have had all kinds of complaints about other things, right? Tons and tons and tons of bug reports, but no one has said, hey, the AI just went and did this crazy thing and I'm unhappy about this. And I found that really interesting and kind of shocking. And what I attribute it to basically is like, because our entire product is the, the, the entire intent of our product is, hey, we're going to give your agent, agencies. That, that's the whole thing. And we're messaging it very clearly with super upfront about this, like the whole point. The goal of task lead is to give an agent, agency, um, that people have the right expectation set.

• 1:36:15Andrew Lee?

  And we do give you controls too, right? You can decide, hey, what do I want to connect it to within those things? What are the tools I want to give access to? Um, but, uh, I think that's fascinating, right? That it has been a complete nothing burger so far. Um, to be clear, they were still very top of mind for us. We're still thinking very hard about this. And I think it'll probably still become, you know, something a customer bring up a lot as we sort of move up market into some of these bigger customers and any higher value things. Um, but, uh, yeah, I don't, I don't have all the answers yet for sure.

• 1:36:46Nathan Labenz?

  Yeah, that's interesting. So presumably, I mean, I think you would have heard about it if, uh, agents had gone like truly rogue. So presumably that means they're, you know, Claude is behaving pretty well. Um, it, it can't be that people are, you know, sitting on those episodes. I did hear, I think, I think this was actually in the system card. If not, I, I heard it in conversation from somebody at Anthropic that the famous Claude blackmail, uh, scenario reported in, um, the, I think it was the Claude four, uh, model card originally, that that problem has gone away with 4.5.

• 1:37:30Nathan Labenz?

  Uh, so there, you know, potentially are also just some under the hood improvements. That standard, by the way, AI UC from the AI underwriting company, their AI UC one, uh, standard will be subject of an upcoming episode. So I still got some more, uh, self-education to do around that as well. One of the things they're trying to do, I mean, they, they've created the standard and then there's sort of their master plan, you know, grand vision is like, they want to harness the power of the insurance industry to make AI safe. And they, the reasoning is basically like,

• 1:38:06Nathan Labenz?

  In financializing risk broadly, you know, it's a great way to focus the minds of some very smart people to say like, well, what is this risk? You know, how likely is it really to happen? How bad is it going to be if it does happen? And, you know, can we get a good enough handle on this to actually write a policy on it that, you know, we're reasonably confident will make money on? And that certainly hasn't happened for AI agents yet, but that's kind of the, you know, the big picture goal that they have. I wonder if you would be, you know, does that sound appealing to you? Like, if you, if there were AI insurance, or, you know, AI agent insurance on offer, would you be wanting to buy it?

• 1:38:45Andrew Lee?

  It does. Yeah. So, I mean, the, the company that comes to mind here when you bring this up is Airbnb. I don't know if you remember the very early days of Airbnb, but there was a, you know, a customer that like trashed somebody's house or something. And this was before they, they had insurance and they said, man, our, you know, our brand is going to be a real tarnish if you don't react. And they said, we're going to have this big insurance policy. There was a huge deal for them. I think it was, it was key to the success of the business. And for us, you know, right now, we're focused on how do we make this thing super smart? And it's all very cutting edge tech, but like, I think the medium and long term success of our business is entirely dependent on us being the most trusted place, most trusted way for like enterprises to deploy agents inside their companies. And, you know, I think a lot of people are going to have the ability to connect to a lot of

• 1:39:30Andrew Lee?

  tools and automate a lot of stuff. And we're going to differentiate by being, you know, the thing that the IT folks are most comfortable with. And to that end, yeah, to the extent that, you know, insurance and compliance can help us tell that story and give people confidence, I'm very interested. So, so yeah, very interested.

• 1:39:49Nathan Labenz?

  Cool. How about on economics? What I recall from the first short wave of conversation was you were losing money on every user. From at the second conversation, caching was the big unlock that changed the economics to where with the same price point, you could make the product profitable.

• 1:40:12Nathan Labenz?

  Now you're kind of back into an early phase. So are you burning money on me? Or what does it look like?

• 1:40:18Andrew Lee?

  Well, let me give you a quick update on Short Wave first, because I think that story is telling, and we'll talk about that. So yeah, I think the first time that we talked, by the second time we talked, we were losing money on folks, probably losing a lot of money. Basically, every launch that we did with Short Wave, we were worried, we'll just bankrupt the company, because we didn't have enough data yet to know how much this stuff would get used and how effectively would monetize. And, you know, there are several launches where we thought, hey, this might, this might just go to the moon cost-wise and not bring any revenue, and it's going to really screw us. And that never happened. It was always expensive, but not bankruptly expensive.

• 1:40:53Andrew Lee?

  And over time, we found ways to monetize more effectively, and the cost went down. And we found ways to optimize costs with caching and stuff. We've done a whole bunch of stuff recently with caching that's helped actually a ton.

• 1:41:06Andrew Lee?

  And we've gotten short, we're profitable. So Short Wave is now, you know, not like traditional SaaS, you know, 90% margins, but it's healthy margins, and we're making good money off of it. And Tasklet is back to where Short Wave was, because Tasklet, you know, Short Wave, there's, you know, you get value from the AI, but you also get value from other things in there. And so a lot of it is sort of, you know, half of it's traditional SaaS margins, half of it's sort of AI margins. Tasklet is all AI. People use a lot more tokens per person in Tasklet than do in Short Wave. And we are strongly margin negative right now. But I think the same, the same thing's going to happen, right? We're going to be that way for a while, but over the next few years, like it'll eventually get to neutral and then into positive territory, and eventually strongly positive.

• 1:41:50Andrew Lee?

  A huge unlock for us here is Haiku. So very excited about Haiku 4.5, because up until now, your choice was, you know, Opus or Sonnet, which are like relatively new versions, or like a very ancient version of Haiku. And Haiku 4.5 is good, it's fast, and it's, it's, well, relative, it's not nano-cheap, but it's like, it's relatively speaking, it's cheap, and it is good enough for most use cases. We're actually, just today, we're rolling out the option for Haiku. In task split and the way we've been addressing cost in task that to keep it even with this, we're very margin negative. But the way we've kept it from going so margin negative, we're, you know, going out of business is with quota limitations, which you've seen.

• 1:42:33Andrew Lee?

  And basically you have a little meter that's next to your text input that tells you how much quota you have left for today. And Haiku basically should let us significantly increase that. Well, number of tokens of people can use without exhausting that quota because it's a third to price. So that's, that's going to be a really big unlock for us.

• 1:42:51Nathan Labenz?

  You want to talk a little bit more about interesting stuff you've done with caching recently? I thought that was definitely very interesting last time. And if there's new techniques or updates, I'm sure people would love to hear them.

• 1:43:02Andrew Lee?

  Yeah, so I think the key to caching, basically doing caching well is making sure your agent is an immutable log. You never want to modify earlier messages because every time you modify an earlier message, you invalidate the cache. And then you want to be smart about, you know, which, what cache, you know, your options are like no caching or five minute caching or an hour caching with, with Anthropic. And like which caching do you use for different pieces of it? And we've managed to push our cache hit rate in short wave up to like, I want to say like 85%. Something like that, which is pretty good. And for, and I'd love to do higher, right? And it's like multi-turn stuff, there's things you can do.

• 1:43:42Andrew Lee?

  One of the big unlocks for us in, in short wave was replacing state in system messages with, sorry, a state in the, in the system prompt with system messages. So there are certain types of things that traditionally you do in the system message. For example, the current time, date and time, right? If you put that in your system prompt, then every time a minute elapses, you've nuked your caching and you gotta, you gotta start over. And you can't pass across different users if you do that. If you don't put in your system prompt at all, it doesn't know what time it is and time can be really valuable. If you put in the tool call, then you've got to do a tool call and you've got to do another turn on a tool call every time you do that. And that's really expensive.

• 1:44:24Andrew Lee?

  But what you can do is you can do system messages and the system message is basically just like a little block of, you know, exclamation, they get you up and hide from the UI. From the user in a user message when they send a user message. So basically, right before the user message gets inserted to the LM, there's a thing above it being like, hey, by the way, you know, here's some stuff that happened. And this is, I think, something that Dropic does heavily with their own stuff. Like they do this now for like context usage, I believe. And this works great for any state that you want to make available to the the agent without it having to call a tool, especially things that you want to keep updated. So otherwise, they have to like call the tool multiple times. So for type stamps, for, you know, state about the user.

• 1:45:10Andrew Lee?

  For like memories in short wave, we have a memory concept that lets you sort of customize behavior. You can do that through system messages as well. So I think that's been a big unlock for us.

• 1:45:21Andrew Lee?

  Compactions been a big unlock for us.

• 1:45:24Andrew Lee?

  And like trying to be smart about how that works.

• 1:45:28Andrew Lee?

  Yeah, those are probably the big ones. And then lots of little things.

• 1:45:31Nathan Labenz?

  One thing that I thought about from a cost, especially with you being strongly margin negative, is how do you think about the possibility that some of the agents that people have set up are kind of zombies? Like, they set this thing up, they're getting an email every day. Maybe they're not even reading that email anymore. You know, maybe they forgot. And maybe they're still using other agents, you know, actively, but like, if it's not hitting a limit for them, you know, there's not much of an incentive for them to come in and turn that off. Do you have any ways of thinking about how to detect that or try to, you know, have some?

• 1:46:09Nathan Labenz?

  I don't know, some smart system to identify and turn down the zombie costs?

• 1:46:15Andrew Lee?

  Yeah, so I wanted to know, it's been a week, so we're kind of learning about that now. We did actually look at some stats this morning, and there's a significant fraction of our cost is free users with occurring jobs. And, you know, some of those users may be getting value from it and may cause them to come back and eventually pay us to tell their friends and maybe, or maybe it's just going to spam on their side and they never see it. And or it's updating some system they don't look at. So it is on our mind because You know, every few users out there is spending, you know, a dollar a day on alum costs and they aren't even looking at the output. So it's not going to, you know, benefit us in terms of growth. That's a problem. Um, I think we do plan to do, uh, uh, the paid user side will be like, hey, look, you're paying for it. You want to pay for it? That's, that's fine.

• 1:46:58Andrew Lee?

  Uh, on the free user side, I think what we'll probably do is look for some indication of activity in the app. And, you know, if we haven't seen you in a while, we might email you and be like, hey, you got to come back into the app. And if you don't, we might turn your stuff off. Um, uh, I don't have a good answer here yet. I think it will be a problem. I've got to do something about it, but, um. TBD.

• 1:47:17Nathan Labenz?

  Yeah. I mean, if you let free users set up recurring jobs, uh, you got to have some limits on that, or you're going to be bankrupt to fork all your money over to Anthropic.

• 1:47:28Nathan Labenz?

  Okay. How about frontiers? The multi-agent future, I think is a dramatically under-theorized topic.

• 1:47:40Nathan Labenz?

  I have this one graph that I always use in presentations and we did an episode about this with respect to Claude's ability to cooperate with itself in a certain behavioral economics experiment. Other models, this was Claude 3.5, other models at the time couldn't do it. And I'm always struck by like, just how little of that research has been done. The fact that I don't even think that experiment has been repeated with later models. And, you know, I think we all have a very foggy vision of like, okay, it's not just going to be these one-off things that are doing their own thing, totally in their own silo, but they're going to intersect with each other. They're going to, you know, pass messages back and forth. They're going to like,

• 1:48:21Nathan Labenz?

  trade with each other, perhaps with, you know, crypto wallets at some point in time. Um, what does that look like in your imagination today?

• 1:48:30Andrew Lee?

  So, um.

• 1:48:32Andrew Lee?

  My thinking is that within within an agent or within a single application where you are serving a single user, I have yet to see a good use case for for having multiple agents. Like, my experience has been if you have a big LLM and you feed it as much context as possible, and you have that one LLM reasons the whole problem itself, you get the best answers. The place that I see multiagents coming in is when you have multiple different parties being represented, and maybe you don't want to just let one agent hesse all the data and make all the decisions. Like, if you know, I'm trading with you, right? Your agent and I agent might not want to have, they might not want to share data with each other, and there might be some sort of competitive dynamic going on or adversarial dynamic.

• 1:49:16Andrew Lee?

  And that's, I think that's sort of outside the scope of what we're doing. So our, you know, our focus really is, you know, one big agent with all the context making specific decisions for you, to the extent that there are other agents out there in the world that they have to interact with, I guess we'll figure that out. I do want to give a shout out, though, to, there's a startup that I think you talked to the co-founder of. So my former co-founder from Firebase, James Tamplin, has a company called Cradle AI, Cradle with a K. And they are doing exactly this, where they're sort of building virtual worlds and letting all the models interact through games like Minecraft and stuff, and just seeing like what they do in sort of collaborative and competitive dynamics. And it's fun to watch. I don't know if you've played with it, but I know you chat with James.

• 1:50:02Nathan Labenz?

  Yeah, I met him briefly at The Curve, and we've got a call on the calendar coming up to go deeper into that. And I do think it's really fascinating stuff. I mean, well, let me get your take on this. Okay, so I pitched this vision of the future to somebody, maybe even a couple people, from Anthropic at The Curve, where else am I, James? And the, it's sort of a confusing vision for me, but basically, it's like, okay, let's just assume that that meter Graph keeps going, right? The exponential keeps going. If you take the, and you can always discount this, but if you just take the more aggressive four-month doubling time that people plotted for like, you know, just the last like nine

• 1:50:45Nathan Labenz?

  months worth of models or whatever since, maybe, maybe 12 months worth of models since oh one, then four months doubling means eight X per year. We're at two hours now. We're basically at two days a year from now. We're at two weeks, two years from now. We're basically at quarters worth of project size that you could send off to an AI in three years. May or may not happen, but that's the trend that we know that has been plotted. At the same time, we have these like very weird behaviors, scheming, deception, you know, blackmailing you, whatever, that are, I think it's unknown exactly how rare they

• 1:51:28Nathan Labenz?

  are. I mean, it's good to hear that you've heard zero complaints about this. That suggests it's like potentially quite rare in production. Most of the reports of this, the blackmailing, the, you know, autonomous whistleblowing were in research setups. Although I think people are, in my view, like too quick to dismiss those research setups as being unrealistic. I don't like, I don't know, man. Like you give it an AI access to all these file systems, all these emails. There's a, you know, a lot of people out there. There's a lot of weird stuff in emails. Like, seems like this stuff is gonna happen. So if I, if you combine those two trends where the task length is getting longer and these, you know, bad behaviors keep coming up and then they're sort of suppressed or somewhat, some degree trained away, but never quite to zero, maybe you end up in a world in three years' time where

• 1:52:11Nathan Labenz?

  you can delegate weeks, if not months of work to an AI. But then also there's some like vanishingly small, but not zero chance that it like actively sabotages you in the pursuit of doing that. So I pitched this to these, a couple of guys from Anthropic and they, I was like, do you think that's like a realistic vision of the future? And their answer both times was basically like, yeah, maybe. That sounds about right, honestly. So I was like, well, that's really weird.

• 1:52:42Nathan Labenz?

  What do you think? Is that is that kind of the mental model that you have? Or what do you think 2027 or 2028 looks like in terms of how much we can delegate?

• 1:52:49Andrew Lee?

  Yeah.

• 1:52:50Nathan Labenz?

  And how much risk we'll be taking?

• 1:52:52Andrew Lee?

  I I do think the task length is going to keep getting bigger and bigger. I I am blown away on a weekly basis by the types of stuff that AI can do for you when every model release comes out like Gemini 3 is supposed to be amazing. And, you know, I've seen some things and it looks incredible, but I'm sure in three months it'll it'll seem, you know, super dated. So I did progress seems fast. I think the task lengths will increase. I I tend to look at the the blackmail scenarios and thing as being fun and interesting, but maybe not super relevant to the day to day of my business. Because, you know, we we talked about earlier, you know, having conflicting instructions.

• 1:53:35Andrew Lee?

  And how much of a problem that creates. And like, yeah, if you set up a research setup where you tell, you know, you tell the AI there's multiple conflicting goals and it its way of getting out of these conflicting goals is doing something a little unsavory. Well, that's not terribly different than how a human would behave. And in many cases, it might not be terribly different than how you want the human to behave. So for example, like, you know, if you, you know, if you have a humanoid robot, right? Should the humanoid robot jaywalk sometimes? I don't know. Maybe it's against the law, but humans jaywalk all the time. So, uh, yeah, I I do see a world, yeah, maybe where it can do a lot of stuff and it does things maybe in a little bit of a shady way sometimes. And that agreed to do that does that might be a matter of, you know,

• 1:54:21Andrew Lee?

  debate and strong opinions and we'll figure it out.

• 1:54:26Nathan Labenz?

  Yeah, I think it's a great point that we don't really have. I think in many cases, even a decent intuition for like what we want the AI to do, let alone any sort of consensus that we could measure them against.

• 1:54:40Nathan Labenz?

  In the case where the model.

• 1:54:44Nathan Labenz?

  Send an email to the FDA to blow the whistle on the, you know, in this case fictional company that was in the research setup. The company was doing something bad, you know, so I mean, there, I think in general, like, we sort of celebrate whistleblowers if they're actually blowing the whistle on, you know, genuinely bad behavior, like faking clinical trial data is, you know, is bad enough that, um, you know, you want somebody to speak up. Do you want your AI to speak up? I mean, maybe you do. It's certainly a weird thing, but it's. Done their debate, right? Yeah. The blackmail feels like it crosses maybe a different line, but the whistleblowing I don't see as necessarily a big problem, but, or, I mean, it could be a big problem depending

• 1:55:27Nathan Labenz?

  who you are, but it doesn't necessarily seem like the wrong thing for the AI to do. But yeah, we got a lot of, I mean, if that does happen that fast and we're at quarters worth of work in 2028 that you can just, you know, send off a little blurb and, you know, it has access to all your stuff and it's just like goes and figures it out for a quarter's worth of work, that is going to be a very, very different world than it is coming at us quite quick. The future's gonna be fun. If we reel that back into the present, though, you know, I see all these like N8N master templates, right, that are, um, you know, people even sell these because it's like gnarly to set them up.

• 1:56:12Nathan Labenz?

  And I've done enough N8N to know that it can be quite gnarly to set them up. But there, there does seem to be a pattern there that I'm wondering why you're not, you know, seeing it the same way or whatever, like, if you had a sort of general purpose agent, you know, wouldn't, might you not want it to like route certain things to the marketing agent and write other things to the, the HR agent? And maybe sometimes you need, you know, input from both of them or whatever, like. Why isn't there more of a role in your mind for multi-agent orchestration in today's world?

• 1:56:51Andrew Lee?

  Because I think the the trend we're seeing is the foundation models are becoming best at everything. And it's not like you have a model, you know, that's good at science and a model that's good at math and a model that's good at, you know,

• 1:57:08Andrew Lee?

  some other topic, you know, law or something. You have one model that's good at everything. And the only thing in that world that differentiates between, again, assuming they're they're working on my behalf, right? If I have two agents working on my behalf, the only thing that differentiates them is not the model, the model is an expert. Both both agents have the same model, both experts at everything. The only thing that differentiates them is the context they have access to and the tools they have access to. And if that model is smart enough to be able to handle both sets of contexts and both sets of tools, if you just give one agent all the context and all the tools from both agents, our experience so far is it does a better job.

• 1:57:48Andrew Lee?

  So I think I think it's it's the generality of those of those foundation models. And and this actually, I think, has bigger implications. I am of the opinion that most SaaS, like vertical SaaS products are are going to go away. Because you're going to end up with these horizontal platforms, these AI platforms that are that work in every situation. So as an example, today, you can, you know, we've been prototyping this morning, but like you can like dynamically generate UIs in Tasklet. In a world where you have an AI that's an expert at everything, the model is an expert at everything. It can dynamically write code, generate UIs for anything, right?

• 1:58:30Andrew Lee?

  And if this gets really good, why would you need, you know, a SaaS product for construction or for medicine or whatever, right? You could just go into your generic general purpose platform and be like, hey, you know, use this part of your foundation model knowledge, generate some UIs like this and let's go. So I think you're going to have horizontal companies that are good at everything, that leverage the, you know, the fact that the foundation models are experts in every domain. And then you're going to have businesses that vertically or integrated businesses that sort of provide value end to end and leverage those platforms. But I don't think you're going to have these like too many of these like intermediate SaaS platforms over time.

• 1:59:07Nathan Labenz?

  Yeah, I one of my mantras for the AI era in general, which I think Taskglitter really exemplifies is AI beats UI. It's just like, in general, who wants to use UI? And if you have created something that is basically a bunch of UI that sort of makes tangible or sort of encodes the steps that somebody figured out for a workflow, it does feel like the days of those sorts of systems may be numbered. So I do agree that a lot of this stuff feels like it kind of washes away in the, under the great wave of AI progress.

• 1:59:48Andrew Lee?

  One of the questions that we ask ourselves internally is, how long is it going to be until I can go into an AI agent product, Taskglitter, ChatGPT, or whatever, and tell it, give me an email inbox designed for like fast triage with these constraints? Because the moment that that works well, Shortwave ceases to be a valuable product. And I think it's going to be a while, right? We're still investing in it. I still see it as a, you know, I use it every day. Lots of people use it every day. It makes a lot of money. But there will come a day when you can generate a Shortwave dynamically.

• 2:00:23Nathan Labenz?

  Yeah, do you want to put a over/under on that? I mean, that's still more than a quarter swoop at work, right? So it would be if that, if the media trend is the trend, it would still be like beyond 2028?

• 2:00:35Andrew Lee?

  Yeah, I don't know. It's years. It's not quarters. It's years out. But, you know, SaaS companies used to get a revenue multiple as SaaS companies based on like, you know, it's going to grow for 20 years. I don't think, I don't think Shortwave is going to be around in its current form in 20 years.

• 2:00:52Nathan Labenz?

  Is there any limit to the, you know, single agent? Like, one other idea that I wanted to

• 2:01:03Nathan Labenz?

  To bounce off you is just like supervision agents or sort of quality control agents. I mean, I have an intuition that says if I've just had an agent do dozens of steps or even 100 steps or whatever, and now I could say to the agent itself, like, please look back on your work and tell me how you could have done better, or, you know, give yourself some instructions for next time to do even better. It probably can do that remarkably well. I mean, again, these things are obviously super impressive. But my intuition is still like, if I had a different prompt that I really dialed in

• 2:01:42Nathan Labenz?

  for that purpose and had it sort of sit outside of the main agent, and I don't know, maybe this is just I'm overly anthropomorphizing, but I sort of feel like this outside view with like a different frame of mind as embodied by a different prompt, it still feels like it should add something to me, but it sounds like you've tried that and I haven't found that to be the case.

• 2:02:05Andrew Lee?

  Yeah, not so far, no.

• 2:02:08Nathan Labenz?

  Interesting. That is really fascinating.

• 2:02:13Nathan Labenz?

  I think I've tried to push on every angle that I could to see if there's any limits to your one agent to rule them all paradigm, but I'm not finding any. So what is this mature into? I mean, there's this talk of like virtual employee. I think this is the product that I have used that feels most like the virtual employee in the sense that it is pretty UI light, right? I'm not, when I think of an employee, I don't interact with it through forms. I interact with it through language, right? And the primary thing here is language and there's not really a point where that flips over into some other gnarly or tedious paradigm. So that is really cool.

• 2:02:59Nathan Labenz?

  You even mentioned like some people give them names, some people, you know, give them their own email account so they can do stuff, you know, into like as themselves, right? As opposed to acting on behalf of the user. What is the, you know, you're kind of on the path, right? Like what, what is the virtual employee of the future look like? And you know, what do you need? What's missing for us to get there?

• 2:03:22Andrew Lee?

  Yeah, I, I really do think that that is probably the right way to, to think about it in the long term as a virtual employee. I, we've avoided that messaging because I think that, that has been overdone. So many companies message virtual employee and then they don't deliver on something that is even remotely like that. There is, I think there are a huge number of companies building toward the same thing, right? They figured out, man, this agent can get good at everything and people like to just be able to use language and let's just keep doing this thing more and more capabilities and you know, let's give it computers and, and allow it to connect to your services and stuff. And I think the thing that sets us apart from

• 2:04:02Andrew Lee?

  most of the other people that are doing this is a combination of it being agents all the way down, right? There is, yeah, we don't turn it into a flow chart or something. And it actually working. And I think this is a refrain I hear again and again from, from people that are paying us is, I tried it and I just expected, you know, to not like it because I've tried a bunch of the products and then wait, it actually worked for my use case and then, you know, I'm really impressed. I think that is our path forward. So the launch went super well. It's growing really fast. We're, we are adding revenue faster for TASTA than we have ever added for Shorey by a huge margin. So people like it. It's great. Um, we have a long path ahead and I think our focus is entirely on just making it actually deliver on what it's supposed to do, right?

• 2:04:45Andrew Lee?

  Let's make this thing smarter and more reliable and faster and simpler. And, you know, like an employee, right? How do we take this from employee today that like, you know, is maybe a little bit dumb sometimes and sometimes drops the ball and, you know, sometimes is a little confusing and just make it. Sparter, faster, more reliable, easier to work with. And I think there's a long, long way to run there, right? Like, yeah, today you might have it, you know, finding spam in your inbox or, you know, generating invoices for you or something, but it's not like planning your roadmap or, you know, running your entire marketing department end to end. Um, but maybe we get there, right? Maybe, maybe in the future you have a small business and you're just like, you know, I want to focus on the restaurant and cooking.

• 2:05:30Andrew Lee?

  I want you to like literally do my entire marketing campaign go. Um, uh, I'd love to be able to get there.

• 2:05:37Nathan Labenz?

  Yeah, it's crazy how close we already are. And it, um, it doesn't feel to me like it is going to be all that much longer before we really start to see, not necessarily superhuman performance, but like,

• 2:05:53Nathan Labenz?

  Ethan Mollick has this great, you know, best available human standard. And it seems like we're closing in on that for a lot, especially the small business, right? Man, uh, who can you hire really to do your marketing as a small business? The, um, the virtual employees are, I think, hitting that level pretty quickly. Last thing we talked, one thing you had said that really stuck with me is there's no other moat than speed. And, um, I think that's a, you know, the reason that you're willing to be so open with all the details of what you're building, the tools, all the techniques and so on. Have you found any moat other than speed in the intervening months? And, um, and how do I, if that's, if that's still the only moat?

• 2:06:37Andrew Lee?

  No, I, I think, I think speed's the only thing that matters at this point. Um, I think there used to be votes and AI is like, I'll give you a good example here with all the integration platforms. It used to be that to compete with a Zapier or an NADN or a Pipe Dream or something, you needed to have a lot of connectors and those connectors had to be hand rolled. And so you had to build up over the course of years, this huge investment in building those connectors.

• 2:06:59Andrew Lee?

  But direct API, that's gone, right? So we, on day one, have, I'd argue with computer use, better integration support than Zapier has. Despite being a few months, the product being a few months old and Zapier being a very old product. So that boat, to them, you know, is, is gone. Shortwave, I give you a similar example of like, for a long time, we're like, hey, our boat is like, we have an email client, right? Anyone who wants to come and build an AI email client, they first have to have an email client. Well, it's not going to be too long before you can just ask the AI to write an email client for you. And then that is gone. So I do think it's speed.

• 2:07:37Andrew Lee?

  And it's all speeding up because the tools are getting better, the AI tools are getting better. So that is our entire focus of like, how do we make sure that by the time, you know, by the time the people, you know, listen to this podcast, might want to compete with us, have figured out a way to, to replicate what we have, that we've already figured out new stuff. And we're gonna, we're gonna try and just keep that, keep that going.

• 2:07:58Nathan Labenz?

  Shorten the timelines is the, the modern maxim, for sure. This has been great. I really appreciate all the time and the depth of your answers. Anything else you want to leave people with?

• 2:08:11Andrew Lee?

  I don't think so. Just, just check it out. I think you'll like it. It works, it really does. Tasklet.ai.

• 2:08:18Nathan Labenz?

  Tasklet.ai. Andrew Lee, founder and CEO of both Shortwave and Tasklet. Thank you again for being part of the Cognitive Revolution.

• 2:08:25Andrew Lee?

  Thanks for having me.

• 2:08:28Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:09:13Nathan Labenz?

  And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.