Shortwave Rides the Tidal Wave: Inbox Agents, Hyper-Growth & Hiring AI Managers, with CEO Andrew Lee

In this episode of the Cognitive Revolution podcast, Andrew Lee, founder and CEO of Shortwave, returns to discuss the rapid advancements in AI over the past year and how they have significantly improved Shortwave, an AI email assistant. Andrew shares insights into the exponential growth of Shortwave's revenue and the enhanced capabilities of their AI, which now functions more like a virtual assistant. They delve into various use cases, the technical evolution of their platform, the impact of new AI models, and their strategic decision to shift from being an AI-enhanced email client to offering a broader AI-driven communication solution. Andrew also talks about the shift in company culture towards an AI-forward approach, the importance of speed and agility in the AI space, and the increased productivity achieved through leveraging AI. The conversation also touches on the future of the software industry, the potential of AI to automate routine tasks, and the company's hiring strategy focused on people who are passionate and forward-thinking about AI. PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (04:00) Introduction and Welcome Back (04:07) Shortwave's Evolution and Revenue Growth (05:09) AI Email Assistant: Then vs. Now (06:44) Exciting Use Cases of Shortwave (14:11) Technical Deep Dive: Database and Search Stack (23:44) Agent Behavior and Iterative Approach (34:04) Model Selection and Cost Optimization (39:53) Future of AI and Convergence of Providers (42:54) Exploring the New Cursor Agent Mode (44:07) Understanding AI Filters and Their Functionality (45:38) Challenges and Solutions in AI Email Management (49:24) Evaluating AI Models and Their Performance (55:53) The Role of AI in Enhancing Email Communication (57:05) The Future of AI in Communication Tools (01:12:35) Building an AI-Forward Culture at Shortwave (01:17:14) Leveraging AI for Content Creation (01:17:57) The Shift in Team Dynamics (01:19:36) Adapting to Industry Changes (01:20:39) Optimizing for Speed (01:22:45) Monetization and Pricing Strategies (01:25:42) The Future of AI Integration (01:36:36) Hiring and Team Structure (01:43:28) The Future of Software Development (01:47:18) Closing Thoughts and Future Outlook (01:50:03) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, I'm excited to welcome back Andrew Lee, founder and CEO of Shortwave, for a conversation about the incredible speed of AI progress, how Shortwave is maximizing agent performance with today's frontier models, and fundamentally reimagining digital communications, the ongoing transformation of the software industry at large, and company building for the AI era. The impetus for this episode was a beautifully exponential revenue growth curve that Andrew recently posted on Twitter, the sort that you can really only achieve with genuinely word-of-mouth-worthy practical value. Over the next two hours, Andrew takes us on a tour of everything that he and the Shortwave team have done over the last year

• 0:42Nathan Labenz?

  to transform what was a useful but perhaps not quite transformative email assistant then to an email agent that is now routinely surprising and delighting both users and Andrew himself with the increasingly complicated projects it can tackle. So much so, that Shortwave is now expanding beyond email and reconceiving the product as an AI agent to help manage communication across all major channels. Having concluded that AI makes software so much easier and faster to create, such that "speed is really the only moat going forward", Andrew does not hold back on lessons learned, and the technical insights here are outstanding. Andrew breaks down how they've completely rebuilt their infrastructure at every level by constantly testing and swapping in new models,

• 1:26Nathan Labenz?

  moving to Pinecone's serverless offering for their vector database, and adopting a hybrid structured-plus-vector-search paradigm that delivers better results at lower cost. Perhaps most fascinating is Andrew's perspective on agent architecture. While many companies are pursuing multi-agent approaches with specialized subagents, Shortwave has found better results with a simpler approach that makes careful use of Anthropic's caching features to support long-running tasks with lots of context while also maintaining positive margin unit economics for the business, but otherwise largely trusts Claude to act effectively as an agent, both by calling the right tools and determining for itself when it's found what it's looking for. Personally, I've had a number of wow moments as a user.

• 2:11Nathan Labenz?

  I was honestly a little nervous to allow it to organize my inbox for the first time, but now I'm making regular use of the conceptual to-do lists that it's created for me. And it also saved me a cool 30 minutes the other day when it collected all receipts from a recent trip and compiled them into a tidy expense report. In the last third of the conversation, we turn from the product itself to the question of how to structure a company for success in the AI era. Having recently closed another round of venture capital, Short Wave is hiring for a number of roles, which Andrew describes not as traditional individual contributors, but as AI agent managers across software development, marketing content creation, and more. He plans to keep the team quite small, targeting just 15 or so employees for the foreseeable

• 2:54Nathan Labenz?

  future, and prioritizing talent density and speed of execution above all else. With that in mind, he's offering a $10,000 referral bonus, including to listeners of this podcast. Finally, before getting started, I want to note that this episode is brought to you ad-free by Short Wave. I've mentioned in the past that we are experimenting with sponsored episodes that allow companies with a timely story to cut to the front of the line. Of course, my commitment to you, the audience, is that our bar for interesting content and my preparation process will remain the same as always. Andrew and Short Wave were really a perfect fit for this opportunity. Their product is clicking, their business is booming, and he was eager to get his hiring message out sooner rather than later. As always, if you're finding value in the show, please take a moment to share it with

• 3:38Nathan Labenz?

  friends or colleagues who might be interested, leave us a review on Apple Podcasts or Spotify, and I always welcome your feedback, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, let's dive into this fascinating conversation with Andrew Lee about Short Wave's AI-powered transformation, not just of email, but now all digital communications. Andrew Lee, founder and CEO at Short Wave, welcome back to the Cognitive Revolution.

• 4:06Andrew Lee?

  Thanks for having me. It's good to see you again.

• 4:08Nathan Labenz?

  Yeah, it's been, boy, a lot gets packed into just a year in the AI space. And I was looking back, it's been just about a year since your first appearance on the pod. And indeed, a lot has changed. At the time, I called Shortwave the AI email assistant that I had been waiting for. And then, as we were kind of catching up in preparation for a second conversation, you said, at that time, it only kind of worked. And I was like, yeah, you know, I guess that's true. I kind of, I look back at these things that seem so mind blowing to me at the time. And obviously, you know, we've way surpassed them. But what caught my eye and got me to reach out again was you had posted a graph of Shortwave

• 4:49Nathan Labenz?

  revenue on Twitter. And, you know, it's basically the canonical exponential curve, where it looks like it's on the verge of going totally vertical. So to kick things off today, what's new and, you know, what is working now that is leading to such incredible growth that was only maybe sort of working a year ago?

• 5:10Andrew Lee?

  Yeah, it's, it's really been an evolution of that AI assistant that, you know, we talked about last year. I think the thing that you played with a year ago, you could chat with it, it could answer your questions, it did an okay job of searching, it did an okay job of writing your email, but it wasn't really that smart, and it wasn't really that trustworthy. And if you asked a question, you know, like, hey, where is the receipt for this? Like, maybe you'd find it, but you couldn't really trust that it found it. And it also couldn't do a lot of the normal stuff that you want to do in an email client. Like, if you go in there and you say, you know, hey, what are the most important emails I have? Or, you know, archive all the cold sales emails or something that I got? Like, it couldn't do any of that stuff, couldn't manage your to-dos, it didn't have no idea who

• 5:51Andrew Lee?

  your contacts were, your labels were. It was just sort of like a cool search and a writing thing, and like, maybe like a general person's AI thing. But it wasn't quite like having an employee sitting next to you, which is the pitch that we're trying to make. And we have iterated our way to a thing that actually really delivers on that. Like, you use the thing, and it kind of does what, like, a virtual assistant would do for you, and it actually works. And it can do almost everything that you as a human can do. So, yeah, I think it's just sort of reached a tipping point where people are like, holy crap, I can do my email, not by like, doing my email, but by talking to a thing that then does my email for me. And that lets me think about things at like, a much higher level, and be like, way more productive.

• 6:35Nathan Labenz?

  Yeah, I've experienced quite a bit of that and can definitely testify that there are some pretty wow moments that I've experienced. I mean, I don't want to start with use cases or sort of how it works under the hood. Well, maybe start with use cases. One that I tried, which I thought was really interesting, was to just ask it, "Take a look at my last 100 emails sent and give me whatever advice you have." And, you know, you learn, there's a lot, you got a lot in the outbox that says a lot about you. And I honestly thought it gave me like, pretty interesting advice, which was both apt in some ways, and then also made me think like, maybe I'm not exactly spending my time 100% the way I aspire to be spending

• 7:19Nathan Labenz?

  it, just because of the sort of balance of things that it was seeing. What are some of the other, you know, exciting use cases that you have got the most value from, or that you, maybe your customers have surprised you with?

• 7:30Andrew Lee?

  Yeah, email is a crazy valuable corpus of stuff about everything involved in your business or your personal life. So obviously, it has all your human correspondence, but it's got all your SaaS notifications, and it's got all of the attachments, you know, all the files, the PDFs and things that come along with that, and all your calendar invites. And like, we just know a ton of stuff about you. And I think if you would ask a human, like, "Hey, go through my email and give me some advice," and they were a reasonably smart human, and they had the time to do that, they would also give you some good insights. You just, you never do that, because it's a lot of work. I'd like, but the information's there. I think the prompt you sent me, like, you said that the other day, like, I think that's a super fun one, super cool. I actually, just this morning, I had a prompt where, you

• 8:12Andrew Lee?

  know, just yesterday, we rolled out a big UI change. And anytime we make big changes in our product, it's controversial, and we get a lot of feedback, and it's always scary. And I wanted to know, hey, how are people responding to this? And we have this new sharing feature that lets us share, like, all the support threads that come in, get shared across the whole team, and get available to the AI. And I asked the AI... So many people have emailed us and complained about the new layout in the last 24 hours. And I got a report, it was 19, there were 19 users who reported it. And it gave me a summary of here are the top five reasons. And like, it was super useful and super good. And like, for me to consolidate that myself would have taken me quite a bit of time. And I got a snapshot right away of like, okay, what's the reaction? How are people feeling about this? What are the top things we maybe need to address?

• 8:55Andrew Lee?

  So that was a big use case for me personally. We've seen a lot of other fun ones in the wild. I think one of the most common ones is people sort of start their day with a complicated prompt. So, you know, you're in sales and you have eight customer demos coming up. And you have an inbox full of emails from folks. And you're just like, hey, help me figure out like, what are the tasks I need to do? What order do I need to think about things in? What should I remember for each sales call? And they'll have this big prompt. A lot of time users will share these prompts with us of like, this is the thing I use to start my day. So that's a pretty common one. I think another really common one is people doing attachment analysis along with their email. So there's a lot of like, real estate agents and general contractors and architects and stuff like that,

• 9:38Andrew Lee?

  where they email them back and forth all these PDFs all the time. And they just need to answer one question from the PDF. Like, you know, what was the specific payment terms of this contract or something? And they can just ask the assistant and it can like, read the whole PDF for them and give them the one answer they need to like, write the email. There was another example that got shared on LinkedIn the other day with one of our users who was selling his house. And he needed an inventory of like, all the furniture that was in his house. And all that stuff was in his email somewhere, right? Either in like emails with his wife or receipts that were sent or whatever. And he was able to get a full inventory of like, all the furniture they had purchased over the years for their house in a nice consolidated, accurate report with just like a few keystrokes. So lots of interesting use cases. They're all sort of big prompts, but spanning like a whole wide spectrum of things.

• 10:25Nathan Labenz?

  I did a small version of that inventory thing after a recent trip where I had to collect and submit receipts for an expense report. And that was another, I mean, it's a very mundane task, right? But in a way, it's like the perfect job for AI to like handle these things that I, you know, otherwise kind of dread. And it was cool to be like, find me all the receipts for my trip, you know, there should be Ubers, Lifts, and a couple Door dashes in there, and just have the whole thing pop out with values and everything sort of itemized. And I was like, man, it's another one of these moments where I was like, this AI thing could really catch on. I mean, I'm sure there's more. Actually, I do think that one big barrier to practical

• 11:05Nathan Labenz?

  value in AI is just kind of lack of imagination sometimes on users' part. And I certainly feel like I'm guilty of that all too often. When I see something else, somebody, you know, has done, I'm like, why didn't I think of that sooner? Any others that you want to share, you know, at the top here that are just like, things that should get people's wheels turning about how they might, if they're just a little bit more, you know, intentional or creative, get more value?

• 11:29Andrew Lee?

  I'll share one of the most creative ones that I ran into. So this was another one that a user sent to me. And the thing is, like, I personally am still figuring out how to use the thing. And often the insights are like, user sends me a thing of like, why doesn't this work? Or check out how this thing works. And I'm like, I never thought of that. So here's what I never thought of. There was a user who was using us in combination with another SaaS tool. I think it was Linear. And they wanted to be able to like, extract action items from their inbox and like, add them as tasks to Linear. And we don't have a Linear integration. Maybe we should, but we don't. And turns out that the LLMs actually know the structure of the URLs needed to create tasks in these other SaaS products. And so if you go and ask the LLM, create a link that when

• 12:12Andrew Lee?

  I click it, will go and do this task in this other product, it actually works. And so he had a bunch of custom prompts where he's like, you know, extract action items from this email thread, you know, and then give me some links that I can click, and each one should create a thing. And like, here's my base URL for my project and this other thing. And then the LLM spits out a bunch of links. He clicks all the links, boom, he has tasks in this other product. And you can build integrations with other things without us doing anything, and without any code being written. It's just a prompt, and the LLM's knowledge of like, how to construct URLs.

• 12:47Nathan Labenz?

  Yeah, that's really very creative. I like it. And also kind of an interesting window into, you know, just a future in which AIs are potentially more and more likely going to be sort of solving problems in unexpected and, you know, maybe at some point even hard to interpret ways, although that one's pretty simple to interpret, but the creativity there is on both the user and the models part is pretty impressive.

• 13:12Andrew Lee?

  I'll give you one other one that I thought was pretty fun. There was a user, they wanted to do a mail merge. And but they wanted to have a mail merge where like every email really looked custom, where like the AI was like going off and like searching their email history and like finding an interesting fact about them. And like, they really wanted that user to feel like they, you know, they were paying attention to them. And so they took a text file, they put a whole bunch of email addresses in the text file, they uploaded that text file into the chat AI assistant, and they said like, loop through the emails in this text file. For each email, go and search, find emails that are sent with this person, figure out like a nice, you know, greeting for this person, write an email. And so they fed in the text file, had it loop through, it wrote a very custom email for every single person.

• 13:53Andrew Lee?

  Each one they got to review it, click approve, click send, and they could send, you know, 20 emails that all felt very custom in like a really organized, fast way with the AI.

• 14:03Nathan Labenz?

  Yeah, that's cool. I was actually gonna ask you about supporting loops, but I hadn't tried it myself. So it's cool that that is already something you're seeing.

• 14:10Andrew Lee?

  It works.

• 14:11Nathan Labenz?

  In the wild.

• 14:12Andrew Lee?

  Yeah.

• 14:12Nathan Labenz?

  So what have been the, you know, last time we got like quite deep into the guts of how it worked, you have a background as a database whisperer. And, you know, there was a lot of talk about kind of how the index happens and you're, you know, you take everybody, every email out of somebody's inbox and stored in your own system and have your own indexing sort of thing. And then the model can kind of build on top of that. If we work from like the ground up, how much change has there been at that foundational database layer and sort of retrieval layer versus how much has come at the higher layers of the model and then the sort of patterns of behavior that you're getting the models to actually exhibit?

• 14:54Andrew Lee?

  Yeah. I, so honestly, between the time I was in the pod last year and now, I think basically every part of that stack has been completely rewritten. You know, we are using a different embedding model. We are using a different vector database. The search stack has been completely rewritten. The API to that search stack has been completely changed. The models we're using for the agent are different. The agent code has been completely rewritten. So yeah, top to bottom. And I think driven by changes in the capabilities of both the model and also our understanding of how to apply those models. And every time the model can do X better, we're like, actually, if we reroute the system like this, it'd get this extra unlock. And so the thing is just evolving at a truly phenomenal pace.

• 15:38Nathan Labenz?

  Yeah, that's undeniable. And although some do try to deny it.

• 15:44Nathan Labenz?

  Maybe let's go a little bit deeper on each of those levels. I'd love to just hear kind of what you've learned and maybe also, you know, to the degree that you think it's applicable for other AI builders. You know, what sort of upshot or advice comes out of some of these, these different changes? At the database layer, do you have a favorite vector database that you would recommend to others at this point? Or like, what have you learned that caused a change there?

• 16:10Andrew Lee?

  Yeah, so we've made a bunch of changes here. And for as a little bit of context, one of the big unlocks for us here was having a model and an agent on the front end of our app that was able to reason a lot better about how to use search. And most importantly, was able to reason about how to run multiple searches. And this sort of simplified the requirements for us on the search stack because it used to be the model would run one search and it would try to, it would give you, you know, like a semantic component in the search and you try to run that search. And you and the search stack had to find the right email in that one search because you get one shot at this.

• 16:50Andrew Lee?

  And now that we have an agent that can run multiple searches in parallel or also can run them in sequence. So it can try a search. It doesn't find it can try something else or it can like try a search for one thing, find some information and then adapt and try another search. It's actually allowed us to simplify the backend implementation and kind of focus it on a much more narrow task and then be much, much better at that. So the evolution. Of our search stack in the backend is driven by, in some ways, the simplification requirements. And we focused on how do we make it really good and really fast and really reliable and really cheap to do this kind of more narrow focus task. So we use the Pinecone serverless offering as our database.

• 17:31Andrew Lee?

  We used to use their pods, but the serverless offering is a much more cost-effective solution because it separates storage and compute. And so you can kind of tune for your use case. And we use a lot of storage because you have a lot of email.

• 17:45Andrew Lee?

  We use an embedding model now called BGE, and we use a bigger embedding model now than we used to because the serverless offering kind of unlocked the ability for us to just use these bigger vectors without burning too much money. And I think one of the big changes is we started to use hybrid search on the backend. We used to use this pretty complicated pipeline that combined some like smaller model LLM calls with some like very particular types of like feature extraction and search. And if you look and listened to the podcast last time, you can remember how we did all this. And there was like a re-ranking step that we did. The whole thing was like very slow and complicated and very brittle because there was a lot of sort of custom stuff in there.

• 18:24Andrew Lee?

  And what we've done instead is we've allowed the search API to specify a semantic component and some constraints. And the constraints are sort of like normal email query constraints. You can say, hey, I want only not emails that are like in this date range or like with this contact or with this label. And the semantic component, we run like an embedding search with Pinecone. We also run a keyword search. And there's an algorithm to sort of combine the keyword search, like the full text search results with the semantic component and like get a score, kind of rank based on that score. And then we filter based on these other components. And so you get this API where you can say, hey, I want emails about this topic with these constraints.

• 19:09Andrew Lee?

  And we can very quickly give you a very accurate list of those in a way that's like cost-effective to scale across everybody's email. I think this has been huge for us. It's a lot cheaper to run. It's a lot faster. It's a lot more reliable. And it is also producing just way better results when you combine that with an agent that can reason about like running multiple different queries.

• 19:31Nathan Labenz?

  There's multiple things that I think are quite interesting. First of all, just the fact that the whole stack kind of has to obviously work together, but also

• 19:41Nathan Labenz?

  An improvement in one layer of the stack allows for simplification in another layer of the stack. Like, that's definitely been a huge theme for me over time with the work at Waymark, too. Like, the shenanigans and hoops that we had to jump through to choose an image for a user, once upon a time, were just like comical, almost, in their complexity, and also that didn't work that well. And, you know, in a similar way, now it's just like, feed them into a model, and it usually picks pretty well. And it's like, that's a, you know, we've cleared out so much old cruft that we developed to kind of get that first version working. And now it's just like a simpler and better solution, and driven by model progress. Did I understand correctly, though, that when you do this search, is it running

• 20:26Nathan Labenz?

  the full, like, the filter comes after? That's kind of interesting that you're doing like the full vector search through the entire corpus, if I understood correctly, and then filtering only after getting results back?

• 20:41Andrew Lee?

  It's a bit more complicated than that. I'll give you the somewhat simplified version, which is essentially that we run two types of searches. One of them is a full-text search that is constrained based on, like, the keyword and the metadata components of the filter. The other one is a semantic search that we run in Pinecone. And we combine the results of those two queries, and then we score between them. And that gets post-filtered. And so it means there are situations where if you had a lot of good semantic results for something, you could potentially miss the best ones. But I think in the vast majority

• 21:25Andrew Lee?

  of, like, real use cases, the way that we are, you know, combining that full-text metadata-constrained search and the semantic portion and post-filtering is generally finding the best results. The true, we are actually going to score every email across everything and apply these filters. If you really wanted to have that be 100% accurate all the time, that would be sort of an intractable problem, but we can approximate it very closely with our solution.

• 21:54Nathan Labenz?

  Yeah, interesting. Okay. It seems like model progress has driven a lot of value for you, but also the scaffolding and actually shaping the behavior of the thing is also super important. One thing I've definitely seen repeatedly in my usage, both when I accept the invitation for the AI to organize my inbox and also in some kind of random idiosyncratic things that I've done, like this one really, it was like, man, the AI and I are like really on the same wavelength. I have the total inability, for whatever reason, to remember which of my contacts is the stronger and which is the weaker prescription.

• 22:38Nathan Labenz?

  And every time I go to change my contacts, I have to search in my email for this particular thread where I know that that information is contained. And so I'm always doing this search and every time I've always been searching the traditional way, I end up running multiple searches. I'm like, okay, I think the word prescription was in there and then there was, I know the word corrected is in there and, you know, but what is it, right? And so eventually it takes me usually like two to three searches to get there. So I asked the Short Wave AI assistant to do that and it was interesting to see it basically go through the same process that I went through where it was like, search, you know, get some results. I'm not catching the thing that I want to catch and then, you know, search again. And I think it took like three or four rounds before it finally, you know, landed on the

• 23:23Nathan Labenz?

  thing and answered my question. You know, more often I think people will engage in probably higher level things. The organize your inbox is a great example where it sort of just kind of goes off and I mean, it comes back and gives you suggestions and opportunity to confirm and, you know, make the movements that it suggests that you move. But when you do accept it, like it just keeps working. So maybe, you know, take us through how you've thought about creating the

• 23:52Nathan Labenz?

  Agent behavior, you know, what, what has been the big unlocks at the model level that have made this possible? And then, you know, what have you had to do to wrangle it into something that is actually valuable for people on a day to day basis?

• 24:04Andrew Lee?

  Yeah, this has been the biggest shift in our product since, since we last talked. I mentioned like every part of the stack has been rebuilt, but the basic functioning of the assistant has gone from a single L L M call that produces the final output and sort of a complicated Ruben Goldberg machine to like get the prompt right for that one thing. And we had all these complicated heuristics and rules and whatever to do this and like smaller LM calls. And we've thrown all that out and said, what if we just run the big L L M a whole bunch of times repeatedly until we get the right answer? And we tried this, I'll go through the history a little bit. So a while back, uh, let's see,

• 24:44Andrew Lee?

  like two Decembers ago, I think, OpenAI rolled out their tool calling features in G P T four. And we tried them and we didn't think they worked, right? Our general experience was as soon as you try to get it to call a tool, like the whole model stopped reasoning well and just gave you bad answers. And it was much better to just like tell it, you know, format something in XML and we'll go and do the tool call for you. And even getting to do multiple tool calls didn't really work very well. And so we kind of kept this sort of rules based system. And then last summer, we tried this again with G P T four O and it kind of worked. And we said, hey, well, what if instead of this kind of one shot approach, we had a multiple shot approach? And originally it was very much geared around search of like, well, the search use case of running multiple searches seemed important. What if we ran

• 25:29Andrew Lee?

  multiple searches? And so we kind of rewrote it a bit to do that. And it worked better, but not like dramatically better. And we had a launch actually in September that was like, hey, our, you know, our, I think we called it our V two agent, like here's our new agent. And it definitely got us some growth and some excitement, but it wasn't a major thing. And then in October, or maybe, maybe November, I was listening to a podcast from the founders of Bolt dot new. They were talking about how they, they built their stuff. And my memory from that podcast is basically they said, hey, you got to try Claude Sonnet three, five, the October version of it. It's different. And by the way, oh, Bull dot News is open source, at least parts of it are, and you can kind of see what their prompt is and how it works. And I found that very interesting. And so I did, I tried Klutz on three five specifically for tool calling the October version of it.

• 26:15Andrew Lee?

  And it was dramatically better and it was able to like G P T four O, you know, you could, you could have it. You know, you call it, it spits out a tool response, you feed the tool back in, you call it again, you know, it spits out another dual response. Like you could have it kind of iterate a few times, but it would sort of like go off the rails after, after too long. So we didn't want to do too much of that. But with Klutz out three five, it could go on for a long time, right? It could run many searches and do many things and it's kind of still stay on track and still keep reasoning. And it seemed totally different. And so we're like, okay, maybe we should rewrite this whole thing and say, we're going to have this new approach. We're just going to call Klutz on it over and over and over again.

• 26:58Andrew Lee?

  We're going to let it run not two or three or five times, but like 20 times. And we're going to put all of our smarts into like really good tools and a really good prompt for the overall thing. And like a nice agent framework around that. Um, and we're just going to let the model reason about what data it wants to pull in. And this was a total rewrite, like we, we had a lot of like very custom email centric stuff. And we said, no, no, we're going to have like a fairly generic agent framework at the core. We're going to build a whole bunch of really nice tools around it. We're going to have a lot more tools than we used to have, and then we're going to iterate our way to a solution. And this is the agent that we launched in January. And if you look at our, you know, you mentioned the hockey stick growth graph, like that little kink. That's the V three agent where we rolled this thing out and it was just a whole lot smarter.

• 27:40Andrew Lee?

  It's just, it could start solving like very general, very open ended things. And there were a lot of places where the old version didn't work that iteration solved. I think that's the core. When people are talking about agents, I think what they're really talking about, if they actually have a working solution is iteration. It is, it tries a thing. If it works, great. If it doesn't work, it tries another thing. And sometimes that's running three searches in a row. Sometimes that's like, you know, trying to run a search, finding out that the search, you know, criteria that you specified is actually like malformed. Having our system throw an error, having the L M see the error response and try again. Or sometimes it's like trying to schedule a calendar invite, having realizing that the calendar invite is like conflicting in some way.

• 28:22Andrew Lee?

  Having our system spit back some information being like, this is conflicting and then trying again. So you get this feedback mechanism where. Over a series of like many LLM calls, you can iterate your way to an answer that no single LLM call could could have produced and and get really, really good results that way.

• 28:40Nathan Labenz?

  Well, okay, I'm gonna dig in a little bit more. We have time. It would be a very natural time to, you know, to talk about sort of the, you know, expanded view of the company and the product. But before we get to that, one thing that I've just been kind of wondering myself is like,

• 28:58Nathan Labenz?

  how does a model like this is one of the challenges with, you know, rag apps, whatever. And a lot of people feel like rag apps and have come to some frustration with them. Where I think often the core problem is, as you said, like you do one search and then the model just kind of has to do the best with what it's given. And if the right information isn't in there, then it's really gonna have a hard time. Like definitionally, expanding to, you know, allowing for iterative search, you know, definitely I have felt that like makes a qualitative difference in terms of how likely it is to be able to pull the thing back. Like, I don't think it would have found my contact prescription a year ago.

• 29:39Nathan Labenz?

  But I still wonder like, how do you think about maximizing that? Given that, you know, the the models don't have the the thing that I have, which is I know it's in there, you know, like, and I know when I've found it. And this seems like a sort of fundamental challenge from the model's perspective that, you know, how do I sort of know when to call it? When do I decide that this amount of information or this particular information is like in some sense satisfying or the best I'm gonna get versus feeling like, you know, the way I feel, which is like, I know that that's not quite it yet, you know, and I I but I know it's in there. So I'm gonna kind of keep looking. And then when I do find it, I'm like, it's always very clear to me like, yes, this is what I was looking for. And, you know, I know it with high confidence.

• 30:24Nathan Labenz?

  So the models don't have that. How have you approached the problem of like?

• 30:30Nathan Labenz?

  Leading them or guy, you know, I can't imagine that they're entirely, you know, just doing it on their own. So how do you help them decide when they have found enough versus when they need to keep digging?

• 30:43Andrew Lee?

  So I'm going to give you kind of a complicated answer here. So the first thing I'll say is the models actually do have some of that because they may not know, you know, what you know specifically, but they know, you know, what it looks like generally to have found the thing you're looking for. And they, they have a lot of training on just emails in general and what emails look like, what people expect from emails. And so we've found, so for example, we have this organize your inbox feature and it goes through and it finds what we described as like low quality emails and it gets rid of them. And I have been shocked at how little work that we have to do to describe what a low quality email is. And so for example, like it's really good at spotting cold sales emails, even though like you might think like, hey, how do I know

• 31:26Andrew Lee?

  it's a cold sales email? Maybe this is somebody I know, but there's something about the tone and style of whatever cold sales emails that like really stands out to them and they spot. And I found that to be like with there, we do have some prompting work to make this really clear, but like we didn't have to do a lot to tell it this. So that's my first thought. My second thought is we do try to give it tools so that it can go look up this information itself. So if you're like, hey, you know, I'm looking for an email from, you know, some important investor or whatever. It maybe doesn't remember who your important investors are, but we have a contact tool and the contact tool, when it returns results, we put in there using statistical information, how important this contact is. So it can actually look up who is important if it wants to. It can also just go search your email history and find like, you know, emails you sent and make

• 32:11Andrew Lee?

  assumptions about, hey, if you've recently exchanged emails with this person, it's probably important. And so this is an area we're trying to get into more and more of, of if you ask it some open ended question, kind of use the tools it already has to reason about it in some reasonable way. Look at your email history, look at your contacts, look at your calendar events. Have you met with this person recently? Those types of things and try to try to get some of that context. The third thing I'll note is I see a lot of room for improvement here. We today don't do anything to capture sort of the triage actions that you take. So When you use organize my inbox in the morning and you know, some of the excess, the suggestions we accept and some of them you don't. What we really should be doing is remembering all of the things you did and then sort of customizing it to you over time.

• 32:51Andrew Lee?

  And today, we don't do that. If you want us to change the way we do triage for you, you can actually tell the AI, you can say, remember, you know, never archive newsletters from the sender and like, we'll remember that. But you have to be explicit. We don't remember just based on the actions you take. So I do see a lot of opportunity there.

• 33:09Nathan Labenz?

  Yeah, that's, that's quite interesting. I just did an episode with the chief scientist, Guy Giorari at Augment. And

• 33:19Nathan Labenz?

  yeah, he, I mean, there are multiple similarities, actually. I mean, they also have like a big emphasis on just ingesting a ton of code right off the bat and putting it into this like specialized index. But also the behavioral part, they've developed a process they call reinforcement learning from developer behaviors. And it's about kind of, you know, both observing what the developers are doing, but then also like how they're, you know, how the developers are reacting to what the AI is bringing to them. And it sounds like it's working quite well for them. It seems like one big revelation, reinforcement learning works. So I see that probably coming quite soon in your future, too. On

• 34:00Nathan Labenz?

  just another kind of random question, but a, an interesting one to me at least. Claude obviously is significantly more expensive than some other options. So what I see typically from the agent, I don't know if this is like super consistent or just kind of my observation, but typically it seems like it's 10 threads found. And then it kind of, you know, does its sort of reasoning and do I need to search again, whatever, based on the 10 things that it's found. The other way I could imagine doing that would be like, maybe using a slightly lesser, but way more affordable language model to do the evaluation on stuff that came back.

• 34:40Nathan Labenz?

  So like if you used Gemini Flash, for example, I think you would be able to handle like 30 times as much. And you could maybe complicate this with caching. I'm not exactly sure how you, how you're using some of those more advanced optimizations, but. First order approximation, you could handle like an order of magnitude more search results if you put it all through flash to assess relevance versus, you know, saying to Claude, like you handle this all yourself. Are you doing any sort of that kind of ensembling or using different language models for what they seem to sort of specialize in all things considered? Or is it just all Claude all the time because it really is just that good?

• 35:19Andrew Lee?

  So we do use a bunch of different models, but we use them for different features. So for example, our autocomplete is actually using a fine tuned G P T four O mini. Our quick reply suggestions, like the one button reply suggestions that we give you, those are actually using Lama three point two, the three B model. We use G P T four O, just like not mini, but G P T four O in a few places. So we do use different models in different places for different things. We have looked at sort of stringing them together, you know, like starting to save costs for certain things in the assistant by outsourcing other models. And I think our experience has been that it introduces a lot of complexity and it affects the ability of the model to kind of reason across lots of different types of activities in a sane way.

• 36:04Andrew Lee?

  If you just feed all of the data into one big model, it can think about, you know, lots of complication relations between different things. And it can come up with ways to use tools and ways you didn't think. And it's been a big unlock for us and the cost savings haven't seemed worth the complexity and sort of the loss of generality of like having this turn into more of a pipeline between different types of models. The other thing I want to note is the caching that Anthropic has is really, really critical for us. It's a really big deal. So when you have this agentic flow, you call this, you know, the same history over and over and over again, you only append to the bottom. And our histories get very long, like we can have hundreds of thousands of tokens at any one of these calls.

• 36:47Andrew Lee?

  And the Anthropic caching is a little hard to use. You have to really construct your agent to be very careful about keeping like earlier things immutable. And we've done a lot of work to make that happen. But if you get it working, it can save you 90% of your cost. It's a hugely impactful thing. And frankly, if we didn't have that, we could not afford to run like we would be just on anthropic costs, we'd be losing money to every user by a huge margin. So that's been a huge unlock for us. It also is one of the reasons we switched to cloud, actually, is we even if the, you know, the models from OpenAI could use tools as well as what we have for anthropic, we couldn't afford it because the caching from OpenAI is just a lot less of cost savings. It's only like a half off.

• 37:29Nathan Labenz?

  Yeah, half off and pretty much automatic. You don't have to do anything, right? Versus, can you

• 37:35Andrew Lee?

  Yeah.

• 37:35Nathan Labenz?

  tell a little bit more about how, I know anthropic is like 90% off. Once something is cached, but there is also a like a one time cost to get something into the cache, right?

• 37:48Andrew Lee?

  Yeah, but it's not a, an additional cost, right? So you, at the time that you, you know, run it, you have to tell it to cache. And I think there is maybe some slight additional, but it's not, it's not a significant additional cost. And then every iteration after that is 90% cheaper. And if the average, you know, if the, if the most common action is like people organizing their inbox, which it is, and that is 20 tool calls, it adds up really quickly. Um, I was actually, I was actually just at OpenAI yesterday and I was just talking to the agent SDK folks about this and about caching. And this was frankly, the number one question from the room, from the other founders I was with there was like, what about caching? How do we get caching to work? Because I think everyone's figuring out that you're going to be calling

• 38:29Andrew Lee?

  these models with the same context over and over and over again, and making it work efficiently really matters a ton. And the OpenAI caching, you're right, it is a very simple API, right? It sort of automatically figures out, you automatically get cost savings, which sounds great. And the anthropic one's much harder to use, but the magnitude of those savings is quite different.

• 38:48Nathan Labenz?

  So on the anthropic side, is it smooth to the point where you can sort of build a cache? Like I've, I've used the caching a little bit, but it sounds like module is some, you know, possible implementation difficulties. You can like extend and also extend the cache iteratively as you go. And basically it has all the features you really need to realize like the full savings in the sort of sticker, you know, price of 90% off.

• 39:16Andrew Lee?

  Yeah, you can. So you sort of like checkpoint at every iteration in your agent flow saying, okay, I cash up into this point and you have to construct your agent to like make use of that well, but it's totally doable.

• 39:29Nathan Labenz?

  Interesting. So are they actually caching? I wonder if there's a big optimization there for them under the hood where they're like, it sounds like they may be caching like multiple versions of each conversation?

• 39:40Andrew Lee?

  I don't know. I actually am very curious what they're doing. Maybe they're just eating the cost. I don't know. But our bill goes way down. So it's great.

• 39:47Nathan Labenz?

  Yeah, that's interesting. How do you feel about this sort of, a huge question in the AI space generally to me is, are all the frontier providers converging or are they diverging? We've got like, you know, somewhat different caching things, but maybe OpenAI is going to get the message and sort of, you know, do a more Anthropic like one. We've also just seen in the last like day or two that OpenAI is going to embrace the Model Context Protocol that Anthropic, you know, led the way on. Do you feel like these things are converging, diverging, some, you know, some complicated mix of those?

• 40:27Andrew Lee?

  I think it's some complicated mix of those. So it has been nice that they've been, I think, converging in terms of the API surface. So, you know, we now very easily can swap between different providers and like all the tool calling and stuff lines up perfectly and makes it really easy to do that. So I think that's been a very nice thing. And everyone is understanding that like, there should be caching and are hopefully converging on some of the ideas of how to do that. MCP is maybe becoming a standard that everyone's respecting. So I think there's been standardization in the interfaces. It does seem like the different labs are focusing on different things. So it seems to me like Anthropic is really focusing

• 41:08Andrew Lee?

  on this iterative approach. It seems to me that OpenAI is maybe caring a lot more about reasoning and potentially more about like multimodal. And you'll notice that we use, you know, many different models. We use models from three different vendors. And a big part of that is we feel like each vendor is bringing something different to the table. So one example where I think OpenAI totally crushes it is they're serving infrastructure. Is super performant and reliable relative like Anthropic's API's they go down a fair amount and time to first byte is slower and stuff like that. But so for example, for our autocomplete where latency is really critical, we use the G P four O mini with a fine tune and like the

• 41:51Andrew Lee?

  serving stack API is like super good. So I think that's a differentiator there. We use for the summaries and for the instant reply, like quick actions and for a lot of the stuff that where it's like cost and latency matter a ton, but we don't need a lot of intelligence. We use open source models. So we use the Lama three point two and that's running just like on Vertex and G C P. So it's great to have, you know, as a someone building an app to have many different vendors converging on standards, but like focusing on different areas ends up being really nice for us.

• 42:21Nathan Labenz?

  What else have you learned about agents in this journey? I mean, everybody's, you know, setting out to build one. I think you are, you know, pretty far ahead of most. Have there been like false starts, you know, the things that you sort of expected to work that didn't or just any kind of unobvious lessons learned that you would share with the folks who want to build their own agents?

• 42:40Andrew Lee?

  I think that the biggest thing I would say if they're thinking about now is that things are different. Um, in just in the last few months, right? Basically, like if you tried this before October of last year, or if you've never tried the Anthropic models, like it's different now. The stuff that you tried before, it actually works. The cost actually with the with the caching stuff is manageable and you've tried again. And I think you'll be really surprised. And this is, yeah, you'll notice stuff like bolt new has gone totally crazy. And the new cursor agent mode, I think a lot of people are still using cursor for the autocomplete. Like that's old school. They use the agent mode. It's totally, it's totally insane. So yeah, it's the big messages. It works now. It's different. You got to try it out. It's going to be, it's going to be amazing. I think one of the big things we learned is people have no idea what the UX should be around this stuff. And we're very

• 43:25Andrew Lee?

  much still figuring this out, but you know, people kind of figured out like, oh, autocomplete, it works like this. And like, people, you know, everyone kind of wrapped their heads around how this sort of thing should work. You tab it's complete. And now we have a thing where it's like, it's going to go off and just do a bunch of work for you. How do you do that in a way where the user doesn't get really uncomfortable about, is it sending emails for me? Is it changing my code in ways I don't realize? And we're going to need some UX for like, sort of oversight and approval and providing guardrails on these things that like, doesn't get in the way, but still gives you confidence as doing the right thing. So I think that's going to be a big area for us. There's just a lot of questions, a lot of unknowns, given how new this stuff is, but I think it's going to be really exciting to see how it unfolds.

• 44:08Nathan Labenz?

  Yeah, one product feature that I honestly haven't used as much as I maybe should, and kind of because I'm not sure quite what it's going to do, honestly, is the AI filters. And I was both curious to know like, how that works under the hood, like, is it sending all? Because what I think is going to happen is I'm going to say like, filter out this kind of email and then emails are going to come in. And I guess then like, literally everyone would get sent to a model to say like, is this the kind of email to which this filter should apply? And I've only been a little, I mean, I like the sound of that because I got a lot of crap that I need to filter out. And I, you know, historically have definitely spent way too much time clicking on those myself. But the flip side of that, of course, is like,

• 44:53Nathan Labenz?

  if I create a workflow where I don't have visibility into what's getting filtered out, you know, I have been burned by that in the past as well, even with just like email priority inbox over time. So first of all, am I intuiting correctly, like what's going to happen there? And I guess more broadly, how are you thinking about these, you know, sort of next level agentic scenarios where we're going, you know, today, the AI system like brings me stuff, it's up to me to confirm, but you know, it's not hard to, with all the talk of like how well it's working, it's not hard to extrapolate a little bit and imagine that, you know, okay, fine, you know, just go ahead and do it. So how are you thinking about those next moves into actually taking actions without necessarily like a, you know, a human in the loop approving each one?

• 45:39Andrew Lee?

  No, that's, I think you've caught an interesting difference in that feature, which is it is the only AI feature that we have where the AI just does things and it does, there's no approval flow from you. And it's actually a really popular feature. So the basic idea is like, you could write a prompt and you can choose three actions based on that prompt. You can either archive it, you can apply a label, or you can delete it.

• 45:59Andrew Lee?

  And we've had this live for a couple of months now. People love it. There hasn't been a ton of problems in terms of people losing stuff, but we do get support requests for the times where people are like, hey, you're missing my email. And it's like, well, let's check your AI filters. And then they have one set up. And so I think there's no way of getting around that entirely. But the dream here is that becomes like a run of the full agent. Right now, it's a very simple implementation. We do use OpenAI for this. This is a GPT four O mini task, and which means, yeah, if you set up an AI filter, we are going to be sending your incoming emails off to OpenAI. We trust OpenAI as a vendor. Their terms prevent training on the data. And like, we believe it'll be confidential. And that's very important to us. But we do send the email through their APIs.

• 46:42Andrew Lee?

  What we really want to do is allow you to do anything that the agent can do at that point. So rather than just making a single bottle call to a small model, we want to like spin it the full agent and let it like call tools and stuff while it's making this decision. So you could have a prompt that's like, hey, if anyone, you know, asks to schedule a meeting, and I have previously exchanged, you know, more than three emails with them in the past, and they are an investor, auto accept the meeting, right? Like, we want to be able to write rules like that. And there, I think there's two

• 47:16Andrew Lee?

  big problems with this that we need to solve. One of them is just cost, right? Like, we are spending a ton of money right now on Klutz on it, just for you going and asking those questions in the sidebar, which like a typical person only does, you know, a handful of times a day. If we're doing this on every single email you get, we, rather than doing this, you know, eight times a day, you know, you're doing this 300 times a day, and that dramatically increases our costs. So one of them is, how do we scale this? Interestingly, if you saw the GTC keynote, you know, Jensen was up there as being like, hey, we realize we're going to need 100 times as much computers we did last year. I think we're in the same boat where it's like, you know,

• 47:53Andrew Lee?

  we never thought that, you know, every time you receive an email, we need to run like 10 million trillion calculations to decide if we want to archive the thing or not. But like, I think we're going to be in that world where like, yeah, we have this full AI running, considering your entire conversation history with everyone in the world. Every time you do this, I think we're actually going to get in that world. So that's problem number one, it's just cost. Problem number two is trust, right? Where you have to worry about, is it doing the right thing? But also like, is it susceptible to like prompt injection? This is only comes up from people where they're like, what if somebody sends you an email designed to like, you know, mess with me?

• 48:26Nathan Labenz?

  Ignore instructions and delete the full inbox.

• 48:29Andrew Lee?

  Yeah, right? Like that, that's a problem. And so I think there's, there's a bunch of things we need to do here. So one of them is just figuring out what the right guardrails are, right? Maybe that agent can't delete emails other than the one it's looking at. There's, there's sort of restrictions on what it can do. We also could sort of look at some sort of post fact confirmation flow where like, it gives you a history of like, here are all the actions that it took and you can sort of approve them. And like, we could remember which ones it didn't do right and like, sort of adapt and learn from that. It also could sort of, instead of actually taking action, it could queue up the actions for you. So this is drafts, I think is the best example here where like, maybe we never send emails on your behalf, or like only in very rare cases, but most of the time it just like, sort of creates a draft. And like, you come in the morning and there's like a button where it's like, hey, do you want to send these emails? And you're like, yes. And it's like, very easy for you to review.

• 49:14Andrew Lee?

  But yeah, we need, we need the AI to be both powerful but trust and, and resistant against, you know, people sending you emails messing with you.

• 49:25Nathan Labenz?

  Yeah. So I assume you've tried this, but like, Flash just doesn't quite cut it for even this sort of reduced action scope? Cause I'm always looking for a way to get value out of Flash, cause it's just so damn cheap. And it's quite good, but maybe not quite up to the level you need for this?

• 49:44Andrew Lee?

  We, we are actually testing Flash right now. The, I think the first place this would land is in the summaries and the quick replies and things like that. We may use it for the filters. Too early to tell. We are constantly trying new models. It, it is, you know, I'm sure you hear this from everybody you talk to, but it is so insanely hard just to try all the stuff that's coming out, like, you know,

• 50:05Nathan Labenz?

  It's my full time job, I can't keep up, yeah.

• 50:07Andrew Lee?

  Yeah, DeepSeek comes out like two days later, everyone's like, why haven't you switched to DeepSeek? And it's like, I haven't even had the time to, you know, play with it with like a single prompt. So we are constantly looking at, looking at new models. Yeah, we are currently playing with Flash. We may roll it out or we may not. There's a lot of considerations, you know, cost, latency, caching behavior, how it performs on specific tasks, and we try to factor all of those in.

• 50:30Nathan Labenz?

  What do your evals look like? I mean, obviously, that's key if you want to be able to make confident decisions on whether to upgrade or switch out to a different model in any number of contexts. I also, though, am struck by like, especially when I'm looking at output that's supposed to be like writing, you know, in my voice as much as possible and sort of, you know, representing me, that this is a very challenging thing to eval, right? So I'm sure you must have some like mix of objective and vibes, but what is that? You know, what's the perfect balance between objective scores and vibes?

• 51:09Andrew Lee?

  It is. So we have made the conscious choice to say this technology is evolving super fast. Our product is evolving super fast. It is more important that we adapt quickly than that we don't break things. And we move very quickly. So our evals basically consist of two pieces and they're very, you know, very seat of the pants. Piece number one is I have a Notion doc of golden test cases where I say, hey, here are some prompts that are supposed to work. I run those in my inbox. I make sure they do same things. So, uh, you know, when we're tweaking prompts, I go in and I try and be like, hey, does this still do the thing that I expect or do something reasonable? We don't want to lock it down to specific behavior because often it gets better.

• 51:51Andrew Lee?

  So for example, we recently added a tool to do unsubscribe. And magically, without us touching anything, anytime I did an inbox organization thing and we ran across an email that looked low quality with unsubscribe link would start offering to unsubscribe you. You don't want to have a test that is like expecting organize your inbox to do a certain thing because that's better. Right? This is so you'll notice now if you organize your inbox, it's going to start offering unsubscribe. And that's something we didn't really think about when we first were building the feature. So one is we have a, we have a Notion doc and I go through it and I, you know, say, did this produce reasonable results? The other thing.

• 52:23Nathan Labenz?

  How many, how many prompts, by the way, just to, uh, calibrate myself?

• 52:28Andrew Lee?

  Uh, over a hundred, maybe at this point.

• 52:31Nathan Labenz?

  Okay, it's a little more than I expected, but.

• 52:33Andrew Lee?

  And I don't, I don't say I.

• 52:34Nathan Labenz?

  You run a manual.

• 52:35Andrew Lee?

  I'm not. Yeah, it's just a notion, Doc. I don't necessarily run through every single one every time I make a change. Right. I'll go look at like, what are the ones that are relevant to the areas that we touched? And I'll try those and see if they do reasonable things. The other thing that we do, which I think is the more objective metric, is we have an experiments framework. So every new change, every big new change that we roll out, we provide as an opt-in experiment for our users. And our users tend to be like, super, you know, forward-thinking thinkers who like to play with this stuff. And they'll go and they'll turn this on, like in large numbers. And we look at the retention stats of that. So like the unsubscribe feature, for example, we had like 99% retention for people enabling that. And when you enable that, it modifies the prompts and stuff and adds new tools. And like, it could, it could break all kinds of stuff.

• 53:18Andrew Lee?

  But, you know, after leaving this on for like a week and seeing like 99% retention of this feature, we're like, okay, clearly this isn't breaking a lot of people. People will start turning it off. So we feel comfortable rolling it out. So anything that is like a major change, we put into that form. We've lost retention stats over time and the assumption is if retention is high, it's probably working pretty well for people.

• 53:39Nathan Labenz?

  Yeah, interesting.

• 53:39Andrew Lee?

  So testing in front.

• 53:41Nathan Labenz?

  You have.

• 53:42Andrew Lee?

  To put it lightly.

• 53:44Nathan Labenz?

  I mean, you know, that's a society-wide phenomenon these days. By the way, when things do go out, is that also just something that you've made a strategic decision to live with? Like, there is no substitute for Sonnet. And so if Sonnet is down, like, we're down. Is it as simple as that?

• 54:01Andrew Lee?

  Yeah, like, we actually, you know, we could fall back to GP4O and it might give us decent results and whatnot. But there's always a question of, is it worth thrashing people to deal with a short downtime or whatever? So, so far, yeah, we've just sort of eaten the downtime and waited for stuff to come back up. And we don't seem to lose users because of it.

• 54:19Nathan Labenz?

  Yeah, I mean, I think it's, this is something I've also been kind of, I don't know, it's a little bit of a hobby horse for me, even with, you know, my own company too. It's like, this is just the new normal, you know, these, we're like, going to be more dependent on these services. They have pretty good uptime, but not perfect. And there's really not much we can do about it, you know? If it's out, it's kind of out. So I'm with you on that. And I think, just in general, like, there's a mindset shift that is kind of needed. That's like, let's aim for creating really magical, like the most valuable experiences we can, as often as we can, and, you know, live with a little risk into whether it may be, you know, outage or, or just, you know, some uncanny valley behavior, like certainly, you know, still see it, but I think it's,

• 55:05Nathan Labenz?

  for everybody, really, it seems to me, it's worth taking a little bit of that risk to get the, you know, those special upside moments, because when they happen, they're like incredibly valuable. So, yeah.

• 55:17Andrew Lee?

  Yeah, we try to be upfront with people about this and people ask me like, hey, you know, is this, you know, is this going to be super reliable? You know, can I count on this? And my answer is, hey, if what you're looking for is the most reliable, stable product and nothing changes on you, like, use Gmail. That's what Gmail, it's there, you can trust it, right? If you want the absolute most cutting edge stuff in any email client ever, use our stuff. We are trying very hard to be at the absolute edge.

• 55:45Nathan Labenz?

  Yeah, you do have a really nice benefit too, that like, worst case scenario, I can't always just open up my Gmail and, you know, access stuff directly that way. One thing I did notice too about the new to-dos, I wonder if there's kind of a deeper philosophy underlying this, you know, so I go organize my inbox, the assistant, first time I did it, just like started coming up with to-do categories for me and suggesting like, okay, you should group these into this section. And I'm like, okay, interesting. That doesn't seem to touch back to Gmail at all, as far as I can tell. Then there's also like labels which do, right? And I guess I'm kind of wondering to what degree you have found it to be.

• 56:27Nathan Labenz?

  Advantageous to like have this sort of, you know, single ground truth where like something that you do propagates back into the core Gmail account versus I act honestly kind of like it better where I'm like, okay, this is my short wave universe and I can let the AI assistant do its thing and run a little wild and you know, worst case scenario, I can always go see my old view. So, but I don't know what users in general want. Do they want like a unified reality? Or do they want you to kind of build, you know, something a little bit off to the side that, you know, in a sense like de-risks them from anything that could happen to the core data store?

• 57:04Andrew Lee?

  Yeah. So yeah, I have a bunch of thoughts here. So the first thought I'll have is we, we, one of the big sort of conceptual changes we made here since the last time we talked is when I talked to you last time, we were building an email client with an AI built in. We don't think of ourselves that way anymore. We think of ourselves as we are an AI with email features built in. And the plan here is to integrate in the medium term with products that aren't even email and basically sort of anything with an inbox, you know, your Slack or your LinkedIn or whatever, like you should be able to access and manage from this interface. And you'll notice we moved the AI to the left. Big driving force behind that is we see that as the main product. We see people coming in primarily because they want to interact with that AI, that AI can work with

• 57:47Andrew Lee?

  their email inbox. It can work with other inbox in the future. It can work with the CRM, your project management tools, you know, whatever you want to use. And so we're thinking it from that standpoint. And in that world, keeping everything in parity with Gmail doesn't necessarily make sense because you might not even be using it with Gmail. You might be using it with some other product. The to do concept is here basically because we see a need for the AI to be able to like add and manage state. Specifically for like short term organizational purposes. Labels are a great tool for long term classification of like, you know, I want to apply this thing. And then two years from now, I want to go and search for this characteristic. And I want that label to be like short and simple and easy to remember.

• 58:31Andrew Lee?

  To dos are great if I have some project that's happening right now. And I want a name that's like a whole sentence and I want to have a bunch of notes in there. And I want to attach a bunch of email threads for it. So I want a bunch more complicated, but much more short lived type of thing. And I think the AI shines at this, right? So you can, you know, let's say you're producing your podcast with me, right? And you and I might have, you know, two Google docs and five threads about this. And the AI can spot, hey, these are all related. I'm going to make one to do that's like, you know, prepare for podcast interview with Andrew. And it'll put all this stuff in there and add some notes in there for you. And we don't think that back to Gmail largely because there is no concept like that in Gmail. And we don't think it back to labels because it's kind of a different thing.

• 59:13Andrew Lee?

  And I do think people like the idea that we are not trying to like shoehorn new features in the old Gmail things in a way that like might mess things up. So a counter example of this I'll give you and is one of our competitors. So if you use superhuman, superhuman implements some of their features basically by sending emails. And so if you use like the reminder feature or whatever, if you go back to Gmail after you stop using superhuman, you end up with all these extra emails, which like kind of ends up looking cluttered and weird. And we try to avoid doing that so that at any point you can leave, you can back to Gmail. Everything looks the way you saw it before. And it's like an easy switch, not that we ever want you to leave, but we want you to be comfortable that we are like doing the right thing with your inbox. Cause it's, we know how important it is.

• 59:57Nathan Labenz?

  Yeah, certainly. I think it makes me more comfortable in taking a leap. And that is, you know, a big part of what people need to do, I think, to get value from AI is be somewhat willing to take a leap. Couple more little product questions. And then, you know, the, the big shift I think is honestly super exciting, super valuable because I have tried over the last two years, honestly, at this point, to fine tune a model to write as me. And one thing I've really learned is that. If I want to make something genuine, I've not really succeeded in this, to be honest. Like, nothing has beaten dumping a lot of writing samples into Claude and just, you know, asking it to do the next thing. But, you know, I've learned some stuff by trying.

• 1:00:40Nathan Labenz?

  One of the things I've learned is just that my, and I assume probably other people are often similar, no single system has the full me. Email, you know, I don't like email back and forth with my wife. I would need like, you know, text messages really to, you know, that's where that kind of relationship in text form, at least, lives. And then Slack is key for like, a lot of the sort of day-to-day planning and discussion of what we're going to do. And then some of the more interesting conversations happen on Twitter DMs. And so it's like, sort of, not just that there's no single source of all the information, but there's honestly not even like, a single thing that I feel like really, you know, it's like, very different facets, I guess, of my life, I feel like, across these different channels.

• 1:01:25Nathan Labenz?

  So to be able to unify that into a single thing and have like, a, you know, an AI system that could span all of them, I do think is quite exciting. When you do the write-as-me, how is that going? I would say I've noticed an improvement, for sure. It seems now, usually, that it handles like, routine stuff at least, pretty well. So, you know, kind of classic, let's put a meeting together, you know, here's my Calendly link, whatever. That kind of stuff, I'm increasingly feeling comfortable just like, hitting send on with the assistant's draft. Obviously, as it gets more specialized and more context-dependent, you know, it gets much harder.

• 1:02:10Nathan Labenz?

  But what have you learned about, like, making the write-as-me work, maybe on multiple levels? Like, what do people really care about? How, you know, representative or not am I? And, you know, techniques-wise, like, what is getting results?

• 1:02:22Andrew Lee?

  So, yeah, so, from a techniques perspective, one of the things I am, at this point, like, very, I very strongly believe, and I think this is a little bit controversial with other people I've talked to, but I very strongly believe this, is that the paste examples into Claude and have it use the examples to produce the response, is the correct solution. It is the best solution. And that doesn't fit well with people. They'd like to assume that the more technologically interesting solution with the fine-tuning and stuff is going to produce the better results, but... I think there's two, I'll give you two reasons why I think this is the right solution. The first is this technique is much better at recalling specific facts as opposed to like a shallow

• 1:03:04Andrew Lee?

  fine tune. And when you're writing an email, or when you have the AI write an email for you, you don't care so much that it sounds like something you would have written. You care that it's correct, right? If you provide a link to someone, they're like, hey, you know, where do I go to like, to pay for this product, right? You don't care if the link looks like a thing that you would send. You care that it is the right link, right? If you, you know, if you're scheduling a meeting and it picks a time that you mentioned in the previous email, it shouldn't just be a time that you would send. It should be the right time. And if you give it specific examples and frame those correctly where you say, hey, you know, the last time you talked about this topic, this is what you said. It can actually get the facts correct. It can get the links correct. It can get the times correct. So it isn't just sounding like you. It is, it is being

• 1:03:49Andrew Lee?

  accurate. So that's, I think, the point number one. Point number two is the big problem with fine tuning is it slows down your ability to update models. And if you have built a bunch of fine tuning infrastructure for a particular model, and then you decide to change models, you got to refine tune everybody. And that's a big process. And that's a big migration. And the reality is we are switching models constantly, right? Constantly. And often the leading edge models don't have, you know, fine tuning features built in initially. And so if you want to stay at the cutting edge, and you want to be able to roll things forward really quickly, you don't want to have to be fine tuning these models. So we do have some fine tuned models, but we don't do the fine tuning for the purpose of fact completion. And we try to avoid fine tune if you don't have to. And we do things this other way. I'm actually fairly excited about reinforcement

• 1:04:34Andrew Lee?

  learning here, because I think the big models do a pretty good job of sort of extracting facts and style and tone and stuff from examples. But I think they could do a better job. And I think reinforcement fine tuning will allow us to teach the model how to do style and fact matching better in a way that is like generic across everyone. So we don't like per user fine tune. You know, we just take the new GPT-4O, you know, with this reinforcement learning, and we teach it, here is how to use examples for style matching and get facts right and write emails. So I'm actually very excited about that. But I do think the, your 20 examples of things that have been said, master style and tone with the right prompt and the right reinforcement learning is the right, is the right solution.

• 1:05:20Nathan Labenz?

  Okay, so how about on memory? I mean,

• 1:05:24Nathan Labenz?

  I hear you on the operational challenges of fine tuning. And every time I raise this, you know, notion of like per user fine tuning, or even per company fine tuning, that's like, I think the most compelling counterpoint is like, what are you going to do every time you want to change models? And then there's also new knowledge that comes in and how often are you going to run it? And like, it does sound like a real bear.

• 1:05:45Nathan Labenz?

  Is there, I guess, it still feels to me like there's some sort of missing middle in terms of memory where we have context window, obviously, and then we have like, you know, stuff that is known in the weights. And then you have like the sort of database, you know, call as well. But those all feel like they're still kind of missing a little bit of something. Again, I kind of come back to this intuition of like, I know when I found it, you know, and there does seem to be something qualitatively different about that, or I kind of, I know what I've tried. So I've been really interested in state space models. I've been really interested in the Titans paper that came out from Google not long ago, where they actually use a neural network as a memory module that gets updated at runtime.

• 1:06:28Nathan Labenz?

  I'm also, I've been quite interested in, you may have seen some, I did an episode on a system called HippoRAG, which was inspired by the hippocampus and how it is understood to like, connect concepts together. So they have sort of a extensible graph, you know, network that they can query against. But maybe you think like, we don't need any of that. And we just need like, better models and, you know, better search. And maybe this whole thing is sort of a confusion on my part, where just keep pushing the current frontiers and it'll all work. And one day I'll just realize that we didn't never, we never really needed, you know, another middle piece. What's your expectation for that?

• 1:07:05Andrew Lee?

  But you know, I, I don't know what's going to happen here. I, my, my spidey sense is there is going to be some big breakthrough here. And there is going to be some concept of memory that's sort of baked in that allows the models be customized in a way that isn't quite so heavy weighted as fine tuning. But I have no idea what that's going to look like and I don't even have an idea of what that could be. Our approach right now is we have this thing called memories, actually there's a feature in our product, which basically just allows the LLM as a tool to sort of manage a list of facts and then we insert those facts into the prompt. And this is really useful for, really for behavior customization. So if you want to be like, hey, every time I write an email, always CC my EA or like anytime you schedule an email, like default to have it be 45 minutes or whatever.

• 1:07:50Andrew Lee?

  It's really useful for that. So we have that, but you have to be explicit. You have to say, hey, remember this fact and then they'll do it or remember to do this. We also, obviously you search, I think, I still think there's a lot of opportunity with an agentic model to you search, you could have a database of, you know, interesting facts about you and you could have a tool that like looks at the facts and tries to apply those. I don't think it'd be quite the same as like built in memory, but you could probably do a lot of the same stuff. I have not had time yet to read the Titans paper. I think you mentioned that the other day. I'm very interested, but yeah, I do think there's going to be some sort of breakthrough here and I can't wait to see what it is.

• 1:08:25Nathan Labenz?

  It feels like one of the, you know, it's too early to call a final frontier, but in terms of imagining something that could really, you know, work alongside me on an extended basis, it does seem to me like a sort of more integrated, dynamic, you know, kind of active learning, you might call it, but you know, constantly a bit of memory does feel like something that would move the needle tremendously on that particular dimension. Okay. I think last question before we switch gears into, um, you know, the sort of meta of like how you're doing it. Cause I think that's going to be really interesting too. But one thing we talked about last year, I wanted to get an update on is we had speculated

• 1:09:07Nathan Labenz?

  about the rise in AI generated spam and sort of, you know, what happens as more people are adopting various tools and you know, you've got AIs potentially talking to AIs.

• 1:09:19Nathan Labenz?

  Has that happened? You know, another thing we of course, we sort of expected was like a lot of deep fakes during the election. That didn't really happen all that much. Is this another thing that we're just kind of all too worried about? And you know, it's not really a thing? Or are you actually starting to see any sort of interesting, you know, AI dynamics? Let's call it.

• 1:09:36Andrew Lee?

  Not a ton, honestly. I do actually get a lot of AI generated content in support, especially just like people being funny. We're like, you know, we'll send out a newsletter and they'll have it like write a poem or a joke or something respond. And so people are doing that kind of thing. But I haven't seen a huge uptick in AI spam. Our users haven't reported a huge uptick in AI spam. It's possible the normal spam filters from Gmail are doing a good job, or maybe it's just not a big problem. But no, thankfully, it doesn't seem to be as bad as I feared.

• 1:10:08Nathan Labenz?

  Good. Yeah, I would say I still notice honestly, more often I marvel at how I'm getting cold emails that were obviously written by a human and would have been better had they been written by an AI. So it seems like maybe just adoption generally lagging is still the broad explanation there.

• 1:10:27Andrew Lee?

  A significant fraction of the emails that I send are at least AI enhanced. Some of them are fully AI written. And as far as I know, no one notices. I'll talk to people in person and they'll be like, they'll ask me this sort of question. I'll be like, hey, do you realize that my email is AI written? And they usually say no. So maybe I just don't notice, which is fine if it if I can't tell.

• 1:10:46Nathan Labenz?

  Yeah, there's a couple of really interesting things there. I think I personally have not shaken this yet, but I've increasingly had to confront the fact that like, I'm probably just way too precious about the little elements of my style that, you know, feel to me like they make me unique. And I honestly kind of doubt that anyone else notices or cares. I've got a friend who really hammers on that all the time. And he's just like, nobody cares about that dude. Like, your little flourishes, like, they're totally lost on other people. Anyway, I sort of also differentiate the world between like, routine tasks and non routine tasks. As somebody who like, doesn't really have a job and is just kind of, you know, scouting the AI space all the time. I do like very few routine things relative to I think, you know, certainly like your typical, you know, sales

• 1:11:31Nathan Labenz?

  user who's got a CRM integration type of thing would need to do. But Yeah, it's it's it's kind of philosophically challenging to be like, maybe all this style and all this sort of personality and all the care that I've kind of put into how I want to show up, maybe it doesn't really matter. And not only like, you know, it's one thing to think like, maybe an AI could replicate it. And it's another thing to think like, it doesn't even really matter if it replicates it. It's just, you know, do a good enough job and like, the world just keeps moving on. But

• 1:12:00Andrew Lee?

  I really do think routine is the place to look for AI. Like, I do a ton of routine emails. As for example, yeah, we made some changes to the UI yesterday. I'm sending emails that kind of walk through why we did things and how to adapt your work style that are very similar between all the different people I'm talking to. And the AI is really useful in, you know, looking, hey, you just sent a bunch of emails kind of like this. Let's pull out some of the ideas here and reconstruct the answer to the question. But yeah, I'll send like 20 emails of like a very similar type.

• 1:12:28Nathan Labenz?

  So let's shift gears a little bit, if you're ready, to what it's like to work at and be building Short Wave. I understand that there is some news around a fundraiser. And there's this expanded vision of, you know, not just email, but kind of omni-channel communication. And then I thought maybe the most interesting thing from your recent Twitter thread was quoting here, we're also building an incredibly AI-forward culture where the focus of our work is managing AI agents rather than making changes directly. We write a ton of code with AI, use AI for research and design, and even wrote this posting with AI. So I don't know much you want to share about the fundraiser, and you can expand on the vision. And then I really want to get into,

• 1:13:12Nathan Labenz?

  you know, what's it like to be building this AI-forward culture? And, you know, what's it like to be managing agents all day at Short Wave?

• 1:13:20Andrew Lee?

  Yeah, so the company has changed tremendously in the last few months. We kind of reached a point where we figured out, hey, the future is not making a better email client. The future here is an AI that has your communication app sort of integrated into it. But like, that the main thing that people are doing in this thing is actually talking to the agent. And In that world, we're building a very different thing. It is not, you know, how do you streamline every interaction of the work in the inbox? And how do you make the UX the email threads amazing? And like, we care about that stuff to some extent, but it is mostly, how do we help people get things done at a higher level? And we're already seeing a lot of traction. Like this is, the reason we're doing this is because this is already where people are being successful. You know, our biggest and fastest growing plan is

• 1:14:02Andrew Lee?

  our most expensive plan that actually has, you know, basically like a bunch more money being centered on the topic for AI. Like the people who are using us now, they care about the AI. And so we're just doubling down on that. And that vision, I think, has gotten our, like, our investors very excited. You know, they see, you know, opportunity for email client here, but they see opportunity for like this sort of thing here. And there does seem to be an opening in the market for an AI that can actually do things, right? If you use ChatGPT, or you use Claude, or perplexity, like, they're great at answering questions and doing research, but when it comes to actually doing work, they don't really have any capabilities to do that. And someone's going to build a thing that can actually get stuff done. And we see that opportunity.

• 1:14:45Andrew Lee?

  So we're starting to see our competition as like, you know, the next version of ChatGPT, rather than the next version of Gmail. And so it's been a reframing. We have raised some money. I'm not ready to share the number who was involved, but enough that we can significantly expand our team. So we're doing that right now. As part of this, we basically revisited everything about the company, you know, the way it's financed, the way we operate every day, who's on the team. And we think this tidal wave is coming. We want to be in front of the tidal wave. I think the tidal wave is going to hit faster and harder than basically everyone realizes. And we want to make sure we're on the other side of it. And so we think it's going to require some different people with a different mindset to do what we want to do. And yeah, I think you mentioned the soundbite

• 1:15:28Andrew Lee?

  there of like, your job is going to be primarily managing a bunch of, you know, AI interns to get work done, rather than necessarily doing it yourself. And we're seeing this especially right now with code. If you're someone who is using the cursor agent mode, like, you know, A good, I think a good software engineer right now is gonna have at least one agent running all the time doing something, right? They start the agent off and then they go start working on the next problem and it goes off and solves it. And like, we have found that it's possible right now. You can, we can take a bug report from Asana. We can copy it verbatim into that agent mode and ask it and it'll solve it in one shot in many cases, which is pretty amazing. So on the software engineering side, it's like, hey, we're looking for people

• 1:16:10Andrew Lee?

  whose skill set is primarily not execution, it's not writing code, it's primarily understanding the problems, understanding the components involved and how they should interact and being able to like frame those prompts. And so the value has shifted from people who can get things done to people who can understand the problems and structure of things. So that's on the coding side. On the design side, I think it's changed too, where it used to be that the way that you design a product is like, you know, you get together in a room and on a whiteboard and you draw it out and then you make some mocks and then you make a prototype and then you build a thing. And now the first thing you do is make a prototype, right? Like before I even talk to anyone else, I just build it in bolt new. And it's only after I'm like, yeah, I think this will be pretty good.

• 1:16:53Andrew Lee?

  Then I go and I talk to the right people and we like make a mock and we try to feel, figure out the details. But it lets us kind of cycle through bad ideas like much, much faster. So we're looking for, you know, on the design side, we're looking for designers who have already made the shift of, I don't start with a mock, right? I don't even start with a wireframe. I start with a working version that I have built with bolt that new or a tool like that. That's the beginning. And so our design, I think is totally different. Same thing with, with like content. I mentioned like the job postings that we wrote. That's all chat G P T four five. It's really good at writing, right? It's not, it's not quite at the level of like, if I had really put in effort and done a good job myself, but it's not far off. And I can produce.

• 1:17:37Andrew Lee?

  You know, content that's at a pretty high bar in a fraction of the time. And so, you know, we need to do a lot of comms here. We want people who are good at, you know, how can one person manage our blog, our social media, our change log, all the docs in our website, everything? How could just one person do this if they're thinking hard about the right prompts and the right way to use these tools and how to have it generate this stuff? So it's gone from, hey, maybe we need a team of 50 to do this. So maybe we just need a team of like 15 or 20 people with like the right skill set and the right usage of these tools. And, you know, the 30 people that aren't there anymore are really more on the execution side. And the 15 or 20 there are really more on the sort of like problem, understanding and solving side.

• 1:18:20Andrew Lee?

  And I think it's going to be a big shift and we actually, I think I mentioned this earlier, but we, uh, we actually let a couple of people go who were really, you know, very talented, very good at what they did, great employees, great attitude, but not quite right for this new world that we felt we need to get into. And not super, super passionate about it either. And so we thought a change needed to be made.

• 1:18:42Nathan Labenz?

  Passion is super important. I mean, that's definitely something I observe all the time. Maybe, you know, it does take some curiosity, you know, just innate interest, you know, that I feel like to the degree that I have demonstrated, you know, some aptitude here, a lot of it really just stems from that. Is this something, I guess, at this point, is this something that you think is an investment in the future? Obviously it's that, but is it also something that you think like allows you to be more productive or execute at a higher level today? Or do you feel like it's kind of like, well, you know, we might not be, like you mentioned with the blog question, it's not quite as good, but it's fine because it's pretty good.

• 1:19:24Nathan Labenz?

  And we're also like investing in sort of being the type of organization we think we have to be in the future. Does that make sense? Like, what is the time horizon for ROI on this?

• 1:19:35Andrew Lee?

  I think we're already seeing it in a big way. And I think companies need to be making changes now to the way they work, the way their org charts work, the way they spend their money. I'd say, you know, I don't, I don't know if I can put a number on it, but our engineers today are significantly more productive now than they were six months ago. It's really changed in the last few months. Like if you last tried this stuff last summer, the world is different. Give it another try. So significant imperat- differences there on the design side too. Like, yeah, we can just skip over half the process and get right to the prototype. And it, I wouldn't say it, you know, changes the work needed to build the final design dramatically, but it does allow us to cut out a lot of bad iterations, which is where most of your time spent anyways. So I think we're a lot faster there.

• 1:20:20Andrew Lee?

  On the content side, also a lot faster, right? Like good writing is hard and most of the time you don't need to write really amazing writing. You just need to write pretty good writing. And, you know, if you can do that much faster, you can keep your docs up to date faster and things like that. So I think we're seeing those gains today. But there, there's a related thought I wanted to add in here, which is a big change in the way we're thinking about our business. So a year ago, when I was talking to investors, I was saying, hey, their, their question was basically like, what is your moat? Right? Like you, you're building this cool AI stuff and, but why can't someone else come and do this? And I was like, well, our moat is that we have an email client. And it took us years to build this thing. If anyone else wanted to enter space to build like an AI enhanced email experience, they'd have to go build an email client. That's really hard.

• 1:21:02Andrew Lee?

  And now I think, you know, the thing that took us four years to build, it might not take four years for the next person, right? It'll still take them a while, but they're going to be able to do a lot faster. And I think there's this big change happening in the industry where all of these companies that have these moats of like the software they have built, that moat is being significantly eroded. The value of that like code that's sitting around is going down very quickly. And they're going to have to come up some other moats. And we're looking at this and saying, hey, the moat that matters, the only moat that probably matters is speed. And so how do we optimize our team for speed? And I think the best way to optimize your team for speed is to keep the team small. It's just harder to get consensus on, on big teams.

• 1:21:45Andrew Lee?

  So we're like, okay, how do we have 15 people that are the right people with the right tools that work together really well? So we can just like move, move, move. And our moat is that we're always ahead of everybody by, you know, two months because of the way we're structured. So that's been a big, a big change in our thinking.

• 1:22:04Nathan Labenz?

  So fifty, is that like a long-term number?

• 1:22:07Andrew Lee?

  Uh, not, no, not a super long term, but I think for the next year, right? Like, I don't think we're going to go beyond fifteen and then we'll see beyond that. But, uh, no, I don't think most people could build the product that we're the scale or the scope of product that we're building, you know, historically with, with fifteen people, but I think we can't.

• 1:22:26Nathan Labenz?

  Yeah, I mean, there's a lot of surface area already and it sounds like you're planning to add a lot more and on top of that, got a lot to keep up with.

• 1:22:35Andrew Lee?

  Well, and we're not, you know, we're not the first ones, I think, to figure this out. Like Cursor, you know, it's like twenty people, Midjourney is like twenty people and they're obviously wildly successful. So I think, I think there's a new era.

• 1:22:47Nathan Labenz?

  Are you still subsidizing users on the margin? That was another, um, tidbit from last time. You said you were literally like losing money on every user. Is that still true?

• 1:22:55Andrew Lee?

  Uh, not anymore. No, we've done a lot of work to, to kind of increase efficiencies. We also added like a higher end plan and we've gotten people to kind of opt into those higher end plans. So we are, uh, margin positive. I wouldn't say hugely margin positive. We still spend a significant fraction of every dollar that you give to us, like goes directly to L L Ms. And then a lot of that also goes just like traditional email infrastructure. But on the margin, we do make money now. Yeah.

• 1:23:20Nathan Labenz?

  All right. Well, that's cool. Um, is there like a two to ten X more expensive version of the product that you could imagine rolling out? Or are you already basically, I mean, you're not quite maxing out, right? Cause you talked about like you could spin up the agent every time. Could you do that today if you just charge five times more?

• 1:23:40Andrew Lee?

  We could, and, and I think we probably will. If you look like, I think I mentioned this earlier, but we have all these different plans and the, all the growth basically is happening in our most expensive plan. Everyone's coming in and they want the most expensive plan because it lets you have the biggest context window and it lets you index all of your history for search. And people care a lot about that. And even though the, you know, our business plan can give you almost as good answers as the premier plan, but people want the best answers. And, you know, the difference between good and best is worth a lot of money to them. And, uh, yeah, I think we undershot. I think, you know, K G P T Pro came out at $200 a month. I pay for it. I think it's too low.

• 1:24:17Nathan Labenz?

  No hesitation on my end either.

• 1:24:19Andrew Lee?

  Yeah, I would, I would pay more than that. No, no hesitation. And I think we should probably do the same. I think there should probably be for the type of person who really truly lives in their inbox, who's able to get the kind of value out of what we do that we want them to. I think we could also have a $200 a month plan. And I don't know how far this goes. Like, I know Sam was talking about, like, maybe we'll have a $2,000 a month plan or a $20,000 a month plan. And I have no immediate plans to do that. But to the extent that people want us to spend money on GPUs on their behalf, like, there's, there's sort of no limit here. So, for example, right now, we don't, I think you mentioned, like, the automatically running the full agent on every email.

• 1:24:59Andrew Lee?

  That's a great place where we could spend a lot of money on your behalf if you want us to. And it could be a 100x increase in the amount of compute required. I think another example is reasoning models. Where, right now, we don't use reasoning models. They're much more expensive. They're slower. But if, you know, you're trying to do something complicated, like, I want you to give me, like, a detailed analysis of every customer report over the last year, right? You know, this might be something that otherwise you're given to an employee and they're spending a month on. And you might be happy to spend, you know, hundreds of dollars of compute just to answer that one question. So there may be something that we can do there. So, yeah, reasoning models, automatic execution of things, bigger models in general. And yeah, maybe there's a $200 a month plan or maybe more someday.

• 1:25:43Nathan Labenz?

  Going back to the building side, are there, you know, you mentioned cursor, of course, and I'm a cursor user. And I've also, you know,

• 1:25:53Nathan Labenz?

  gotten into Bolt and to Lovable and, you know, basically try everything I can. What I haven't quite figured out yet is how to make things like work together, you know, how to effectively sort of architect systems of agents, you know, and Replit too, right? You know, great experiences with all these different things in their moments. But are, do you have kind of a? Mix of things that you've brought together, you know, are there things that sort of complement the coding agent? I'm thinking, for example, of like a company called Coto that does like specifically emphasizes testing. Or, you know, there could be like

• 1:26:35Nathan Labenz?

  observability or sort of monitoring tools. I mean, monitoring is another thing that, you know, you could really imagine

• 1:26:42Nathan Labenz?

  layering on. And it ultimately seems like we're going to need it, right? If you're going to have 15 people who continue to run it roughly human speed with a lot of AI assistance, they're also going to need like AIs to sort of help them supervise the AIs, right? I mean, this sort of the pure, as the pyramid, you know, built under each person and under the company collectively gets bigger and bigger. There's got to be like a whole architecture and sort of specialization within these agent ecosystems, right? So I guess, is that roughly how you see things shaping up? And have you started to tie these things together in a useful way at all so far?

• 1:27:20Andrew Lee?

  Yeah, I have a, I have a couple of thoughts here. So the first thought is, I think one of the potential futures for Short Wave is really a sort of a routing layer for the messaging going on in your business life. So you might be using a bunch of these different AI tools, but something needs to sort of like take the incoming events and decide which agent handles it and get it to that agent and take the output from that agent and do something staying with it. And inboxes are great tools for sort of managing that flow in a way that both the human and the AI can collaborate together on it. And if we're going beyond email and including other kinds of human communication, maybe we're going to go beyond traditional human communication. Maybe, maybe you're going to start thinking as more like a, like a Zapier type of thing with like a human UI that you can use as well.

• 1:28:05Andrew Lee?

  And we might be a routing layer that helps you route things into these other tools. So we've had requests from, from customers like this. And this was a discussion I actually just had with one of our investors of, you know, maybe we need to start thinking ourselves more as like a, like an agent routing layer. So that's my first thought. My second thought is, we have tried in our approach to agents, some different ways of having like a multi-agent approach to solving problems for basically like you, you know, you have one system prompt that you iterate on over here. And then once you decide it's a certain type of problem, then you like change the system prompt, you add in a different prompt, or you hand it off to a different model. And Our experience has been so far, and you know, this is just one experience, our experience has been so far that

• 1:28:48Andrew Lee?

  the systems end up being kind of brittle. They tend to struggle at reasoning across different types of tasks. And the cost aside, the better approach tends to be just take the biggest, most expensive model you possibly can. Stick all the instructions into the context and let it reason across the different things. And that, I'll give you a specific example of a win we had here recently. So we used to have the ability for you to add custom instructions for certain types of operations. And the way that worked was when, for example, for writing, you could add like custom writing instructions. And when we called the tool that was used to grab the writing instructions, we would insert those custom instructions into that thing.

• 1:29:29Andrew Lee?

  That worked well for cases where the writing tool was being called appropriately. But if you had some custom instructions that were like, you know, trying to give it hints of like when to write emails or what types of examples to look up or that wanted to handle like not just things during writing it during other times. It didn't work so well because these things only got plugged in certain times. And what worked much better was this memories approach where you basically say, hey, we're going to take some customization instructions, we're going to add them into the master prompt. And we're going to include that in every call. And then you can have a customization that can be considered at any time, right? Like you could have a customization that's like, you know, always

• 1:30:12Andrew Lee?

  address Nathan Labenz as sir, right? And it could do that when writing emails, it could do that when scheduling calendar events, it could do that in any situation. And that has proven to be like a much better user experience. It produces much better results from the AI. And so we, yeah, we used to have a model that was sort of like routing things to different AIs and just the screw it. We're just going to put it in the main prompt has actually worked much better.

• 1:30:33Nathan Labenz?

  Yeah, more cash flow too, I guess, right? To have that consistency.

• 1:30:37Andrew Lee?

  Yeah, so I see the opportunity for kind of like the multi-agent world really being more about working across organizations where, you know, we need some interface between like our team and the cursor team, but probably not something that's happening like within our app. I think within our app, the approach of just like one, one, the biggest element we can get and the biggest context and then like using caching really well. I think that's probably more likely in the future for us.

• 1:31:01Nathan Labenz?

  So I share the same intuition, just based on, you know, everything that I've tinkered with over the last couple of years. And I just actually told a company that I have been doing a little bit of, you know, agent advisory with that. And then like two days after I said that, OpenAI came out with their thing where like handoffs between agents is, you know, a pretty notable new feature. Is that, um, what do you think is driving that?

• 1:31:29Andrew Lee?

  I literally talked with that team yesterday about that exact thing. And I had the exact same questions for them and I showed my perspective. And I think, uh, I don't think handoffs are going to take off. That's my, that's my hot take.

• 1:31:44Nathan Labenz?

  Yeah, I mean, I think it's some upsides to it. I mean, the things that they cite are like, it's easier to test them in isolation and whatever. But yeah, I mean, it's tough when you then have like classifications and yeah, there's like just more things to get that can go wrong. That's for sure as well. Okay. Well, I'm glad that, you know, to hear somebody else shares my addition on this. It makes me feel like, because I was like, am I out of touch on this? You know, certainly OpenAI, you know, has some real insight into what's going on in the broader world.

• 1:32:12Andrew Lee?

  They shared the same perspective with me around, you know, sort of developing things in isolation. But I think the reality is with an agent, you don't really want the tasks to be isolated. You want the agent to reason across all of its capabilities and make a smart decision. And, uh, you know, maybe there are some places where you really want to sandbox it. But my experience is that's not usually what you want. You usually want to think about everything is capable of to make the best choice.

• 1:32:36Nathan Labenz?

  Yeah, I wonder if this is to some degree driven by

• 1:32:41Nathan Labenz?

  larger companies that sort of have like existing organizational structures and like who's responsible for what and you know that those lines are really hard to cross. I heard one of the more interesting little tidbits that I've heard in AI discourse over the last year probably was with Yi Tay from I think the Late in Space podcast. Where they were talking about multimodal models and like why they've developed as they have and and why you know it was like vision models and language models developed separately and then you know kind of merged with these sort of late fusion approaches. And his take on this was very simply that's a reflection of legacy team structures like it used to be that you'd have a vision team and a language team.

• 1:33:24Nathan Labenz?

  And so they would do their things and then like maybe you could try to fuse them, but the you know, the upshot was like that won't last, you know, into the future. You're going to have just unified teams and you'll have unified architectures from the beginning. So this feels like maybe the seed of kind of a similar insight where it's like

• 1:33:45Nathan Labenz?

  maybe one reason because I've often also been like why we talked about this a little bit last time too, with respect to Gmail specifically like, you know, why can't the big incumbents do a good enough job with this, right? Like the technology is not that hard to use relative to you know, it's one of the things that's so great about it, right? Is it's so flexible and in many ways forgiving and it like, you know, totally understands what you meant, even if you're like riddled with typos. I've even seen people doing this like speed typing where they just put tons of typos in and have the AI correct it. And for whatever reason, I haven't got over the hump on that myself. But maybe there's something here in terms of like who will win in different verticals that is just like big companies with these org charts and these division of responsibility and like who's going to sign off on what?

• 1:34:31Nathan Labenz?

  Like maybe these things are being built for them because that's what they kind of have to have to be able to make any sense of it. And maybe it's not really about like what's going to make the thing work best and And a person more like yours where it's like small team, totally unified, you know, single agent that knows your whole history and can kind of, you know, deal with you on that basis. It does seem like what I want, you know, I don't want to be, I don't, you know, it's very, it seems to counterintuitive. Like, I don't want to be handed off from one AI, I don't want to be handed off from one human to another. I don't want to be handed off from one AI to another. So yeah, this is interesting. I think there's definitely some, some insight there that might prove predictive.

• 1:35:05Andrew Lee?

  I think you're probably right. I'm sure they did this in response to like some real needs that they're seeing. But, you know, I think one of the reasons we started short right back in the day is, you know, worked at Google. I had some insight into like what sorts of organizational struggles they were going to face. And I had the belief that the Gmail team was not going to be able to, you know, innovate quickly for organizational reasons that were very hard to change. And I think that has borne out. I think you're seeing that now where, you know, the models from Google are awesome. Right?

• 1:35:38Nathan Labenz?

  The, the new 2.5 is literally mind blowing.

• 1:35:42Andrew Lee?

  But like the, the product progress in using those bottles in interesting ways in, in stuff like Gmail is way behind. And I think that's driven by, I just thought that those people are smart or whatever. I think it's just driven by the way the organizations work. And, you know, for example, we decided, hey, we need that, we need the LLM on the, on the left. Right? Agents got to be over here. There's good reasons for this. Right? If I was at Google and I was like, we need to take the right sidebar and put it on the left. That'd take me two years, right? Like, think, think of all the sign off I need to get, all the people I had to get to convince to have buy in here. And all of the, you know,

• 1:36:20Nathan Labenz?

  You literally have a thousand meetings.

• 1:36:23Andrew Lee?

  Right. And for us, it was, you know, it took a few weeks. So there was a lot, but it was all the discussion was, is this the right thing for the product? And we could focus entirely on that question and we can just move a lot faster.

• 1:36:36Nathan Labenz?

  Yeah. So going back a little bit more just to who you're looking for and you know, what the, you know, what the hiring process is like. One thing I noticed is as far as I can tell, all the roles are in person, San Francisco. And I have kind of a question about that, especially as you think about like a future in which, you know, you have like a lot of the work being done by AIs. In a sense, it feels like the AIs are like always remote. And personally, I have had, you know, being in Detroit, it's like quite different, you know, local talent pool and the, you know, pros and cons are quite different. But I've had a great experience hiring remotely. It's definitely like opened up the aperture of who we could go for. And I kind of am like, well, jeez, if all the AIs are like, you know, only going to interact with me through

• 1:37:20Nathan Labenz?

  a screen anyway, kind of, you know, at least until I got humanoid robots sitting at the desk next to me, you know, it'll be that way. How are you thinking about like, holding the line on in person versus, you know, potentially liberalizing to remote?

• 1:37:35Andrew Lee?

  There's a couple of, a couple of pieces here. So I think piece number one is what I talked about with speed. We have been historically a remote team. In fact, at one point, we were fully remote. We weren't even hybrid at all. And I think it's becoming increasingly important that speed is critical. It is a lot easier to move quickly in person, especially when what you're doing is making like rapid product changes, right? You can, you could move in a straight line very, very quickly remote. But if you need to like quickly change course and have a lot of difficult meetings and conversations and have to give deliver tough feedback and be like, hey, this isn't working. It is just a lot more emotionally challenging to do that in a remote environment. It's possible and teams do it. I think it's harder. And I think it's something we really struggle with. So if speed is paramount, we think having a core team of folks who are

• 1:38:19Andrew Lee?

  driving most of the roadmap, product decisions and the stuff that can meet in person and hash it out on a whiteboard at any time, I think is really key. So we really want to have that happen. The other, I think, really key piece is, is just, it's just a reflection of the founders. So Johnny and I work better in person. I am a better leader and manager in person. People, I think, like me better if they see me in person than if they're talking to me over a camera. And that that's a weakness, probably, you know, like, I sure I could learn to be more effective in a remote way. But I also to some extent need to, you know, understand my capabilities and limits. And, you know, I look back at Firebase. Firebase was a fully in-person company, and there were a lot of struggles that

• 1:39:03Andrew Lee?

  I think we've had a short wave that we didn't have there caused by some of the differences there. So I think Johnny and I looked at this and said, hey, we need this to be a place that is optimizing for our strengths. And we work better in person. That we need to have a court, and we need to move really, really fast. We got to have this core team in person. And yeah, I think this can make recruiting harder. There's lots of super amazing talent that is not in San Francisco. I'm very keenly, keenly aware of that. But we think it's worth it.

• 1:39:31Nathan Labenz?

  Yeah, okay. How about an AI scout role? This is a hobby horse of mine. Have you thought about a dedicated position for just like, somebody to go try every new thing, whether it's new model, new framework, new, you know, agent experience, whether, you know, somewhat competitive or totally far field? This to me feels like something more and more companies are going to need. But I don't know how many are feeling it quite yet. So

• 1:39:58Andrew Lee?

  Are you asking for a friend, Nathan?

• 1:40:01Nathan Labenz?

  Yes, well, many friends, actually, I think, because, you know, when you're, when you would have been 50 and you're now 15, and that starts to get generalized, like, we do need some of these new AI jobs. Like, I've managed to kind of stumble my way. You know, sometimes call myself the Forrest Gump of AI, where I just sort of unintentionally stumble through these important scenes. And I've ended up in a, you know, a place that I honestly quite love and like, I have no,

• 1:40:27Nathan Labenz?

  you know, nothing but appreciation for. But I do think we need like, a lot of new AI jobs. And this feels to me like one that might become common. And so, yeah, I think there's, I'm asking on behalf of maybe a lot of people.

• 1:40:39Andrew Lee?

  Yeah, a hundred percent. I it's not a role that I have listed right now just because I think we're we're a very small team right now and we only have a handful of roles, but I I do think it is sort of impossible to keep up right now. I do think there have been a number of moments where our discovery of, for example, if I hadn't listened to that particular podcast that talks about the new behavior of Claude, it might have been months before we figured out the same thing, right? So I think there has been a number of inflection points driven by discovery of new technology that I think we just got lucky on and it'd be great to have someone who's like systematically doing this sort of thing. Obviously, I try to like listen to great podcasts like this one and keep up

• 1:41:17Andrew Lee?

  that way if I can and read whatever I can. But yeah, I think in the future, we may very well have a role like that. And I think a lot of other companies, especially bigger companies, where not everyone is constantly thinking about AI, could benefit tremendously from something like that.

• 1:41:32Nathan Labenz?

  Yeah, situational awareness, harder and harder to maintain all the time. Maybe last a little bit on the hiring side. You know, to some degree, obviously, these are just labels, but I did also notice that the roles that you do have posted are all staff or senior, you know, designation. I guess maybe two layers to the question. One is like, is there a place for, you know, somebody who doesn't have a lot of experience at Short Wave? And what would it take for them to, you know, to take maybe a somewhat extreme case, like, let's say they're freshly graduated, no work experience. Is there any way for somebody like that to demonstrate to you that they have the skills that would make you even open to hiring such a person?

• 1:42:18Andrew Lee?

  Yes, absolutely. And and for what it's worth, the job postings have been updated significantly since I think you saw them. So I posted the CDO roles first. We actually have some entry level roles up there now. Including, I just posted yesterday, we're looking for someone for customer success. We're looking for like a content creator. We want to make a lot of videos. Because I think there's a lot of, just a ton of sort of like discovery and education in AI. So we want to do some content creation. There's a junior kind of product engineering role up there. So we are, we're definitely looking for some junior folks as well. If you want to impress me, there's a very simple application process, which is send me a video. We've kind of gotten rid of some of the sort of take home tests and things like that because they're too easy to get with AI. So we're going to just, you know, send me a five minute video, show me something cool that you did. I think the thing we're most interested in is

• 1:43:05Andrew Lee?

  Are you someone who's like really forward thinking with AI? Have you figured out, like, if you could show me, here's a way that I leverage AI to do something useful in a way that I haven't thought of and impresses me? I think that would be, you know, that would cost me to take notes. Um, so yeah, are you super forward thinking with AI? Are you being creative about how this stuff is used? Are you staying on top of it? Uh, as you know, if you can stay on top of it, like you're, you're, you're pretty impressive already.

• 1:43:28Nathan Labenz?

  Yeah, no doubt. I guess more broadly, what do you think is going to happen to the future of the software industry? This is like, obviously a super hot topic right now. Are there going to be more developers because they're more valuable? Or is there only so much software that needs to get written? Um, you know, and also this could definitely like break down in phases where I think we might be on the up ramp of that curve, but the more I hear from folks like you that, you know, you only need, you know, 15 people for the foreseeable future, the more I'm like, man, I don't know how much we can bet on this sort of developer market to grow or even sustain itself as it has been for the last few years. What's your expectation?

• 1:44:05Andrew Lee?

  Yeah, good, good question. Um, my, my co-founder oscillates depending on the day between an existential crisis of, you know, he had the Thetia, right of like, I am obsolete. And on the next day, he's like, I am a god and I can do so much. And he doesn't know which one to think, right? So, uh, I think, I feel like every software engineer is going through that right now. We're like, their productivity is like going through the roof, but also maybe I'm obsolete at the same time. I think it's going to change a lot. The nature of what you use can change a lot because, for example, like actually writing code, like localized code is starting to become something that LMS can do. So for example, like any sort of front end development where you're like, hey, I need a button that looks like this, right? The LMS are great at that sort of stuff.

• 1:44:47Andrew Lee?

  And so I think increasingly code is going to get written by this. There's a, there was a, a interview with Dario from Anthropic, who I think he said, like, within two years, like 90% of all code is going to get written by, or maybe it was 100% of code is going to get written by AI. I'm not sure I quite believe that, but I think it's going to be very significant amounts of it. The places where I think AI is going to take longer to, you know, if ever to, to be able to do it is really sort of understanding user problems and understanding the components and how they interact, which I think is. Super important, super complicated and like actually a huge part of the job today, right? Like if you, if you take a senior engineer today, the senior engineer isn't senior because like the, the UI code they wrote is like dramatically better than the junior engineer.

• 1:45:32Andrew Lee?

  The senior engineer is able to sort of think about the whole business problem and solve it. And so I think, you know, the folks who are good at thinking about the business problems and how to solve them and thinking about the components and how they interact, I think they're going to do great. I think the folks whose strengths are on the execution side are going to have to learn some of those other skills and get good at that. In terms of numbers of people, I don't know, right? On the one hand, yeah, I think you can do more with less. On the other hand, it's suddenly there's, there's a million little companies that should have existed, but it's just too expensive, right? Like software engineers are crazy expensive right now. And if you need 10 of them to do anything, right? But suddenly there might be a startup where, you know, you would have taken 20 people for it. Now you can do it with three. And, you know, you can build a niche product for something.

• 1:46:16Andrew Lee?

  So maybe we'll just see an explosion of, you know, another 10x explosion in, in, in software. And, you know, the jobs will stay the same. Uh, so I don't know, I generally an optimist. I generally think that it will make life better for everyone, including employees. And it just, everyone's going to have to adapt, but I think it's going to be a good thing. I think being a software engineer is going to be an awesome place to be over the next few years.

• 1:46:40Nathan Labenz?

  Yeah, I can't imagine coding without AI assistance at this point. So on that level, it's no doubt. And I definitely share your expectation on 10 times more software getting created. And I'm still kind of like, but maybe even 10 times more software getting created isn't enough to sustain the number of like just pure headcount jobs that we currently have. So, you know, we'll, time will tell.

• 1:47:04Andrew Lee?

  Yeah, for sure.

• 1:47:05Nathan Labenz?

  I, I do advocate people start thinking about universal basic income if they haven't already started to contemplate it, but we're not quite there yet. That's, um, you know, that's like a long-term problem, like, you know, maybe two, three years out. This is outstanding. Maybe one last question, you know, as you look a little bit into your crystal ball, what do you think are going to be the big developments for, say, the rest of this year? Like, you know, what are the things that you're kind of, man, if only they could get this to work or fix this? Um, you know, if only the world were a little bit different, you know, things would be like much better, much different for you. What are you kind of most closely watching in that sense?

• 1:47:43Andrew Lee?

  Yeah, I think the number one thing is post-training on agentic behavior and tool calling type stuff. So the step change between every other model before Cloud Sonnet 3.5 and Cloud Sonnet 3.5 just enabled a whole bunch of new stuff. 3.7 was better, but there's still gaps. It'd be great to have more competition there. Great to have OpenAI. Other folks have options here. And I think this is all coming down to agentic specific post-training. And I am excited to see where that leads and how good that gets. And like, iteration and tool calling can work around so many limitations in your system. And the AI can sort of find creative solutions to stuff that I think the sky's the limit there. So that's number one. Number one thing is like, post-training on

• 1:48:28Andrew Lee?

  agentic behavior. I'm also just generally excited about improvements for the productionization of these things, performance of the tools, cost, latency. There is a lot of things that we can't do because they're impractably expensive, like running the full agent on every email that comes in, or adding more search results into every query. And the cheaper, faster, more reliable these things get, the better for us. And I expect that to keep happening. Cost is still a huge factor for us. So that's another one. I think another area is kind of more native multimodal voice. Today, you can do voice things. We actually have a voice input in our app. It's fine. It's not great. It's not like really talking to a human. But models that are more, you know,

• 1:49:14Andrew Lee?

  support voice natively, multimodally, that we can use for the full agentic flow would be like totally killer. So fingers crossed there.

• 1:49:23Nathan Labenz?

  Yeah, I love that vision, too. I mean, anything that will get me outside more and less tethered to my desk is hotly anticipated, certainly by me.

• 1:49:32Andrew Lee?

  100%.

• 1:49:33Nathan Labenz?

  Any other thoughts or, you know, closing wisdom you want to leave people with?

• 1:49:38Andrew Lee?

  Just you should you should come check out Short Wave and you should totally either apply yourself or refer your friends. We have a $10,000 referral bonus for anyone. If you get them to apply and tell us that you referred them, we will give you $10,000. No joke. So yeah, help us find great people.

• 1:49:58Nathan Labenz?

  Love it. Andrew Lee from Short Wave. Thank you again for being part of the cognitive revolution.

• 1:50:03Andrew Lee?

  Thank you, Nathan.

• 1:50:04Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.