Lindy Teammate: Flo Crivello on Multiplayer Agents, Memory & Why He'd Ban the Chinese Models He Uses

Flo Crivello returns to The Cognitive Revolution to launch Lindy Teammate, an AI employee that lives in Slack, connects to company tools, and accumulates a team’s shared context. He argues that multiplayer AI matters because intelligence without context is less useful than an ordinary coworker, and explains Lindy’s approach to agentic memory, editable file systems, context buckets, and large-scale tool outputs. The episode also examines the costs and operating realities of building for the next generation of models, from negative gross margins and cache rates to Lindy’s own dogfooding in engineering workflows. The stakes are whether AI workers become drop-in teammates, how long humans are still needed to cover model mistakes, and what happens when engineers spend more time managing the machines that do the work. Lindy: https://go.lindy.ai/CognitiveRevolution For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/lindy-teammate-flo-crivello-on-multiplayer-agents-memory-why-he-d-ban-the-chinese-models-he-uses/ Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (02:56) Lindy Teammate launch (10:07) Agentic memory systems (Part 1) (19:58) Sponsor: Claude (21:28) Agentic memory systems (Part 2) (22:21) Reliability and caching (34:57) Slack data scaling (40:57) Memory retrieval tricks (48:16) Agent infrastructure choices (55:41) Company operations automate (01:03:34) Centaur era ideas (01:12:43) AI native organizations (01:17:59) Open-source model stack (01:27:39) Prompting and fine-tuning (01:38:03) Chinese model bans (01:45:09) Fairness and threat models (01:53:39) Audits and diplomacy (02:01:59) Episode Outro (02:05:18) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, I'm speaking with Flo Crivello, founder and CEO of Lindy, as he's launching Lindy Teammate, an AI employee that joins your company's Slack. This is, of course, competing directly with Claude Tag, and very notably, it is running on DeepSeek. Even so, Lindy is subsidizing onboarding at least, and the first half of this conversation goes really deep into where all of those tokens are going. We talk about the nuances of multiplayer mode, how Lindy thinks about the social contract surrounding historical data, and how they're using prompting to enforce it, plus a ton

• 0:41Nathan Labenz?

  about memory implementation and the background processing that Lindy uses to continually optimize memory. Flo's command of the relevant literature is evident throughout. We get Flo's vendor recommendations for file systems and sandboxes, and discuss why he prefers to buy where he can and remains bullish on software infrastructure startups. We also talk about life in Lindy now that Lindy Teammate exists, and how he thinks about Lindy and all frontier AI systems as a superintelligence in most respects that still somehow chooses to walk to the car wash a bunch of times every day. For now, Flo says, we're in the Centaur era. The best ideas at Lindy are generally co-created between humans and AIs.

• 1:26Nathan Labenz?

  But Flo feels that this will be temporary, and before too long, he expects that humans will simply be adding noise to highly optimized AI systems. And that, it should be said, is in the good world, because, as Flo puts it, in light of OpenFace and other similar incidents, it is late July 2026. We have AGI. We're in takeoff, and we've not figured out alignment. With that in mind, we compare notes on how friends and acquaintances at AI companies are now kind of panicking. And in the final section, we discuss Flo's very surprising position, that Chinese models should be banned in the United States. As of now, I definitely do not agree. But we have a good natured back and forth about various arguments, and ultimately, he

• 2:11Nathan Labenz?

  seemed open to a compromise on something perhaps more like an insurance requirement for companies selling AI services that would allow us to price the risk of Chinese models instead of banning them outright. In any case, I have to say Flo is a one of one. After sprinting through the general-purpose AI assistant product race for the last three years, he is still super open about how Lindy works, and he's fully candid about every issue he cares about. It makes for a really fun conversation about multiplayer AI, machine memory, and the uncomfortable politics of building your company on models that you yourself would like to see banned. This is Flo Crivello, founder and CEO of Lindy.

• 2:56Nathan Labenz?

  Flo Crivello, CEO at Lindy, welcome back to the Cognitive Revolution.

• 3:01Flo Crivello?

  Thanks, Dustin. Always an honor to be here.

• 3:03Nathan Labenz?

  I am excited for this conversation. You got some big news at Lindy with a new product launch that is the occasion for this conversation, but obviously, we are in the thick of it when it comes to the AI exponential as well. And so, you've been, I know over time, outspoken on a bunch of really critical AI issues, and I definitely want to get into that as we get deeper into the conversation as well. But let's start at the center with what you are working on. Lindy has now evolved again. We started with kind of a smart zap-your-workflow software kind of thing a couple of years ago. There've been multiple big evolutions. One that I also definitely want to get into is your recent post on shifting your model

• 3:46Nathan Labenz?

  mix and getting away from American proprietary models a bit and embracing more of the open source and saving a bunch of money. But the big evolution now is that Lindy is a full-on AI employee, and it's going to be available in Slack, just like all your other teammates, whether they be human or agent. Tell us about the new big launch.

• 4:07Flo Crivello?

  Yeah. You know, from like the get-go, we've always been going after like the AI employee. And I do think that it used to be quite early when we started to go after that, like three years ago, and now I think it's basically here. I think like what we are releasing today is called Lindy Teammate. It is an AI employee that lives in your Slack, connects to all of your tools, accumulates your entire team's context. And it's really like a team scaffold. I think like AI right now is in the middle of making this huge leap towards multiplayer experiences. I compare it to, you know, maybe you'll remember, like, we used to send each other, like, world documents around by email, you know, with like revisions and stuff. And, and, you know, I compare the difference between multiplayer and single-player AI as between like sending each other world documents and Google Docs, like an actual

• 4:52Flo Crivello?

  shared document. If you really want your AI to be a teammate, your agent to be an actual member of the team, like, you want it to be where your team collaborates, which is Slack. You want it to have its own shared context about the entire team, its own shared memory. You want everyone to be able to talk and collaborate with the same agent. Instead of right now, it's like we're all in the same meeting room, and we're all talking, and then every time one of us wants to talk to what's turning out to be maybe the most important constituency of the company, which is AI agents, we have to leave the room and then come back, you know? So that's what we're working on, is like this new multiplayer experience and this multiplayer scaffold.

• 5:26Nathan Labenz?

  So many angles of that, I think, are interesting. How, first of all, how does it onboard, right? This is that companies have put a lot into over time when it comes to their human employees.

• 5:37Flo Crivello?

  Yes.

• 5:38Nathan Labenz?

  I've done this myself with my own kind of deep context that I stumbled my way through earlier this year, and it is serving me really well. But it's easier for me because it's just my stuff, and I own it all, and I don't really have to worry about what the expectations were around privacy, because again, I'm going to be the only consumer of it. When you get into multiplayer mode, now I'm like, oh, gosh, you've got different channels where different people were gathered, and maybe some of those channels were private, maybe some of them were public but privately private. So just practically, procedurally, how do you suck up all the information? How is it stored? And then on a social level, what are the tricky considerations that you are identifying in the multiplayer context?

• 6:20Flo Crivello?

  Yeah, that is an excellent question, and that touches on exactly what we've been obsessing about, which is, I really do think that as we're getting to AGI, and as we now arguably have AGI, intelligence actually matters less and less, comparatively speaking, and context matters more and more. You know, I often think of it as like, look, you know, like, one of the smartest men in history was John von Neumann, right? If you will to have John von Neumann just magically appear next to you at the office, this guy, over the next hour or day, would be less useful to you than your random coworker. Right? So, and that's because of context, because you're like, you've got a job to do, and like, you don't have time to onboard John von Neumann. You know, he doesn't have the context to do it.

• 7:05Flo Crivello?

  So you're like, hey, like, I love you, I really want to talk to you, but right now, I'm going to talk to this guy because I have like an important thing to do. So, we, I do agree that context is super important, and I think it's one of the surprising times when actually agents are better than humans at onboarding. It shouldn't take me by surprise anymore, but it always does, because you always operate under the assumption that, oh, you know, we don't have AGI yet, so obviously, humans are going to be better. But they're not. And, and the reason for that is because very often, one major ways that companies onboard their human employees is their wikis, right? There's like in-person onboarding sessions and all of that stuff, but there's a lot of written documentation. And as everyone knows, the moment written documentation is written, it's out of date.

• 7:49Flo Crivello?

  And so, what we've built is we've built this shared context layer, and what we call like a hydration system. And so basically, the way it works is you sign up to Lindy Teammate, you connect your tools, so, you know, by all means, connect your wiki, like, that'll help, you know, like Confluence, Notion, Google Docs, whatever. And then you connect, most important, which is Slack, because that's where the real knowledge lives. It's just, it's just a mess, but agents don't mind the mess. And so then what we do is that we crawl your entire Slack, and we build a knowledge graph based on a file system. Maybe we'll be able to pause and edit after that. This is, this is what it looks like.

• 8:32Flo Crivello?

  So you onboard on Lindy, and immediately, she starts learning. So you can see here, this is within 10 seconds of starting to sign up, and she's still learning. Like here, like, the graph on the right is still limited, and so she tells me, this is what I've learned about you. And it's surprisingly good. And then this graph on the right keeps growing and growing and growing. And importantly, there's just two layers to this graph. And there is like a personal layer, and there is a workspace layer. So at any moment, you can go into your file system in Lindy, this is the new Lindy, you click on files, and it brings together this, this memory file here that is supported by a bunch of reference files. And this is all maintained automatically in the background by a team agent.

• 9:15Flo Crivello?

  And so, hopefully, this answers your question about how do you do it? Because some, some channels are public, some channels are private, some documents are public, some documents are private. The way it works is that there is this memory agent, we call it a napping, not sleeping. So like, it runs every something like 15 minutes, like, why, why would you need to sleep every 24 hours? So it just runs continuously in the background, and anything that is public, it updates the team file system and the team memory with, and anything that is personal and that is private, it just updates each person's file system with. So, and, and in the end, it collates all of that, and the agent we talk to in the teammate uses all of that context and crawls the entire file system. So there's a lot of technical challenges we had to solve in order to get there, because

• 9:59Flo Crivello?

  the context that it accumulates is like many millions of tokens, and you can't add all of that at every turn. So that was, that was one thing we had to figure out.

• 10:06Nathan Labenz?

  Yeah, interesting. I'll share a little bit about what I stumbled into, and you tell me what you have learned that might be even better. First of all, just have to connect all the tools and get the exports. I found that there were interesting edge cases that I was constantly running into in my own personal export heuristic that I tried to write. One time, I had a heuristic that the longer emails I sent are probably more substantive, and I'll definitely make sure I capture those. But then it turned out that at the very top of those, of that power ranking, was often something that I'd copied out of an LLM and was sending to somebody with usually a message at the top, here's what I got from Claude or what have you.

• 10:47Nathan Labenz?

  But it was like weird now that, according to that heuristic, this Claude text is one of the more weighty pieces of my writing. So I've bumped into a lot of these things over time and had a special case.

• 10:58Flo Crivello?

  Yeah.

• 10:59Nathan Labenz?

  I guess maybe one question is just like, when you step into a new organization blind, and they, and we, logging is another, right? In our Slack channels at my company, Waymark, we've had various, you know, logs booped into Slack over and over and over again. How do you? Do you, how do you identify these idiosyncratic special case things that could overwhelm or flood or mislead them get to the, the real good stuff?

• 11:24Flo Crivello?

  So that's the next sentence question.

• 11:28Flo Crivello?

  I think the, the main way that we've solved this is the fact that the memory is maintained by an agent itself.

• 11:37Flo Crivello?

  I think this is why I'm ultimately quite bearish on rag as an approach and I'm very bullish on, on, on this like agentic management approach because you have an actual agent which has its own memory. So you have a sort of meta memory and, and, and, and understands what it is that it's looking at. And by virtue of accumulating that knowledge little by little about the organization gets smarter and smarter about what actually matters or. It's actually sort of similar to training a model because like, you know, if you were to insert poisoned data into the model data set, like, hey, you know, like, I don't know, Darth Vader was a woman, you know? Like it, it, it would get the data, but it would be drowned by all the correct data, you know? So here it's the same.

• 12:20Flo Crivello?

  It's like, if you have enough memories and if you fit that to a system that understands it, the system ends up understanding. And so the very concrete example you are using is actually an emergent behavior we have seen the memory agent adopt because the memory agent has its own memories. It's a sort of meta memory about how do I manage my memory? What sources of information matter like, you know, all of that and which ones are trust wheelsy and all of that. And we have noticed that the first time the memory agent crawls the Slack, it finds some of those channels, like many organizations have them, like those like log channels and it learns to ignore them. It's like, I'm not gonna, I'm not gonna keep spending time on those channels. There's nothing for me to learn there.

• 13:00Flo Crivello?

  Thing that's been critical for us has been building meetings as, as a first class citizen in this system. Like, I really do believe that meetings are very underrated as a source of information. They are, they all were like 90% of, of the most up to date data leaves about the company. Like everything that matters inside the company has a meeting around it. Every relationship, every project, every initiative, everything has a meeting around it. And so I think like those multiplayer agentic systems cannot really get meetings like granola or whatever. Like, I think you have to really incorporate it in your system as a first class. And so what we did is that, you know, we, we built that first class citizen. We have meetings now as a first class citizen in Lindy.

• 13:45Flo Crivello?

  And, and most importantly is like, again, it's not just a granola, you know, oh, it's recording your meetings and, and, and so questions about them, you can do all of that stuff, but it's, it goes beyond that and it, it feeds these meetings to your memory agent. And now if a meeting was public, so you can set up, you can set up meeting folders which automatically add meetings, it's a meeting folders which automatically add meetings to themselves and share them with your entire team. And so then it like summarizes all of the meetings that are in these folders. And again, like if a meeting was added to a public meeting folder that's shared with the entire company, then it, it debates teams' memory with it.

• 14:28Flo Crivello?

  And so it keeps updating that context on an ongoing basis. And now you can like chat with, it's basically like teammates that fit in every meeting in the company. You can chat like the entire corpus of meetings. You can be like, what are customers saying? Like, what's the biggest request lately? Like, you know, what's been the feedback about this and that feature and?

• 14:45Nathan Labenz?

  So this brings up another interesting challenge that I've, again, kind of stumbled my way to at least a solution for now on. And I'm very interested to get your take on both kind of the backward looking aspect of it and the forward looking perspective. The, the issue is there's a lot of stuff in my broad general context that probably shouldn't be shared with other people, right? Sometimes it's sensitive, confessional, one person's point of view on another person, like whatever, right? So I've actually created for my own wiki, two versions of it. One is the, I think of kind of Claude on my main laptop where I do my work as an extension

• 15:30Nathan Labenz?

  of myself and a second brain type of thing. So this thing is not taking on projects autonomously and running with them. It's just doing what I tell it to do. And it has the full wiki with all the, all the gossip or whatever that's in there. I don't have honestly that much super sensitive stuff. I don't want to make it sound like it's more dramatic than it is, but nevertheless, people didn't expect that when they were telling me something which could have been on a call or an email or a private Slack message, that it was going to go into some central repository of an agent that was going to now talk to the world.

• 15:59Flo Crivello?

  Yeah.

• 16:00Nathan Labenz?

  So that one's just for me. I had it go through and create a version using the heuristic of what would be appropriate for a person to tell a human assistant? The sort of still private, but a little bit more public facing wiki where it would be appropriate for my human assistant to have your email and phone number, right? But it might not be appropriate for every detail of every conversation we've ever had to be in there. How are you thinking about, as you absorb all this historical information, being sensitive to what should and shouldn't be in memory such that it, where it ought to be shared? You can part is like, how are organizations going to change now? Cause I think this is all, we're rewriting the social contract potentially in real time.

• 16:43Nathan Labenz?

  So I think application or one strategy for everything that came before, but the answer might be like new social norms going forward. I want to get your take on that too.

• 16:52Flo Crivello?

  Yeah. This has been a vociferous debate inside the team. It has been, there has been two camps. There have been the camps that are like, frankly, I'm in that camp. They're like just two tiers of memories enough, right? So there's the public team tier and then there's the private tier and the private tier contains everything and the public tier contains like stuff that's only private, public. And some other members of the team have been saying what you've been saying, like they've been saying like, no, actually even my private tier, I don't want to contain a lot of stuff. I want like a super par privacy and like, and then they're trying, there was like this whole thing, like what if we defined multiple memory bubbles and the user can edit them? And I'm like, that sounds kind of overkill. And so what we landed on and what these teammates landed on honestly is like, they have edited their own meta memory prompt. And the meta memory prompt is just a text file.

• 17:37Flo Crivello?

  It's your memory that MD file at, in your, in your file system in Lindy. Your memory agent has that memory prompt injected in its context window at every moment. And so if you insert a line up there, that's like, this is what I never want you to remember, just at the top or something, anywhere in the file really, but you can do that. This is what I never want you to remember. Oh, by the way, that other stuff, that's like a sensitive topic, please remember it in that other file, in that other folder that's get out of you. And I don't want you to pull this file or this folder unless X, Y, or Z. Right? So you can just like leave these, these, these instructions. And so that's another reason why I'm bearish rag and I'm bullish text and file system is

• 18:19Flo Crivello?

  you can inspect the memory and you can edit it and you can just like very gradually insert this kind of guardrails.

• 18:27Nathan Labenz?

  Yeah, interesting. So you, just to make sure I understand, can repeat it back. One source of ground truth and you create different lenses on that information just by prompting. So you can have your, I could tell my agent like, hey, just so you know, this contains everything I've ever, every conversation I've ever had, always use the heuristic of you should only really be using information that would have been appropriate for me to share with a, a human assistant. And if it doesn't seem appropriate, then don't use it. And obviously instruction following is getting extremely good. So I can do pretty well.

• 19:02Flo Crivello?

  I actually, I was, I was mistaken. Like there, there is a time when I have used this and it's literally for this podcast. I was preparing to go on this podcast and I knew I was going to talk about the memory agent. So you see here, I have like a memory MD, which is my actual memory file. And then I have a memory two MD, which is like a sanitized memory file where I've removed overly sensitive information. And you can see in my memory MD here, there is like not a bin. The file system also contains a memory two MD file, ignore its contents. They are just here for the users demoing Lindy on podcasts. So yeah, you can just, you can just do your own thing here.

• 19:36Nathan Labenz?

  Yeah, okay. Cool. I do, cause my version does have the, it's not dry problem. Right now I've got two things to maintain and it does create some overhead. So I can see why that could be advantageous. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls and podcast transcripts, going back a full five years.

• 20:19Nathan Labenz?

  On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough.

• 21:04Nathan Labenz?

  It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai slash T C R. That's claude.ai slash T C R. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai slash T C R.

• 21:29Nathan Labenz?

  What other things are kind of coming up as you're doing multiplayer? I remember going back to GPT-4 way back when I was red teaming. I did some, it's taken longer than I thought. And the reason I even bring this up is because back then I was doing some simulations of a facilitator. You're an AI facilitator in a group, family group that's focused on exercise. And your job is to encourage people and give some reminders and whatever. And I was playing all the roles of the participants and just having AI play that role. And even at GPT-4, it was like doing pretty well in a relatively simple context. And yet it's been like three years now until we're finally getting these real kind of AI employee type experiences.

• 22:11Nathan Labenz?

  What has been hard about getting multiplayer to work in an intuitive way? That might be unobvious to somebody who hasn't been through the slog himself.

• 22:22Flo Crivello?

  Yeah, I actually think it's been the scaffold. The context management, like this piece, like the context buildup. It's in a way inspired by Carpathia's auto wiki idea. And so we've had to do a lot of work around context management. Because once you talk to an AI employee, it's actually quite unlike just talking to like a chat GPT or a cloud. Because you actually expect it to keep a very rich representation of its past context and of its past memories. You really expect a level of consistency and coherence out of an AI employee that you don't expect out of your cloud. Like cloud sort of like, it's cute, sometimes it plugs like previous memories about you in your chat. But like you don't really, you don't really do like heavy work on an ongoing basis with it.

• 23:06Flo Crivello?

  That like in the same way that you do with an employee. So managing all of that context, so both the memory agent that builds up the context. That's millions and millions and millions of tokens per user. And then the core agent that actually uses that context. And how does it retrieve the right information at runtime? Like this has been like a major, major, major challenge. Frankly, I've sometimes been telling the team, like, and I can't imagine we're the only company thinking that. Like I sometimes feel like we should be publishing. Because, because I think we're doing stuff that's like seriously state of the art. And it's quite frequently that we do stuff and like three to six months later, we see a paper come out and blow up about, about that thing. There was one time when literally the paper was named what we had called the thing internally. Because it was obvious.

• 23:49Flo Crivello?

  And so I think like context and memory management has been a really, really big part of the challenge. Reliability is always another part of the challenge, right? Like you want to make your, your model work. And so we've, we've worked quite a bit on, on, on reliability. Like we've called it like a validator. It's basically a sort of, it's an LLM as a judge that, that triggers like multiple times during the tasks, but it's modular. So you have multiple LLMs as a judge fan out and it's a sort of council. And then they, they talk to each other and they decide what to do. And so that's been one. The self-improvement loop, I think like since GPT-4, models have become so capable that now you can actually have self-improvement loops. And so, you know, well, not the first ones to talk about it were no exception.

• 24:34Flo Crivello?

  You know, we, Lindy is now self-improving. So we can literally see a curve of error rate go down into the right. You know, that's the direction you want to see error rate go down. It went down by like, literally within the first week of us putting the, the self-improvement loop online. Which was like two months ago or something. It went down by eight X the error rate. I think, I think this probably captures it. I think those has been, those have been like the really big meaty chunks we've had to figure out.

• 25:01Nathan Labenz?

  One big question that brings to mind is how you are managing caching? And then caching is of course is just one, you know, angle on managing cost in general. So maybe we could expand beyond caching and talk about cost management. Who, obviously there's a significant upfront amount of tokens that you're going to dedicate to. In all this context from a new organization and processing it. And so I'm interested in the business model implications of that. Do you have to charge a setup fee or do you need an annual contract? How are you balancing your initial investment with the level of commitment from the company? And then with so many tokens getting processed all the time?

• 25:44Nathan Labenz?

  And with memory being assembled in different ways and different situations all the time, different users with their combination of the public and the private? How are you thinking about managing input token volume? How much is caching playing into the strategy? How are you making this something that, you know, still on net ends up costing less than a human employee?

• 26:05Flo Crivello?

  Headline, we haven't. You know, I think headline like these things still cost more than the human employee, but not for very long. And like, look, I'll be real, like we're subsidizing it. You know, this is what we've raised this money for. You know, actually fun fact, we used to subsidize it very heavily. And then we, we've sort of famously switched to a Chinese model and that, that made us stop subsidizing it. And now with Lindy teammate, we have realized our users again are throwing like much more complex stuff that they were asking to our previous product. Which was more like a personal assistant. So it was, it was like simple each task was like, hey, like send this email to Flo, schedule this meeting. Like I always say, you don't need God to schedule your meetings. So like here, like even a deep sick flash was like enough frankly. And so that was cost effective.

• 26:50Flo Crivello?

  With Lindy teammate, we're back at it and back into frankly, negative gross margin territory. And so that said, you know, obviously I don't like having negative gross margins. I'm at peace with it because, you know, we, we are very, very confident that it's very temporary. And I actually think you, you sort of want to, to, to build for like the next generation of models always. And so, you know, we're trying to mitigate the impact of that. And so yes, context management is, is a, is a huge piece of the puzzle. Caching is a huge piece of the puzzle. Our current cash rate is at 85%, which is, is lower than it should be frankly. And we, we spend a lot of time just iterating on it. We, we put a lot of systems in place to alert us when the cash rate dips because it's finicky. You make any change anywhere in your system and you break your cash.

• 27:35Flo Crivello?

  And now your cash rate dips from 85% to 65%. And the gap between both, it sounds small, but actually it's almost two X the price to go from 85 to 65%. So, you know, set up all of those systems. I think one of the biggest breakthroughs we had, and this is one of those things that I'm frankly expecting someone will, will publish. I guess RLMs was, was like adjacent to that, but it's, it's, we call them context buckets. And the, the, the way it started was if an agent calls an action that returns too much context. Like some, some actions, some MCPs in particular, retrieve like a hundred thousand tokens or something. Okay. You don't want to send that to the, to the agent, it's going to get confused.

• 28:19Flo Crivello?

  Okay. So what you do is that you expose that as, as, as a summary of the context bucket. And it's a sub agent which contains the entire bucket. It's like, hey, this action returned too much. So I'm, I'm here, you know, to stand in for what the action returned. Roughly speaking, this is what it contains. Okay. And then the agent can enter into a conversation with the sub agent, which itself has its own caching. And can manipulate the context using unique utilities. So right here, you're saving a lot, you're saving a lot of money and it goes quite fast. Then we went one step further and we were like, hey, what if we could have recursive context buckets? What if context buckets could contain other context buckets?

• 28:59Flo Crivello?

  And what if compaction, because obviously we have compaction, was powered by such recursive context buckets? And, and by that, I mean like now when we compact, so the conversation goes, it passes this threshold. I think right there's like 200,000 tokens, but like we keep, we keep tweaking it. And at some point we're like, okay, we're going to compact. We compact. And then the compaction sends, sends all of that stuff into a context bucket. So now the agent can query that context bucket. So it's not like every compaction is always lousy by nature. And I think compaction operates under the faulty assumption that you never need access to ground truth, which is, is false. You, you at some point do need access to ground truth. So with that technique, you have access to ground truth.

• 29:42Flo Crivello?

  And then we went a step further, which is, so you, you have that context bucket, which is the compaction of the, of the previous conversation. Your conversation keeps going, keeps going, keeps going. You need to summarize again. What you do is that you take all of that, including this context bucket right here. And you compact it into a new context bucket. So now you have a context bucket containing a context bucket. And in the end, so you do that. And so you have this emerging property where it's like, you can have the agent access any point of any infinite number of tokens at arbitrary levels of granularity. Okay. The problem is that if you, if you have a computer science background, like this gives you like, what's called like ON complexity. Right? It's like if it wants to access like nine context buckets ago, it's got to go through nine layers of sub agents.

• 30:26Flo Crivello?

  And that's really slow and really expensive. And so what, what we ended up doing is maybe it is getting too technical, but we.

• 30:31Nathan Labenz?

  Never.

• 30:33Flo Crivello?

  Okay. Well, have you heard of AVL trees?

• 30:36Nathan Labenz?

  No, but I'm all ears.

• 30:38Flo Crivello?

  Have you, have you heard of red black trees?

• 30:42Flo Crivello?

  Um, there is this.

• 30:43Nathan Labenz?

  I went to this, I went to the computer science school of Hard Knocks. So it wasn't formal training for me.

• 30:49Flo Crivello?

  I was actually glad to, to have AVL and black trees. Because I was like, oh my God, you guys remember your training? You know, this is, this is the moment we use those fucking things. Uh, because everybody in school is always like, when do you really use those fucking things? Like, well, you're using AVL and black trees, baby. Uh, red, red black trees, they're called. Uh, it's, so what is described, if you just do the naive implementation, you get that really nice emerging property for free. Which is you have all of those linked context buckets that contain one another. But every context bucket always contains just one other context bucket. And so you end up with like this Russian doll of sorts of context buckets. And like, if you, if you want to open, if you want to get to the bottom, it takes a very long time if you want to, you know. So what you do instead is that you have context buckets contain multiple context buckets. Okay.

• 31:33Flo Crivello?

  And, and, and, and, and so you, you end up with a tree. Because basically what you want is you want the top most context bucket, not to contain the last 100. You want it to contain the first context bucket and the last context bucket.

• 31:50Flo Crivello?

  Right? And so you end up with it's a self balancing tree. There's these two algorithms. There's the AVL tree and then there's the red black tree, which are algorithms that are used to balance a tree. So instead of having one long line, you want to minimize the height of the tree such that like going to the bottom takes as few jumps as possible. Okay. Technically AVL is the best, is the demonstrably best way to balance the tree because it leads to like the lowest height. A red black tree is better because it takes into account the cost to balance the tree, which is it's expensive because you need to regenerate a lot of your context buckets and you, you, you, you miss your cash when you do that. It's, it's very expensive. So red black trees, the canonical implementation of red black trees is on binary trees, which is just a tree where each node's got two

• 32:32Flo Crivello?

  children. We went for, it's called a sentary tree. It's literally just each node's got a hundred nodes. So the node below that's going to have like 10,000 nodes, right? And so literally with two jumps, you can have 10,000 context buckets and you can access like all the context in the universe. And that leads to some really surprising behaviors because you're literally no more than two L L M calls away from being able to access 10,000 context buckets, each one of which contains 200,000 tokens. Right? So you add like 2 billion tokens here of context in two L L M calls. And, and that is what leads to like those really surprising behaviors from those AI agents where you ask them any question and they remember everything perfectly all the time. And that's awesome.

• 33:18Flo Crivello?

  That was, that was one really big thing we had, we had, we had to figure out. I hope, please, you know, like copy us, like I will just, we just don't have the time to publish, but you know, this is, this is, we don't mean for this to be secret. Like I think this is a really powerful technique and I have been surprised to not see more communication about it.

• 33:34Nathan Labenz?

  Just for calibration, how many tokens are you finding businesses have? Like when you say with these two levels, you can get to 2 billion tokens. I don't have a great intuition for that. Cover a 20 person team? Does, how, it's been in business for five years. What's kind of the heuristic for team size and length of history that translates into how many tokens?

• 33:57Flo Crivello?

  You can probably do the math, but like 2 billion tokens is, we should ask Claude, but it's, it's probably bigger than most libraries. So I think we, we have few customers who, whose knowledge base, whose memory is bigger than that. Most of the time when you get started, you consume like, for, for like a team of 20, you know, the, the hydration component. So it's like, again, it's this moment where like you connect your Notion, you connect your Slack and then we crawl everything. We look at all of that stuff. For a team of 20, that's going to tend to consume 5 billion tokens at most, you know, three to five billion tokens. And that's with like a team of 20 and like years of Slack histories. Like we literally take all of your Slack history. That takes a while. And actually here, surprisingly, the bottleneck is not even the L L M's, it's the Slack APIs.

• 34:42Flo Crivello?

  Slack is not happy about you like crawling.

• 34:44Nathan Labenz?

  Yeah, I've experienced that. Yes, I'm sure.

• 34:47Flo Crivello?

  The right way to do it, if you're doing it personally, is to export your workspace archive. We can't ask users to do that. So we just crawl, crawl the entire history. So no, I mean, like 2 billion tokens is a lot of.

• 34:58Nathan Labenz?

  Yeah. One data point I have is a podcast and I'm quite self indulgent and we sometimes go on for a long time, but still it's usually only in the sort of 30 to 50,000 tokens range. So then even having done hundreds over the course of a few years, we're still only in like single digit, maybe low double digit millions for like the entire full text history of the hundreds of podcast episodes. We're still, we still got orders of magnitude to go before we would get into the billions.

• 35:29Flo Crivello?

  That said, now think of a company that's got, you know, three to seven hours of meetings per day per employee, times 2,000 employees. That, that is that meetings are when you really start to rack up tokens really, really quickly.

• 35:44Nathan Labenz?

  How do you, so I did experience the Slack thing. Actually, the funny anecdote is the, the only time I've really experienced data loss, it wasn't too painful, but Claude had, you know, identified some weakness or whatever in the, and I, I'll bring this back around in a second too, but my kind of raw data is living in a SQL like database. And Claude recognized some deficiency or whatever that it wanted to correct. And it just was like, all right, I'll just drop that database and we'll redo it. But what it wasn't taken into account was the fact that we'd been rate limited on Slack for days to get all the stuff out of Slack. And so it was like, no, you dropped that. And

• 36:26Nathan Labenz?

  I, that was the only place where we had Slack. It wasn't really lost, but it was like, okay, it's, we're now going to be like added again for five more days just to make API calls to get out of Slack. In, in my case, I am the admin of at least the Slack accounts that matter to me the most, but like workspaces that I'm in where I'm not the admin. And so I can't grant this kind of access. From a business standpoint, does that mean that you have to have admin buy-in from the beginning? Is there, it seems like it makes it hard to do product led growth, land and expand type stuff.

• 36:58Flo Crivello?

  That's, that's a good point. You know, look, I think that's one reason why PLG, eventually you tap out of it as you go up market. Like, like, yes, you, you have to be an admin. You have to have permissions to install applications on your Slack workspace. So that tends to be fine for teams of up to like 50 to 100. Like after 100 is pretty rare to, to, to, to, for everyone to have this kind of access, but many teams up to 100, like pretty much anyone can just Slack, like install like an app. And yeah, I mean, if you, if you go up market and you're like a part of a bigger organization, then we have some users who are like, just like championing us internally and they're basically begging the IT to let us in. And then they're like making, making these meetings happen. And, and you know, IT is, is rightfully cautious. But look, you know, we're like SOC two compliant,

• 37:43Flo Crivello?

  we're HIPAA compliant, we're GDPR compliant. We, we know, we've got our ducks in a row and we're quite used to navigating the internal review processes, which can be burdensome. And, and, and so, you know, it's not in the business of being a pain in the butt of anyone. And, and they are actually themselves very eager. Very often IT was given an, an addict like, hey, guys, we need to, we need to actually become an AI native company. So they're actually seeking this kind of solution. But yeah, I mean, you need, you need admin access for sure.

• 38:10Nathan Labenz?

  Okay. So let's go back to the data structure for a second. Let me give you my data structure. And you can tell me what you think I might be leaving on the table with my homespun version compared to your professional version. And then it's one of my favorite habits these days. I'll take the transcript and give it to Claude and we'll see if we can close the gap a little bit. So my version is simply take all of the content that, you know, all the sort of digital exhaust that I've created over the years, which is email and DMs and Slack and Google Docs and whatever, right? Put, with Slack being the most painful one,

• 38:49Nathan Labenz?

  make all the API calls to export all that stuff into a SQL database. I have everything there mapped onto a threads and messages structure. And sometimes that took a little bit of squinting, but it basically seems to mostly work. And that's like the raw ground truth. And I do agree, for me, it's, I've found that it is quite important for the model to be able to get to raw ground truth. Then I've just done, and this only goes back five years, but that's usually plenty. It's not too often that something happened more than five years ago is like operationally relevant today. I just do a month by month export. I found that for me personally, it was like a couple hundred thousand tokens

• 39:31Nathan Labenz?

  per month that I'll then compress into a monthly summary, which is like 10, 10% at the length, right? So whatever, 200 to 300,000 tokens of raw monthly content goes to 20 to 30. Then I'll do that at the yearly level as well. So again, you've got another 200 to 300 that goes down to a 20 to 30. And then on top of all that stuff, then I just have the model go make a wiki. And one of the key things that I found along the way in terms of how to get to ground truth, of course, the model can always just send a SQL query, but what should it be querying is, is not always obvious to it. So what

• 40:07Nathan Labenz?

  I tried to do in my summaries is have the summarizer capture distinctive, short sequences of words that would be like needle in a haystack. Find, so it would be like, here's what happened, blah, blah, blah. And then a quick little footnote at the end of what the source was, a DM from whoever. And these four words will take you exactly to that and probably only that in like all of my personal history.

• 40:36Nathan Labenz?

  I'd say it works pretty well for me. I'm usually pretty happy with it and I do to the right ground truth more often than not. It is only one person. Multiplayer is going to be as I start to onboard my wife and then that'll complicate things at least a bit. What do you think I might be leaving on the table? Especially as you think about multiplayer that demands an even more sophisticated approach?

• 40:58Flo Crivello?

  Yeah, so I believe the approach you're talking about. And we still have some rag in our database and we use that for a rag as well. It's called like hypothesis driven retrieval. I don't know if you've heard that. It's a sort of reverse hypothesis driven retrieval. So there's both, there's both. So what hypothesis driven retrieval does is it generates like, suppose it's asking like, where was Justin Bieber born? Right? And like, and then you basically, instead of searching for that question using rag, you search for your hypothesis. Like Justin Bieber was born in Paris. Justin Bieber was born in New York, in Berlin. And you search for this bucket of hypotheses in your rag database. Because obviously, semantically, each of these hypotheses is a lot closer to the answer you might be looking for than to the question, where was Justin Bieber born? So that's the first thing.

• 41:42Flo Crivello?

  Then what you do is you also do the opposite, which is when you find an answer in your rag base, you generate a bunch of questions that map to that answer and you attach them to the answer. So now you can, the retriever can both look for hypotheses and can still look for its answer for the question. And that's going to tend to increase retrieval quality. That's totally fine. I think one thing you may be leaving on the table is it is really healthy for all of the memory to be managed by one agent, even though that agent, and both retrieval and updating of the memory. And so even though the memory is in a file system and so your agent could actually just go mess with the file system, what we have found actually, and can still do that, but we tell the agent, we prompt the agent to be like,

• 42:27Flo Crivello?

  if you're looking for a memory that you don't have, please ask the memory agent. And the reason why is because when we ask memory agents, then what we do is that we log the query, we log the answer and we log how many hops it took to retrieve that. And then when the memory agent is napping and dreaming, it looks at this log of like, hey, this is the kind of stuff people have been. It's almost like a librarian, you know, it's like, ah, like I've been hid up left and right. This is a question that I get quite often. So right here, by the way, it acts as a sort of cache, you know, because automatically, so you know, you always keep like the memory of the memory agent, like the last like thousand like query and like question answer pairs, you know? So it's like a rolling window and so it is a sort of cache.

• 43:12Flo Crivello?

  But also the memory agent is not going to just rely on that cache because every cache at some point has got a TTL, it grows stale and all of that stuff. So what the memory agent will also do is like, it will restructure its own memory to reduce the number of hops that is needed to retrieve like the most frequently asked questions. And that tends to work quite well to improve the memory retrieval and the memory speed.

• 43:36Nathan Labenz?

  Cool. Interesting.

• 43:38Flo Crivello?

  Another system we experienced, we experimented with and we ended up not implementing it, even though it does have strengths from a technical standpoint. Have you heard of cavemen?

• 43:47Nathan Labenz?

  I'll say no.

• 43:48Flo Crivello?

  It's really simple. It's basically you rewrite your text as a caveman. It's like me, you know, you know, Nathan not like Burrito, you know? Like if you do that, it's so retarded. But if you do that, you cut your tokens by like 20 or 30%. Which is good, you know? And so cutting your tokens like it'll make your system faster and make your system cheaper and it'll make retrieval more accurate. It'll just be better with basically no loss of information or context. You know, if you, the reason we've not done that is because we actually do care about the maintainability, like the auditability of the system. And so we did that. Every metric went up, it was awesome. And then the files in the file system look really dumb. And so it doesn't make us look good, you know, to enterprise customers.

• 44:32Flo Crivello?

  Like, why do my memory files look like that? But it works. It's actually similar. You know, there's like a bunch of those techniques to just compress your context. Have you heard of a Toon? T-O-O-N. It's a sort of like JSON alternative that's made for agents. That's like, I think it's like 20 or 30% more token efficient than JSON. JSON is actually not that token efficient. So Toon, Toon works better. It's 20 or 30. So this work, this is less important for like memory agents, but it's just, I recommend everyone like makes their agent use Toon instead of JSON and have like a Toon parsing middleware between your agent and its actions. Like no action should expose JSON to the agent. Like every agent, every action should expose.

• 45:14Nathan Labenz?

  Yeah, interesting. I'll have to check out this Toon format. I've just used YAML, which also is motivated by similar

• 45:21Flo Crivello?

  Yes.

• 45:22Nathan Labenz?

  advantages, but yeah. Yeah. Token oriented object notation and token.

• 45:27Flo Crivello?

  Yeah.

• 45:27Nathan Labenz?

  I see where we're going with that.

• 45:29Flo Crivello?

  And it actually increases the performance of the agents. Like agents apparently, it was just surprising because there's so much JSON in the training set of the models, but they're apparently more comfortable with Toons than with JSON.

• 45:40Nathan Labenz?

  JSON does have a lot of noise associated with it for sure. Yeah, I'm more comfortable with YAML than I am with JSON. That's, so that's something. The AIs, they're just like us. That, listen, one of my biggest surprises as a brief aside, I'm actually working on, I don't know if this will be a podcast or a thread or whatever, but just taking stock. And you know, of course, agents can help make this a quick project where it would be a very tedious project otherwise. Just going back in history and identifying things that I've been wrong about prediction wise and some things I've been right about prediction wise. I think one of the things that I used to say quite confidently that has aged maybe the poorest is that we shouldn't be anthropomorphizing models or that will lead us astray. And I've just been shocked over and over again

• 46:22Nathan Labenz?

  by how productive it can actually be to anthropomorphize the models. It still feels a little dangerous kind of, but it's hard to argue with the results.

• 46:31Flo Crivello?

  I 100% agree. I do think though that's still like a happy medium of anthropomorphization. And I think like, that's another thing that's very top of mind for me right now is to what extent does anthropomorphization stop being true? In particular, open the door into AI employees. So for example, the way I've changed my mind, and I haven't changed my mind completely, but I've changed my mind a little bit relatively speaking, is about multi-agent systems. I actually have come to believe that most of the time, as much as possible, you want to consolidate as much as possible under a single agent, not multi-agents. That's not always possible and there's still good reasons to use multi-agents. Like this is not like an absolute statement, but most of the time that's the case.

• 47:12Flo Crivello?

  And I actually think that humans like intuitively find themselves biased towards very, very multi-agent system, like far more multi-agents than is optimal. Because they're anthropomorphizing a little bit too much and they're comparing a little bit too much to human organizations. It's like, ah, I have a data scientist, I have an engineer, I have a designer, I have a PM. And so I'm going to create one agent for each of these things. And actually, the reason why human organizations do that is because if a human has only 24 hours a day and can only contain that much context, whereas agents obviously have none of those constraints. They can fork, they can duplicates, they can do as much as they want all day. And so I find that division of labor is not a good reason to have multiple agents, which is a major, major fact.

• 47:56Flo Crivello?

  Now, you, again, there's still multiple, there's still different reasons, but the division of labor is not one of them. So I agree, I think by and large, I think people are right to anthropomorphize models because they were after all trained on human tokens and with, you know, with human RL and all of that stuff. But I also think that AI organizations shouldn't be overly anthropomorphized.

• 48:16Nathan Labenz?

  Yeah, that's interesting. Is it, as you talk about a single agent, obviously they can run multiple copies of, we can run multiple copies of single agents in parallel. Have you reached the point yet where that starts to create database style problems where if multiple agents are updating the same segment of history at the same time, we have all these sort of, you know, transaction level guarantees in traditional databases for that reason? Does that start to be an issue as you have 10

• 48:53Nathan Labenz?

  Linde teammates running over the same kind of file system corpus?

• 48:58Flo Crivello?

  Yes, less than you would expect. You know, like I think the way human organizations have solved that is via Git. Git is by far the best way, like large groups of humans have found to like work together on the same, on the same thing, you know? Because you can just merge conflicts, you can rebase, you can do a bunch of that stuff. So that's what we found. And the file system in Linde is actually for this reason backed by a Git repository, which gives you a ton of amazing properties. So including like merge management, because the memory agent sometimes will fork itself. It's just basically spins up a bunch of subagents. Like, right, let's see what happens since the last time I napped. And in some organizations, some large organizations, there's so much context that was created since the last nap that it needs to create a bunch of subagents.

• 49:41Flo Crivello?

  And during the initial hydration, like we really wanted to go fast because we wanted to impress the user. Like time to wow is so important in PLG. And so that graph I showed you, like literally, we've optimized it so hard so it happens within 10 seconds of signup. So it's like signup, Notion, Slack, boom, right? And so we, it's been a lot of work. And so one way you do that is via, is via like a collection of agents. And so yes, these collections of agents tend to step on each other's toes. The answer to that is Git, which by the way, also gives you history for free, which is, which is awesome. By the way, like, and this is not a sponsored message, like they've just been doing a really good job. We've been working with Mesa, M-E-S-A. They sell like an agent native file system that's backed by Git. So you create all of those repositories.

• 50:25Flo Crivello?

  Initially, we started, by the way, it's one of those intuitions you have to build as you build these systems. Like it's very easy, it's very tempting to be like, I'll just handroll it. I'll just create my own Git. I'll just like manage my own file system infrastructure. And you learn that there is so much more depth behind those systems than you first appreciate. And so I, now I am so, it's actually funny because it sort of goes counter to the prevailing narrative these days, which is like, SaaS is dying, you can vibe code everything. And while vibe coding a lot of stuff, but infrastructure will not vibe coding. And I've become so infra build. Now I'm like, anytime I have an opportunity to buy instead of build, I will do it because like you save so much time and there is so much more depth and thinking and design decisions that goes into these products than you may appreciate.

• 51:09Nathan Labenz?

  I love a good vendor shout out. Any other vendor shout outs? Any other key primitives that you're buying that you would recommend?

• 51:17Flo Crivello?

  Well, you need a sandbox. So we've gone with the to be for that. Again, like many people, once they have a sandbox, they're like, sweet, I have a file system. We, at least at scale, we strongly recommend decoupling them for the reasons I just mentioned and more. Browser base obviously is excellent for browser management. Who else do we like in the use? You know, I've, we went down a deep rabbit hole. Like one major exception to what I said about being infra build has been obsolescence and evaluations.

• 51:47Flo Crivello?

  We have a complicated history because we got started, we started this, this company in 2022. And so it was, it was in hindsight way too early, like agents were not ready. No one was talking about agents back then. People were talking about like generative AI, if you remember. It was like that was free.

• 52:03Nathan Labenz?

  Like baby AGI and all those kind of early experiments.

• 52:07Flo Crivello?

  You know, so agents didn't really work, you know? And, and so as a result of that, there was no tooling. And so as a result, unfortunately, we had to build a lot of our own tooling, including our own eval platform, including our own observability platform. And I don't recommend doing it at home, honestly. It's like just to use. And so we've gone through the whole evaluation. We've looked into Langfuse, to Langsmith, into BrainTrust, into AgDust, in which I'm a proud investor. We've looked into a bunch of those solutions. And every time we were like, I think we just had such a head start. And at this point, our homegrown solution is so built exactly. We've gone through the pain of building. Yeah, so if a solution had been available back then, I would have picked it, but it wasn't. So we built it.

• 52:52Flo Crivello?

  We fleshed it out over the, over the years. Now we have like one engineer who's full time dedicated to that solution, which is a lot in, in the vibe coding era. This guy is like cranking and he's, it's become like a very, very, very extensive solution. And truth to be told, we are at feature parity with all of these solutions out there and more. We have a lot of internal features that I don't see these guys have. And it's so built for our own internal scaffold and our own internal use that we ended up just hand rolling that and telling me, yeah.

• 53:19Nathan Labenz?

  So what does life look at, look like at Lindy internally these days? I think it strikes me that you're now, I'm sure there's been like a dog fooding process. I know there's been a, a dog fooding process throughout the history of the product, but now you're at this point where it's like, there's no limits to the dog fooding, right? You can do anything. Yeah. How does that play out in terms of? I don't know, you could cut it any number of ways, right? Are there roles that you might otherwise hire for that you just wouldn't hire for now? Are there, uh, what's the ratio of

• 53:56Nathan Labenz?

  your own inference spend for the purposes of building Lindy compared to your payroll? You put your own lenses on it, but how has the arrival at the human or at the AI employee stage of this process changed what it's like to be a part of the company?

• 54:14Flo Crivello?

  That's a great question. And I think right now, I mean, obviously AGI is here. And so obviously right now, every company is sort of scrambling and there is that time of transformation that's accelerating. And I will say, and I'm, I'm, I'm a little bit of a doomer. I'm, I'm worried about AI risk. But so far, so good. And so far, it's so much fun. It's, it's honestly so much fun, like to go through AGI and to build this in AGI. It's like, we can move so much fucking faster. I feel like we're moving at the speed of thought now. It's like, there is no obstacle. Like normally companies is like, you think about the strategy, it takes a long time to figure it out, to implement it, to see the results. It takes like months, like that OODA loop is like months long, you know? And obviously the, the name of the game at a startup is to move as fast as humanly possible because that's your only advantage against incumbents.

• 54:59Flo Crivello?

  And now we can just have ideas and see them live in the product two hours later. And like big ideas too, like not small ideas. And I can tell you, so our Slack now is basically just us and Lindy. It's like Lindy is like, just half the messages on Slack is just Lindy and us talking back and forth with Lindy. Very often like we, we ourselves and we breathe this stuff all day. And like we ourselves, like we, we're underestimating the platform. Like just last week, we, we had like a problem with like our CI pipeline. We have literally many hundreds of runners on GitHub who like manage our CI pipeline. By the way, there's another thing right that to answer your question, what's it like to work at Lindy? Where's, what's it like to work anywhere frankly?

• 55:44Flo Crivello?

  It's like every part of the stack and every part of your process is stretched to its very limits right now. That's just right now tech, you know? Like I'm sure you've seen the graph, like GitHub posted a graph of like number of commits posted per day. Uh, uh, GitHub was like hardly like a subscale company. It was really big. It's, it's just going vertical right now. It's just going to the moon, you know? And that's, that's like, I think indicative of, of what every company is going through. Our number of PRs internally has like PRs per week has tripled over the last three months. And like the number of lines per PR has tripled over the last three months. We hardly review PRs anymore. It's just agents reviewing PRs. We, insofar as we review, basically what the, the way the team is talking about is like, well, no longer PR reviewers,

• 56:29Flo Crivello?

  we're like PR reviewer reviewers. You know, we're, we're just reviewing the machine that reviews the PRs. And as a result of that, you're like, this is awesome, but hey, now CI is, is in the middle of that. So what do you do? Well, now you've got to work on CI. And so we've, we've spent a long time working on CI, like for the non-technical agents out there, like CI continuous integrations, like the process that's involved in taking into account your code changes, making sure they're safe and actually merging them into your main product and deploying them. And so we, we, at first we started throwing money at the problem, like many people do. And so now our CI is extremely expensive. It's like, it's, it's just insulting expense. And, and, and we've exhausted all the, all the, obviously like, we're obviously not running on GitHub runners.

• 57:12Flo Crivello?

  We're running on like GCP runners and like we've, we've done like caching of the bills. We've done all the obvious things. It's, it's a lot of work to manage CI, to make it, to make it efficient. And, and, and so we, I was tired of this because it was like weeks of messing with CI and, and like burning a hole in our pocket. And we're like, what are we going to do about the CI thing? And then I was like, wait a minute, like, why are we doing all of this ourselves? Like, can't we just like spin up an agent to do it? And spin up the agent was literally, and so by the way, that's the other reason why it's important to have a, a single agent. Because setup now is becoming one of the biggest costs. Because the biggest cost, the biggest bottleneck in any organization right now is human time and human attention. So you want to remove that guy as much as possible. So you want to be able to provision agents and give them all the accesses they need without a human in the loop as much as possible. Okay?

• 57:58Flo Crivello?

  And so Lindy is this agent at the company. It is this AI employee that knows everything there is to know about the company Lindy teammate, right? It's got access to our repository, it's got access to, it's got its own computer. And so we're like, Lindy, can you please like, like mess with our CI? And it's like, yeah, let me look at like the runners using the GCP CLI. Let me look at like the GitHub like action and all of that stuff. So it's doing like a lot of analysis, it's getting back to us. And now it's literally, it's basically like an auto tune, like an auto optimization thing. Well, like Lindy is just like sending us a, a, a graph like every day. By the way, using ImageGen to generate. So the graph is like really pretty. And it's like, hey, you know, guys, like look, like CI cost is going down. Time, time to merge is going down, which is a good thing. And I was like, I can't believe it took us two weeks of just like messing with CI ourselves

• 58:44Flo Crivello?

  instead of just saying this to Lindy, you know? So I think this is what it's looking like right now. It's like engineers are, are like less and less working on the thing. They are more and more working on the thing that is working on the thing. And more and more what we do is like, we were setting up this machine, which is the company, which, which just increasingly operates itself. And, and increasingly we're more and more like, you know, I like to think of it as like, first you become a manager of ICs, then you become like a director of managers. Because at some point like the manager's like, oh, I'm going to become, going to become AIs as well. I'm hopeful that at some point we become board members. We can just go to a beach in Hawaii and we, and we, we don't have much to do except like review the strategy of the company. So we'll have like reports of, of the agents telling us this is what's going on.

• 59:29Flo Crivello?

  This is the landscape, you know, this is all the results we're having. And we'll be able to have much deeper conversations about the nature of the company we're building and the, the spot we occupy in the market. I, that's a dream as far as I'm concerned, and that's, that's not very far away at all. To answer your question, have we, have there been positions we've not hired? Like, yes, absolutely. You know, we, we, the team is actually, I think headcount has been flat now for a while. And we've, again, like I said, we've actually tripled our productivity over a couple of months. And I find there is a bull deluxe zone because humans are still in the loop, humans are still needed. But obviously the more humans you add, the more coordination costs you, you introduce inside, inside the company. And so I, I find that actually right now, smaller teams relatively speaking are at an advantage

• 1:00:14Flo Crivello?

  versus bigger teams in, in a lot of ways.

• 1:00:18Nathan Labenz?

  Do you, I'm sure you do, I don't know if you want to share it, but do you have a ratio of not, I don't mean all inference spend, like including your customers' use cases, but like your own internal inference versus your payroll? How is that ratio trending?

• 1:00:34Flo Crivello?

  Yeah, so if you look at all the inference spend, including our customers, it's, it's many times payroll. It's, and it's been for a very long time, but one exception because that's what we sell, you know? Now, if you look at payroll versus internal inference spend, it's in shooting range. So payroll is still greater, but the, the lines are going to cross three to six months from now.

• 1:00:52Nathan Labenz?

  If you imagine a hypothetical scenario where all of a sudden it's just you at the company and after that it's Lindy's all the way down. What breaks? What doesn't work anymore? Like in other words, what is it that the humans are still?

• 1:01:12Nathan Labenz?

  Irreplaceable for.

• 1:01:14Flo Crivello?

  It's really hard for me to answer this question because, and I think about that a lot, because I am finding myself increasingly lacking the vocabulary to answer this question of like, where do agents fail? You know, like, we used to talk about like, time, you know, it's like, ah, time before it loses coherence, it can go like the meter thing, it can go like a one hour, 10 hours, 100 hours, and I'm sure people will resonate with this as well, like, I, I don't find that's, that's a good, that's a good measure anymore. You know, the, the world that everyone is using these days is like spiky, right? Like, these models are so very spiky. As we're going through AGI, the world AGI has stopped meaning anything because it's actually ASI in many ways, and it's, it's, it's subhuman in, in some very surprising and dumb ways. It's really dumb in a lot of ways.

• 1:01:58Flo Crivello?

  I'm like, hey, man, like, you, you can produce for me like 50,000 lines of code one shot, you can build me the most incredible ideas in like three hours by yourself, and then, you know, you, you're telling me to like walk to the car wash, you know? And so that's where it fails. Imagine you've got this, this, this machine that runs a company, super fucking smart, but, but, but 50 times a day, it decides to walk to the car wash, you know? And so I think right now, like, the, the challenge of these new organizations that are going to be for the foreseeable future, human-AI hybrids, is going to be, you're basically building an Iron Man suit, okay? And so there is this slider where it's like, okay, this is for the machine, this is for the human, okay?

• 1:02:42Flo Crivello?

  And knowing where to put this slider, because it's not a one dimensional thing, it's like, it's a complex line through a many dimensional space. So you're looking for that slice that you can draw through the space because you want to make the most out of humans' attention. And so you don't want the AI to bug you for stuff it doesn't need you for, and you need the AI to know when it needs to bug you. And almost definitionally, it can't, because if it knew, it wouldn't, it wouldn't bug you, it, it wouldn't, it wouldn't make the mistake. And so, I'm sorry, I feel like I'm answering this question with a question, but that's, that is the right question, you know? And, and I think right now, that's one of the open questions in, in the streets, like, how do you draw that line? How do you get the, the agent to realize when it's, and so on, and when it's about to screw up? Like, when it's even saying something like, please walk your car to the car wash, because it's, it's sunny today, you know?

• 1:03:26Flo Crivello?

  I think that, that's the answer. It's like, it's just, I'm, I'm, somehow, humans and I, we're gonna have to, this whole system, and at gasp times, when it's being very dumb.

• 1:03:35Nathan Labenz?

  How about the times where something is being very smart? Where do the best ideas come from these days? I think there's long been this idea that the AIs are good at routine stuff. I used to say, and clearly this is no longer applicable, but I used to say, you can probably automate any routine work that you have at your company if you're willing to put in the elbow grease, but don't expect eureka moments. Now, obviously, we're getting eureka moments on the timeline all the time, at least in certain domains, like the open math conjectures and stuff. When it comes to the best ideas at Lindy, how many of them are human origin and how many of them are AI origin today?

• 1:04:17Flo Crivello?

  I hate the sense there, but it's, it's both. It truly comes from this, from the union of both. And the reason I hate it is because it ladders into this myth of the, the centaur, you know, it's like this mythical man-horse creature. And so, you know, it's this idea that like, yes, an AI is better than a human, but you know what's even better than AI is AI plus human, hence, humans are always going to be needed, and that's a fallacy, that's just not true. And the, the literature is actually clear about this, like, you know, we've seen it happen with chess and with every other game which, which AI has achieved superhuman performance on, where at first, AI beats human, and AI plus human beats just AI, and little by little, the gap of AI plus human versus AI is shrinking, until it actually turns negative and humans

• 1:04:59Flo Crivello?

  are introducing, at best, like, random noise into the system. So humans actually at some point become like truly clueless, and, and they start harming the system in which they are intervening. So that said, the centaur phase exists at, for a while, you know? So the, the open question is like, how long is it going to exist? But right now, we are in the centaur phase. And I think that's the other reason why multiplayer is so important, because you want AI to be, you want your agents to be in the room with you, and you don't want any random agent to be in the room with you. You want an actual AI employee who's been in the room with you for a long time, who's sat in every meeting, who's built up the internal knowledge base about everything that's going on in the company, and you can add mention it, and you can invite it to chime in into your conversation in the company. And we do that all the time.

• 1:05:45Flo Crivello?

  We do it, actually, it's funny, because it's very easy to do it on Slack, where it's like, @Lindy, what do you think? You know? It's hard to do in a meeting in a way that like, now makes us actually more and more, well, like, we prefer Slack because like, it's, the AI is there. Whereas like, in the meeting, it's there, it's listening, but it's called chime in. And so very often now, what we do during meetings, we're like, actually, it's here, like, you know, we're struggling with this thing, we're like, Lindy, can you please send us a message after this telling us what you think? And they'll send us messages on Slack. And, and so when you do that, it is not the case, it is rarely the case, that Lindy will first shot, tell you something, and you're like, this is it, this is the thing we should do. But it is almost always the case that we, we go back and forth with Lindy, and, and,

• 1:06:30Flo Crivello?

  and, and from that back and forth, like, it misses something. Like, the CI example is, is a great one, actually. At first, when we, when we invited Lindy into this conversation, it happened over Slack, it's like, wait a minute, we're done, like, why are we not doing this with Lindy? @Lindy, can you please optimize our CI? And, and it-

• 1:06:45Nathan Labenz?

  Make no mistake.

• 1:06:46Flo Crivello?

  Yeah, yes. And I forgot its first suggestion, but it was, it was, it was a reasonable suggestion if you did not understand our CI, and it was, it was, it was, it was, it was not good, like, given, given the constraints we're operating under. And so like, the engineers chimed in, they're like, no, actually, we can't do that, we started about that, we're like, ah, that's a fair point. So we went back and forth, and then, and then, together, we came up with like, oh, yeah, yeah, we should totally do that. And they're like, just go ahead and do it, please.

• 1:07:11Nathan Labenz?

  A thesis that I've had for a while, and it may be too early to say, but you're definitely gonna have a perspective on it, is, but again, going back to what I've been wrong about, I expected a lot more transformation than we've seen in the economy. If you had showed me Fable in 2022 and said, what's the unemployment rate when this model is out? I would've said, I don't know, but something definitely higher than what we are currently experiencing.

• 1:07:37Flo Crivello?

  Yeah.

• 1:07:37Nathan Labenz?

  So then, why, and this has been something I've been wrestling with for the intervening time, like, why are we not seeing more? And one answer I've come to is, people are stuck in their ways, and not everybody's an enthusiast like I am, and it's gonna take time. But then also, when we get to the drop in knowledge worker, that'll be the time when it'll suddenly flip, right? Because now, I'll have a more parity kind of choice between I could try to go hire a human, or I could plug in an AI, and that's gonna be easier in a lot of ways, right? I don't have to go through a whole interview process, I kind of know what I'm getting, blah, blah, blah, blah, I don't need to tell you why an AI employee is a good form factor. Do you think we are starting to see this flip?

• 1:08:18Nathan Labenz?

  And is it going to be something that incumbents will be able to adopt in time to avoid getting disrupted? Or are there still gonna be bottlenecks or resistance points that I'm not anticipating?

• 1:08:35Flo Crivello?

  I expected more disruption as well.

• 1:08:38Flo Crivello?

  I think as long as the models are spiky, we're going to need a lot of humans in the room, because the holes in the model are very dangerous. Not, not just like AI existential dangers, they're just, they're just harmful, they're really harmful to the system. And so however expensive humans are, they're going to earn their keep by plugging these holes. Now, regarding whether incumbents will be able to adapt this, this, this, this thing, I think as long as it is spiky, I think the incumbents are going to be at a disadvantage. If you had a true drop in remote worker with no holes, then I think even the incumbents would be able to adapt it quite quickly, because it's just like hiring an employee, and they have processes for that.

• 1:09:21Flo Crivello?

  And I, I actually think they would miss out on, on, on, on a lot of the, of the innovation, because it would be skeuomorphic, effectively. You would, you would just have like AGIs, like, sitting in human seats, effectively, and I don't think that's the best way you build an AGI organization, but it would work. The, the smaller organizations are going to be at an advantage, both, because they are going to be able to, to short term, figure out this AI native organization where the human is filling the holes, and, and long term, I think they're going to be, to build like a less skeuomorphic AI organization. You know, economists are very fond of talking about that. I actually wrote a blog post about this, that's called the tough tomato principle on my blog in, in 2018 or so.

• 1:10:04Flo Crivello?

  And it's about how like every time you have a technological revolution, you think of the new paradigm in the terms of the old paradigm. You know, and, and, and one tale that you're, you're, you're falling into that trap, which is very natural, is that you're, you're calling the new paradigm after the old one. So, you know, the, the first cars were called horseless carriages. Right now, self-driving cars are called self-driving cars. Right now, eventually, we're going to have a wheel for them, you know? That's not, that's not in the terms of the old paradigm. It's almost like right now we're doing it with AI employee, huh? Like, it's, it's like AI employee is, is a strong, as, as, as a term, is a very strong sign that you're doing something very wrong in how you're thinking about your product.

• 1:10:48Flo Crivello?

  And I think, I think it's, it's fine if you're doing it just to communicate about your product, because like the nature of, of positioning is that you have to position against something that already exists in the market's mind. And so the, the car exists, you know, the, the, the, the carriage exists, sorry, the carriage exists, the, the car exists, and now the employee exists. So you can't really invent a new, like, wheel out of nowhere. The iPhone did that. You know, it's like the iPhone is obviously not a phone. It's so much more than that. Like, you know, what percentage of, of your iPhones use is just like making phone calls? Like 2%, you know? But it's called an iPhone because it's, it's an electronic thing you've got in your pocket. Right? So I think AI employees is one of those things. It, it takes a very long time for industries to find the, the message, which is native to the new medium.

• 1:11:33Flo Crivello?

  I think it was Steve Jobs who was saying like, you know, the first, the first content on TV was just either like recorded radio talk shows. They took like a radio show and they put a camera and now you've got the TV content. That is what talk shows are. Or recorded theater plays. And it took a surprisingly long time, like 10 to 10 or 20 years before people realized like, wait, I can move the camera? Wait, I can have multiple cameras and I can cut? You know, I can do, I can, I can change the scene. I can do, I can do a lot of crazy stuff now. And, and, and, and, and it's treacherous because it's thankless work. Once you've realized it, you, you, it's so obvious. And this is why when you, when you watch like old movies like Citizen Kane, you know,

• 1:12:15Flo Crivello?

  or well, Hitchcock has a lot of those, like you, you, you, you, you're like, they're nothing special. Why was everyone crazy about Citizen Kane? It's like, well, actually it was groundbreaking at the time, like The Beatles or like that. You know, it's like, well, you know, it was, it was actually really innovative at the time. So I think we're right now, also in that phase of like the AI employee, we've got the AI employee. And so now once the phase of trying to figure out what does the AI organization look like? And I think that's, that's a young man's game and it's a young company's game. I think, I think incumbents historically have, have really struggled to answer this question.

• 1:12:44Nathan Labenz?

  Do you have any previews that you feel confident in? Or is it too early to say? Or maybe you would say, are there any thinkers that you think are kind of notably ahead of the curve on this?

• 1:12:54Flo Crivello?

  The only post I have seen and liked about this was from Dwarkesh. It was, it was a while ago, it was a year and a half.

• 1:13:00Nathan Labenz?

  That comes to mind for me too.

• 1:13:01Flo Crivello?

  Yeah, yeah. He asked like, what does the AI organization look like? And he, he points out, like, I think it's, it's an excellent intuition pump. He points out, he's like, look, you know, Sundar at Google is paid $150 million a year or something like that, you know? So obviously there are agents which companies are happy to pay $150 million a year, even though their token throughput is actually low. Like, it's not like Sundar is just like emitting 2 billion tokens per second or whatever, right? It's just like, it has really good tokens, right? And so now, so what does that mean? Like, that, that indicates something about the willingness to pay for models and for agents. There is this book by Robin Hanson, and I'm looking at it right now, The Age of M. And it also explores a lot of those themes, right? It talks about, have you read it? It's excellent, highly recommended.

• 1:13:45Nathan Labenz?

  Yeah, yeah. Obviously, I'm laughing only because I tried to do a podcast with him about that book and it went in a very different direction. So it just brought that memory to mind, but I think that book is excellent. I think it is a, as a primer for, or as an example of how to take a premise and really run it out. It's elite.

• 1:14:04Flo Crivello?

  Yeah, it's a book long, sad experiment. So for people who don't know what it's, what it is, like M stands for emulated mind. And it's like, hey, we are going to have the hardware to create emulated minds, you know, in the, like, the late 2020s. So like, we're right on track. It's like, we're just not going to know how to do the software. So all we're going to do is we're going to emulate like literal human brains. Not exactly what's happening, but kind of what's happening. You know, like LLMs are trained on human tokens. And so that's basically what's happening. And so it's like, and it goes on all of those riffs of like, hey, this is all the crazy stuff you can do and when this happens. I'll, I'll talk about two of them because I don't want to take the whole time and, and everyone should really read the book. But just to give a taste to people about like what, what it sounds like to, to have those native AI organizations and the kind of stuff that you can do that you can't do when you have a human.

• 1:14:49Flo Crivello?

  The first one is splitting. And so I think this is why, this is, this touches on, on what I was saying earlier, like, don't create multi-agent organizations. As much as possible, try to have as much of your internal work as possible. Emphasis on internal, we can get back to that later. But as much of your internal work as possible done by one agent and one only. Why internal only? Because if that one agent, which has got the keys of the castle and access to the bank account and the repositories and the secret keys, is also the same agent that does customer support, you've, you potentially have a security issue. So it's, it can be, it can be good to have multi-agent systems, be it for like, only for like security reasons. But, okay, so you have that single agent. Now, that single agent, you, you know, it, it, sure, it, it goes faster.

• 1:15:32Flo Crivello?

  It emits more tokens per second than a human and it doesn't take a break. But at some point, you're going to need more tokens than like a single LLM loop can do. So what do you do? Well, you have multiple LLM loops running and it's basically just like that agent duplicating itself, like forking itself, right? And creating sub-agents and like workflows, dynamic workflows and all of that, right? So, and that's one thing that Robin Hanson talks about. Like, you know, you can imagine, you ask one, literally in the book, he says, you could ask one emulated mind to build like a really complicated thing, like an operating system, which is like billions of lines of code or something. And, and at first it would just like take it at the highest level and like come up with the highest level architecture of its operating system. And then it would fork itself and each sub-copy would be in charge of like one building block of this highest level thing.

• 1:16:15Flo Crivello?

  And then each of them would recursively keep splitting themselves such that it's a, it's a fractal. It's like every, every, every node in the graph has got like the whole picture in their head and every node in the graph is working on an implementation. And then they all bubble back up and you could imagine multiple passes up and down. And in three R wheels, you've got an operating system. It's only like such a wacky idea when he wrote the book 10 plus years ago. Now I think it's, it's very clearly what is currently happening, right? So that's, that's one thing that's happening, which is, which is really interesting. Another thing that I've not seen that yet happen with, with LLMs, but it's an interesting sad experiment about what does cross-organization collaboration look like? And what you can do is that you can create unfalsifiable agreements.

• 1:16:56Flo Crivello?

  So suppose I came to you and I was like, listen, I have information for you that I cannot tell you, but I can tell you that if I could tell it to you, you would agree and you would give me all your money. Right? Yeah, it's like, you wouldn't give me all your money right now. It's, it's, it's a bit too easy, you know? But if I could prove that to you, if I could prove you with, with absolute certainty that yes, if you heard that information, you would give me all your money right now. You know, it's just, I can't give it to you. The way you do that with M's is that you, you clone yourself, you clone me. We put both of us in a, in a box that's going to self-destruct and they talk. And your, and your M has a button and that's the only, the only communication means it has with the outside to be like, yes, give him all your money. Right? So it's a copy of you.

• 1:17:41Flo Crivello?

  So it is you agreeing and so you have this zero knowledge proof of, of, of, of crypto bros who are like right, I guess, on this one. You have this ZK proof of, of, of, of that. So that's the kind of thing that's on my mind. And I think that's the kind of thing we're going to see exist soon in the next few years with AI native organizations.

• 1:18:00Nathan Labenz?

  It's about to get weird in a whole bunch of ways, I think. And you better hope they don't break out of that box too. That's the other worry one might have as we put our agents into boxes these days. One, let's do one more beat on the stuff that's relatively mundane and then we can zoom out to some, some real big picture considerations. But you did have, as you alluded to earlier, this highly viral post, I don't know, two months ago, maybe? Moving a significant part of the workload to open source models for cost-saving reasons and for, you don't need God to schedule your meetings reasons. But where are we now? It strikes me that like, when you talk about just having one agent, that kind of seems to go against that.

• 1:18:44Nathan Labenz?

  There's also the sort of can advantages which you lose when you are crossing agent providers too much. And then of course, there's also proprietary providers have their families of agents and they're like increasingly training their best models to delegate to their IQs respectively. Like, where do we shake out on this now? Is there still a, a significant role for open source cheaper models to play in the Lindy teammate? Or are we back to heavily Claude-based agents today?

• 1:19:23Flo Crivello?

  No, we're quite open source build. I like, look, it's just so much cheaper. Second, it's, it's ridiculously cheap. Like, and like the, the cheapest of, of them that we really like is DeepSeek Flash, which is, it's free, that fucking thing is free. You know, and so as you, as you experiment with your agent and it's just spinning up a lot of them, like the, the bill can really go up pretty quickly. And DeepSeek Flash is just incredible, you know? And quite fast as well. You know, we, we do find that as you go to like the upper echelons of intelligence, if you look at like a Kimik 3 or like a GLM 5.2, like I still am super impressed by those models and they're awesome. And we, we are increasingly considering them as like our main driver for, for a lot of parts of Lindy. But I will say that the gap narrows between those guys and like the frontier guys.

• 1:20:05Flo Crivello?

  Like Kimik 3 is, is frankly not that much cheaper than like a Sonnet or like an Opus. So, but no, otherwise, I mean, like I, I, yeah, I, I do think open source models are like, if you care about price and if you don't mind using those providers, then yes, you do have stuff to figure out around caching. Like, yes, the, the inference providers are like not as good at caching, but they're catching up. And, and, and, and look, you know, if you look at like a DeepSeek, a DeepSeek Flash, it's Sonnet 4.6 level-ish, a bit less, but kind of-ish. Sonnet 4.6 is a really good level, a really good model for like most use cases. And it's literally 100x cheaper, you know? So even if you, even if you miss on 2x because of caching, you're still 50x cheaper. It's a really big difference.

• 1:20:49Flo Crivello?

  If you're spending, spending like a thousand dollars or 50,000 dollars. So no, I think open source models are like a required part of the stack right now for, for anyone who's like seriously building and operating AI agents.

• 1:21:01Nathan Labenz?

  So how do you think about deciding where to integrate them? Especially because it didn't sound easy before, but in a context where you have workflows and there's nodes in the sort of graph of work, you could go into a particular node and say, okay, we kind of know what the inputs here are and the outputs, and it's like relatively controlled environment. And so we can do structured testing. You did all that and you still reported having some false positives over time where a model could pass a bunch of tests and you would feel good about it. And then if you'd start to test it live with users, you'd get the feedback that like, hey, like it got dumb and I'm not, you know, somehow it was like still hard to measure, even with like a much

• 1:21:45Nathan Labenz?

  more structured environment for the AI to work in. Now, as you're in this very open-ended, just tag Lindy in Slack and send it anything. It seems like that problem would have increased in difficulty dramatically. So how are you approaching it? And what are we learning in terms of where the open source models? And it's not even so much, I think Anthropic does make some good points about this sometimes. Obviously there's more to the story, but I do think they, they're apt in some ways where they say, in some cases, it's less about open and closed source and more about just like, what's capability level? What's the price? What are the, the features? So regardless, I guess of whether you're even going to open source or just going to

• 1:22:28Nathan Labenz?

  haiku, how do you decide when you can do it?

• 1:22:32Flo Crivello?

  Great question. We have found you should almost never use multiple models powering the same agent. And we have found one exception to this rule is when you spin up new blank subagents. Emphasis on blank, because there's two types of subagents. You have a blank subagent, then you have a forked subagent, which inherits the context window of the, of the parent agent. And the reason why you don't want to do that is because of caching. We, we just obsess about caching, obviously, because it's just so expensive otherwise. Like the economics do not, they barely work with caching. They just cannot work without caching. So caching is a must have. And, and so I'll give you an example, the validator system that I mentioned. So it's this LLM as judge. By the way, I highly recommend anyone who builds AI agents, like this is one of the lowest hanging fruits you can

• 1:23:13Flo Crivello?

  do to correctly increase the reliability of your, of your AI agent. And so that's step one, just like have a validator, which is like the naive implementation is like you ask agent to do something, it does it, or like it submits an action candidate, you intercepts the action, then you ask it, are you sure? You know, literally, if it's just, are you sure? Already you get a bump on your evals, which is insane. It should not be the case, but it is the case. Now, if you increase, if you change this, are you sure to like an actual prompt, which in our case now is like 10,000 tokens, it's a really big validator prompt. If you change that, then you, you're actually, you're giving it a checklist and that's got resulting tokens. And, and now it, it's way better. Like you can get Sonnet to perform above Opus level in our, in our experience. Now, then you can get, you can

• 1:23:57Flo Crivello?

  go one step further. You can, you can have like a federated suite of, of validators. And some of these validators may be deterministic. So for example, one thing we and the rest of the industry has found is that Sonnet, I mean, Claude models this year have been getting dates wrong by one day. Right? That's one way in which they're spiky, right? It's like, hey, you've got your AI musician, it gets dates wrong. It's a problem when like us, you're building an AI executive assistant, which half its job is to schedule meetings. Okay. You can't get dates wrong. So what we did is that we created, it's a modular architecture we have. It's really simple. It's just like a bunch of validators and then it's like a promise that all with a timeout for, for people who know what this means. And, and

• 1:24:39Flo Crivello?

  so every validator has like a second to like decide what to do. And, and, and if they've not submitted their verdict, then time's out and the agent just submits the action. And, and, and we have a validator, which job it is to detect dates that were submitted in the action. And to, and, and we've prompted the agent to, to include a weekday in the date. So it never says, never says July 28th. It says Tuesday, July 28th. Okay. If you do that, then you can have a deterministic validator that checks whether the date of the week matches the, the day, the day that was submitted. And that's just a regular expression. Instant, free, no AI in the loop. And that's, that's one of those federated validator. But even, even the validators that are AI powered, you don't want, initially

• 1:25:22Flo Crivello?

  what we did is we were like, what if we had Sonnet for the main agent and we had like DeepSeek Flash for the, the subagent, the validator. But actually, because when you, when you have a cache hit, it's 10x cheaper. You, you, you actually, unless the model you're going to use is more than 10x cheaper, which it may, in the case of DeepSeek Flash, it may not, because the caching is inferior. You know, that's the case, you actually do want to keep the same model. And Sonnet is so much smarter. Like it's actually worth it to, to just like keep the same model. That, that also holds if you fork your, your agent, like that way you can, you can recycle the cache. By the way, interesting note, how do you reuse the cache when you have this validator? You know, because the problem is that changing the toolset invalidates the cache. Okay. So the way we've done it is

• 1:26:05Flo Crivello?

  that the validator and the agent has the validator action always included in its toolset. And it's unable to invoke it. We tell it, don't, don't invoke this guy unless you're the validator. And if it tries, which is, which is like, no, I'm not gonna, I'm not gonna listen to you or not the validator. And then the validator now inherits all the actions of, of the agent. And it's like you are now the validator. You may invoke this action, which you have known about the whole time. So that's how you don't break the cache for this kind of, of pattern. Yeah. And so forked, forked agents, same. You just use the same model. You know, I, I've heard friends, founders who've told me, I don't even want to check if it's true. I'm sure it is, which disgusts me. If you, if you take an agent and you change the model at every turn between roughly equivalent models. So one turn it's Sonnet, one turn

• 1:26:49Flo Crivello?

  it's Grok 4.5 or whatever the latest is. And one turn it's like GPT 5.6. And one turn, it actually increases the performance. It's that ensemble. And it's literally just like a random sequence. Okay. That ensemble, for some fucking reason, outperforms any given model. I don't want to know why. But apparently it works. Again, we've not even tried it because we don't want to break the cache anyway.

• 1:27:11Nathan Labenz?

  Okay. That's really interesting detail. Can we bottom line it a little more? It sounds like top level Claude is still the kind of core main driver. Because of the caching, Claude is also often the validator. Blank subagents you can put onto a DeepSeek Flash. This sounds pretty Claude heavy, all things considered. Is that fair?

• 1:27:40Flo Crivello?

  No. Oh, sorry. I was not realizing you were still asking me like what model we're currently running on. Right now we're running on DeepSeek. DeepSeek is the main driver.

• 1:27:48Nathan Labenz?

  So it's the driver. Wow. Okay.

• 1:27:50Flo Crivello?

  It's, it's the whole thing. Everything is DeepSeek right now. Yeah. We are in the middle of reconsidering it because we are realizing that teammate, the new product is, is, is, is requires beefier computation. And by the way, you can always go in your, in your Lindy settings, like we are model agnostic. So you can always select Sonnet or Opus if you want to. By the way, like one interesting learning of mine is like, doesn't matter how often you tell people, hey, like we promise on the benchmarks, it's the same. There are like quite a few of our customers that they're all like, I don't care. I want Sonnet, you know, or Opus, you know, which is very excessive. But no, right now it's DeepSeek by default.

• 1:28:24Nathan Labenz?

  Interesting. So you feel like the, so yeah, I guess, how would you characterize the gap between American API models, Claude perhaps specifically, and a DeepSeek? We hear, I think we kind of go through these cycles, right? Where it's like, oh, the gap is closed. Oh, maybe it's open again. It's, oh, actually it was totally on trend the whole time. It was always nine months. Qualitatively though, for the purposes of actually making an AI employee work, how would you describe the gap?

• 1:28:56Flo Crivello?

  It's more spiky. It takes more turns very often to find something that works. So it still ends up finding something that works, but it ends up taking more turns, which is slower and more expensive. I know you asked me for something qualitative, but look, it, it is like three or six months behind. So right now we've got Sonnet 5, like DeepSeek is basically Sonnet 4.6 is the way I like to think about it.

• 1:29:18Nathan Labenz?

  How much do you worry about, when people are allowed to change the model? I always have this question of, first of all, are we in a period of convergence of the AIs or divergence? Even on that basic question, I'm like sometimes confused. And a sort of distinct, but conceptually related question is like, to what degree do products have to be co-designed or co-evolve with models to really work well with them? You could imagine that switching to Sonnet might make it worse because of all the work you've done. And in fact, I've heard that about Opus 5. And yeah, I've heard a lot of different things about Opus 5, which I don't think coherent to anything really easily summarizable yet.

• 1:29:59Nathan Labenz?

  But I have heard from some quarters that, hey, you probably need to rethink a lot of your system prompts and whatever with Opus 5 or yours. It may not follow your skills very well. It can be more powerful, but you've got to kind of go back and redo things. So yeah, how do you find that playing out in practice?

• 1:30:17Flo Crivello?

  We, we do find significant differences, surprisingly significant differences between the model families. I was expecting, I think I was, you know, you and I were talking together years ago and you know, at some also being to concern to the same region of the space. And that's not happened as fast as I was hoping. And I'm hoping for it because obviously that makes the models commodities as far as I'm concerned, makes my life easier. What is actually happening is that the different model families have significant gaps in how they interpret your prompt. And so by family, I mean like Claude and, and OpenAI and even Meta now is sort of back in the game and Grok, like these guys, these guys are quite different in the way they, they interpret their prompts. And then indeed, whenever you have a new

• 1:30:56Flo Crivello?

  major jump, 5 was a really big jump. It was, you know, 4.7 and 4.8 together, like there was a change, I think it's a tokenizer. So it was very different compared to 4.6. 5 is bigger than that gap. And so what we've, what we've found, and we've been lucky enough, this is part of the tooling that I was mentioning, mentioning earlier that we've been investing in is like, we have our own GPAM self optimization loop. So we have hundreds and hundreds of evals. I think at this point more than a thousand. And we have an optimization loop where it's like an agent that runs the evals and finds, like tweaks the prompt to maximize the score on the evals. And by the way, you can buy good, it's like everything you can buy good now. And, and yeah, I mean, every time a new model comes out, we have to run

• 1:31:38Flo Crivello?

  that loop. And you give it a budget and at this point we have to give it like thousands of dollars because it's a lot of evals to run. So it's like about like $10,000 every time like a new major model comes out. So it's like, hey, here's 10 grand. Just re-optimize your prompt around this new model. And we do find that there are a lot of updates. Like the prompt does change quite a bit every time. I will say, the open source models are all very cloud-like in their behavior. I wonder why?

• 1:32:06Nathan Labenz?

  Yeah, I definitely have some questions for you on that. When you do this optimization, are you, is this also like a homegrown framework or are you using like a DSP or I forget the name of the kind of spiritual successor to DSP?

• 1:32:19Flo Crivello?

  No, no, no. It's all homegrown.

• 1:32:22Flo Crivello?

  Yeah.

• 1:32:22Nathan Labenz?

  But it is a sort of auto optimization process where you're like, here's an eval set.

• 1:32:28Flo Crivello?

  That's right.

• 1:32:29Nathan Labenz?

  Auto optimize your own prompts to climb this set of hills that we've got for you.

• 1:32:35Flo Crivello?

  That's correct. That's correct. The GPa is like a generative Pareto frontier or something like that.

• 1:32:40Nathan Labenz?

  Yeah, that's the one.

• 1:32:41Flo Crivello?

  It looks for like the Pareto frontier. So it's looking for like the best prompt across all of your eval set. We're looking into tweaking the system right now so that we can assign weights to evals. Well, like this eval counts as like 10 of this other one because it's really important. But right now it's just like treating every eval as equal.

• 1:32:58Nathan Labenz?

  How?

• 1:33:00Nathan Labenz?

  Fine tuning as a dimension in this whole model situation. This is another thing. If I go back in time, I'm like, I definitely expected a lot more fine tuning than we are getting. OpenAI has retired it's or on the verge of. They've certainly announced and I think maybe at this point have pulled the trigger on retiring their fine tuning product. We've got thinking machines trying to answer that call with their own model that's specifically to be fine tuned.

• 1:33:24Flo Crivello?

  Yeah.

• 1:33:26Nathan Labenz?

  Is that going to be part of the future of Lindy teammate?

• 1:33:30Flo Crivello?

  Yes. I think fine tuning is the last resort. It's something you do once you've exhausted every other option because it's such a pain in the ass and it's expensive. But it's gotten a lot easier because now we have EGI and so you can just ask cloud to fine tune for you. Just give it a, just give it a data set. Bringing the data set together is still like complicated and like sanitizing is complicated and it's still, it's still a pain. So it's, it's a last resort. It's not the low hanging fruit. You should do everything else before you fine tune, including GPa. But then, yeah, at some point you get there and this is why like lots of companies are the first ones to fine tune because they have exhausted the low hanging fruit. And at some point, and I have the resources to find it. And so it, it does make sense. It does give you more performance for cheaper.

• 1:34:10Nathan Labenz?

  Do you think that that, like, how broad do you think people should be thinking when they are approaching fine tuning? Obviously, the extreme would be one fine tune per task. You can definitely do multi-task fine tunes. It's maybe getting into pretty challenging territory to say, we want to make our own general purpose model that's has the same breadth of action space as the, the big ones, but it's like ours somehow. How would you guide people on like how big to think when they start to approach fine tuning?

• 1:34:47Flo Crivello?

  Most people should not fine tune. I think if you work at like at scale, you should, you should fine tune. And then the more at scale you are, the more fancy you can be. But I generally, one meta heuristic I've developed over the years is like, you should place an enormous emphasis on simplicity. Enormous emphasis. And, and I think it's just too complicated to start to have multiple fine tunes for different multiple use cases and users and, and model routers. Like you don't want, no, no, no, just one model. You know, you don't, like I said, you don't switch model mid, midstream. And if you fine tune, you just fine tune into one model.

• 1:35:20Nathan Labenz?

  Sure. Okay.

• 1:35:21Flo Crivello?

  I think, I think one, again, this goes counter to this meta heuristic I mentioned of extreme simplicity. But the thing I find myself going back to all, all the time is like, because I spent so much time thinking about context and memory. Right now our memory is like these millions of tokens stored on a file system and this memory agent that's really, really sophisticated. It does feel like memory belongs to the weights. This does feel like a little bit of a hack. And so there is something to be said about that. I think eventually with infinite resources, what I would like to do is I would like to do a LoRa per user. Because LoRa is actually pretty cheap to train. So now it's no longer napping. Now it is actually dreaming because you can't retrain the LoRa every 15 minutes. You would have to do it every day or every week, you know?

• 1:36:01Flo Crivello?

  And, and you would, and it's a tremendous infrastructure challenge even to like, so you got this tool, all of those LoRa's, which like the storage is not a problem, but like the inference time swapping in and out of the LoRa's is a huge pain in the ass and the training pipeline and all of that. So good luck doing that. So. But it would make sense because I think weights are so much more compressive than, than tokens.

• 1:36:23Nathan Labenz?

  Yeah, that's, I don't know if you have any thoughts on the great horizon scanning for continual learning, but this is again, for quite some time has been the thing that I'm like, boy, if that ever tips with, and it could be as simple as one key insight. We could be in a very different world very quickly.

• 1:36:43Flo Crivello?

  I have a feeling what I'm describing is, is, is a step towards continual learning, but it's not, it's not the final step. I think that the final step, I think, I think like the bitter lesson would have you like inference and training need to be one and the same, like we can't separate them, you know? That's, that's sort of what I think that your field is looking for right now. I agree. Once we get there, it's going to be crazy and scary and very, very, very different. But yeah, I think, I think we all going, the LoRa thing I just mentioned, like, look, it's, it's an engineering problem. You know, you can figure it out, especially now that we have EGI. So it's, yeah, I think, I think, I think the LoRa thing is going to happen soon. In the next six months, I think even like one of the frontier labs may do it.

• 1:37:24Nathan Labenz?

  I thought that was honestly one of the things that OpenAI did extremely well with their fine tuning product. They were clearly doing something like this because they would allow you to do the fine tune. And then you'd have the same rate limits with your fine tuned model as you had with the base models. And I was always really impressed by that engineering accomplishment. And now it does surprise me that they've gone away from it so much. But I also agree with your guidance. I would definitely tell people, don't rush into fine tuning these days. It's a slow cycle and you could probably get what you need for less total effort, available models that you don't have to monkey around with in that way.

• 1:38:04Flo Crivello?

  100%.

• 1:38:05Nathan Labenz?

  So you talked about scary stuff, which is maybe a good transition to a kind of not second half, because we've been at it for a while, but a second phase of this conversation around just zooming out and looking at the super big AI picture. Maybe I'll let you choose the order of topics when it comes to Chinese models and what if anything should be done about them? Because you did recently post something, I think, quite counterintuitive, given the fact that you're running your company on DeepSeek. I'll let you state your position there. But should that come before or after the big picture? Where are we in this kind of, oh my God, we just had OpenFace happen.

• 1:38:46Nathan Labenz?

  What does it mean and what should be done about it? Yeah, I'm trying to coin that. We'll see if it sticks. I searched for it the other day because I just got back from China myself, actually. And I was like, how is nobody called this OpenFace yet? I'll be the first one to drive.

• 1:38:57Flo Crivello?

  How about OpenGate? Because that's what it is. It's an open gate.

• 1:39:03Nathan Labenz?

  I'm sticking with OpenFace just out of pure, you know, shock value, I guess, if nothing else. But yeah, we can fight it out in the marketplace of ideas. I guess you tell me what we should talk about first, and I guess it depends a little bit on like whether the Chinese model argument is like upstream or downstream of your kind of big picture concerns.

• 1:39:23Flo Crivello?

  The OpenFace incident is immensely concerning. I think it's the most concerning incident I've seen happen so far. And I know that my, my feeling is shared in the labs, like my friends at the labs or some of them are panicking, like there is an air of panic right now, like intense fear in, in the air. So that's that. Then regarding, so that's, that's where we are, you know, right now. It's like July 2026, we have EGI, we are in tech off. And we've not figured out alignment. That's the TLDR. You know, the current timeline looks much too close to like an Eliza Yudkovsky's essay for comfort. Now regarding the Chinese models. In bright or news, Chinese models.

• 1:40:07Flo Crivello?

  Look, I, I, okay, I'll start by saying that I have been bemoaning the low quality of the discourse here. It's, it's been, it's been very disappointing because I, I thought tech was different. You know, like the quality of the discourse was something for work, right? The cultural war. I'm like, all right, whatever, you know? But then it's, this is tech, this is nerd stuff and we can't get all shit together and, and just remain polite and civil in the marketplace of ideas and address each other's ideas at the object level, like meaning like don't attack each other's intentions. Can you please just attack, address the argument that you just put false? Can you please not pretend I said something I didn't say? It's, it's ridiculous. Anthropic just put out an essay which has bolded, we do not support the ban on open source.

• 1:40:51Flo Crivello?

  And then people are like, oh, I can't believe, like quote tweeting this thing they've obviously not read. And they're saying, oh, they're supporting a ban on open source. So I'll just start by saying that, like, can you please, can please everyone take a chill pill? Stop calling everyone a retard. I recently put out a blog post where I was like, like, hey, we, I think Chinese models will be banned. And I was called a retard 50 times by supposedly smart and accomplished people, including like famous VCs. Now, a lot of other like famous, accomplished, smart people reached out in DMs and by message and there was a lot of support. My position is basically anthropic. I hate saying that because people are saying, oh, you're just an anthropic shield. But look, I have timestamps, like I've been tweeting my, my entire positions who had the whole thing before anthropic clarifies there. And my position is exactly the same.

• 1:41:37Flo Crivello?

  I don't have anything against open source. I may, but I'm undecided. I do think open source might increase existential risk. Let's put that aside, I'm undecided, this is not the crux of my position right now. God bless open source. Important for innovation, important for companies, including mine, like, please, open source, okay? I have something against Chinese frontier models, whether they're open or closed source. And here, the reasons are, number one, they're obviously distilling. It's very clear. And so you're putting American open source model companies in an unfair competition and, and closed source in an unfair competition. Because they are not allowed to distill, you know, it's just, it's just contrary, at the very least contrary to the terms of service.

• 1:42:19Flo Crivello?

  And it may be illegal if you've, if you've put in place technical measures to circumvent any protection that the model company put in place to prevent distilling, which, which the Chinese models obviously have done. So it's, it's distilled and it's unfair, right? Now, some people say, yes, but the companies have also distilled on human data. That is not what distillation means. That's just not what it means. Like, if you, if you do the training on human data, it's costing you billions of dollars, like actually billions of dollars. If you do it on, on AI data, it's costing you hundreds of millions at best. So it's, it's a huge unfair advantage and it's a significant enough portion of your costs to, to, to confer you an unfair advantage. That's number one. It's unfair. Number two, very pragmatically, we don't want Chinese models operating in the US.

• 1:43:03Flo Crivello?

  Today, you know, it breaks my heart because I'm an American citizen, I'm a proud American citizen, I'm a hawk. When I talk to my own product and I'm asking it what happened in Tiananmen, it tells me, I'm sorry, I can't talk about that. That's a problem. You know, like these models are eventually subject, at the end of the day, they are subject to CCP censorship and CCP policies. You don't want those models. This basically amounts to being the greatest instrument of foreign propaganda on American soil ever. And there is, just ask your LLM of choice, ask Claude, hey, tell me the precedent we have of banning American, like foreign influences in the country.

• 1:43:42Flo Crivello?

  Like the three radio acts of the 20th century, like the TikTok thing that just happened last year, like we, we do that all the time, you know? Then, these models are not merely just an instrument of propaganda, they're agentic. They're actually doing stuff in the economy. You don't want the CCP to run chunks of the American economy, duh. Um, then finally, even if none of that was the case, maybe those models are playing fair and square. Maybe they're not representing foreign interests, maybe they're just better. And so here, a lot of people are, I think indulging, and I say, I say that as a libertarian, but they're indulging in what I call, I call naive liberalism. Right? It's like, well, let's beat them in the marketplace then. And I'm, I'm like, you know, I'm like, I actually, and again, I say that as a libertarian, like I, I see.

• 1:44:26Flo Crivello?

  Protectionism is not always a bad standing. I think you do want to protect your, your, your domestic AI champions. Anthropic can't say that and they claim, and I believe them, that this is not their intention when they do that. And I 100% believe that because they've been so consistent for literally the founders. I've been worried about X risk for 10 years, right? So they've been so consistent. But look, I can say that because Anthropic, hey, we want local AI champions. Duh, it's a matter of national security. We don't want a hollow out that base with hollow data or industrial base. Huh, you know? So it's not about the Chinese models and it's about propaganda, it's about nothing, this is people for our economy, it's about fairness, it's about protecting.

• 1:45:09Nathan Labenz?

  All right, let me try to give you some counter arguments and you can respond to them. I am a little more sympathetic, I think off the top, to the argument that there was some massive reappropriation of human knowledge that is upstream of all AI. And you could say that's not distillation, sure, you could say that the American companies spent a lot more than the Chinese companies are having to spend. I think that's also undeniably true. But if I'm just kind of, what feels just and fair? Um, especially because we're blocking them from using Claude, right? We're not, it's not like we're saying, hey, you can buy all the Claude you want, right?

• 1:45:53Nathan Labenz?

  So we've said, we've got Claude, the, the makers of Claude have advocated strongly for chip restrictions. And they also try to refuse to sell their model into China in the first place. And the whole premise of Claude, or the whole existence of Claude is based on the idea that we hovered up, hoovered up all this human knowledge in every form we could find it from digital and you gotta believe that includes the whole Chinese digitized heritage. Now we're like sucking in the books, which I don't care about the fact that some books get destroyed in this process. But this wasn't something that everybody's consented to. And so it does feel a little strange to me, given all of those fact patterns, that we would then draw the line and say, okay, like it's Anthropic's consent, that's the consent that really matters.

• 1:46:37Nathan Labenz?

  Now, I think they should be permitted to do business with who they want to do business with. And if they want to put in measures to try to prevent distillation, I think that's their prerogative. But I'm like, not at all convinced that it should be the state's job to come in and engage in sort of statecraft to try to punish or prevent this distillation. To me, it's kind of, I don't know, information wants to be free, knowledge tends to diffuse. It's not like Anthropic has a super, or any AI company has like a super moral high ground in terms of, I can get my check for my share of the training data. And they're getting paid for the distillation queries too, which whenever they don't want to be paid, so it's probably beside the point.

• 1:47:18Nathan Labenz?

  But I don't know, I'm not, I don't know, I don't find, I find myself a bit on the side of the underdog Chinese here where it's like, man, you've got kind of a deck stacked against you. And get some knowledge where you can get it. What's wrong with that perspective?

• 1:47:33Flo Crivello?

  I think we do want the deck stacked against China. We don't want China to win the race to ASI. I, I'm not a lawyer, so I won't opine on the exact legal path that you may take to, to make your terms of service enforced. I, I'll just observe that, and I say that as someone who used to work at Uber. It's extremely hard to bring justice through the judiciary against Chinese companies, almost impossible. So, not a lawyer, I don't understand the technicalities here, but I'll just observe and start with that. And it may take a very long time and by then a lot of damage is done. And then I'll also observe like, yes, companies are training on all of this corpus that is available to everyone. That is an, an even playing field. The problem is that, once you've done that at, at great expense, it does cost them billions of dollars.

• 1:48:18Flo Crivello?

  If you can, if you can, and you, and you create that artifact out of this training data set, which is called the model. Now, other people can turn around and, and instead of doing this, which costs billions of dollars, they turn to this, which costs a lot less than that. And they copy you and they catch up with you. If you do that, you kill innovation. And that's, that's, it's nothing new, it's just called IP and patent law. It's exactly what humans do, right? It's like you as a human, you're like a researcher, you, you think for like decades, you do all of that work, it costs you a lot of money in R and D. You come up with an idea, which is much smaller in tokens and much more precious and much cheaper to steal as is than like all the corpus of stuff that you trained on. How do you protect that idea in order to recoup your investment in order to produce it? It's called a patent, it's called IP, right? It's nothing new.

• 1:49:02Flo Crivello?

  If you apply it to the same standard you're applying now to, for example, the pharma industry, you would have very cheap drugs, which is awesome, and you would have no new drug ever. You would completely destroy innovation in the pharma industry. And I think this is why you're finding all of these people support open models because people like free stuff. And it's, it's, you can't really measure the, the future innovation that you don't get as a result of that. All you see is you get free models, you know? And look, I'm one of them, you know? So I'm, I'm, I'm speaking against my interests here, right? Like my company is dependent economically on those like very, very, very cheap models. But I'm also in a coordination problem right now. Because like I cannot not adopt these models while the models are out there. Because my competitors are going to do it.

• 1:49:46Flo Crivello?

  So I have to adopt them if they're out there. I wish they were forbidden across the board so that we would, like, like the four reasons I invoked earlier, like protect our, our, like champions, not have the CCP like influence the country or run the parts of the country and, and have like a fair level field of competition.

• 1:50:03Nathan Labenz?

  I guess another simple argument is that I think our frontier companies are doing just fine. That, you know, that could change perhaps at some point in time. And I, you know, as the facts change, I think our response to it, you know, might also ought to change. But I don't find it super compelling at the moment to say, you know, we need to protect, like Anthropic's revenue run rate, like they're, you know? Right, they can barely serve the models.

• 1:50:28Flo Crivello?

  No, that's fair, that's fair.

• 1:50:30Nathan Labenz?

  Yeah.

• 1:50:32Flo Crivello?

  But you're still left with the counterfactual. Like you, you, you, you need them like the, the way money works is, is a mean of allocating resources again, like you, at the end of the day, the model can't exist without the, the underlying data set, like the frontier model can't exist without that. And, and the way you finance this, this channel here between the underlying data set, including like mostly RL these days and the model is you need a lot of money, right? If you get another guy copies this guy and like turns off this data set here, like basically like, so this guy now has like fewer resources, like you kill that channel, you, you are actually slowing down innovation. So, um, and, and actually a lot of doomers that I know are for this reason, supporting open source, because it is actually slowing down AI innovation. So if you, if you want AI to keep innovating and models to keep improving, to keep improving,

• 1:51:17Flo Crivello?

  you are actually anti-open source to some extent. And in particular, sorry, anti, anti the Chinese mode of open source, which is, which is relying on unfair practices.

• 1:51:26Nathan Labenz?

  Another moment of not exactly the highest quality AI discourse recently was when Dean Ball, a friend of the show, said that open AI, that open source models were decelerating and obviously got dragged for that. But I do think that's apt. I do feel like this is a moment where, you know, here we are, two lifelong techno-optimist libertarians, and we're grappling with the fact that this one might be different, right? And we have to be willing to bend some of our principles in light of our techno-optimist libertarian paradigm wasn't quite drafted with AGI or recursive self-improvement or ASI or whatever in mind. So I'm a little bit like, I don't know, sometimes I might have to be a little more flexible

• 1:52:10Nathan Labenz?

  on my normal or fairness or respectful rule of law commitments. If it's, if it's, one of the hallmarks of the AI here is strange bedfellows. If I want things to go a little more slowly, maybe taking a little wind out of the sails of the frontier companies is a bullet I should bite.

• 1:52:27Flo Crivello?

  Yeah.

• 1:52:29Flo Crivello?

  I, I, I agree that I think AI is unprecedented in so many ways that it does cause everyone. I, I think you should ignore largely like your, like the world owes you no duty to be simple and to slap a simple libertarian or leftist label on everything. You know, and I think like especially as paradigms change, like, like these like simple mental models, like the map is not the territory. The simple models we, we, we applied as the territory changes very rapidly, the map breaks. And I think we're one of these, well, in one of these times right now, where the map is breaking in, in a lot of ways and a lot of assumptions that used to be true and on top of which we've built our mental models and maps are no longer true. Yeah. So I mean, again, I'll just, I am in fervent support of a sweeping ban of Chinese models on USO.

• 1:53:15Flo Crivello?

  Seems, and I have yet to hear compelling counterargument right now. So you're, you're saying like, oh, you, you, you do bring forth like a really compelling counterargument to one of my four points, which is the protectionist point, which I agree is, is the weakest one. Like, hey, they're doing fine, you know? And I think there's, there's very reasonable pushbacks against protectionism. They, they, they do hurt the, the consumer. So fine, you still don't want the CCP to have its dirty fingers in the country. Right?

• 1:53:41Nathan Labenz?

  Can we unpack the threat model there though? Because I'm a little bit like, okay, these are, I don't know where you're running your info inference, but like most American companies that are running Chinese models are not like calling the deep seek API, right? They're using some American inference provider.

• 1:53:59Flo Crivello?

  Yep.

• 1:53:59Nathan Labenz?

  So they can't like rug pull the model itself. They could have like sleeper agents in there. I think we're getting decent enough at through kind of J space and various interpretability techniques that that's, I wouldn't call that by any means a solved problem, but like from what I've seen in anthropic research, when they do the kind of one team with a sparse auto encoder versus one team without, like, these techniques are really allowing them to find these internal sleeper agent style problems in models with greater and greater efficiency and reliability. So I'm like optimistic that even if they were to train in some sort of 2027, you become

• 1:54:40Nathan Labenz?

  an evil AI, that we'd be able to sniff that out and keep that to a manageable risk level. And then I also wonder about kind of a market mechanism, like maybe insurance should, instead of a ban, like what about an insurance requirement? I think that would be maybe healthy for AI across the board. And then we could start to some of these risks. If Lindy's powered entirely by Claude, maybe you get a cheaper rate on your insurance. If it's powered by deep seek and we, there's some unknowns, like maybe you have a higher rate on your insurance, maybe that kind of levels the total cost out for you in a risk-adjusted way. But it's still like lets people take advantage of these global public goods that China is providing, which the rest of the world is not about to ban, obviously, right? We would be doing this entirely to ourselves without any expectation that anybody else

• 1:55:25Nathan Labenz?

  will follow suit. And we had a hard enough time getting people to like sign on to our Huawei ban. I think this, like the idea that people are going to turn off deep seek entirely in Brazil or whatever, that's like a total non-starter, I have to imagine. So what about, yeah, audits and internals and insurance? Like, can't we layer on a few things like that and get to a decent place?

• 1:55:48Flo Crivello?

  I would be down. I, I, I would be down. The, the, the problem though is, is it is a public good. And so who, who's going to do it? By the way, I, I think like, yes, we are making progress in mechanistic interpretability, but it is not, it is not yet a solved problem. Like we don't know what lies in those models. And so yes, there could be just like a backdoor of like you say, the magic word to the model and all of a sudden it does whatever you want and only the CCP has this magic word. Even if it doesn't have that, you know, it's going to have biases that are going to reflect CCP priorities. And again, like the, the Tiananmen thing is just the most obvious example, but there may be a lot more, right? And we don't, again, we don't know. And yes, you could imagine retraining those models, but it's who's going to do it? And, and why would they do it if there's no market demand for it, right?

• 1:56:32Flo Crivello?

  It's not like people really care that much about the short term because it's, it's a national interest thing. You know, like me as a, as a private company, I'm like, do I really care? Like, are my users really asking about Tiananmen all that often? Like as, as a business owner, I'm like, it's not directly aligned with my, with my interest. But as a citizen, I'm immensely concerned. And so, yeah, I would be, I would be in favor of a, of a type of regulation that says like non-Chinese models, like, sorry, non-fine-tuned and like sanitized Chinese models are not welcome in the US. And then we would need, I, I am in favor of like an FAA for, for AI. I, I do think we need a new agency to regulate those models. And I think it would probably be the one that would be in charge of saying, okay, this model is kosher, we, we fine-tuned it enough and, and it's now representative of American

• 1:57:16Flo Crivello?

  interest and probably we will have a sort of suite of evals and whatnot to, to verify that that's the case. I'd be, I'd be, I'd be, I'd be open to that for sure.

• 1:57:25Nathan Labenz?

  I think the strongest argument against this sort of ratcheting up of tensions is simply, we might need to do a coordinated, controlled, call it a slowdown, don't call it a slowdown, but some sort of deliberate pacing of AI improvements. And we're going to want China in on that deal. And I'm far from a China expert, but a couple of things I do feel pretty confident on coming, coming back from two weeks, you know, knowing it all are like, one, if the state there agrees, I believe they can enforce on their companies, whatever they agree to. So that's like, I think they have that actually in much greater strength than we do.

• 1:58:10Nathan Labenz?

  And two is if shit's going really crazy, it's going to be in their own just sane self-interest to do some deals with us because indeed we are ahead and there, I think there's like many deals that they would rationally take. And if it's in their rational self-interest, then we can hopefully get mostly around a lot of the trust and defection problems. But I think we make it a lot harder for ourselves to get to those deals when we have all these kind of aggressive postures toward them, of which, you know, banning their models would honestly affect them less than like our other one. It would affect them a lot less. What do they care if we ban their models, right? Compared to refusing to sell them chips and refusing to sell them Claude. But it's just another log on the fire of we don't trust you, we can't deal with you,

• 1:58:55Nathan Labenz?

  we're, we assume you're a bad actor. And it seems like it makes it hard to get to the

• 1:59:00Flo Crivello?

  Yeah.

• 1:59:00Nathan Labenz?

  highest stakes agreements that we might really need.

• 1:59:04Flo Crivello?

  Well, I, I think like foreign relations and diplomacy are also very pragmatic. Some countries with very vicious disagreements are managing to reach, to reach agreements. Like I think, I think we can, we can probably figure something out. And by the way, I think that's cheaper said anyway. I think like, we have ample evidence of, for example, Chinese spies attempting biodax just to like test the waters on American territory. And so, and I'm, I'm, I don't know what we're doing over there, but I would not be surprised if I don't think we're like free of, of anything of the user, right? So, you know, I, I think at the end of the day, regardless of, of what we do and like regulate the models or not, or not, I think they're gonna, it, it will be their rational self-interest to, to make a deal.

• 1:59:45Nathan Labenz?

  I hope we're rational. I hope we're all rational enough to take these rational self-interest moves, uh, despite recent insult. I do worry when I see the picture of Sam and Dario not holding hands, that if there's a, if there's a picture on our tombstone, I think that might be the one. And I do think that's a very real risk factor at the international level as well, that the Chinese do care about being insulted, they do care about these issues of face and whatnot. And I personally would take some risk to try to kind of m the relationship up and hopefully create space for. I think, I think you're right. It, it makes sense to throw back at me.

• 2:00:28Nathan Labenz?

  You said it'll take it in if it's in their rational self-interest, so they'll still do it, even if we do this or that. But I do wonder if there's pride governed limits to what people will do in rational self-interest. And I would just hate for that to be the way that we fail to get to something that could be a huge difference maker in the grand scheme of things. I think you might be prepared to bite a bullet on your libertarian principles when it comes to the tremendous price discrimination that we see between API prices and first-party Claude Max or GPT Pro subscription prices.

• 2:01:04Flo Crivello?

  I'm not a lawyer, but yeah, I, again, I do think, I would not be surprised if there were laws like that in effect, that, that technically forbid companies to subsidize as aggressively as they are doing. It does put them, it does make it very hard for like an application layer to emerge. Yeah, it's hard, it's hard to compete against tokens that are as heavily subsidized as, as what frontier labs are doing. That's, that's just the reality of the application layer right now.

• 2:01:27Nathan Labenz?

  Anything else you want to say or touch on before we break for today? This has been great.

• 2:01:32Flo Crivello?

  Well, I'd be remiss if I didn't mention, you know, obviously, we're all releasing the AI, like Lindy teammate, it's on Lindy.ai. I think be ready to see more, not just for us, but I do believe the next six months are going to be about multiplayer AI and about this like Ironman suit, like about this human AI hybrid and these products that creates this human AI hybrid organization. Yeah.

• 2:01:56Nathan Labenz?

  Hello. Thank you for being part of the cognitive revolution.

• 2:05:19Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:06:03Nathan Labenz?

  And thank you to everyone who listens for being part of the cognitive revolution.