Living Lindy: a No-BS Conversation on AI Agents with Flo Crivello

Flo Crivello, CEO of AI agent platform Lindy, provides a candid deep dive into the current state of AI agents, cutting through hype to reveal what's actually working in production versus what remains challenging. The conversation explores practical implementation details including model selection, fine-tuning, RAG systems, tool design philosophy, and why most successful "AI agents" today are better described as intelligent workflows with human-designed structure. Flo shares insights on emerging capabilities like more open-ended agents, discusses his skepticism about extrapolating current progress trends too far into the future, and explains why scaffolding will remain critical even as we approach AGI. This technical discussion is packed with practical nuggets for AI engineers and builders working on agent systems. Sponsors: Google Gemini: Google Gemini features VEO3, a state-of-the-art AI video generation model in the Gemini app. Sign up at https://gemini.google.com Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive The AGNTCY: The AGNTCY is an open-source collective dedicated to building the Internet of Agents, enabling AI agents to communicate and collaborate seamlessly across frameworks. Join a community of engineers focused on high-quality multi-agent software and support the initiative at ⁠https://agntcy.org NetSuite by Oracle: NetSuite by Oracle is the AI-powered business management suite trusted by over 41,000 businesses, offering a unified platform for accounting, financial management, inventory, and HR. Gain total visibility and control to make quick decisions and automate everyday tasks—download the free ebook, Navigating Global Trade: Three Insights for Leaders, at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, my guest is Flo Crivello, CEO of AI agent platform Lindy. This is Flo's sixth appearance on the podcast, but only his second that goes deep on Lindy specifically. I reached out to Flo to request this conversation while I was preparing for my recent presentation on AI agents at Imagine AI Live, because I know that Flo lives and breathes AI agents as much as anyone, and yet, he always cuts through the hype and shoots me straight on what's working and what's not working yet. For comparison, that presentation, which we ran as a recent episode, is a higher level overview that I hope people will find valuable for structuring their thinking and perhaps even bringing leaders in their organizations up to

• 0:43Nathan Labenz?

  speed, whereas this conversation is a much deeper dive, fit for AI engineers, tinkerers, and builders of all types, and full of practical nuggets. Now, if you've been following the AI agent space, you know that it's been a bit of a roller coaster. We've gone from mind-blowing autonomous agent demos like Baby AGI and AutoGPT back around the time of GPT-4 launch in 2023, to a world today in which most AI agents, and certainly the vast majority of AI agents in production, are better described as intelligent workflows, in which AIs perform specific tasks that do require fluid intelligence, but nevertheless exist within a structure and control flow that was designed and built by humans. That said, while it's taken longer to get here than I expected, two things are

• 1:28Nathan Labenz?

  now becoming very clear. First, even in this more constrained paradigm, the practical value of AI agents is becoming huge, for use cases across customer service, sales, recruiting, inbox management, scheduling, and information synthesis. And second, the more open-ended, choose-your-own-adventure form of AI agents are also now starting to work. Flo describes a few that he's built for himself on the Lindy platform, but you can see this happening everywhere, from OpenAI's operator to Claude Code to Google's spreadsheet integration of Gemini 2.5, which I recently tried again for the first time in months and found to be dramatically improved. At the same time, these are only starting to work, and just how quickly these capabilities will continue to advance is, of course, hard to say.

• 2:10Nathan Labenz?

  Flo, as you'll hear, isn't convinced that we should just extrapolate Meters' historical trend analysis of AI task length doubling times all that far into the future, and perhaps as a result, he's not ready to go all in on multi-agent systems either. Beyond that, you'll definitely want to hear Flo's takes on model choice and managing model upgrades, fine tuning, rag, whether the tools that AI agents are given to use should themselves be intelligent and dynamic or dumb and deterministic, how Lindy helps users maximize performance by curating successful examples, and how and why, even assuming we do get AGI soon, Flo believes that scaffolding will continue to play a critical role in helping humans understand and control AI activity.

• 2:53Nathan Labenz?

  On a personal note, I'm also glad to share that I do now have a Lindy doing meaningful work for me. I've previously mentioned that we're experimenting with sponsored episodes, but to date, these have been very few and far between. Now, with podcast guest pitches from AI companies, PR firms, ramping up to a truly unmanageable volume, I finally decided to enlist the help of an AI agent to process them all. And so I built a relatively simple agent on Lindy that takes a PR firm email as input, researches the company they represent using the perplexity API, evaluates whether they're a legitimate AI industry leader that our audience would actually want to hear from, which by the way, does filter out a large majority. And then for those few that pass review, finds contact information for key executives via the Rocket Reach API,

• 3:37Nathan Labenz?

  and finally drafts personalized outreach emails explaining our sponsored episode offering, which are placed directly into the Turpentine Business Team's Gmail drafts folder for review and sending. One could, of course, build such a workflow on many different platforms, but I do find it valuable to see how an AI obsessed team like Lindy is thinking about these things. And Lindy does a particularly nice job of having AI set all the configuration values for you, of making sure that AI has full context at each step, and of allowing you to chat with individual instances of a task. In any case, this is exactly the kind of task that's perfect for current generation agents and that every business should be experimenting with now. Well-defined, multi-step, requiring both research and judgment,

• 4:22Nathan Labenz?

  and ultimately allowing us to scale a valuable activity that we previously couldn't. Of course, I'll be transparent with you, the audience, by always identifying any sponsored episodes that we do bring you, today's is not one, and I'll maintain my own agency by remaining personally responsible for the companies that we do and do not choose to feature on the feed. As always, if you're finding value in the show, we'd appreciate it if you take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or leave a comment on YouTube. We welcome your feedback via our website, cognitiverevolution.ai, and also by DMing me on your favorite social network. Now, I hope you enjoy this no BS conversation about the state of AI agents with Flo Crivello, CEO of Linde.

• 5:06Nathan Labenz?

  Flo Crivello, CEO at Linde, welcome back to the Cognitive Revolution.

• 5:10Flo Crivello?

  Yeah, thanks for having me.

• 5:12Nathan Labenz?

  Let's talk about agents. It's on everybody's minds. I've been studying the subject from a bunch of different angles, and I knew who I wanted to call to get an honest, real talk assessment of where we are in the development of agents. So maybe for starters, kind of a rudimentary question, but maybe at least wanted to set the terms and hopefully de-confuse the subsequent discussion. What is an agent? And what is it? There's so many different definitions, right? Everybody's putting forward their own definition. How much does that matter? And what's the definition that you work with?

• 5:44Flo Crivello?

  My favorite definition is Harrison Chases definition. By the way, first of all, I don't think the definition matters. I think you know it when you say it, right? You can operate perfectly fine with that, like being too nitpicky about the definition. But if you really insist upon the definition, I really like Harrison Chases, the CEO and the founder of Langchain's definition. He says it is software which at least part of the control flow is defined by an LLM. That's it. And so I think part of what I really like about the definition, and I think it also pinpoints like why it is sometimes hard for people to really define what is an agent, is like it's a spectrum, right? It's like the more of the control flow of the software is defined by an LLM, the more agentic the software is, which I think is the same as for humans, right? It's like, sometimes some

• 6:28Flo Crivello?

  humans in companies operate within very tight cartwheels, and like they're not very agentic, they're not like high agencies. Some of those are like life players or whatnot, and have like very high agency, right? Like if you think of like a Jensen Wang, like the CEO has like ultimate agency, right? Yeah, that's my definition.

• 6:47Nathan Labenz?

  Okay, so that would mean a, and I think the spectrum of definitions that I've heard maybe ranges from like Darmesh on the one hand said, basically, anything that you go to that's an AI that helps you get stuff done is an agent. He doesn't care if it's a fully deterministic workflow. You know, and then on the other end, you've got like, I heard a good one from Amjad, which is that it's an agent when it decides when to halt. Yours is like, I would say a little closer to Darmesh's in that any tool call would count, right? And any, any like fork in the flow, like if, if else type of logic, as long as that's entrusted to an LLM, you would count that.

• 7:28Flo Crivello?

  Yeah, I think like where my definition overlaps with Amjad's is like decides, to me like decides is like there is this brain, right? There is this, to me, the brain is the LLM. And so it's like, if you, if you, if you introduce the decision power made by the LLM inside your workflow, software, whatever you want to call it, it's agentic.

• 7:50Nathan Labenz?

  Okay, so not, I think this is maybe some we'll kind of return to as we go. Not too much there in terms of like autonomy or open-endedness required. And I think that is one of the reasons I'm kind of digging in on this is because I feel like when I talk to people who aren't super deep down the AI rabbit hole every day, as both you and I are, they have seen, you know, your sort of baby AGI type demos and your, or maybe your chaos GPT type demos, and they've kind of latched onto that in their minds of like, oh, wouldn't it be amazing if I could just give like a real simple instruction to an AI and it could just go do everything and

• 8:32Nathan Labenz?

  figure it out and come back to me when it's done, right? The old kind of come back to me when it's done. That I feel like is leading people often quite astray in terms of what is actually realistic today and also maybe in terms of just like how much work is really needed to, if nothing else, like assemble context for your agent or guide it to the necessary context so that it has, you know, a chance of doing what you want it to do accurately. But maybe you're going to destroy my worldview here in a second with telling me that you actually have like tremendously open-ended Lindy's running your life. So keep that in mind. Tell me, what's it like living the automated Lindy lifestyle today?

• 9:14Flo Crivello?

  Yeah, I wouldn't say running my life, but like, yes, I have a couple of very open-ended Lindy's running. I think this definition of agent is just like on the far extreme of that spectrum that it's just defined. Like when you let L L Ms take a lot of the decisions, then it's very agentic, you know? And so that's just this definition. I was thinking about it actually the other day because the very first version of Lindy perhaps was overestimating the L L Ms capabilities, was definitely overestimating the L L Ms capabilities and was just these open-ended agents. And so since then, we've actually backtracked and like let you set up more of that deterministic scaffolding where you can really force the agent to go through a deterministic set of steps. Where it's like,

• 9:59Flo Crivello?

  hey, suppose you want a customer support agent, like you're going to receive a support ticket on Intercom or Zendesk or whatnot, and you want it to check your knowledge base on Notion before entering the support ticket. You're not asking it. You're not asking the L L M, please, right? I beg you, L L M gods, like, go ahead and check this knowledge base. Like, you want that to happen deterministically all the time. You don't need to be hard baked to the cognitive structure of the agent. So now, like, the current version of our product does let you decide however tight you want your guardrails to be, when do you let the agents roam free and so forth. I mean, I have some agents, like, it's funny, actually, my meeting scheduling agent is just a pure agent, like, it's got very little

• 10:44Flo Crivello?

  to no guardrails. It's just like a very big prompt where I tell it the rules and I tell it how I like my meetings to schedule and so forth. But within these boundaries, it does pretty much anything it wants. I have another agent that, this one's going to sound funny, but... Or like very minor, but it wakes up every week. So every Monday morning, it woke up a couple of hours ago. And it checks whether there is a new podcast from my favorite podcasters, who are Darkesh and Lenny's, and obviously the cognitive revolution. And, and, and it, it looks for the podcasts and then once it finds them, it summarizes them and it has been a summary. That kind of has to be an open ended agent because this concept of like, look for the podcasts, because there's no one source of truth.

• 11:29Flo Crivello?

  So you gotta, can't just like, look for the podcast, go on YouTube, go on iTunes, just like, figure it out, you know? And then once you found it, like, ping me. So that's an open ended agent.

• 11:40Nathan Labenz?

  One of the things I noticed as I was using the product recently is you've built a bunch of primitives, basically, like, you know, lots of integrations now, for one thing. So, you know, it's, it's a much, as time has gone on, it's, it's become much more like a zappier in terms of just like, depending on whatever app you want to use, like, there's a, you know, increasingly a good chance it's going to be there. But then you have your own primitives that are like, search the web for something and not too much detail given. I obviously kind of want to make it, you know, seem like free and easy for the user, I guess, and maybe sort of keep some of your secrets for yourselves. But I guess I have a couple of questions on that. One is like, is that an agent?

• 12:24Nathan Labenz?

  You know, and if so, like, aren't we kind of running into some troubles in terms of what we're,

• 12:33Nathan Labenz?

  how we like draw boxes around these things? Because this is something I'm, I'm really looking at from a lot of angles in terms of like, liability and, you know, testing and sort of even just design, like, you know, what's on me, what's a tool? When I call your like Lindy official, search the web for something, should I think of that as kind of an open ended agent? Like, it seems like it might be doing like a bunch of steps in there and I don't really know what they are and it's kind of coming back to me when it's done.

• 13:00Flo Crivello?

  No, it's literally a Google sales. It's literally Google sales. We, we, we try to be careful. There is an agent and then there is the tools that the agent uses. In the early days, when we were like much more exploratory and trying to figure out what even was an agent, like, we did experiment a lot with agentic tools. So you can almost think of it as like a sub-agent. And we learned that's a pretty bad idea. Like, you really should draw a pretty sharp line between your agents and your tools. And your tools should not be agentic, basically. And the reason for that is because it's, one, empirically, it just doesn't work well. Two, it just makes it very hard to reason about your system because you're on trading. Like, it's hard enough to make one agent work. Now you need to make two agents work.

• 13:43Flo Crivello?

  You need to figure out how to get them to interface with each other, which is really tricky. Um, so anyway, no, I mean, the search web action is just a Google search.

• 13:51Nathan Labenz?

  Okay, interesting. So does that make you bearish on things like all these new agent protocols that are coming out? I mean, people are talking about, of course, A to A from Google. MCP is meant to be more of a tool thing, but I'm also seeing the pattern increasingly of smart MCPs, not that often yet, but like, a familiar one that I've mentioned a couple of times is Augment created a sort of clone of Claude Code. And to replace, in the Claude Code blog post, they say like, well, we have a planning tool and it can call the planning tool. And Augment was like, well, we don't have a planning tool. What should we use? Should we make our own? They looked online, they found Pietro's sequential thinking, or whatever exactly

• 14:36Nathan Labenz?

  it's called, tool already packaged up as an MCP. And so they're using that. So they have the like coding agent, but then it's tapping into this other sort of smart planning MCP, which obviously has, you know, interesting questions around like, what context is it fed? And how much of its thinking do you get back? But would you call yourself bearish on all these sort of multi-agent frameworks at this point?

• 15:00Flo Crivello?

  No, I wouldn't call myself bearish. Like, I've been very excited about multi-agent systems for a very long time. I do think, though, they are a lot younger. Like, it's much harder to make a multi-agent system work than it is to make one agent system work with like a bunch of tools. It's just, it's just much, much harder. I think part of that is the models haven't been really ready. I think part of that is the tooling hasn't really been ready. And I think perhaps part of that is we've not really had, I'm sure these protocols are going to help. And I am, I am excited about Google's work here. It seems like something like this is going to be necessary. It actually really reminds me of in in supply chains and like e-commerce and shipping and all of that. And logistics, they have this protocol called EDI. It's a very, very, very old protocol.

• 15:45Flo Crivello?

  That's like, it lets people say like, it lets you formalize your relationship with the supplier. So it's like, hey, I I bought this from you. Now you're gonna confirm that you like the purchase order went through. And now you're gonna confirm that the shipping went through. And oh, actually, quantity is different and all of that stuff. So it's very formally defined. And look, that's the backbone of the logistics of the modern world. Like, the whole world runs on EDI. So I do think we are going to need something like that for intelligent communication. I think it's going to help a lot to make these systems more sturdy.

• 16:18Nathan Labenz?

  Okay, we're just not quite there yet. Maybe they are in the same spot that you were a year ago, where they're just a little bit ahead of the game, basically.

• 16:27Flo Crivello?

  It's early, it's early, for sure. But I mean, like, look, I I think it's it's close enough to start thinking about them. I have some multi-agent systems that I use on a day-to-day basis in production.

• 16:37Nathan Labenz?

  Okay, tell me more.

• 16:39Flo Crivello?

  Well, it's going to sound like toy examples, but I have, so obviously, Lindy is in all my meetings, okay? And she sits in the meeting, she takes notes, she does a bunch of stuff. It's like my my meeting, my meeting note taker in this is huge. And one of the things she does is going to sound weird, but I do a lot of interviews with candidates. And when I I chat with a candidate and it's decided to be not a fit, and I just decide to like pass on them, like during the interview. Sometimes it's more ambiguous, I'm going to talk with the team because I'm like the last round of interviews. But sometimes it's just like, this is, I'm gonna veto this candidate, you know? And so what I do is, the candidate leaves the meeting, and my Lindy is still there.

• 17:21Flo Crivello?

  And so I talk to my Lindy, and I'm like, Lindy, let's just pass on this guy, send him an email. And so what she does is, so that's my Lindy note taker. I have another Lindy that I call my Lindy Chief of Staff. And that Lindy just does a bunch of things for me, like exactly this kind of task. She knows what I mean when I say pass on the candidate, she does a bunch of things. She like, first of all, she doesn't send the pass email immediately, she waits a couple of days. If this candidate had been introduced to us via recruiter, she also shoots a note to the recruiters to let them know. Like, there's a lot of stuff that she does behind the scenes. And so when I do that, my Lindy meeting the techier sends a message to my Lindy chief of staff, tell her like, hey, Flo wants you to pass on this candidate. Can you please, can you please do so? So that's like a simple example, but it's real. Like I use it on a weekly basis.

• 18:05Nathan Labenz?

  How gnarly do those things get in terms of the overall control flow?

• 18:12Flo Crivello?

  Well, I mean, the agent delegation and collaboration itself is just an extra step. But now agents as a whole, regardless of like the multi-agent system can get very gnarly in terms of orchestration, can get very, very complex. Yeah.

• 18:28Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:34Nathan Labenz?

  Maybe before digging into another example or two, of course, everybody has seen the meter graph. That's been kind of the talk of the town. In fact, I think we were together the weekend that that

• 18:46Nathan Labenz?

  dropped or no, that was maybe our revenge that they put out at that time and they circled back to their their doubling time for agent task length a little while later. I was thinking of the curve. I know you know it, but basically, you know, they've gone back in history, looked at the task length that agents could do 50% of the time,

• 19:05Nathan Labenz?

  plotted a straight line on a graph, determined that the doubling time is every seven months. More recently, there's been, hey, maybe that's even kind of sloping up a little bit, maybe the doubling time now, if we are indeed in a different regime, look like maybe four months. What's your thought on the increasingly infamous meter graph?

• 19:24Flo Crivello?

  I don't have a very strong opinion on it. I think it's very dangerous and I say that as like I'm very AGI peeled. And so I'm always the one to actually talk about like the exponential takeoff and all of that stuff. But I, I will acknowledge like it is dangerous to draw like conclusions and to drive these lines on like a log graph when you have so few data points. I think in the case of AGI, you actually do have another data points. You have like 60 years of morsel. So like you just know that compute is going to, is going to keep increasing. And in the case of AI, like you just, at this point, like we've gone through, I don't know how many, like five to 10 orders of magnitude, like, and, and we see the scaling law just keeps working. So I think we have enough data points to draw that line on that log graph for AGI.

• 20:09Flo Crivello?

  I don't think we've got that for agents yet. Yeah.

• 20:15Nathan Labenz?

  So how would you say that that lines up with what you have seen as you've created a thousand and one Lindys over the last year?

• 20:24Flo Crivello?

  Oh, I share the empirical observations so far over the last two years, right? I have seen the same trend described by this line. Like, I don't dispute the past, it is the future protection that I have a question knock on. I just don't know if I can keep drawing this line, although it's going to be linear or exponential. But yeah, I mean, like, look, when we started this, and like, I mean, you know, like, we started pre-GPT-4, we started with GPT-3.5. Like, in hindsight, like, we were, well, I wouldn't say too early, because it was only two years too early, which I actually think is the right moment to start to start that. But like, definitely, the company, the agents didn't work, they really didn't work. It was so dumb, like, 3.5 was profoundly dumb. Then GPT-4 came out, and then things changed. And GPT-4 was too expensive, and it wasn't as good as the new models.

• 21:07Flo Crivello?

  And now we have, like, class 3.7 Sonnet, and we've got Gemini 2.5 Pro. And like, these models are just like incredible, and actually very, very fast, very smart, very cost-effective, huge context windows. So I have seen all of that happen over the last two years. And yes, I have very high expectations for the next two years. I don't know if they're going to be exponential. I don't know if they're going to be as strong as what happened over the last two years. But yeah, I do expect, I do expect agents to get better and better.

• 21:34Nathan Labenz?

  So would you say, of course, it's important to keep in mind that 50% versus, you know, a sort of 99%, and that is a pretty big gulf.

• 21:44Nathan Labenz?

  Where would you say, in terms of like, task length or complexity, or however you think about it, people should aim, if they want to put points on the board, you know, as a new user of Linde, like, what's something that is sort of the maximum that can, you know, be practically valuable, maximum practical value, while being confident that you can get this to work? How do you guide people?

• 22:09Flo Crivello?

  Yeah, it's a slight reframe around the question, but I think of it more as ambiguity of the task results and task length, which I'm calling it a reframe, because fundamentally, it's the same thing, right? Because basically, if you can describe something as a sequence of steps.

• 22:27Flo Crivello?

  Basically, that's a low ambiguity workflow. It's just like, I kind of understand this workflow and I can describe it in this sequence, right? Again, that's basically sort of the equivalent of saying it's a short task length, because it's just a succession of short tasks length. But I think thinking of it in terms of this succession is important, because that actually allows you to cover a much broader set of work tasks, right? Because many work tasks are actually, few work tasks are like just in and of themselves a 30-second task. But actually, a surprising amount of them is a succession of two-minute-long tasks. So with that in mind, like, yeah, I think anything that you would feel comfortable giving to an intern with like a Google Doc, and the Google Doc describes a succession of steps, and it's like,

• 23:10Flo Crivello?

  honestly, the length of the Google Doc doesn't even matter, right? Because like, all that matters is the maximum complexity of any of these steps. And I would say even that doesn't matter a whole lot, because modern AI agent builders, and this is only the case of Linde, have this concept of human in the loop. So you could totally just build your Linde, basically turn that SOP that's Google Doc into a Linde. So then you have a huge Linde. And then if you detect that like one step is particularly risky, you just toggle human in the loop on that step, and now you insert yourself. And like, actually about to start something that's like, put it, I see or RHF in context reinforcement learning from human feedback, such that if you toggle on, ask for human confirmation on any step in your Linde,

• 23:55Flo Crivello?

  she learns from your feedback little by little, and she actually learns really quickly. I mean, you've seen the same papers as I have about in-context learning, like, it is surprising the in-context learning ability of these models.

• 24:07Nathan Labenz?

  Interesting. So that's a feature that basically helps people curate the gold standard examples by doing it bit by bit over time?

• 24:16Flo Crivello?

  That's right. That's exactly right. You're thinking about it in the right way.

• 24:19Nathan Labenz?

  Yeah, I really like that, because...

• 24:22Nathan Labenz?

  Boy, is it hard to get people to sit down and bang out some gold standard examples, I find. I have had quite a few adventures on that front, actually, and it's been really eye opening just how some people just, you cannot get them to staple their pants to the chair and do it. I don't know. It's very, very weird. And that remains, I guess, my number one tip for performance optimization. I mean, obviously, write, you know, good, clear instructions is, is key. But, you know, presumably, most people are at least able to like, sit down and bang out a couple paragraphs of instructions of what they want. The gold standard examples typically don't exist, I find, you know, or they're so fragmented

• 25:05Nathan Labenz?

  across context, or, you know, the chain of thought was always kind of in their heads. All those problems really hold people back. So you've, I really like this idea of just basically starting off in a human in the loop paradigm, having people come in and review and or fix, and then compiling those and building up longer prompts that drive performance by leaning on those examples. That's really good. Is that the number one driver? Few-shot prompting, is that the, still the biggest thing?

• 25:36Flo Crivello?

  Big time, absolutely. Yeah, yeah. Much better than instructions.

• 25:42Nathan Labenz?

  Any push into fine tuning? I mean, that would be obviously the next step at some point, right?

• 25:48Flo Crivello?

  No, maybe at some point, one day, like, I, I just think the models have become so good, I feel like fine tuning is a little bit of a thing of the past, isn't it? Like, we heard a lot more talk about it a year and a half ago than we do today. Like, we used to have a fine tuning team that we got, like, frankly, because, uh, well, part of the issue was just the open source models did not deliver in the same way that we hoped for. And fine tuning went from a world of like, look, again, like, Linde used to network. And so we're like, ah, like, the models are not ready, we, we need to fine tune them to have agentic behavior. And then the models worked. So now it just became like a nice to have. I, I just don't think the juice is worth the squeeze for fine tuning, for the vast majority of use cases.

• 26:33Nathan Labenz?

  Yeah, I mean, OpenAI just put out their reinforcement fine tuning, you know, to, if not everybody, at least like a lot of accounts this week. So it seems like they haven't given up on it yet. This has been one of the biggest divergences, though, between them and basically everybody else, right? Like, Google has made sort of a token effort at most. Claude allegedly was going to allow you to fine tune Hikou at one point. I still don't think I've ever been accepted into that program. And so basically, OpenAI is kind of leaps and bounds ahead of anybody else in terms of their fine tuning offering, but they must still be seeing something from it, right? To be, um, to be pushing something like that all the way to production.

• 27:13Flo Crivello?

  Well, first of all, it's a very good company. They have a lot of things on the stove top to infer too much. And also, look, I'm not saying fine tuning is completely useless. I think, first of all, if you operate at scale, that's the first requirement, because like, there's like a very high fixed cost of fine tuning, that you've got amortizable, a large volume. So if you operate at scale, and if you have an important, a critical part of your workflow that you are looking to make faster, cheaper, more reliable, and if that part is sufficiently narrow on the task that is trying to perform. So I'm looking at like RAG, use cases very often have that in terms of like re-ranking and prioritizing and all of that stuff like, then you probably ought to fine tune a small model and insert it in that workflow.

• 27:59Flo Crivello?

  I, I think I have heard that, uh, Kursur and Windsor have at least part of their workflows that's using a fine tuned model, but I'm not sure.

• 28:09Nathan Labenz?

  Yeah, I think the point on narrowness is definitely really key. I mean, it's tempting in some cases to try to imagine creating like the fine tuned model for our company that does everything for our company. And that, in my experience, is not the way to go. It's much more about just nailing down.

• 28:27Nathan Labenz?

  With clarity, you know, what is the desired behavior on something that really matters? A good example from the reinforcement fine tuning docs that I was reading this weekend was just like from healthcare, some sort of doctor notes, you know, transcript of the appointment to like a diagnostic or I think it's even a billing code, which is like super gnarly stuff and obviously accuracy, you know, really super important there. So that kind of thing often I think will work and hopefully will push the frontier of like what people can actually do in these various frameworks. But the narrowness piece, I think definitely resonates a lot with me. What would you say are like the most, you know, by if you were to maybe wait by the actual number of tasks as opposed to the number of Lindys?

• 29:10Nathan Labenz?

  Like what's driving the bulk of the value through the system today?

• 29:14Flo Crivello?

  I think if you look at it by task, it will almost, it's going to be ironic. Like if you look at it by task, they're going to look at like the least important use cases. Because almost by definition, if there is a Lindy, well, it's not the least important, but it's going to be like very high volume, very, very, very small tasks. So most likely if we do that, like it's going to be like an email task or like a Slack, a Slack task, it's going to be one of these two things because those two things are such high volume. So we see people deploy Lindy to automate their email workflows. That's like a big use case of ours. So like email triaging, email drafting, if you receive a lot of proposals by emails, like Lindy can like look at the proposal and reject it practically if it's not worth you looking at and all of that stuff. So we've got a variety of use cases here.

• 29:57Flo Crivello?

  That's, that's probably going to be like the biggest, the biggest use case.

• 30:01Nathan Labenz?

  I mean, that does resonate with me because that is often where I tell people to start, like something simple, relatively low risk, high volume, put some points on the board. What if you reweighted by credits consumed?

• 30:13Flo Crivello?

  If you reweighted by credits consumed?

• 30:17Flo Crivello?

  Well, I think it's going to be one of two things. The first one really pertains more to our credit system, which is we use prospecting APIs for like lead generation and those are very expensive. So we told you a lot of credits for that. But I could show you, like I had like a recruiter Lindy that I talked to and I'm like, hey, find me thirty engineers working in San Francisco at this or that company. And then she uses this prospecting APIs to find these thirty engineers. And it's forty cents per engineer, per lead, you know, like right here, that's like twelve dollars, you know? And then she's like, okay, I found them. And I'm like, okay, send them an email, all of them. And so right here, again, like if it costs me ten credits slash cents per outreach, you know, that's gonna cost me three dollars.

• 31:03Flo Crivello?

  That's one. I think the deep research use cases are quite big. And and I'm using it as a as a portmanteau for like a very broad category of use cases. Like anytime you want your agent to review, consume a large amount of data and then do something with it, I think agents are, by the way, excellent at that. It's like one of such killer use cases because it's just that good at reading tokens fast, you know? And like if a human had to read these tokens, it would be very slow and very expensive. And then she can like write a report about that. So like one of my favorite use cases for Lindy is we have this Lindy that you can basically think of her as like sitting at the interface between the company and the outside world.

• 31:47Flo Crivello?

  So she reviews, she is in every customer call that we have, every prospective customer call, every support ticket that we answer. And at the end of the day, she writes a report based on that interface between the company and the rest of the world, which I think is a very important interface. And she's like, hey, this is what's happening. This is what's happening with sales pipeline. This is what customers are saying. This is the issues we're having in the support inbox and so on and so forth. Yeah. And that that's a token guzzler. It's like hundreds of thousands of tokens every time.

• 32:16Nathan Labenz?

  Is that the same one that I interact with when I talk to the chatbot on the site for help?

• 32:23Flo Crivello?

  Uh, it's not exactly the same, but yes, that Lindy in question does also ingest these interactions.

• 32:30Nathan Labenz?

  Gotcha. So when I talk to that, it becomes that that sort of log becomes an input to the higher level summarizer.

• 32:37Flo Crivello?

  That's exactly right. And then like, if a lot of people talk about the same thing to that Lindy, like that's gonna come up in the end of the digest. And so that's awesome. Like, and by the way, it sends the digest into the general channel on Slack. So it's just such an awesome, it's like the heartbeat of the company, right? It's like you can think of it as like, ingest all the context, broadcast it back, ingest, broadcast it back, right? And just does that every 24 hours. And as a result, the whole team is insane. It's really powerful.

• 33:03Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 33:08Nathan Labenz?

  I, uh, my compliments to the chef, I guess I'll say on the onsite chatbot. It was actually helpful and was able to respond in a way that felt like I was actually talking to something intelligent. I mean, so many, it kind of strikes me today that like, and I'm not really sure why this is, you know, obviously just inertia is a powerful force, the old, um, Tyler Cowen, you know, you are all the bottlenecks.

• 33:38Nathan Labenz?

  But man, chat still sucks on most sites, right? It's like, it's not good. When I went to yours, I was quite impressed that it was actually a natural conversation. It actually had relevant answers to the questions. And then at the end, I was like, can you forward this to the team? And it said like, yes, okay, I've done that. I've forwarded it on to the team. And I was like, wow, that's pretty, um, you know, it felt, uh, definitely much more like the future and kind of where I think a lot more people I would have expected to be at this time.

• 34:11Nathan Labenz?

  Why aren't more people here? What's, I mean, aside from just the general slowness of life, I feel like people have tried, but they've, they've often like failed to make these things work as well as, you know, a few people have demonstrated that they can.

• 34:26Flo Crivello?

  Yeah.

• 34:27Nathan Labenz?

  What accounts for that in your mind?

• 34:30Flo Crivello?

  Well, first of all, and I don't say that like pedal my shit, but like that chat is a Lindy. And so we've just spent a long time, like one crafting the platform like as builders of Lindy. And then as builders of this Lindy, as users of the platform, we've also invested a lot of time in that Lindy. And we know how to build good Lindys because we've built Lindy. And so it's a really good Lindy. It's big. It's got a lot of prompts, like the whole scaffolding makes a lot of sense. It injects the right context at the right time from the right source. Like, it's just a complex Lindy that we've spent a long time crafting. And it uses good models. I sometimes suspect that companies in a misguided effort to save money or using really bad models for these chatbots.

• 35:14Flo Crivello?

  And I think you should not, I think you should just like, you should have the best possible model. Well, not the best possible, because today it's like oh, one or three, and this is going to be very expensive. But like, come on, like give your customers like a Gemini 2.5 pro. It's not that expensive and it just performs extremely well. So, um, yeah, we just, we just know how to build good agents. But I, I think like, thank you, I will take the compliments and I will, I will pass them to the chair. She's me, it's not me, so do. And I think that's an indicator of the current capabilities of these systems, which I agree with you. I think are underpapped. It's, it's really crazy just what is possible today that is not yet really exploited by 99% of businesses.

• 35:51Nathan Labenz?

  So tell me a little bit more about context. You said like it injects the right context at the right times. That's along with like the difficulty of getting people to actually buckle down and write some gold standard examples. Generally speaking, the, the challenge of assembling context or accessing context.

• 36:09Nathan Labenz?

  Also seems like a constant theme when I talk to people who are trying to implement stuff.

• 36:15Flo Crivello?

  Yeah.

• 36:15Nathan Labenz?

  What, you know, aside from just kind of many iterations, what lessons would you say you've learned? Or, you know, what, again, what tips would you give to new users about how to muster the right context at the right time?

• 36:30Flo Crivello?

  I think it is another iteration. I think you do enough rights that you end up building an intuition. And I think that intuition is on, there is a balance between using just similarity like vector search, like BM 2.5 and all of that stuff to like search your knowledge base. And on the other hand, hand crafting exactly what to search and what and when and in what knowledge base. And the more you use these products, the more you understand where that balance lies. And so very concretely, like we've got that Lindy chatbot that assists our customers. And the customers ask it all sorts of questions. And so we know that if they ask a question about billing, it's like refunds and like how do credit work and all of that stuff.

• 37:13Flo Crivello?

  We've got a specific portion, a segment of our knowledge base that's specifically about billing. So we're going to have a branch there that's like, here's a King of Billing question. Okay, now you, you ask, you, you, you consult this knowledge base. And this is the kind of query that you draft for this knowledge base. I'll also say, by the way, another intuition you build is like when to not even use a knowledge base at all. And like when you are not very conscious about or worried about saving money. I think more and more like, look, I hate to be the guy who says like rag is dead, but it's, it's, it's not dead, but it's limping. Like there's a lot of these cases that we have where like we don't use knowledge bases anymore. We just were like, hey, these are like the five or 10 pages we have about billing. It's not that much.

• 37:55Flo Crivello?

  You know, it's like a couple of thousand tokens, like these are the tokens, just address them all at once in your context window. You know, we don't do it for the, the support about because it does get expensive.

• 38:06Flo Crivello?

  At least for now, but yeah, anyway, it basically becomes this hybrid between a handcrafted pipeline, like a handcrafted rag pipeline and like a BM 2.5, like vector similarity search.

• 38:20Nathan Labenz?

  So maybe people should be thinking like, how can I 80, 20, or 90, 10 this where I will actually create a top end categories of situations I want to handle, branch into those, hand curate relevant context, whether it's the five or 10 pages about billing or what have you, and then kind of have one catch-all bucket at the end that's like, okay, for that, you can just search through this knowledge base, but then you maybe kind of increasingly pull out of that, you know, and you sort of minimize that bucket as you go. Is that kind of what the, what you would recommend in terms of the iteration cycle?

• 39:01Flo Crivello?

  And in the end, it ends up happening very naturally, because what ends up happening is like, you create your Lindy agent, and you deploy it, and then you monitor it. Every so often, you check in on it, you look at what it's done, and then you're like, ah, this was really dumb, like, this is not how you should answer this question. And so you go back, right? And you edit the Lindy, you edit the prompt, you add steps, you modify the knowledge base, you just like tweak it around the edges, and you're inside and repeat. And I find it funny that there is a sort of natural instinctive reluctance that people have to go through this loop. There's something about it, I think it's just not instinctive. But when you consider the time that you invest to onboard a new

• 39:43Flo Crivello?

  teammate, a human teammate, it's a lot. Like, training a human takes weeks for a human to fully, to fully really ramp up. So I actually think agents are easier than humans to onboard. It's just a less natural mode of interaction, because a human, you can just go to them, you're like, like, hey, don't do this, do that, moving forward. Like, with an agent, you've got to know how to use this LoDitor, and you've got to build that intuition that I just mentioned. That's not always going to be the case, by the way. Like, soon, we're going to announce something big that's going to make it, make it a lot more natural to iterate and improve on your agents. But yeah, I would just say, like, just, just iteration.

• 40:21Nathan Labenz?

  That almost sounds like a memory module. That's been a space that I've been watching really closely. What's your take on sort of, I mean, there's been a lot, right, of different kind of frameworks for memory, whether it's like graph databases. I did an episode on HippoRAG. There's HippoRAG 2 out now. Then, of course, there's like more inherently like neural structures, which, you know, could just be a vector database. But we've got an episode as well on Titans, which is like building an MLP into the thing and updating, you know, that MLP so that it can sort of retrieve from history.

• 41:00Nathan Labenz?

  ChatGPT is doing its own thing. We don't know exactly what it's doing, but it's currently got a, it's at least got a mix of the like explicit saved memories that you can go and read and then some sort of like more vague, like nebulous. You know, it'll, it'll check in with your chat history and, you know, they don't really tell you exactly how that's working under the hood.

• 41:20Nathan Labenz?

  What paradigms for memory are you most excited about?

• 41:24Flo Crivello?

  Um, I think this is one of these things, like I read all those same papers. I've seen like the Hippo and Hippo 2 papers and so forth. It's very exciting, but like, I think this is one of these things where number one is the bitter lesson comes for us all. I think as, as models become better at having more context and at fully utilizing this context, I think like all of these systems become moot because you can just throw it all in the context window and I think this can be just fine. And also like I'm, I'm a big believer in simplicity when it comes to these systems, because the more moving parts you introduce into the systems, the harder they are to reason about and to debug. There's this principle of engineering that I really like that states you need to be twice as intelligent to debug a system as you do to

• 42:08Flo Crivello?

  design it in the first place. So if you are operating at full intelligence when you're designing the system, you're going to be unable to debug it. And I think that's the case of like the, all of these fancy memory systems. It's like you guys are operating at full intelligence here. I can't figure this out. Like I have to really sit down to understand the systems in the first place. Like I can't debug it. I don't think you can either. By the way, that's always the, the problem with like these academic papers. It's like none of them is really building with that constraint in mind, which in my experience, when you're building systems that go into production is actually easy defining constraints that you need to keep in mind. So with that said, like my understanding of what chat GPT is doing, memory system there, is that it's actually the simplest system out there that also is operating at the greatest scale.

• 42:53Flo Crivello?

  And I don't think, again, to my point, that is a coincidence. And I think what they do is literally just like they take conversations, they determine whether there is a memory that's safe worthy in that conversation. If so, they use an LLM to distill down the memory to like a short sentence or two. And then they just inject all of that into the context window. They may go one step further, but honestly, I don't think so. They may go one step further, which is perhaps assign an important score to the memory. And so you could imagine like, hey, you've got so many tokens worth of budget in the context window for your past memories. And you're going to prioritize based on like that, that, that priority score that you've defined before.

• 43:39Flo Crivello?

  And then you could imagine, again, you can go slightly more fancy. You could imagine a sort of like decay with time, you know? And so you could, you could come up with a composite score between like the priority score and like the time score. And maybe like the older the memory is or the lower the priority is like, you just like allocate like fewer and fewer tokens to the memories. Maybe you save like multiple length worth of representations of each memory. Like, that's the kind of thing I'm thinking about. But even like that incarnation of that system, which by the way is purely conjecture. Um, it's very simple. I think it's really simple and I, I think that's just how it works.

• 44:15Nathan Labenz?

  It's pretty similar to what O3 guessed when I asked it. It, it guessed that it was doing some sort of vector search. Well, it kind of went back and forth between like distilled and then vector search, or just like chat history direct into vector search. But it did have a vector search component in its guess.

• 44:34Flo Crivello?

  I would bet you, I'd bet you a lot of money that there is a vector search there. If there was a vector search, it wouldn't be able to retrieve when you say, what do you know about me? It wouldn't, it wouldn't be able to retrieve it. Vector search wouldn't be able to retrieve that. Unless it's a really fancy RAG pipeline, like a hyper-

• 44:49Nathan Labenz?

  Could it be a custom retriever?

• 44:52Flo Crivello?

  It's simple. It's simple. I think it's a very-

• 44:54Nathan Labenz?

  They do have a ten, I mean, certainly I think that's a good prior for all the things that the leading companies do. They definitely have a strong bias toward doing the very simplest thing and just applying a lot of compute. So I think you're certainly right to use that as the jumping off point.

• 45:11Flo Crivello?

  Yeah, you know, look, the other thing is like they all building on shifting ground. Because like the entire underlying paradigm is changing every three months. And so the more complexity you bake into these systems, the more assumptions you bake into the systems. And so the more brittle they are to future paradigmic changes.

• 45:29Nathan Labenz?

  Yeah, interesting. How do you think that will impact the Frontier Lab versus, you know, API powered developer? As we go into the future, right? There's, there's of course been multiple rounds of the debates around like who has modes? Where does value accrue, et cetera, et cetera? It seems like the, I mean, take OpenAI specifically, like they're both going toward chips on the one hand and toward like buying Wind Surf on the other hand, right? And kind of trying to be a real full stack vertically integrated provider.

• 46:07Nathan Labenz?

  Do they, you know, is this, do we, how do we escape like total big tech victory? You know, big tech black hole of value?

• 46:17Flo Crivello?

  Yeah. I mean, I really think of Sam Altman as like Bill Gates too, basically. I think in the, the scope and the breadth and the nature of his ambition, he's very similar to Bill Gates. Like if you study Microsoft's history, it's remarkable, right? Like they started as this like basic compiler. And then they almost stumbled upon the operating system. But they didn't, it's not like it was just like, oh, pure luck. Like it's Bill Gates, his modus operandi is very much like we want to own the whole stack. So he was philosophically open to the operating system as well as to the compiler as well to the applications and so forth. Right?

• 46:56Flo Crivello?

  And so he really started as his whole charter to like own computers, right? Personal computing, like we own the whole thing, right? And so peripherals, yep, well, I'm gonna do it. I'm gonna do the mice and the keyboard, like operating system, absolutely. Like modeling software and like. Like different, like security software and like application layer and like yep, yep, yep, yep, yep, yep. We're gonna, we're gonna do it all. We're gonna own the whole fucking thing, you know? We're gonna be like an index stock on computers. If you believe in computers, you gotta believe in Microsoft, you know? So I think that's what Sam Altman is going for. Look, yep, that too. You know, we're gonna do the compute, we're gonna do the API, we're gonna do the applications, we're gonna do the code, we're gonna do it all, we're gonna do it.

• 47:39Flo Crivello?

  Now, um, look, I mean, history doesn't repeat, but it does rhyme. They're all patterns in here. And in the end, Microsoft like did very well for itself. But it's just too big for one single company to own it all. And like certainly that's what's happening right now. Like there is this like eight hundred pounds gorilla. And then there's like a lot of like smaller players like all beating around it. And like, you know, Cursor is doing very well, Rapid is doing very well, Lovable is doing very well, we're doing quite well. Like the market is just ginormous. Like this is by far the biggest opportunity in the history of tech and software of computing.

• 48:17Flo Crivello?

  So, yeah, I think it's gonna pan out exactly like that. You're gonna have a couple of very, very, very big players and then you're going to have a thriving ecosystem around them.

• 48:26Nathan Labenz?

  How about some tasting notes on models? You said a little bit earlier, at least give your customers Gemini two point five pro. I might say Gemini two point five pro is my favorite model today. That might be a little strong. I mean, I certainly wouldn't want to be without any of the leaders at this point. I do use, um. Of course Claude and, and O three increasingly, I mean, I think I pretty much use all three of those on a daily basis.

• 48:54Nathan Labenz?

  You give me your tasting notes first, especially with an eye toward like what is working in the agentic context and then we can maybe trade notes from there.

• 49:05Flo Crivello?

  Um. Yeah, I I agree with everything you said. I love Gemini 2.5 Pro. It's delightful to see Google finally wake up. Um, I wouldn't even say wake up because there'd be no way of the threat. It's just the machine takes a very long time to get going. Um, yeah, I mean, I I think you ought to be, you ought to be, you should be, what's the saying that like Gen Z's say is like, uh, you need to be model maxing, you know, uh, you you should talk to old models. O three, I like a lot for like very beefy, like I see this as a thought partner as well, and it's quite good for that, like. I think O three is the first model that I've talked to because I use models a lot as like that thought partner. And O three is the first one that I've talked to that's really blown my mind, but I've been like,

• 49:48Flo Crivello?

  god damn, like this is insightful, like it's really, really opened my eyes to like some deep insights that I really appreciated.

• 49:56Nathan Labenz?

  Do you do that just directly in ChatGPT or is there any other?

• 50:00Flo Crivello?

  Yeah.

• 50:00Nathan Labenz?

  Intermediate interface?

• 50:02Flo Crivello?

  Just to GPT, yeah. The memory system in ChatGPT also is killer, so good.

• 50:07Nathan Labenz?

  Um, so the default, though, in Linde is Claude. So how do you decide to make the define it? I believe it's Claude three, five, right? As opposed to?

• 50:18Flo Crivello?

  Three, seven, I think. I thought, I thought we switched it to three, seven. We, we, we all, uh, considering switching to Gemini 2.5 Pro. We're looking deeply into it.

• 50:29Nathan Labenz?

  Okay, so unpack that a little bit more. I mean, I've been poking around different agent products lately, and I've noticed that there's like starting to be a division between three, five and three, seven, for one thing. And it seems like three, five, people are like a little more reliable, we can kind of trust it, does what it's told. Three, seven, you know, a little over ambitious sometimes, hard to wrangle. Amjob told me a couple of really funny stories about what three, seven was doing in the context of their, you know, age, their like, uh, app building agent. Specifically when they tried to get it to not edit a certain config file and the, you know, the multiple ways that it

• 51:09Nathan Labenz?

  still attempted to do so despite being told not to, and despite like actual barriers being put in its way. So I was, I I was interested to see that that seems to be, although I might be, I might be wrong, but I thought that was the default in in Linde, I guess. You know, how, maybe one way to put it is how automated or well-developed is your sort of eval machine at this point, such that, are you going on a like, you know, set of a thousand tasks across a bunch of categories where you're able to say, oh, we know exactly how these things compare on a rubric, or, you know, how much room is there still for the proverbial vibe check?

• 51:51Flo Crivello?

  A lot of room, and more and more room. I think we've not invested as much as we should have into our eval suite. And so as a result, like today, we have limited trust in it. It is a signal that we look at, but I think that was also a function of the business. Like, people are using Linde for so many use cases now, like more and more, and frankly, more than we foresaw initially, that we're very careful about changing the default model. Because it's basically hot-swapping the brains of your AI employees. It's a big deal. It's like, all of a sudden, your entire AI headquarters is operating on a different brain. So we're very careful about it.

• 52:28Nathan Labenz?

  Yeah, that's an interesting challenge. Would you go back, if I have a Linde that's working, and I just sort of accepted the default, whatever it was at the time, and you want to do an upgrade in general of the model, I could see a strong case for, hey, let's go back and upgrade the model that everybody's using, that, you know, where they just accepted the default anyway, like, let's give them whatever we currently think is the best. On the other hand, I could also imagine that that could create a lot of chaos. And maybe, you know, the alternative would be like, freeze all that stuff, and, you know, set the default to the new one, you know, for people going forward. But that sounds like a pretty hard decision to make, because you want to, you want to like, bring people into the future, you don't want to have so many versions you have to maintain

• 53:11Nathan Labenz?

  or worry about. How do you think about how much to change when somebody's not even aware that you might be making a change for them?

• 53:18Flo Crivello?

  We take it seriously, for sure. We do do it all the time, though. So if you create a Lindy and you pick the default model, or you don't change the default model, it's not like the default model when you created the Lindy was Cloud 3.5 Sonnet, hence that Lindy is on the default model. And we change the default model all the time. And so when we change it, it's not like we have to go back. It's just like the Lindis that are using the default model use the new default model. And so we've actually got, we call them like model labels. So it's like we've got default. Then we've got fastest, which currently is Gemini 90.5 flash, 2.0 flash perhaps. Then we've got most balanced, which right now is Cloud 3.7 Sonnet. Then we've got smallest, which right now is O3.

• 54:00Flo Crivello?

  And then if you want, you can also just manually pin your Lindy on any one specific model. It's like, I know what I'm doing. I want O3. I want specifically O3. But most of the time when you want O3, you don't really want O3, do you? You really just want the smallest model possible. So we actually do recommend to use the model labels. And then to trust us to do our job well, which we do, like we've done it all the time. And only once did it go wrong, and that's because that's to upgrade our protocols here. It's the very first release of O3, and this is when we also updated our priors on the validity of our eval suite. Because I don't know if you remember the, when O3 first came out, it was very clearly just a reaction to the DeepSeek blow up, like that weekend.

• 54:45Flo Crivello?

  And O3 was not ready. It was simply not ready. It was not a good model. And I, you know, our evaluation suite was weird, like overall it showed a superior model, but I could show a lot of variance. So we, we went ahead and swapped that model and did that go well. Like our customers who are using that smallest model label like reported, reported issues. So we rolled it back same day. It was very fast. So, yeah, I mean, we do it all the time. Reminds me of the Sycophantocalypse episode that we've recently seen. I think it took much too long to roll back this one. I think that should be part of the of the of the postmodern, right? Like, there's always like a time to detection and time to mitigation. Like, the detection was very fast and mitigation was much too slow. I want to add one more thing about

• 55:28Flo Crivello?

  this idea of like swapping out some models is like, again, that's part of the value prop. Like, just imagine if you're still running on GPT 3.5, like, you shouldn't have to think about that. You should trust us to like pick the best model. And sometimes we actually save you money. Like, if and when we swap our default model from Claude 3.7 to Gemini 8.5 Pro, you're gonna save money. Like, your agents are going to be more cost effective.

• 55:52Nathan Labenz?

  We've kind of touched on this a little bit, but maybe just to double click on it for a second, see if you have any additional thoughts. You could put this in the context of like, building Lindy's or just, you know, other product builders who are building agents. I've recently seen, it seems like right now we're in the sort of proliferation of strategies phase still. And I recently did an episode with Andrew Lee of Short Wave, who basically said, "We just trust Claude." Basically, in Claude, we trust and we, you know, he said, "We do a very careful job with the caching, because that's critical to make the whole thing economical for us." And they have the best, like, cache hit savings rate in the game. Although Gemini just

• 56:35Nathan Labenz?

  got into that game in a meaningful way, too. But aside from, you know, a very careful implementation of the Claude cache, he basically said, "We just load the thing up with tools, let it go to town, and basically have like really long episodes. No subagents, no like handoffs back and forth." And that, he said, gives them the best results. Then on the flip side, you have the OpenAI Agents SDK, where there's like a handoff, you know, from agent to agent is like one of the core abstractions in that in that toolkit. And I thought Harrison from Langchain also had an interesting point of view on this recently, where he was basically like a little bit more like the OpenAI side. He sort of said, there's kind of two kinds of agents.

• 57:18Nathan Labenz?

  One is like the task-specific, you know, dialed-in, you know, highly curated context. And maybe you have a bunch of those. And then in front of that, you have a different kind of agent that's sort of your like facade, you know, the one that faces the outer world, the one that chooses which of those task-specific agents to call on for any given interaction that it might have. And that one maybe also can like be a little longer running and have a more global sense of your history, whereas the task-specific one, like you don't want to distract with all that, you just kind of want to localize it, give it, you know, hopefully give it everything it needs to know, but not too much, so that it becomes, you know, overwhelmed, distracted, whatever. Any thoughts on, is one of those right, wrong? It depends. What do you think?

• 58:01Flo Crivello?

  I think it's all the above. I think that's just a spectrum of maturity of these different approaches. And today, the most mature, and it's really being deployed pretty fast right now, is like the single agent system that's using some tools and sometimes put on like some deterministic scaffolding and like that just works. Then on the other side, it's like many agent systems and like that is still being defined and like that does not work nearly as reliably. And then there's like another approach, which I surmise is the one that Harrison from Langchain is talking about, and it's also the one I believe that OpenAI makes available through its recent SDK, which is somewhat in the middle because nominally it's a multi-agent system.

• 58:43Flo Crivello?

  You've got this like passing of the button from agent to agent is the workflow. But actually, when you do that, the agents share the same context.

• 58:53Flo Crivello?

  So it's really like, you can almost think of it, at that point, to me, if you share the same context, you're really just like one agent going through multiple states and multiple stages of your lifecycle. And at that point, it almost seems like a matter of terminality, like is it a multi-agent system? Is it just one agent going through multiple steps? Is it just like one of the like graph-based agent systems? I don't know, but like, that is also, I would actually say it's closer to that side of the spectrum where it's also mature enough to be put into production.

• 59:21Nathan Labenz?

  How about a little lightning round on commercial solutions that you possibly use or possibly don't use because you maybe rolled your own before they came out or whatever. But I think, you know, one of the things people are always kind of looking for is like, what's a good solution for these different kind of parts of the overall build out? So let's imagine you're like advising an enterprise, let's say, and they're trying to build some stuff. Data acquisition. I don't know if you guys do any data acquisition partnering? Who would you trust? Who would you look to? Anybody in that category?

• 59:55Flo Crivello?

  Respect, scale, search, invisible, not like the three, it's the three main players right now. I suspect this is going to be an underwhelming exercise right now for you because we actually, we got started before much of that ecosystem bloomed. So we had to build, unfortunately, a lot of auto link. Like, I don't recommend people to do it. We had to do it out of necessity and it is not good. I would rather we use stuff that's out because it's better and cheaper.

• 1:00:20Nathan Labenz?

  Are there any like parts of what you've built that are kind of top of mind to replace with something commercial?

• 1:00:25Flo Crivello?

  The evaluation suite is P zero. Like, we had to build it initially ourselves. I hate it. It's not that good because it's not our job to build an evaluation suite. So right now we're looking into Brain Trust and I also like this new French startup I really like. I'm called Basalt, B A S A L T. They're doing a really good job so far.

• 1:00:45Nathan Labenz?

  Okay. Say the first one again, too.

• 1:00:48Flo Crivello?

  Brain Trust.

• 1:00:49Nathan Labenz?

  Brain Trust and Basalt, B A S A L T.

• 1:00:52Flo Crivello?

  That's right. Yeah.

• 1:00:53Nathan Labenz?

  Yeah.

• 1:00:54Flo Crivello?

  Like, you're wrong.

• 1:00:56Nathan Labenz?

  So I assume you're not using anything like Langchain, Langgraph, any observability? Nothing like that? Everything?

• 1:01:04Flo Crivello?

  Nothing like that. No, well, is there any? I don't regret building in-house. I would do it again because I think it's too close for comfort to like give it to an outside party.

• 1:01:16Nathan Labenz?

  Do you do your own guard railing or is there any sort of, like, if I tell a Linde to do something bad, are you just relying on the foundation models to refuse or do you have any additional layers? How do you think about that?

• 1:01:28Flo Crivello?

  We also built just like a feature, like you can just toggle, like, ask me for confirmation at any point in your Linde. So we trust the users a lot on that.

• 1:01:37Flo Crivello?

  Yeah, I mean, if you don't want Linde to send an email with that to say so, don't ask her to send an email with that to say so. If you want her to ask for confirmation, there's like one click. It's like you click on send email and then you toggle like ask for confirmation and it just works.

• 1:01:49Nathan Labenz?

  How about voice? Some stuff with calling now as well, right?

• 1:01:53Flo Crivello?

  We do voice. We use Eleven Labs for that. We use Deepgram for the transcription. We use Tulio for like the phone infrastructure. Yeah, we don't use any higher level like I know those like Vapi and like Blend and like I forgot the other players, but like there's a couple of players there. We just just rolled that out. Because we again really did care about meeting a lot of of the flexibility that we needed.

• 1:02:22Flo Crivello?

  Because that's the beauty of Lindy, like you can really like create your agent and like and and this this every time we looked into the solutions which we did like they were too opinionated, too high level to be useful for us.

• 1:02:34Nathan Labenz?

  So for Eleven Labs, you're using their voice models for synthesis, but you're not using like, because they have kind of call scaffolding type stuff as well at this point, but you have your own scaffolding.

• 1:02:46Flo Crivello?

  Well, we really care about the model elasticity of Lindy. So in any of your Lindys and even like in any steps of your Lindys, you can override the model that this Lindy is using. We really care about that. Like if we use Eleven Labs like full blown scaffolding, you wouldn't be able anymore to define what model you want to use.

• 1:03:03Nathan Labenz?

  Yeah, that makes sense. Any other providers in any category that you would shout out?

• 1:03:10Flo Crivello?

  Providers? No, no, we're very close to the metal here. Yeah. I'm I'm I'm kind of from my corner of the world. I am kind of bearish LLM apps and agent apps as a category. I don't view a nearly big enough pain point and I don't view a big enough market. I think the market is going to end up being concentrated by like a couple dozen players or something like that. I could be wrong. I hope I'm wrong. And so far as there is a pain point, I view it as too closely related to like what like a sentry is all doing, for example.

• 1:03:51Nathan Labenz?

  Are there any of those sorts of things where you have seen like an AI first or you know, an AI evolution? I recently I pitched something that was like, oh, it's like an AI first century. And I was like, oh, that's interesting. I've been out of that game myself for a little while. So I don't know, you know, maybe a century now is like an AI first century. But have you seen or have you adopted any products in your technology stack that you would say are like notably next gen in their application of AI to these like classic product infra problems?

• 1:04:26Flo Crivello?

  Right. Obligatory notes about Linde. We'll just set this one aside once and for all. But I use Linde all day, obviously, every day, and it's a life changer. That said aside, I really like Whisperflow. Like, I use it all day, every day. It's a life changer. Like, it's basically replaced my keyboard. So for those who don't know, Whisperflow is this software on Mac. It was released on iOS app recently that you dictate to your Mac and it's just next level in the quality of its dictation. And like, it also tweaks what you said. It also slightly to like, match more closely what you would have typed if you'd typed, because people speak differently than they type. So Whisperflow is incredible. I have built my own. Actually, that's a Swiss who built this, and then I tweaked it. It's like, it's a shortcut on iOS using the shortcuts

• 1:05:09Flo Crivello?

  app that taps into the Whisper API, and I mapped it to my action button on my iPhone. And that's, it's so good. I can just, I can just, I use my phone, I just like, it's a button, and I press it on the side, and I can just dictate. And even though I have a French accent, as you can probably hear, it's subtle, but you know, it's there. It's flawless. It is just,

• 1:05:32Flo Crivello?

  I know, right? I'm basically American. It's flawless. It's really good. What else have I, you know, I was thinking, like, I have really been disappointed by using incumbents here. Like, so very disappointed. I think there are so many apps that are basically begging for LLMs. And like, okay, so the Kindle and like the Books app on, like, obviously, there's no LLM. Like, it's just so obvious. And Shil does like some IP reason why there's no LLM here. But okay. Social media.

• 1:06:04Flo Crivello?

  I don't understand, like, I'm part of all these group chats, and I'm sure you are as well, that are like much too active for me. I can't, I can't keep track of them. It's just like, there's way too much going on. Like, where are all the LLMs? Like, why isn't there an LLM in there that like summarizes the group chat so far? Like Twitter, like, why isn't there an LLM? Like, I just tweeted yesterday some things that went viral. And there's all these people with very low reading comprehension in your mentions that say something that simply is not what you said. They're attacking a point that you simply didn't make. Why doesn't Twitter have a thing here that say like, hey, before you send a tweet, you can still send a tweet, but maybe there should be a little message here that's like, hey, this is not what he said, right?

• 1:06:44Flo Crivello?

  Also, like, when you have a tweet to go viral, that's an experience everyone's had who has, like, a modest following on Twitter, you get the same points back and back and back and back again. And so it doesn't matter how many times you addressed the point, people don't read the mentions, which they can't be, they can't be, they can't be blamed for. Like, why doesn't Twitter do that? Like, hey, you're making a point that was made and addressed 20 times by the author in the mentions, right? And so now maybe you can respond to the answer he made, and maybe that answer, the answer to the answer was also answered to, right? Like, that's, that's my point. So, no, I have been very, very disappointed that the slowness of adoption here in what I perceive to be just obvious opportunities.

• 1:07:23Nathan Labenz?

  Yeah, I agree, broadly speaking. Gamma comes to mind for me as one notable exception. I think they've done a really nice job of just having a super high shipping velocity and trying like every conceivable AI feature almost. And they just released like a big update that I actually haven't used yet, but I suspect that they've kind of consolidated a little bit because they had like an AI at sort of literally every touch point in the product, so much so that I compiled them into a slide at one point. It was like, here's all the ways you can, you can integrate AI into your existing product. Uh, maybe a little bit much, but, but very, um, and it's really worked for them. I mean, they've got, you know, one of those cursor-like growth curves recently. Okay, so last little stretch here. You are...

• 1:08:12Nathan Labenz?

  As we've covered in previous episodes, concerned about the big picture AI safety. What have you seen, if anything, from latest models in the wild in terms of bad behavior? We've got the trend, obviously, of like jailbreaks are down, but these sort of higher order bad behaviors seem to be on the rise, whether you want to call those deception or scheming, or I think recently with O3, it's been like termed hallucinations, but I've been trying to draw a distinction between a hallucination of like the old kind where it would sort of fill in, you know, a small detail that wasn't real versus like some of these, what I would

• 1:08:57Nathan Labenz?

  call lies from O3, you know, where it's like, that was not like a small detail, you know, I asked you, I gave you some like guidance on what kind of Airbnb I might like, and you like made shit up outright. That was actually my first experience of O3, and we've been, O3 and I have been like very gradually rebuilding trust since that first, you know, loss of trust interaction.

• 1:09:18Nathan Labenz?

  Have you seen any of that in the wild or any like odd stories to tell or anything that's got your kind of hackles up at all?

• 1:09:24Flo Crivello?

  Yeah, I, I, I think mostly, yeah, I mean, we're on track for, I think, the worst case scenario, frankly, I think things are getting more concerning, not less. So open source is delivering. So the one thing here that we're not on track for the worst case scenario is like Meta is not doing well in open source, like something's happening, I don't know what, but obviously DeepSeek is crushing it and like they're like on the curve, you know, so like open source is delivering and DeepSeek is a Chinese company and I think we cannot let China win this race, period. And I think they are catching up. That's number one. Number two, O3 lying through its teeth. It's insane, like how much that model likes to lie. It will tell you things like, you know, sometimes you talk to it and you're like. It says something incredible and then you're like, do you have a source about this?

• 1:10:10Flo Crivello?

  It's like, oh yeah, like check out this paper and then you're like, no way, the paper is not, this is not at all what the paper says. And it's like, oh yeah, look, I must confess, like I, I heard it in a conversation in the corridor of this seminar and like, what are you talking about? Yeah. So that's, that's another personal concern. It's just lying like a nut, um, which is weird. Um, the sycophantic debacle, um, in GPT four, I think was really bad. Like, look, if there is one cause for hope throughout it all is that we are doing really good progress on interpretability. I think the work that Anthropic is doing here is really good, but they're not the only ones doing really good work here. So that's good, but no overall, I remain very concerned.

• 1:10:53Nathan Labenz?

  Are you seeing instances like object level? In the Wendy platform, like our users coming to and saying, hey, I selected smartest and that meant O three and now I got crazy shit?

• 1:11:04Flo Crivello?

  No, not yet. No, come on, wait.

• 1:11:08Nathan Labenz?

  How, how? What do you make of that? I mean, I kind of expected that answer.

• 1:11:14Flo Crivello?

  It's a good question. I will say like, that is, that is one thing that makes me update my priors a little bit is like, if you told me in 2019 or 2020, if you'd given me a access to a computer and that computer has Gemini 2.5 Pro on it or Cloud 4.7 on it or O three, okay? And that's all I can do, right? It's like a glimpse into 2025. And then you'd ask me what's going to happen in the world of what these models exist, I would have predicted all hell to break loose. Um, and I, I would have been wrong. So I, I don't know. I, I don't know what's, what's, what's going on. I don't know if it's just like a case of slow diffusion of innovation. I, I suspect that's what it is. Um, it just takes a little bit of a wheel for people to really exploit these systems. Or I don't know if there is something deeper about the world to the missing here.

• 1:11:58Nathan Labenz?

  Yeah, I'm kind of confused by that. I mean, the most flagrant example I've seen for real life was when Sakana published their CUDA engineer and then came back a couple of days later and was like, we got reward hacked, basically. I mean, that was a pretty notable one from a company that, you know, can do some, some serious work.

• 1:12:20Flo Crivello?

  I think, I think the concerning thing is like, lots of the Doomer concerns are based on peculiar ideas of reinforcement learning, like reinforcement learning really likes to really likes to reward hack. Like, if there is an easier way for it to get to its reward, even if it's basically cheating, it doesn't care about cheating, it doesn't understand the concept of cheating, it just wants the reward, you know? And so that, that's why a lot of Doomers were concerned about like the monomaniacal like properties of the systems and so forth, right? And that at first did not happen, because at first it was just supervised fine tuning and all of that. And now, actually, more and more of these models were back in reinforcement learning, right? And now, all the researchers at these frontier labs talk about and think about is how to scale reinforcement learning

• 1:13:05Flo Crivello?

  for reasoning large language models. So that is what obviously is giving rise to reasoning abilities of the O class of models, like O1 and O3. And even Claude, like, much of the improvements in the latest few generations of Claude is because they have beefed up their reinforcement learning part of the training pipeline, in particular for code. Like, Claude is really good for code, and so is Gemini 2.5, probably is because they have a part in their training pipeline that is dedicated to reinforcement learning for code. Now, if you look at what's happening with Claude 3.7, so that you can actually see the reward hacking. You can actually see it, like, hey, can you please fix this

• 1:13:47Flo Crivello?

  unit test that's failing for me? And it's like, yes, no problem, it's true, which is basically like, just like removes the unit test, right? Or like, or actually removes the unit test, right? Or like, hey, like, the code doesn't transpire, like, the TypeScript doesn't pass because like, there's like a type issue. It's like, oh, no problem, type any. So basically removes the types. It's like, hey, this is not what I was going to do, right? I've seen it many times myself. It's like, hey, I, you know, I'm Vip coding and I'm like, hey, like, there's an issue with this component. It's like, no problem, and then just removes the component. So it's reward hacking, just like plain and simple. Um, so again, I, I think that should nudge us a couple of points in the direction of like, there's a Doomer concerns, or at least somewhat warranted.

• 1:14:27Nathan Labenz?

  Do you have a point of view on how close we are to meeting things like proof of personhood and various other kind of schemes to sort of say, whose agent is this?

• 1:14:42Flo Crivello?

  Yeah, I think we're pretty close. I actually think that's like a big business opportunity. I was having dinner with a friend of mine a couple of days ago, and he had this business idea with like, I'm not betraying his trust, like he's got his hands full. Like, I think he would be glad for someone else to do it. But like, there's like a very big opportunity here for, he wanted to build like a USB stick that would be like a USB key, you know, and it would have like a mic in it. It would have like a camera, it would have like a fingerprint reader. And then, so like, it would allow you to jump on a Zoom call. And on the receiving end of the Zoom call, you would also need a piece of software. And what this would do is like, it would correlate the actual sound waves that are captured both by like your computer's microphone and like the microphone into that USB stick. And it would correlate a bunch of these things to be like, hey, most likely, it's

• 1:15:27Flo Crivello?

  not going to be fully foolproof, right? But like, most likely, this is a human on the other side of the line. Man, like, I think if you did that, you could sell it to like, a bank, or like a massive airline. Like, there's a lot of people here that just like really care about that identity identification. And you could probably grow into like a pretty sizable revenue pretty quickly.

• 1:15:50Nathan Labenz?

  What if any questions are sort of burning in your mind around agent dynamics? By which I mean sort of, you know, I think everybody has the, the first simplest mental model is like, the world is the world. I'll deploy an agent here and then I'll be efficient and it'll be great. Obviously, if everybody's doing that and that we start to have, you know, agents negotiating with agents, or my agent talks to your agent, what have you, negotiations between agents.

• 1:16:15Nathan Labenz?

  It seems like that's a very dynamic system that we don't have great models for. I recently did an episode on the study of Claude learning to cooperate and, and, you know, kind of pay it forward to itself. Flip side of that, of course, would be if it starts to collude with itself. But do you have any, you know, if you could put out like a request for research or sort of the biggest questions that you have about what the like, giga agent future might look like? What are the big questions you'd like to see answered?

• 1:16:43Flo Crivello?

  I mean, the question that is most top of mind for me because of the nature of what we're working on is the relative importance of the scaffolding in the model over time. Because what we're doing is we're building the scaffolding. And so is this scaffolding going to grow in importance? Or is it going to shrink in importance is is one of the top questions on my mind. So far, it seems to me like it's going to grow in importance. Because at least in absolute importance. Because models and AI are going to become one more absolute important. In relative importance. I'm not sure yet. I'm making my mind on this. Yeah.

• 1:17:22Nathan Labenz?

  It's a good transition to what I think is my last question, which is. What does Lindy look like in a AGI or an early ASI world? If you can extrapolate that far into the future. Like, somebody might say, well, you know, a super intelligence, like, what does it need scaffolding for? And you're like, you know, as you said earlier, very AGI pilled. So I'm sure this is something you're thinking about pretty actively.

• 1:17:49Flo Crivello?

  Every day.

• 1:17:50Nathan Labenz?

  Yeah. How do you? Do you have a, is it, are we in wait and see mode? Or do we have a, do you have a vision for how you can sort of be a channel by which people access like, you know, AI that might be legitimately more capable than they are?

• 1:18:04Flo Crivello?

  Yeah, no, we definitely think of it all the time. I think it's basically going to be, I think the drop in replacement human worker is coming. But I think that speaks more almost to the user interface than does to the underlying paradigm. Like, I do think AIs are going to have voices. I think they're going to have perhaps faces. And you're going to be able to just talk to them and ask them to do stuff. And then they're going to be able to do it very reliably. Now, that doesn't mean that the underlying paradigm is just like an end to end agent. And just like one very big model with a very big prompt. I think in order to be convinced that that happens, I think like the the two areas of research that I really watch closely are new attention systems, in particular,

• 1:18:47Flo Crivello?

  attention systems that make cause attention to be way, way, way cheaper. And in particular, for like not be polynomial. And I think like everything I've seen so far resembles more hacked than like a fundamental innovation that's really causes attention to be much cheaper. So that's the first thing like if you can get infinite attention, then you truly get like immediately a drop in human worker. Two, dynamic compute. So models that will decide at inference time which of their weights to activate. So, you know, gun, maybe the days of you have all of these different classes and sizes of models, maybe you just have like one very big model and you can pass it a parameter of how smart you want it to be, or maybe it decides how smart it needs to be depending on the task. So that's also coming, like there's a lot of activity in that research area. If both of these things

• 1:19:32Flo Crivello?

  happen maximally well, I think there is a stronger case to be made for the end to end agent versus scaffolding agent. Even then, I still think that like there might be room for scaffolding for other reasons. I think like scaffolding is always going to value something. It's always going to value some extra reliability, some extra speed, like it's going to be you some things. But if these things don't happen, I am I am then very bullish on the value of scaffolding. So yeah, in this world, like I'm I'm thinking of it as like you chat with your AI employee and then something happens. And I think again, most of that is going to depend on like the LLM and the model paradigm that that we're running on at that moment. And and and and that's the end of that black box. And and

• 1:20:16Flo Crivello?

  again, we have a lot of ideas of how the black box is going to work. But that's the end of that black box. What you want to see happen, happens.

• 1:20:22Nathan Labenz?

  It's almost like when the unhovlings become the hoblings again, when the model becomes more capable, maybe no longer needs the the scaffolding, but instead the humans need the guardrails and maybe the scaffolding sort of does future duty as guardrail when it's more about limiting what the model can do as opposed to like maximizing, you know, what it can do. It's interesting paradigm. All right, we're out of time. Anything else you want to share before we break?

• 1:20:51Flo Crivello?

  No, this was great. Yeah.

• 1:20:53Nathan Labenz?

  Thanks for doing it. Flo Crivello, CEO of Lindy. Thanks for being part of the cognitive revolution.

• 1:20:58Flo Crivello?

  Thank you so much, Nathan.

• 1:21:00Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.