⚡️Composio: 10,000+ tools that evolve for Agents — Karan Vaidya and Soham Ganatra

Watch the full show on YouTube: https://www.youtube.com/watch?v=aI5xMKxXU6c

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Alessio Fanelli?

  Hey everyone, welcome back to a Lit and Space Slimy Pod. This is Alessio, partner and CTO at Decibel, and I'm joined by swyx, founder of Small AI.

• 0:10Speaker 01

  Hello, hello. And today we're so happy to invite Karen Vitia from Composio, who just announced its release. Hey, welcome.

• 0:16Speaker 02

  Yeah, thanks for having me, Alessio.

• 0:19Speaker 01

  Yeah, Karen, you, I think we met a few months ago. I think honestly, you were sort of evaluating a move to SF. And Composio, I think at the time was sort of like the bundle of tools for agents instead of building that agents platform. How do you introduce Composio today?

• 0:35Speaker 02

  Yeah, we are kind of self-evolving skills now for agents. So people building or using agents want to like basically make agents interact with their apps. They can use Composio. We manage all the authentication and user account related stuff for them so that they don't have to take care of all that and provide agent-friendly skills, which can be anything from like a direct API call to somewhat more complex, like directly agent giving national language tasks to an app. And the idea is that kind of like the skills evolve over time. So we analyze the patterns of how the agents are using these skills and kind of like the specific usage patterns of a particular agent

• 1:18Speaker 02

  and improve these skills over time so that the agent take reliability is on the higher side over time.

• 1:23Alessio Fanelli?

  And you started this company before the MCP spec came out. It's been about two years. Maybe talk people through how that felt, you know? Were you excited when that came out? And did you have to throw out a lot of stuff and kind of move on to the new paradigm? Or were you just happy that there was now a standard?

• 1:39Speaker 02

  So yeah, I think firstly, MCP is really amazing. And but it has its own quirks of how like it is good for some over like like universal clients and not that great for like people building on top of. So like MCP is great, for example, if you want to connect your chat GPT Claude cursor, sort of a universal client with different apps. It provides a ton of like servers that people have already built. So like you can connect with all the tail end of applications. But even in the AI Engineering conference, I was kind of like looking at the MCP stream and the MCP creator mentioned that they are optimizing the protocol. So that like the ease of server creation is on the higher side and not so much for the client.

• 2:23Speaker 02

  So what that means is like kind of people building on top of MCP can't customize a lot of things. And like it's kind of like the DX of creating on top of MCP is somewhat difficult while we are optimizing Compose you. For the like server side of like the client side of things, so that like people building has have a very easy experience of connecting with tools. In addition, I think MCP is one of the protocols, there might be a lot more protocols that might come over time. But there needs to be an overarching learning infrastructure that needs to be that needs to be developed, be it MCP or function calling. So that when one of the agent discovers an edge case, let's say while connecting to sales force. Like other agents also benefit from that. So that's what Composure is. We are over like we connect via.

• 3:08Speaker 02

  People connect to Composure tools via MCP, via function calling via Kodak, multiple formats and like all of them have the same learning infrastructure.

• 3:16Alessio Fanelli?

  When you say there might be more protocols, do you have ideas on how to make it better? Or requests or MCP improvements or what should people build if they wanna get a new protocol going?

• 3:26Speaker 02

  So I think MCP, as I mentioned, is great for like the server side of things. But they can do a lot of improvements on the client to make it easier to connect to different servers. In addition, we have now seen a lot of like Vite code coded platforms. So like Kodak is becoming somewhat more real. We have seen like recently a surge and how people use Composure tools via like directly code generation. So I think that still is somewhat new. So like there might be some changes that needs to that need to be done to improve on that.

• 3:58Speaker 03

  So I can I can take last couple of like maybe just draw it some more context on last couple of questions. Uh. The first answer like how MCP could be but so so some of the limitations that we have been observing in production is like. Um MCP, uh let's just say you have a tool that has really long context, a really long sort of response. Currently, if you use MCP for those kind of tools, it ends up sort of messing up, right? Because let's say I'm trying to fetch like hundred emails on Gmail and my sort of context overflutters. And I'm not able to sort of post-process the output that is coming out from MCP. I'm not able to sort of play around with it before it reaches my sort of L L M. And and a lot of like potential failure scenarios like that exist. Some of the other things that we have observed is like, hey, maybe I want to sort of change the tool descriptions.

• 4:41Speaker 03

  But like in MCP, the tool descriptions are controlled by the server itself. And so essentially for me as a developer who's sort of using an MCP to build on top, like messing around with tool descriptions and messing around with tool schemas becomes a very, let's just say, painful experience uh to actually do that. So a lot of like this client side thinking, just like how do you optimize the whole thing so that for a developer who's sort of building on top, is kind of missing today. And truthfully, like when you talk about like new product, like so many thoughts emerged in my brain. But the but the the way I would put it is, I think we are like really sort of in the emergent phase. Few different lines of thinking I could get into, unlike how new prot- protocols could look like.

• 5:23Speaker 03

  But I don't really have a concrete answer to like where where it should be, right? Uh, that.

• 5:29Speaker 01

  Yeah, I I think this is an evolving skills over time. Um, any ways to eval this stuff?

• 5:37Speaker 02

  At least we have an agent essentially which builds a lot of these skills. And kind of like does a lot of like testing as well as eval. So it's like you can say like agent as an eval where our internal agent uses these tools and kind of like, we figure out where it is messing up and improve over time, even asking agent what we can do better, et cetera. So that like that's one. Then obviously we have like how the agent is using it, where it's getting stuck, that data to improve over time. Does that answer your question? You're looking for something specific?

• 6:07Speaker 03

  Uh, I can add maybe a couple of more things and maybe that also. So I think like MCP, by the way, uh, so one of the really cool things that it ended up doing was it ended up sort of uh, creating a very simple way to sort of integrate with universal slides. So for composer tools to be used in cursor today, it's super easy. Uh, anyone can go start using it. And that creates a very sort of interesting way for us to actually create evals and stuff. Because then I can allow anyone who's using composer tools via cursor to essentially give me feedback right inside the cursor chat window. Saying, hey, were we able to sort of execute on what you wanted to execute on? Uh, and, and so we are, we are getting a lot of like human feedback through this.

• 6:48Speaker 03

  And we are able to sort of use somehow accumulate some part of that feedback into like a internal sort of eval. Uh, so again, tool calling has a lot of different evals. You have seen the function tool calling evals, you have multi-function like, like multi, multi done the tool calling ones. But like, none of them are actually like production call it, like, like none of them are actually something that we are able to rely on truthfully. Because like they don't really represent the situations that occur, like the orchestration issues that occur uh, in a complicated sort of scenarios uh, that, that you often see in production. And so this like getting the feedback through MCP clients and essentially using that in some way is something that we are actually finding it way better for at least our internal setups and tracks.

• 7:30Alessio Fanelli?

  Are you guys doing any autonomous maintenance of this integration? So you get this data, what happens next? Like you guys manually kind of review it and decide where to work on it?

• 7:39Speaker 03

  So ninety-five percent of all our integrations are completely built using agents, maintained using agents. Uh, we, I think at this point, we have a repository with more than three million lines of code, completely built using agents. Uh, all, all the integrations out there that you could think of.

• 7:55Speaker 02

  And even the maintenance or sorts, like if kind of like something goes wrong, we figure it directly via our kind of like observability platform and like the agent that self-corrects it.

• 8:04Alessio Fanelli?

  So what's the five percent that you're gonna maintain?

• 8:07Speaker 03

  Uh, the, the enterprise tools that have really shitty documentation. And you, you basically just wish that if they had. Taken like maybe ten percent more time documenting that cases, I would have solved it. And like more than that, the problem is that you you need a dev environment to sort of run this agents on. Like a lot of this enterprise tools don't really have that type of environment. Like a very simple example would be, okay, I want to sort of integrate with Replink, but like my dev instance of Replink has no HR data. And so what am I going to do? And so like you do your best case analysis of like what the API is like looking like and how to go about it.

• 8:45Speaker 03

  And and then you try that. And like real sort of the applications that I'm talking about are like what they like, they they they really turn right in terms of the documentation they uh sort of put out, right? So yeah, tho- those kind of applications.

• 8:58Alessio Fanelli?

  Can you maybe give people some numbers about how many integrations you have, how many developers you have? Maybe the growth rate of integrations, like how many do you add? Is there like a big backlog that people are requesting? I think there's a lot of people launching MCP servers. Like you said, it's not very easy to use them. So I feel like sometimes it's like a marketing launch to do it, but then there's not a lot of usage. Like also, what are you seeing the most popular ones? Just give us some numbers.

• 9:24Speaker 03

  Sure. So we have at this point more than five hundred different applications on top of Composio. We are adding close to at this point, like like we we're adding like close to twenty a week. Like a month back, now that like a lot of sort of improvements have been done on the agent side. We added like last month has been like basically thirty five a week. Uh we're planning to certain times that up to like close to hundred a week in next month. Planning to reach like a target of like five thousand applications by end of the sort of year. Uh which would like put us in the similar lines of like all the sort of like which would essentially encompass all the major applications you could think of. Um in terms of developers, we have like more than hundred thousand developers.

• 10:05Speaker 03

  Uh but uh when building on top of first um and like trying our product out. Yeah. I I I yeah, what what are the the numbers seem kind of up? Up to us.

• 10:15Alessio Fanelli?

  Oh, that's kind of like the Composio product platform. I think there's maybe the other ninety percent of, uh, once you have the infrastructure, how do you kind of make it successful? In your product, there's a lot of talk about how many tools an L L M can support. Whether or not, you know, you should give different tools, whether or not you should put all the tools in one MCP, which you guys also offer with the Composio MCP. Can you maybe give people kind of state of the art today, where do things go wrong? Uh it may be the slope of improvement of the models?

• 10:45Speaker 02

  Sure. So I think like we have seen over like a lot of practice that like more than like twenty twenty five actions like just confuses the server like it's not able to kind of like figure out which tool to use. And same goes with like if the schema of the tools are really complex, that also confuses the agent. So we try to keep our kind of like schemas simple, like some of the things are like flapping the schemas, etc. In addition, I think like basically, I think it's like agents like because they are L L M's inherently. So they like to think in natural language. So some of the thing that we are planning on that print is like a single natural language execution of a tool. So that like the agent does just gives like like agent has just given a single tool and can call

• 11:28Speaker 02

  any of the other kind of skills that we have like inside Composio with just natural language. So like create an ocean dock and send a meeting invite just in natural language and we'll do that in the backend.

• 11:37Alessio Fanelli?

  And when you do that, do you have, does the Composio M C P have a search tool that the agent can use to search what tools to have or like you put like, you know, when you have an M C P, it kind of exposes what tools are there? Is there just like a big list or do you use search for that?

• 11:51Speaker 02

  We do have a composer search tool, which kind of essentially does exactly what you said, like searches across all the like we have around like fifteen thousand plus actions that we support across composer tools. So it can search all across depending on the natural language use case that you provide and kind of like then execute that specific action.

• 12:09Alessio Fanelli?

  What do you guys use for benchmarking? So there's kind of evals, which is like internally what works. I know for a while there was a Gorilla, kind of like a benchmark function calling thing. Is there anything new that you kind of keep track of or is it maybe something you should do?

• 12:26Speaker 02

  Yeah, soon do you want to?

• 12:27Speaker 03

  Sure. Yeah. So so essentially, so there there are like few, right? Like Gorilla was one, but then like a bunch of others emerged, right? And they were all sort of more on the base of like, is an L L M sort of good with tool calling, right? We are trying to create a different kind of benchmark. It's like, how good is this tool for L L M to be sort of like calling it like in the correct fashion, right? And this is more like in the lines of D S. If you look at like how they're like, like they sort of engineer the prompt and stuff, you're trying to engineer the sort of tool in a way. That like it becomes easier for L L M to sort of call it like internally, we have essentially the, the way I would go about it is like you can break it down in two terms like we have figured out the right heuristics.

• 13:11Speaker 03

  That we have seen essentially one better for L L M's and then we have also sort of given like a A B testing kind of infrastructure. For L L M's to go ahead and like test out with. So there are like two kind of optimizations we do, right? One is like, is the tool to the A P I transformation, right? It is like, is this Salesforce tool getting transformed into the right request payload so that it is hitting the right endpoint in Salesforce? Right? And that is like the major source of error today because a lot of documentations are not great. A lot of like edge cases are not mentioned and you kind of keep seeing that issue. And then the second feedback that we kind of optimize for, which is like, is the Salesforce tool really sort of simplified so that an L L M can call it nicely?

• 13:55Speaker 03

  And that optimization essentially uh is built on a bunch of sort of internal like benchmarks that we have, which is like, which are around. Here's like a query that we saw, here's a query that L L M couldn't really transform into the right set of sales force call. That is the reason Salesforce C P I endpoint resulted into failure. How do we sort of like make sure that we are able to get those kind of queries working nicely, right? And like we keep seeing the meta patterns emerge here and then we do A B testing, which is like. If we would have built the tool in this way, and like all of this is through agents, of course, we can't do it manually for all the tools. But we start doing A B testing that way. So like if we had sort of built a tool in this way, maybe this query would have been easier to resolve.

• 14:35Speaker 03

  Right? And the, the reason sort of M C P helps here is because if you know, M C P allows you to do dynamic tool sort of like uh exposure. Which means essentially like based on the query, I can decide what tools to expose and how to go about it. And so we can do that. So we, we can essentially decide that for the same Salesforce, uh for the same query, we can expose two different kinds of Salesforce tools. To the M C P client and just see like what tool is being better sort of used here. So, so that's the infrastructure moving building.

• 15:03Speaker 01

  I was aware about tools discovery in MCP, but I didn't know that it was that dynamic. Can you say more about that kind of behavior?

• 15:11Speaker 03

  So essentially, what you can do is like you can decide what tools to expose, right? So since the sales force server is with us, like the MCP server, we can decide like in 20% cases, we want to expose this tools. And in the other 80% cases, you want to expose this tools. And then we can expose like different tools for different people. And so we can know like what tools are essentially getting better results. And then the dynamic tools, that there's also one more layer of dynamic city that is coming, which is like in MCP, you can actually change tools. And you can notify the client that you have changed tools while MCP is like, like I can share that with you, but like we have, we have been building around this also.

• 15:53Speaker 03

  The idea here is that eventually we want to sort of create a new year where like we can like keep like changing tools and like recommending better and better tools. Let me just quickly share this with you, like that where it's coming essentially.

• 16:07Speaker 01

  In the MCP spec, I, I sort of the, yeah, the tools page.

• 16:11Speaker 03

  Yeah.

• 16:12Speaker 01

  Cool. Well, I didn't, I didn't know that, but I haven't tried to sort of extend it to as many tools as you have.

• 16:18Speaker 03

  Yeah.

• 16:19Alessio Fanelli?

  I was gonna say the other thing at the engineer world fair in the MCP track was a sampling, which was pretty new. I don't think many clients supported it. Is that interesting to you guys? Like having the MCP server also do some LLM inference before returning to the, to the client? And that, that seems like one way where you can take over more kind of like the, the value that happens in the, in the value chain.

• 16:41Speaker 02

  Yeah, I think as I was mentioning, kind of like one of the tools like or kind of like servers that we support, which we internally name as like agent API. Is essentially that we're kind of like the whole end to end, the use case to tool formation, like kind of like tool calling formation is done by us. So the client just gives like, for example, that it wants to schedule a meeting with like five people, probably just names. And we figure out all their email addresses beneath the, kind of like the server and kind of creates, create their meeting invite with that.

• 17:12Speaker 01

  We're gonna probably zoom out from just the product itself. Anything that people are missing about Composio? Like underrated parts of it? Or obviously, you're growing so fast, you just moved to SF and opened a new office. So everything is super exciting.

• 17:26Speaker 02

  I mean, like, a lot of features that we have, like, including, like, one of the most used features, which kind of people discover after they come to Composio is, for example, triggers, where it's not just like the agent can take action, but they can get notified over time. So like, essentially, when you, like, when you receive an email, or when you receive a Slack message, or like, anything that you can use to spawn an agent, that it can work on top of. The other thing that I'm like really excited about, and we are optimizing our tools for is like Codeact, where a lot of, like, a lot of tool calling is happening via generated code. So you can use Composio's actions, not directly in like MCP or tool calling, but like expose them as functions which the LLM can use while generating code.

• 18:11Speaker 02

  So that's very interesting, and we are doing a lot of optimizations on that thread.

• 18:15Speaker 01

  Yeah, awesome. One of the things I want to call out is I was looking at the tools list, and you can use Composio inside of Composio, which is very meta. I wonder if that's the ultimate end state of everything?

• 18:28Speaker 03

  Yeah, that is. We built it like a year back. Nobody used it for a very long time. Today, it's probably one of the most used tools. The idea here was that we just wanted anyone to just install Composio and inside Composio, and that's pretty much it. You don't need to think about anything else.

• 18:45Speaker 01

  Yeah, but like, this is, this is then like sort of the, the tools discovery layer, or like the one MCP through all the MCPs, depending how you think about it.

• 18:52Speaker 03

  Yeah, like, one way to think about it is like, okay, this is basically like me sort of just talking, living about it. But essentially, why do you need like a layer of tools? Because you want to sort of execute stuff on the client, right? But let's say if a server is smart enough, and like, basically, if I build something very interesting, and I say, hey, you don't have to worry about anything, just send me the query, and let me do all of the stuff, then like, you don't need a layer of tools. Then you just need one tool, like, hey, here's the query, go do it, right? And like, that's the idea, which is like composite.

• 19:21Speaker 02

  Eventually gets to that point where like you just give us the query and we just do it.

• 19:26Speaker 03

  That's it.

• 19:26Speaker 01

  Yeah, fair enough.

• 19:28Alessio Fanelli?

  Awesome, guys. So you you're offering people a Porsche if somebody refers. Somebody to work at your company. If they apply after watching this, do we get the Porsche? Or what? How are you doing attribution on on these things as they work as they got in you a lot of traffic? I'm curious.

• 19:43Speaker 02

  It has, it has, I think.

• 19:46Speaker 02

  Yeah, I mean, I mean, like, obviously, like, I think it's anyway hard to kind of hire really good engineers in SF. You have like a lot of competition all around. So kind of like that was the idea where we want some of the best engineers to work at Composio. So I mean, yeah, like if somebody like sees this and applies via the latent space, I'm happy to kind of like, uh, give you guys a Porsche.

• 20:09Alessio Fanelli?

  Please write that, write that in the application notes. I heard about it from from latent space.

• 20:14Speaker 01

  Yeah, I I'm actually in the market for a car, so this would be really nice.

• 20:18Alessio Fanelli?

  All right, get to work, Sean. Other hot takes, Karan, you had a tweet about Grok CLI being better than Cloud Code? So maybe we want to talk about coding stacks and why do you think that is?

• 20:30Speaker 02

  Uh, so yeah, I I mean, like when Grok four came out, we were like experimenting with it and like just developed the CLI because there was none like for Claude, you had Claude code. Uh, you can use probably Codex uh with Grok four as well, but I think the idea was, I think Grok four actually has some of the really cool agentic capabilities. So kind of like, I think I really love Claude, the Sonnet, for example, because of the same. And Grok four had like some of a mix of Opus and Sonnet, which I really like. So kind of like wanted to use a CLI to use that on my code base. So kind of just developed it overnight.

• 21:04Alessio Fanelli?

  Are you still using it?

• 21:05Speaker 02

  Not that much. I'm a cursor fan, so I think move back to cursor.

• 21:09Alessio Fanelli?

  Oh, awesome. Awesome, guys. This was fun. Are you? Obviously, you're hiring any engineering roles, any other open recs?

• 21:17Speaker 02

  Yeah, I mean, we are hiring across like a research engineer to work on some really cool RL stuff that we just kind of like to build skills on top of tools that we just discussed. We are hiring for product engineers. We are hiring for DevRel, I think. Hopefully, somebody listening here might be interested to work in one of the coolest AI companies. As a growth person as well, yeah, any kind of these roles would love.

• 21:45Alessio Fanelli?

  Awesome, guys. Congrats on the CRZ. Thanks for coming on and I'm sure we'll see you around the NSF.

• 21:51Speaker 02

  Yep, for sure. Thank you.

• 21:53Speaker 03

  Thanks a lot for having us. Bye bye.