Open sourcing AI app development with Harrison Chase from LangChain

Companies are employing AI agents and co-pilots to help their teams increase efficiency and accuracy, but developing apps that are trained properly can require a skill set many enterprise teams don’t have. This week on No Priors, Sarah and Elad are joined by Harrison Chase, the CEO and co-founder of LangChain, an open-source framework and developer toolkit that helps developers build LLM applications. In this conversation they talk about the gaps in open source app development, what it will take to keep up with private companies, the importance of creating prompts that can be compatible with many API models, and why memory is so undeveloped in this space. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil |@hwchase17 Show Notes: (0:00) Introduction to LangChain (1:45) Managing an open source environment (4:30) Developing useful AI agents (10:03) Sophistication and limitations of AI app development (14:17) Switching between model APIs (17:10) Context windows, fine-tuning and functionality (21:37) Evolution of AI open source environment (23:53) The next big breakthroughs

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Hi listeners and welcome to another episode of No Priors. Today, we're talking to Harrison Chase, the CEO and co-founder of LangChain, a popular open source framework and developer toolkit that helps people build LLM applications. We're excited to talk to Harrison about the state of AI application development, the open source ecosystem, and its open questions. Welcome, Harrison.

• 0:25Speaker 01

  Thanks for having me. I'm excited to be here.

• 0:27Sarah Guo?

  LangChain is a really unique story, and it started actually as a personal project for you. Can you talk a little bit about what LangChain is and what it was originally?

• 0:37Speaker 01

  Yeah, absolutely. So how I would answer the question, what LangChain is, has kind of evolved over time, as has the entire landscape. LangChain, the open source package, started, yeah, as a side project. So my background's in ML and ML ops. I was at my previous company. I knew I was going to leave. I didn't know what I was going to do. This was in September, October of 2022. And so I went to a bunch of hackathons, went to a bunch of meetups, chatted with folks that were playing around with LLMs, and saw some common abstractions, put it in a Python project as a just fun side project. Turned out to strike a chord, be fantastic timing. You know, ChatGPT came out like a month later. And it's kind of evolved from there.

• 1:19Speaker 01

  So right now, LangChain, the company, there's really two main products that we have. One is the LangChain open source packages. I'm happy to dive into that more. And then the other is LangSmith platform for testing, evaluation, monitoring, and all of those types of things. And so, you know, what LangChain is has evolved over time as the company's grown.

• 1:42Sarah Guo?

  One thing that we talked about the last time we saw each other in person was just how quickly like the AI ecosystem and research field is evolving and what it means to manage an open source project through that. Can you talk a little bit about what you decide to keep stable and change when you both have like big ecosystem of users now and like very rapidly changing environment of applications and technology?

• 2:06Speaker 01

  That's been a fun exercise. So, I mean, if we go back to the original... Version of Langchain, what it was when it came out was essentially three kind of like high level implementations. Two were based on research papers and then one was based on Nat Friedman's like Natbot type of agent web crawler thing. And so there was some high level kind of like abstractions. And then there was a few like integrations. So we had integrations with I think like OpenAI coherent hugging face to start or something like that. And those two layers have kind of like remained. So we have, you know, 700 different integrations. We have a bunch of kind of like higher level chains and agents for for doing particular things. I think the thing that we've put a lot of emphasis in, to your point around kind of like what's remained

• 2:47Speaker 01

  constant and what remains and what's changed is like a lower level kind of like abstraction and runtime for for joining these things together. One of the things that we pretty quickly saw was that as people wanted to improve the performance, go from prototype to production, they wanted to customize a lot of these bits. And so we've invested a lot in a lower level kind of like chaining protocol. So Langchain Expression Language and then in in a different protocol, Langgraph, which is one something we're really excited about. And that's more aimed at basically graphs that are not DAGs. So you know, all these agents are basically running an LLM in a loop. You need cycles. And so Langgraph helps with that. And so I think what we've kind of seen is

• 3:29Speaker 01

  the underlying bits of, there's all these different integrations and like, you know, there's there's LLMs, vector stores, and sometimes they change, right? When chat models came out, like that was a that's a very big change in the API interface. And so we had to add a new abstraction for that. But those have, especially over the past few months, remained relatively stable. We've invested a lot in this underlying runtime, which emphasizes a few things, streaming, structured outputs, and the importance of those has remained relatively stable. But then the way that you put things together and the kind of like patterns for building things has definitely evolved over time from like simple chains to complex

• 4:10Speaker 01

  chains to then these kind of like autonomous agents to now something maybe in the middle of like complex state machines or graphs or something. And so it's really that upper layer, which is like the common ways to put things together, that I think we've seen the most rapid kind of like churn.

• 4:25Speaker 02

  What do you think is still missing from really getting to performance agents? There's a number of companies that have been started recently that are really focused on sort of the agentic world and pushing that whole thread and certain types of automation forward. What do you view as the big components that you all don't have or that maybe the industry more generally doesn't have that still needs to come into place to help drive those things ahead?

• 4:44Speaker 01

  Yeah, that's a that's a really good question. I think there's a few things. One, I think like like figuring out the right UX for a lot of these things is still an open question in my mind. And, you know, that's not necessarily something we can help with. I think there's a lot of exploration that applications need to do to figure out how to, you know, communicate what these agents are good at and bad at to end users and expose ways to maybe let them course correct and see what's going on. And so, you know, I think we try to emphasize a lot of this observability of intermediate steps and even correcting intermediate steps. But but there's a lot of experimentation around UX that I think needs to happen. Another big part, I think, is is basically the planning ability of the

• 5:26Speaker 01

  underlying LLMs. I think that's probably the biggest. I think when we see people building agents that work right now, it's often breaking it down into a bunch of smaller components and and kind of like imparting their domain knowledge about how information should flow through these components. Because I think the LLMs by themselves still aren't able to to reason fully about how that should happen. And I think we see a few kind of like A lot of research is actually around this, I would say, in the academic space. Specifically, I think there are two different types of research papers around agents that we see. We see some around like planning for agents. So there's a bunch of papers that do kind of like an explicit planning step upfront.

• 6:10Speaker 01

  And then there are other research papers that do a bunch around reflection. So like after it, after an agent does something, is this actually right? How can I kind of like, you know, improve upon that? And I think both of those are basically trying to get around the shortcomings of L L Ms. And that in theory, they should do that automatically, right? Like you shouldn't have to ask an L L M to plan or to think about whether what it's done is correct. It should know to do that and then it can kind of like run in a cycle. But we see a lot of shortcomings there. And so I think planning the ability of L L Ms is, is, is a big one and that'll get better over time. The last one is maybe a little bit more vague, but I think even just as builders, we're still figuring out the right ways, how to make all these things work. What's the right information flow between all the different nodes?

• 6:54Speaker 01

  In order to get those nodes, which are typically an L L M call to work? Do you want to do few-shot prompting? Do you want to fine tune models? Do you want to just work on improving the instructions and the prompts? And so I think there's a lot of, how do you test those nodes? That's a big thing as well. How do you get confidence in your L L M systems and L L M agents? And so I think there's a lot of workflow around that to kind of like be discovered and figured out.

• 7:19Speaker 02

  One thing that's sort of come up repeatedly relative to agents has just been like memory. And so I wasn't sure how you think about memory and implementing that and what that should look like. And because it seems like there's a few different notions that people have been putting forward. And I think it's super interesting. So I was just curious about your thinking on that.

• 7:35Speaker 01

  I also think it's super interesting. I have a few thoughts here. So I think there's maybe two types of memory and they're, and they're related, but I'll draw some distinction between kind of like system level procedural memory and then like personalization type memory. So system level memory, I mean more like, what's the right way to use a tool? What's the right way to accomplish this objective? Independent of of who exactly the person is and how I'm different than Sarah or something like that. And then for the personalization bit, I think it's like, OK, you know, Harrison likes soccer and he likes basketball and I should remember that when he asks questions.

• 8:16Speaker 01

  And so I think there's there's maybe slightly different ways that we see teams thinking about both of these. So on the procedural side, I think the main thing that we see people doing and that we think is pretty effective is few-shot prompting and maybe fine-tuning for how to use for how to use tools, because that's basically what it comes down to. What's the right way to use tools? What's the right way to plan? And we see few-shot examples being really, really impactful for that. And so that's something where, and so I think there's this really interesting data flywheel of like monitoring your application, gathering good examples and then and then plugging those back into your application in the form of few-shot examples that we're pushing really heavily with Lang Smith right now. And then the other side of it is this like personalization level memory.

• 9:02Speaker 01

  And I think there's a few different ways to do this. Like, I think OpenAI implemented it in their chat, in their chat GPT, where in the way I think it does it under the hood is it basically has functions that it can call to say like, remember this fact or delete this fact. And so that's a really interesting like active active loop that the agent is engaging in where it explicitly decides what it wants to remember and what it doesn't want to remember. I also think one thing that I'm bullish on is a more kind of like passive background process that kind of looks at conversations and almost like extracts insights. And then you can use those insights in kind of like future conversations. And I think there's pros and cons to each. And I think it speaks to the memory in general, I feel, is like a field that's just like super, super nascent.

• 9:47Speaker 01

  Like, I don't, I actually am underwhelmed at the amount of like really interesting stuff that's going on there. And so I think, you know, a bunch of different approaches, no, no kind of like overwhelmingly best solution.

• 10:00Sarah Guo?

  Has the sophistication shape type of application that you see people building with Lang Chain or just generally in the ecosystem dramatically changed over the last few months? I do think that there are more examples, kind of as Elad mentioned, of agentic applications that are much more productive. And more sophisticated like multi-step RAG systems with much more useful ranking. Like, does that match with the patterns you're seeing? Or like, what are you, what are you seeing that excites you the most, that you think is most useful?

• 10:29Speaker 01

  That does generally match. I think LangChain from the beginning has always been focused on those types of applications. And not only the open source, but also LangSmith, the platform. So I think, you know, a lot of the emphasis that we put into like the testing and the observability is really focused on these like multi-step things. We've always been focused on those. Probably it's generally true in the market that there's been more of a trend towards those. But from our perspective, we've always been focused on those. And so I think, you know, that hasn't been as dramatic. I think there has been like interesting things within that that have emerged. Just calling out like a few things. Within RAG, I think we've seen really interesting and advanced query analysis start to come into play.

• 11:11Speaker 01

  So, you know, you're not just passing the user question directly to an embedding model. You're maybe doing some analysis on it to figure out which, which retriever should I send it to? Or like, what is the bit that I should search? Is there kind of like explicit metadata filter? So some, and then so that retrieval is like a multi-step process and more there. And explicitly around query analysis. Few-shot prompting and that whole data flywheel, I think we're starting to see come into play more. On the agent side, I kind of alluded to this earlier, but I think, you know,

• 11:42Speaker 01

  the way that we've kind of thought about things is there's kind of like chains, which are sequential steps. You're going to do this, and then you're going to do this, and then you're going to do this. And you're always going to do those in this exact sequence. And then, you know, last March or April or whenever, AutoGPT came out. And it was like, we're literally just going to run this in a for loop. And it's going to be, you know, this autonomous agent. And I think the things that we see making it into production and informed a lot of the development of LangGraph is, or is something in the middle, where it's like this controlled state machine type thing. And so we've seen a lot of that come out recently. And so I'd maybe call out that as like one thing that we've really updated a lot of our beliefs on over the past few months.

• 12:22Sarah Guo?

  Yeah, I think a combination of that and tree search and just like trying to be efficient with like your sampling at every step has shown like a lot of really interesting effective applications recently. And I think the cognition as one example of like

• 12:39Sarah Guo?

  A surprisingly amazing agent has come out. Like, where else do you think agentic applications will begin to work? Or that you've already seen?

• 12:48Speaker 01

  I think on the customer support side, that's a pretty obvious use case. I think Sierra, you know, has emerged there and is doing, is doing quite well there. I think, yeah, the cognition demo was very impressive. I think they did a lot of things right. I think they really nailed a really interesting UX. And that was maybe one of the things that I was most excited about. And then obviously, it seems to work very well. And so I don't know exactly what they're doing under the hood.

• 13:13Speaker 01

  But, but those type like coding, coding problems in general, we see a lot of people working on. I think you, there's a really nice feedback loop that you can get by just like executing the code and seeing if it works. And, you know, as well as the fact that the people building it are developers and so they can, they can test it. Coding, customer support, there, there's some interesting stuff around like recommend, like recommendation chatbots almost. So I draw a distinction between that and customer support. Where with customer support, you're maybe trying to explicitly kind of like resolve a ticket or something like that. And the, and the recommendation bit is a bit more focused on like a user's preferences and, and what they like.

• 13:55Speaker 01

  And I think we've seen a few, I think we've seen a few things emerge there. But I'd say customer support and coding are the two. Klarna as well, you know, they came out and had a pretty good release.

• 14:06Sarah Guo?

  One pattern that I think is very popular, and I can't tell if it is real or transient, is whether or not companies will be able to switch between different L M models. Right? Whether it's a, you know, self-hosted, like dedicated inference, you know, instance for, for them, or if it's an actual API provider. But for any given application, take your prompts and go from, you know, Anthropic to Mistral to OpenAI to something else. In, in reality, it feels like, you know, the way an application

• 14:47Sarah Guo?

  Uh, response is probably going to be sensitive to the fact that these LLMs are actually going to predict differently. Like, what do you think about this? Can you, can you switch? Is that a real pattern?

• 14:55Speaker 01

  It's not as easy as it seems like it should be. And I think the main thing is that the prompts still need to be different um for each model. I do think um the prompts will probably start to converge in the sense that if you think the models are getting more and more intelligent, then like hopefully these small idiosyncraticities don't matter as much. Um, and as more and more model providers start supporting the same things, um, then that will make it easier. And what I mean by that is, you know, so many prompts for OpenAI, which is, you know, the leading and most used one, use function calling. Um, and you know, up until some period ago, like no other models did. And so you just like couldn't use those prompts at all. Uh, but now like Mistral has function calling and, and, and Google has function calling.

• 15:40Speaker 01

  And so I think they're a little bit more transfer- transferable there.

• 15:42Sarah Guo?

  What else is on that list? There's function calling, there's visual input. Like, what else is going to differentiate these um model APIs?

• 15:51Speaker 01

  Context windows one as well. So I think this gets to like, yeah, what's the right context that you can be passing? If it's longer, you know, if, if that changes, then that changes, that doesn't, like, that changes the whole architecture of your application. Um, modality is one.

• 16:06Speaker 02

  Prompt injection for safety.

• 16:10Speaker 01

  Yeah, I, I think that's interesting. Um, I think that's a real enterprise concern. I think a lot of the agents are still just figuring out how to make agents work. This is a different axis almost, but to the point around like switching models, I do think we see a desire for this, especially when you start going to scale. Um, so I think it's like make something work with GPT-4, but then, okay, you're rolling it out. Are you really, is that like, you know, are you really going to eat that much cost with GPT-4? Can you use GPT-3.5? Do you want to fine tune? Uh, and so I think that's, that like that transition is where we really start to see people um thinking about switching models. Um, there's definitely some switching models at the beginning, like if you just want to play around with different models and see their capabilities.

• 16:53Speaker 01

  But I think the most like pressing need to switch models happens when you go from prototype to, to scale. Cost and latency would be differentiators there as well.

• 17:02Speaker 02

  One thing you mentioned that I thought was really interesting is just context windows. And obviously Dem and I launched with um. A million token context window, and I was just curious how you think about context window versus RAG versus other aspects of the model and how all those things tie together. And, you know, once we get to very long context windows and the tens of millions of tokens, like, does that really shift things radically, or how does that change functionality? And so I was just curious, since you've thought about how all these things piece together, I was just curious how you think about those different factors and what they mean.

• 17:33Speaker 01

  Very good question that a lot of people are thinking about who are a lot smarter than me. I think, I mean, a few thoughts. I think like longer context windows definitely make like, single-shot things much more realistic, like extraction of elements in a long PDF. You can do that one-shot. RAG over a single long PDF, or like five long PDFs. Okay, cool, you can do that, you can do that one-shot. There, I think, there are definitely things at scale that don't fit, you know, into a single context window. There are also things where it requires iterations. You need to like, decide what to do, interact with the environment, get that back. So this whole idea of chaining and agents, I don't like,

• 18:16Speaker 01

  that's, that's less around context windows and more around interacting with the environment and getting feedback. And so I don't think that's going anywhere. I think with respects to RAG in particular, because I think that's where it often comes up, like, you know, did this kill RAG? I think there's a few things. Actually, just today, one of our team members, Lance Martin, there's that, like, everyone's doing the needle in the haystack thing, and now all these models are like, green across the board, for whatever reason. They've all figured it out. But I think, like, that, that actually really doesn't reflect a lot of RAG use cases, in my opinion. Because like, that's, the needle in the haystack is like, okay, given this long context, can I find a single information point? But oftentimes, RAG is about seeing multiple information points and then reasoning over them. And

• 18:59Speaker 01

  so I think with the benchmark he released is exactly that. Like, as you increase the number of, of needles, you know, performance goes down, as you might expect. And then also, when you ask it to reason, rather than just retrieve, the performance drops as well. And so there, I think there's more work to be done there. And then I think another thing is just around the ingestion for RAG in the indexing process. Like, a lot of attention has been paid to like text splitting and chunking and all of that. And I don't know exactly how that will change. Like, will you still do that, but you now just retrieve the whole document? Like, we have a concept in Lang Chain of like a parent document retriever, which basically creates multiple vectors for each document. So maybe you just do that. Maybe you still, maybe you chunk it up into

• 19:43Speaker 01

  larger chunks and just retrieve those larger chunks. Maybe use a traditional search engine, like Elasticsearch or some. I'm not, I'm not sure. That's probably the place I have the least confidence in.

• 19:53Speaker 02

  The one other area that I see a lot of people talking about, and I see fewer people actually doing, is fine tuning. And to some extent, I think that's because with fine tunes, you lose generalizability. And so people just start focusing on prompt engineering or other ways to effectively get the same performance without the actual fine tune. But it's something that feels very awkward on, and people talk about it a lot, and people talk about doing it a lot. You probably have a great perspective, since you see so many different types of customers. Are you seeing a lot of fine tuning happening in the wild? And if so, there's specific common applications or use cases for it?

• 20:24Speaker 01

  We see people experimenting with it. I think the only real place where they're doing it is when they've reached like really critical scale, which I still don't think is that many applications to date. I think there's a lot of difficulties with it. One's like gathering the data set for it. And so I think a lot of the things we have in Lang Smith tackle a lot of these issues, but like gathering the data set for it. So like having that data visibility and starting to curate that data set. Evaluating the fine tuned model. So like evaluation and testing is a huge pain point there that we're trying to tackle in a few ways. The third is just like, yeah, back to this point of people are still just like experimenting so rapidly. It's much harder to change a fine tuned model than it is to change a prompt, or even change Fushat examples. And so I think

• 21:09Speaker 01

  we're seeing more and more people use Fushat examples, but not a ton graduating to the fine tuning. Just because, yeah, I think much harder to just like iterate quickly on.

• 21:21Sarah Guo?

  In terms of other major changes in the landscape, it's been, it's been a big year. The first commit to Lang Chain, I think, was in October of 22, which is like when I launched Conviction as a fund as well. At that time, we didn't have Llama2, we didn't have Mistral. There were not nearly as many open source models with

• 21:43Sarah Guo?

  What people would consider to be more useful reasoning ability. Has that changed in terms of like what you see application developers do with LangChain?

• 21:52Speaker 02

  Gemini too.

• 21:53Sarah Guo?

  Oh, and Gemini, yeah.

• 21:55Speaker 01

  Fun story about that, the original models that we launched with OpenAI actually deprecated like a month ago. So the like actual original LangChain, you can't run because the models don't exist anymore. But yeah, like there's, I think we see increasingly interest in open source. But the reasoning abilities are still just like lagging behind Claude three or GPT four. And I think like for a lot of the applications that again, it kind of, it probably depends on the types of applications that you're building. But a lot of the applications that LangChain is focused on with this kind of like reasoning aspect, those are just so crucial.

• 22:34Speaker 01

  And I don't think we see super compelling.

• 22:41Speaker 01

  I don't, I still don't think we see super compelling reasoning abilities in the open source models. And maybe that's one of my hot takes, but I think for a lot of the LangChain apps, the open source models maybe don't live up to a lot of kind of like the, the Twitter hype or Twitter excitement, at least not yet.

• 22:56Sarah Guo?

  Zooming out, like you have a really broad view, like what do you feel like that no one is working on that's going to enable better applications that should be?

• 23:05Speaker 01

  I think the most exciting stuff is at the application and UX layer right now. I think that's where the most exciting stuff is there. One of the, I don't know if this is, this, this is maybe, this isn't more the capabilities side-ish, but like memory, I think is super interesting, especially like personalized long-term memory. I don't know if, I don't know if it's necessarily tooling so much that needs to be built there as it's just like an application in a UX that's really focused on that. And, and you know, if I, if I wasn't doing LangChain, if I was starting a company right now, I'd probably start something at the application layer and it would probably be something that really takes advantage of like long-term memory.

• 23:45Speaker 02

  I guess at the, the high level, someone really, is there anything that you view as like a major prediction or things that'll change over the next year that nobody's really? Paying as much attention to.

• 23:54Speaker 01

  Memory is a big interest of ours. And so I hope that we'll have some kind of like breakthroughs there. I think a lot of the specifically around, yeah, learning from interactions, incorporating that back in at a user level. In a similar vein, also this type of more like system level memory, I think is really interesting and building up, building towards this idea of almost like continual learning. So there's, you know, like, can you learn from your interactions? And you can do that in a variety of different ways. This may just be where we sit in the ecosystem. But one exciting and probably under talked about ways is just the idea of of building up few shot example data sets and really using those. I think it's much faster and cheaper than fine tuning models.

• 24:37Speaker 01

  It's easier to do than trying to like programmatically change the prompt in some way, like that's still kind of like a bit of a art.

• 24:47Speaker 01

  And so, yeah, continual, towards continual learning with few shot examples is maybe one like really interesting area that that we're excited about.

• 24:54Sarah Guo?

  Can you help our listeners like just imagine like a little bit more viscerally, like what type of application experience that would enable? Like, you know, a consumer application or a business application of what that type of continuous learning would allow you to do?

• 25:10Speaker 01

  Yeah, absolutely. I think at a high level, it would basically allow the application to automatically get better over time. And it could get better in the sense that it's just more accurate. So, you know, it's it maybe, you know, it first does a mistake. You then like tell it that it made a mistake and it automatically kind of like incorporates that either as a few shot example or update to a prompt, but it starts learning from its its mistakes and its successes as well. Right? There's a really cool project called DSPY or DSPI. I don't know how to pronounce it, but it's out of Stanford.

• 25:39Sarah Guo?

  I say Dispy.

• 25:40Speaker 01

  Oh, no, there's, there's three ways now.

• 25:43Speaker 02

  I say Diaspa. No, I'm just kidding.

• 25:47Speaker 01

  So and I think that actually tackles like I actually see a lot of similarities between that and Ling-Chen Ling-Smith in some way. And I think it's all towards this idea of like, so DSPY or DSP or whatever, it's basically this idea of like optimization. You have kind of like inputs, outputs. You then have your application, which they similarly think as like multiple steps. And you basically you basically optimize your your application through a variety of different ways. The main one of which I would say is probably few shot examples, although we'll probably do a webinar with Omar and he can correct me if I'm wrong. And I think the idea of like continual learning is basically doing that optimization, but in an online manner where your feed, you don't have like ground

• 26:32Speaker 01

  truth necessarily, but you get feedback from the environment, thumbs up, thumbs down if things are good. And so I think, yeah, that kind of like optimization loop, whether offline or online, is really, really exciting. And I think a similar thing could maybe, I think you can think of like personalization also as like what this would look like to end users and maybe like consumer facing apps. So you start with like a generic application that does the same thing for everyone. But then it maybe learns to to search the web differently for for me and ELAT or something like that. And so I think that's like concretely how it could it could manifest.

• 27:05Speaker 02

  Cool. Thanks so much for doing this. It's obviously a pleasure to have you on.

• 27:09Speaker 01

  No, thank you, guys.

• 27:10Sarah Guo?

  Good to see you. Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.