Ep 28: LangChain CEO Harrison Chase on the Current State of Eval and Agents and The LLM Apps that Will Define 2024

Last week LangChain announced a $20M Series A led by Sequoia and released the paid version of LangSmith, which has already been used by 1K+ teams and driven 80K signups. On this week’s episode of Unsupervised Learning, we sat down with LangChain Co-Founder and CEO Harrison Chase to talk about the current state of LLM evaluation, observability, and the agent landscape.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This is Unsupervised Learning, Redpoint's AI podcast. I'm Jordan Siegel and I'm joined by my co-host Jacob Effron. Today, we had on Harrison Chase, the founder and CEO of LangChain. LangChain is the most popular framework for working with LLMs. It has incredible traction, including 38,000 Discord members and incredible companies using it, like Elastic, Dropbox, and Snowflake. Harrison gave his take on a number of topics, including the agent landscape, the fact that despite being in the center of the ecosystem, the only thing he can predict is that no one will know what will happen in AI. And finally, he gave his advice to startups building in AI. I've been following Harrison for a long time and I was super excited about this episode. And stick around for a debrief with me and Jacob.

• 0:39Jacob Effron?

  Well, Harrison, thanks so much for coming to the podcast.

• 0:41Speaker 02

  Thanks for having me. I'm a listener, so it's exciting to be here.

• 0:44Jacob Effron?

  We've been looking forward to this one for a while. When Jordan and I first started, I feel like you were on our set of guests. We were like, we need to have Harrison on at some point. So it is very exciting to have you on.

• 0:53Speaker 02

  I mean, we were talking earlier and you said this started about a year ago. That's like right around when LangChain started and all the craziness started. So I think we're kind of joined in some way, right?

• 1:02Jacob Effron?

  Exactly, exactly. You may be a little more in it than us, but.

• 1:05Speaker 02

  Yeah, I don't know.

• 1:07Jacob Effron?

  Well, to kick things off, you know, one thing I thought was interesting in your background is it feels like a lot of your journey in the AI world, you know, initially was motivated via the sports world and sports analytics. And I think that's how you initially got interested. I'm sure your mind has gotten racing on like the different potential applications of LLMs and like the sports world and sports analytics world. Anything you've seen that you thought was particularly interesting or things that if you weren't building LangChain, you're like, that would be pretty cool for the sports world?

• 1:33Speaker 02

  Yeah, fewer than I would have wished, I think. Like, I would have loved for there to be like a really clear connection. One that I did see, I'm forgetting the name of who did it, but it's basically natural language to query stats about players. So I think there was a site that was doing this before LLMs, I think. I forget the name of the site, but basically it was natural language. You could ask a question like, who leads the league in three pointers or something like that, and it would query a database and give you an answer. And so I saw an application of doing that with LLMs, basically making, making all these stats more available for people. I think that's interesting. I think like, I think, yes, stats obviously drive a lot of fantasy gambling, just general interest around the sport.

• 2:18Speaker 02

  And so having that more accessible is always interesting, but I wouldn't call that like super unique or super revolutionary.

• 2:24Jacob Effron?

  Yeah. Anything in your like dream, the dream, you know, future of, of what could be done?

• 2:27Speaker 02

  Oh, one thing that I think would be really cool is basically like generating commentary for sports. So I think, if you look at like a lot of the cool applications of this technology, I still think, to me, a lot of the really cool ones are the creative ones. So I think like mid journey, character AI, Suno, all of them are really creative. And so I think like, yeah, if you could generate commentators for a game without having to have them there, or I think this is the next level, like personalize it to the viewer that's watching or something like that. I don't know how close we are to that, but I think that would be really cool.

• 3:04Jacob Effron?

  Yeah, it'd be like this three pointers, like that shot you watched last Friday night, you know?

• 3:07Speaker 02

  Yeah, yeah.

• 3:08Speaker 00

  That's super cool. Do you know if like NFL or NHL or anyone is using LangChain?

• 3:13Speaker 02

  We, yeah, we did chat with one NBA team. I think it was mostly for like internal, internal ops, so letting people kind of like query a bunch of the data within the organization. So I don't think anything consumer facing.

• 3:26Speaker 00

  I think it would be great for the audience to understand sort of the breadth of LangChain at this point. You guys are just doing so much and covering so much surface area. So it'd be great to go into that a little bit more as well as LangSmith and sort of how that fits into everything, too.

• 3:39Speaker 02

  Yeah, yeah, we do a lot, so I'll try to remember it all. I mean, maybe just backing up, like I think LangChain started as, I mean, a framework for building LLM applications, which itself is very vague and horizontal because LLMs themselves are very vague and horizontal and good at general purpose things. And so I think a lot of what LangChain, the open source package was founded around was this idea of an orchestration layer to kind of like facilitate building of some of these applications. And I think one of the things we saw pretty quickly on is there's such a wide breadth of applications that you can build. So I think like, Probably the most common one that people are building is some sort of like chatbot over

• 4:20Speaker 02

  your data. And so a lot of the functionality that we have is around chat, is around streaming, is around retrieval. That's a big part of it as well. And so there's a lot of functionality there. But there's a ton of other use cases as well. So extraction is a huge use case in enterprise. There's some cool, I don't know if, yeah, like assistant-like things. And so that's a very, that itself is like horizontal. So I guess maybe trying to get like a little bit more specific, like access to some sort of search engine. So like the typical kind of like retrieval chatbot is like retrieve things, put it in context. But you know, you could mess up the retrieval step. You could need to do multiple retrieval steps.

• 5:04Speaker 02

  And so this idea of like an assistant is useful there. It's also useful for like SQL, text-to-SQL is a big use case, text to like an arbitrary API. So I think a lot of what LangChain is, is connecting LLMs to external sources of data and computation. And there's tons of forms of data and computation. And so it's a really wide breadth. And that's like the orchestration layer in LangChain. I think one thing we realized pretty early on is that in order to, you know, with the mission of building or making it easy to build LLM applications, you don't just need an orchestration layer. There's a lot of other things that you need. Some of these are vector databases, which has emerged as kind of like a big category. And we're not doing anything there. One of the areas where we did see a big need and where we are doing stuff is around kind of like the

• 5:48Speaker 02

  observability and testing and evaluation layer. And so that's LangSmith. That's a separate kind of like SaaS platform that as of recently is generally available. We've kind of been iterating on it for the past six months. We saw this be a huge issue as teams were kind of like go from that prototype to production. And the fun thing is, I'm sure there are other areas that are also needs that we'll discover as we go along. And maybe we'll build some, maybe there will be other companies that come along and build those. But yeah, general purpose of LangChain, make it as easy as possible to LLM applications, started with orchestration layer, now have observability testing, a lot more to be built.

• 6:25Speaker 00

  Awesome. And then, you know, I'm sure a lot of people give you advice to focus just on one thing, let's say retrieval or something like that. What made you decide to sort of go across, you know, a lot of different surface areas?

• 6:36Speaker 02

  I think they're all related in some sense. So if you look at like retrieval, a lot of the interesting retrieval techniques that are coming out are very agentic. So there's kind of like self-rag and corrective rag, which both have this idea of like doing a retrieval step and then looking to see if it was good enough and then correcting itself. And so from the, and then, so, okay, so I'd say like three main areas that we focused on from the beginning are retrieval, agents and evaluation. And I think they're all super related and there's like a flywheel between them. So like agents can be used to do retrieval. Retrieval are maybe the most popular tool to give agents. Evaluation is needed for both of them. Agents can be used to do evaluation. So I think there's a lot of, getting back to this idea of like, language models are this amazing new technology because they're so general.

• 7:21Speaker 02

  Like that's, that's a lot of the value in them. And so, yeah, I think there's just a lot of connectedness between all these different components. You know, I would say we've tried to focus a little bit on those three main things. Obviously, Langsmith does a lot. There are more focused packages that do a great job at some of the smaller niches. But with retrieval, evaluation and agents, we see those being so core to so many different things. It's tough to just say do one because I don't entirely know what that would mean.

• 7:51Speaker 00

  I'm curious on the Langsmith side, a lot of people are super excited about that. What type of production sort of applications do you think are most suitable for Langsmith? And what do you think sort of the, the main user would be for that?

• 8:03Speaker 02

  The, the obvious ones where there's just so much value are when you're building things that are multiple L M calls or multiple steps. Because I think the quickest kind of like way to see the value of Langsmith is, is with the tracing and observability that provides. So it logs all the steps of your chain or agent, the inputs and outputs in the exact sequence. And so when you have these more complex things that are, maybe you don't even know how many steps they're going to take. Or if you do, like there's multiple of them, having that is just super valuable and being able to understand what's going on, debug it, dive into it. So that's kind of like the clear, like if you're building more complex applications, I think you pretty quickly get value from, from the observability part of Langsmith.

• 8:47Speaker 02

  I think there are a lot of aspects where it's a single prompt and there is still value. And so specifically, I would say, when you have prompts that maybe have three or four variables that you're templating into it, even if it's a single call to an LLM, it's still nice to be able to see what exactly it is, what it looks like, what's the arrangement of everything. You maybe have conversational history, like what parts got trimmed or not by the time it goes into the LLM. And so even for like, single calls to LLM, there's still value there. I think if you're doing like, if you're just doing a straight pass through to OpenAI, there's probably less value there. And then on the testing side, on the testing side, it's interesting because we see people testing all across the spectrum.

• 9:30Speaker 02

  So what I mean by that is, we see people testing end-to-end applications, but then also the individual components. And I think, I think for both, I wouldn't say like, either one, we test better. Like, I think, I think testing is kind of similar across those two. And by, what I mean by that is like, if you have, if you have an assistant, you could test like, just the user input and then what the user finally sees, or there's maybe like, a routing step where it chooses what tool to use. And so you could test that. And so for those different types of tests, I think we support both of those pretty nicely.

• 10:01Jacob Effron?

  And obviously, like, you know, I feel like a big part of Lang Smith is eval, which is something that, you know, feels like everyone's talking about these days. And I'm curious, like, you know, how do you characterize like, the current state of eval? Like, what can and can't we do today? You know, obviously, the dream is just to have LLMs evaluate themselves. Like, where are we kind of in the eval space as a whole?

• 10:19Speaker 02

  This is a great question, and we're spending a lot of time talking with teams about this. I think there's a lot of questions that teams are grappling with. And so, maybe just doing a quick rundown, and we could probably do a whole podcast on this. But, but I think there's a few key questions that we see teams thinking about. One is like, what, so the, the basic premise that most people are realizing is, you need some data set, and you need to test it against your system. And so the first question is, how do I get that data set? And so typically, what we see teams doing is hand labeling, coming up with the first 20 or so examples, then turning to kind of like production data and seeing edge cases that fail and incorporating those into the test set.

• 11:01Speaker 02

  And so actually, similarly to the open source, like I'd say Lang Smith does a lot and it's an all in one platform. But we see a ton of value in having that connectedness because in one of those right off the bat is this connection of like production traces that fail, either marked by thumbs down or get flagged through some other system, pull that into your eval set. So the first thing, or first questions that we see teams thinking about is how do I gather this data set? Then there's how do I evaluate like a single data point? And for some, it's easy, classification, you can use traditional ML metrics. For a lot, it's not as easy. And so using LLMs to judge is emerging as a popular technique. Definitely, this adds an interesting wrinkle because it's not perfect. And so there's like, there still needs to be kind of like a human in the loop component.

• 11:46Speaker 02

  And so actually, I mean, maybe even backing up, I think like the goal would be to get to these automated metrics. We still see a lot of teams just looking at it and compiling it there or scoring it and maybe counting them up and comparing things side by side. So we've invested a lot in this like human in the loop. How can humans just interact and facilitate this eval process? Then there's an interesting question of how you aggregate those metrics. So do you want to have everything perfectly scored? Do you want to have, do you just want to see that it's better than the previous prompt that you were testing out or something like that? And that it completely depends, people are doing, you know, both if they have, if they have like a set of data points that they know they really need to get right, like 100% accuracy.

• 12:29Speaker 02

  Other times there's a more general thing and they kind of just like measure that. And then the last component is like, how often do you actually do this? Like, because it's expensive to do this and it's slow and there's error, it's error prone. And so I think we typically see people doing this largely before kind of like releases because there is still a big manual component. To it, one of the things that we're thinking really heavily about is how to reduce that manual component enough so that you can run it. The goal would be in CI, like you run software unit tests. I don't think we're quite there, but I think like pushing towards that, we're working on a bunch of stuff there.

• 13:04Jacob Effron?

  It's interesting. It feels like the manual part of actually reviewing a lot of the exceptions is where so much of the value comes from. We were talking with Linus from Notion, and he's like, that's literally where we figure out everything about how these models work when they do things that we don't expect them to do. And so it feels like that, you know, today is kind of in tension with like this idea of, well, it'd be really nice if it just kind of automated in the background and did what we needed it to do.

• 13:24Speaker 02

  A hundred percent, yeah. And I think there's like, I think there is some stigma like, oh, you have to look at data points, like, why don't you have an automated eval or something? But I think there's so much like value that like, again, like, it's just so early on in the space and everyone should be trying to like, figure out how to, how to work with these models, how to grok these systems. And so looking at things provides not only the benefit of like, knowing how it did on this data point, but also gives you kind of like a deeper understanding into the system and all these new things that everyone's still trying to learn. And so, yeah, we've invested really, really heavily in making it easy to see the thought processes of these agents and the, and the inputs and outputs of each step precisely for that reason. So it's easier for people to grok it because we don't think it's, we don't think it's bad. Like you get, you get a lot of value from doing that.

• 14:08Jacob Effron?

  Totally. We have a lot of listeners who are like thinking about how to do eval on their own products, you know, besides using Langsmith, which obviously they should do. Is there anything that like you've seen like best practices emerging or folks that you think you're like, yeah, that, you know, company X does it this way and that's, that seems super smart?

• 14:22Speaker 02

  I was just chatting with Linus today. I think, I think looking at data is underrated. Similarly, just coming up with an evaluation data set is so valuable. Besides the fact that it gives you something to actually use, it also forces you to think about what you want the system to be doing. Sometimes I think, you know, again, LLM is this great general purpose thing. You can come up with an idea, just throw it at it. Fantastic. You get started quickly, but like, what are you expecting it to actually do? Like, what are the educations it should handle? How do you expect users to be interacting with it? These are all questions you should be thinking about and coming up with an evaluation data set is a really good forcing function to do that. So I would say, yeah, just look at the data and a big part of that is coming up with this evaluation data set.

• 15:02Jacob Effron?

  Yeah, it feels like almost at the start of your like product building journey. That's such a key part of figuring out like what you actually want to build.

• 15:07Speaker 02

  And it's really funny because in traditional ML and data science, you had to do that before you could build a model. And now you don't. And so I think it's, you know. You still should do it.

• 15:17Jacob Effron?

  And do you feel like we're, you know, is this kind of manual, you know, reviewing of data? Is that like a moment in time thing? Cause we're still not like, you know, these models will get better and better. Like when we get to like GPT seven, like, how do you think eval ends up working?

• 15:29Speaker 02

  Yeah, that's a really good question.

• 15:32Speaker 02

  I, I, I don't know is the honest answer. I think though there's, it's definitely probably more valuable now as we're trying. As it's still just so early and so fast moving and, and probably as these things mature, get more reliable. Hopefully we start seeing like more complex systems being built up around them. You know, you'll, you'll have to, you, you won't be able to do that to some extent. So, so I would say, um, it's probably more important now than it will be in a few years. But I still think it will be somewhat important in a few years. But I don't know, it's really impossible to predict things. So.

• 16:14Jacob Effron?

  Totally. And as you built an eval product, like, how have you thought about like, to what extent eval is generalizable? I imagine like, you know, across different use cases, there's some things that hold true. Like you need to come up with your, your own data, testing data set. You know, on the other hand, use cases can be so different. How have you thought about that?

• 16:31Speaker 02

  Yeah, that, that's a, that's a great question and a great point. And before this, I was working at an ML ops company, which faced a lot of the same issues, arguably more so. Because I think in ML ops, you, you know, at least with LLMs, they're text or messages. And so, well, now they're multimodal. So maybe that, but, but I think there's a slightly more uniform, kind of like input, output schema. With ML ops, you're really doing there. It's, it's really tough to come up with kind of like unified things. But yeah, I think it comes down to figuring out what the, uh, I, I, especially at this point in time, I don't think you can afford to be, have your abstractions too high level.

• 17:08Speaker 02

  But I also think you want to be somewhat opinionated in how people should do things, or at least like what's important. And so I think practically what that means for us is like, low level, really focus on what we think are the low level important things. Those would be kind of like gathering data sets, seeing what's going on. We're completely kind of like code first. So, so we do have some UI components, but everything's exposed via API. We're very much targeting developers here. Framework agnostic, so you can use LangSmith with or without LangChain. What we assume is the system that you're scoring, we really just treat as a function. So like, we don't make any assumptions about the number of LLM calls or anything under the hood. And what you provide as an evaluator,

• 17:50Speaker 02

  we just treat as a function as well. So, you know, we're, we're building more scaffolding around certain things. We're building more scaffolding around LLM as a judge assisted things because we see that being popular and easily parameterizable by a prompt. But at, at the core, yeah, just, just really simple and hopefully, you know, generalizable components. But it makes it tough because, because I absolutely, if we think about LangSmith, there's two components to it. One's this like tracing observability and the other is the testing. And comparing them is, is really, really interesting. People can get value out of the tracing and observability, like it within seconds of setting up. Like, it's just like super clear magic moment. Testing and eval, like you still have to do a lot of work to get there.

• 18:33Speaker 02

  You have to get that data set. You have to think about the metrics you're using. And so there is still a lot of burden on, on the user. And so, you know, if there's too much burden on the user, then like, what value are you even providing them? And so I think there's some aspect there. I think with a lot of this like human in the loop component as well, and like reviewing and just making it easy to understand that there's value that we can provide there. But it's definitely, it's definitely tougher than the observability space, I think.

• 18:57Jacob Effron?

  Do you think you ultimately, like, I feel like there's all these folks independently coming up with their own testing data sets and their own edge cases. And there's probably a lot of like great learnings that are, that are happening in that process. Like, do you feel like you end up facilitating some like best practices across that? Or like, how do you see that kind of getting shared today? Or is everyone independently kind of coming up with their own?

• 19:15Speaker 02

  I mean, so everyone's definitely coming up with data sets that are specific to them. Like, I think you have to for serious application. There's probably some sharing of metrics in that, like, people are using LLM as a judge. There's just, there's probably some general prompts of like student-teacher. I don't know if they're exactly the same prompt or the similar ones.

• 19:36Speaker 02

  But, but then, within data sets, for example, if we talk about RAG, for example, there's a bunch of common kind of like RAG failure modes and failure cases. I think there's some discourse around what those are. I think honestly, that's one of the things that we can maybe do better. And I think we probably, I mean, my, my view on the landscape as a whole is like, it's, some best practices are starting to emerge. And I think the first, like, six months, just tons of experimentation. I think the next six months, people were like, all this experimentation is cool, but how do I actually get this into production? And so I think now we're starting to see some things go into production. Some pretty impressive ones, too, like Elastic had an assistant that we worked with them on. And like, it's an assistant in production by a very large company, right? And so that's awesome. And so we're starting to see things come into production.

• 20:20Speaker 02

  So then I think we're starting to see some of the best practices emerge. We should probably do a better job of facilitating some of the discussions and sharing those learnings. And we have some things in the work there. But that's my, yeah, that's my general view on, I guess, the hype cycle of AI over time.

• 20:34Speaker 00

  I guess, switching gears from the eval side to the agent side, I'd love to hear your thoughts on the agent landscape right now. What is sort of possible, what is not possible at this point? And when do you think that actually changes?

• 20:45Speaker 02

  Agents are obviously super interesting. We've done stuff. I, I think people think of LangChain maybe as an agent framework, kind of. I guess we do so much, it's tough to know what people think of us as. But I think we were, we were always doing agent-like stuff from the beginning. We were never full-on, kind of like AutoGPT or anything like that. And so I think what's kind of happened is... There was an explosion of interest in agents, AutoGPT leading the charge there. And I think the, I think what people realized, and so maybe there's two types of agents. There's this super generalizable autonomous agent, and then there's more focused ones. And so I think a lot of interest in the autonomous ones to start.

• 21:29Speaker 02

  More focused one seems more practically kind of like ready today.

• 21:32Jacob Effron?

  I feel like we had a full hype cycle with generalizable agents.

• 21:35Speaker 02

  There, yeah. Baby AGI, yeah. And so I think people maybe relatively quickly realize, you know, hey, like you got a, you want like a custom prompt. You want it to have five tools, maybe really focused on an objective. And I feel like people were talking about that and realizing that maybe summer, late summer, early fall. One, one kind of like interesting thing that we've seen is just, and then maybe adding in another thing. There's this idea of multi-agent frameworks as well. So we've seen AutoGen, we've seen Crew AI kind of like pop up. And I think some reaction to that is like, well, if AutoGPT didn't work, how is putting five of them working together going to work?

• 22:17Speaker 02

  And I think the important distinction there is that a lot of what these multi-agent frameworks are, are really kind of like controlled flows between a specific prompt that maybe has access to some specific tools. And maybe jumping ahead to some of the stuff we're doing there, like we think of this as kind of like a state machine that people are putting together. So if you think of agents as a state machine, then you can enforce more control over some of the transition probabilities between where it is. So if you're in a chatbot, if you're in a customer support chatbot, you can have a really dedicated prompt for if it's just kind of like starting the conversation with the user and trying to gather, you know, their name, their account ID. But then if you transition, you can transition to something where it's like they're trying to debug an issue or something. And so there's this kind of like controlled flow between things.

• 23:01Speaker 02

  And I'd say like a lot of the things that we see in production or starting to get reliable are this, these more kind of like state machine type things. So one of the things we released recently is LangGraph, which is a framework for constructing agents as graphs, agents as state machines, same, similar things.

• 23:21Speaker 02

  And yeah, I'm pretty bullish on this as a future direction, just because I think, and like, I think all the multi-agent frameworks are state machines as well. And so I think this, if you think of it as state machines, I think it's more palatable because you realize kind of like how you can enforce specific transition probabilities, what the specific states are. It's a nice, it's a very nice mental model.

• 23:41Jacob Effron?

  I feel like so much of the power of LangChain, and like when I hear you talk about it publicly, is like, it really shines in like these chain of calls examples, or using agents, or more kind of complex use cases. It feels like, you know, last year, there were a lot of developers building basic like chat your docs, or things that might be a little more like simpler than that. What have you kind of observed in the, like, does it feel like we're starting to turn the corner on like more complex applications? And anything in particular you've seen that you're like, you know, I guess you mentioned the elastic one, any other ones you've seen that you're like, that's a really cool, like sophisticated use of all the stuff we built at LangChain?

• 24:10Speaker 02

  Yeah, I think my, yeah, so, okay, so my, my view on this is there's maybe like three different categories of builders that we see, and it's a little bit different between them. I'd say on one, there's like the super early gen AI native startups and companies, and they're being like super like forward-thinking about the types of things that they're building. So they're building agents, they're making them consumer-facing. They're kind of putting them out there. They're all super early, so it's a little bit tough to tell. Then there's maybe like digital native startups somewhere in the middle. There, I think the main things that they have

• 24:53Speaker 02

  largely shipped have kind of been like single LLM calls, but I think now is when we're starting to see more of, I mean, even like Notion QA, I think is really well done, and that's maybe the, like, that's a really production-ready, kind of like at-scale, individual personalized, kind of like chat over your knowledge base. And yeah, I think that's super well done. I forget when they shipped that, late last year, something like that. So I think we're starting to see things like this come, come online from companies like, like that. I think the Elastic Assistant would be another great example of that. And then there's larger enterprises. And one thing that I think is really interesting here is a lot of the work that's happening is internal. And because it's internal, you can, you can take a little bit more risks in terms of what you're building.

• 25:37Speaker 02

  So we're seeing like assistant like platforms being built for internal purposes, similar to like the, the GPT store type thing, but hooked up to your data and, and your tool, your APIs, running whatever language model you want. So, so Open GPT is a project we did that basically recreates the assistant, the GPT store like experience. And we're seeing some companies adopt that for internal purposes for the, and so I don't know exactly like where you'd put that in the classification, right? Like it's still internal, but it is pretty advanced. Like it's a lot, they're building platforms to allow people to create their own chatbots with their own data and with their own APIs. And yeah, I think we're seeing more and more of that as well.

• 26:17Jacob Effron?

  And like, do you have like a go-to kind of like starter kit you recommend to folks? Like when you're talking to an enterprise that's maybe early in their, in their journey, besides reading through all the Langchain docs, like where do you kind of point them?

• 26:29Speaker 02

  Um, we've tried to put together a bunch of templates to get started with like, uh, uh, specific use cases. Because one thing that we have seen is that Langchain is, is very broad and, and has a lot of different components. And so we've tried to put together a bunch of templates and then also a bunch of use case documentation for Q, question answering, SQL, extraction, general chatbots. We've also tried to do kind of like, uh, use case repos almost. So like chat Langchain is a really good resource for a particular thing, right? For like question answering over a single knowledge base. Open GPT is a really good resource for a particular thing. And, and so I think, uh, it kind of depends on what they're interested in.

• 27:12Speaker 02

  But basically, I feel like over the past six months, we've done a lot of work on the basic kind of like orchestration layer of Langchain Expression Language and Linggraph. We've also done a lot of work on. Some of the table stakes, observability and testing stuff. And so now we're starting to focus more on these use case things so that when we talk to larger companies and they're wondering where to get started, we can easily point them to something that's a bit more specific than go read about the, you know, thousand recent advances in AI or something like that.

• 27:40Speaker 00

  You mentioned that LangShang has a lot of different parts. Are there any parts that you think are underutilized today?

• 27:45Speaker 02

  One of the parts that I think is underrated or underutilized is the example selectors. So we've seen few-shot examples be a really powerful way to improve the performance of applications, particularly in cases where you need structured output or where you have really complex instructions that you want to follow. And oftentimes what you can do is you can hard code like one or two or three examples. But another thing that you can do is you can dynamically select like what the top, what like three similar examples to the current question are. And I think this is a bit, I think this is a bit of a pipe dream. We're not seeing this yet, but I think the holy grail of a lot of ML ops, I would say, is kind of like continual learning. And so I think this is a really interesting way to try to achieve that.

• 28:27Speaker 02

  Basically, you see, maybe you see things that the user thought, this can also get into personalization. You see what a user likes, doesn't like, you pull them in as few-shot examples. You then, you build up this data, database of like 200 few-shot examples. You dynamically select the most recent ones or relevant ones, put those into the prompt and use that to like improve the application over time in a super lightweight way. And so this is, that's a really cool flywheel that I think we want to be building towards. You can still use example selectors in a more kind of like simple way, and I don't think people are using those enough.

• 28:58Jacob Effron?

  I mean, I imagine like one interesting part of building this business is you have this like tension where the space moves so fast that you could kind of stop at any one moment in time and be like, well, most people are building, here's the set of things, like techniques that are out there. Most people are building this way, let's optimize around that. Or you could be like, well, in three months, you know, there's going to be a whole host of other crazy different things. Let's not like solidify too much because it's, you know, and I imagine that's something you're constantly thinking about, like, how do you think about balancing kind of building for what's here and now versus staying nimble and flexible for what crazy advances are, you know, around the corner?

• 29:31Speaker 02

  Yeah, I think, I mean, it's a balance. I don't have any great insights. Maybe one is like, you have to build. Like, I think like, early on, people are like, oh, like, RAG's a hack, or these MapReduce things are a hack. It's like, yeah, it's a hack, but it's needed now. So like, what am I supposed to do? Like, not build my application because it's going to change? So I think like, you have to build, would probably be like, first and foremost.

• 29:53Speaker 02

  And then, yeah, I think like, a lot of, a lot of what we spent time on over the past six months is thinking carefully about kind of like, what the right abstractions are. And I would say, like, as a framework, early on, we had, we didn't have the flexible kind of like orchestration layer of Langchain Expression Language, or of Langgraph. It was very much higher level chains that had different parameters. And if you wanted to change the internals of the chain, you had to modify the source code, or add a different parameter, or something like that. And we saw, we saw people want to customize that. And so we spent a lot of time in these more flexible, lower level things.

• 30:34Speaker 02

  There's also, there's also like, the individual components themselves. So like, the retriever abstractions, and the vector store abstractions, and the model abstractions. Those, I think, we've done a better job at. I think the retriever abstraction, and I think what we've aimed for there is really kind of like, I was actually chatting with someone about this today, but we've aimed for a kind of like, really dead, simple, base class abstractions. And you could argue that's bad, because then for a specific implementation, you have to do more work. So the example that Andrew provided today, when I was chatting with him, was like, for language models, there's this common concept of like, retries. We actually have, we don't have retries in the base class. We have it in individual classes. And that's a little bit annoying, because if I want to add a new class, then I have to add retries

• 31:16Speaker 02

  to it. But that's kind of nice, because we don't make any assumptions about how retries are handled, or the logic for that in the base class. And so if you wanted to do it a different way, you could do it there. And so I think, for the base classes, we've really optimized for like, really, really simple.

• 31:32Speaker 02

  And I think we did, I think we did a pretty good job at that. I don't think we did an amazing job at having like an actual flexible orchestration layer until about six months ago when we released like LangChain Expression Language and we've improved it a lot since then. And I'd also say, you know, we very consciously waited, we released LangChain 0.1 early January, we very consciously waited until after multimodal kind of came out. Because we were worried it would mess up all our abstractions. When chat messages came out, you know, we had to rewrite a lot of the internals and luckily multimodal wasn't actually that large of a change in some of our abstractions. But yeah, I think that's a little bit about how we think about it. And also, you know, we've released LangChain 0.1. We think some of the abstractions are more solid now.

• 32:15Speaker 02

  Hopefully OpenAI doesn't completely prove me wrong right after I say this.

• 32:19Speaker 00

  I saw the 0.1, I was like, I thought they're at like one or two at this point.

• 32:25Speaker 02

  Yeah, we wanted to wait until after that. And I think, yeah, we'll see. I mean, it's, but this is so tricky as well because like we have a few things we want to do for 0.2 and kind of like push ahead, but there's so much work that we can kind of like do to improve the existing state of things. And I think right now we're probably more focused on that. We're more focused on kind of like the improving the documentation, improving the use cases. I think again, like OpenAI is gonna probably prove me wrong, but I think things are more stable now than over the summer. So yeah, over the past few months, like or over the summer, over the past year up until a few months ago, super hectic, we were heavily, we were way more heavily on the, on the like,

• 33:06Speaker 02

  don't solidify anything, kind of like be, be very abstract and vague. We do think things are more stable now. We moved to a stable release. We're moving in that direction. Could be wrong, but that's kind of like our read on this, on this state of things.

• 33:20Speaker 00

  You mentioned the space is fast moving and we're seeing some parts of LangChain and other AI companies where there's sort of an equivalent in the traditional applications. We can take observability as an example. How do you sort of think that will evolve over time? From a competitive landscape and, you know, them potentially going into the LLM side of things and, you know, if not, how would you sort of articulate the LLM problem space being vastly different than the traditional application side?

• 33:44Speaker 02

  Yeah, I think there's.

• 33:47Speaker 02

  I think there's definitely parallels. I think the space is super early, super fast moving. Not to repeat it for the seventh time, but that is like when people ask like, I don't disagree with you there. When people ask like, hey, what are your thoughts on the space? Like, that's the only thing I have confidence in. Like, I can't predict what OpenAI is gonna do, but I know the space is early and it's super fast. So I really do think there's a ton, and I think, you know, I think it will be a hugely impactful space. And so I think there's a ton of opportunity to build not only applications using this technology, but I think it is a pretty different technology in some key ways.

• 34:25Speaker 02

  It's non-deterministic compared to ML. It's a lot of APIs and prompting, and these are new things compared to traditional observability. So this is kind of interesting because Datadog, you know, awesome player in that space. They came out with kind of like an LLM product, I think like right before we released Langsmith, and they had an integration into LangChain.

• 34:47Speaker 02

  And we've heard some company, we've basically heard some companies want to use both for different things. Like, I just think the value props are different at this point in time. I think we're really focused, like, we see the blockers being, people don't understand what these applications are doing, and so they can't iterate fast enough and with confidence. And so we want to give them the confidence and the tools to iterate quickly. Datadog, I think, is fantastic for kind of like some of the system level monitoring and stuff, and even like just aggregate metrics of latency and things like that. And so maybe, you know, as we push more and more into monitoring, maybe there'll be more overlap there. But I think right now, the value props are different enough. We've seen people use both of us kind of like together.

• 35:30Speaker 02

  I think in general, we're not super focused on competition because the space is just so early. We really think there's so much value to be created here. And we think there's a bunch of different ways to do it, and we're doing some of it. Some other people are doing some of it. There's probably like 80% that no one's doing because no one really knows what that is yet because it's still so early.

• 35:50Speaker 00

  You recently, we talked about Langgraph, Ling, LingSmith, and you know, I'm sure there's a ton of other Ling products. One that you launched recently was LangServe to deploy apps. Would love to hear sort of why you decided to, to release that and sort of where that fits in relative to other ways to do it in the space.

• 36:09Speaker 02

  Yeah, we, we want, we want that to be the easiest way to deploy, kind of like LangChain applications. I think the, like, we didn't release this for quite a while, I would say, like, I think, like, one of the first, you know, when, when you think about how to make a company around LangChain, one of the first things you think about is like a hosting platform for that and deployment. We honestly saw like the, the observability, the testing and eval being a bigger pain point for people. And so that's why we focused more on that.

• 36:36Speaker 02

  Then kind of like, as we saw people want to get in production, we did, we did see some pull there. And so that's kind of like why we invested in it. I'd say like it's, there's a few things that are a part of it. It's a wrapper around Fast API and some other technologies there. So it plugs into a pretty common kind of like Python stack. We, you know, we can introspect the, the LangChain runables. Now that we have this common kind of like orchestration layer, there's a common kind of like input-output schema that we can easily define. They all have the same kind of like invoke, batch and stream endpoints. So, so after, like, if we did this in, if we did this last May, it would have been really tough because like, you know, we didn't have that really solid foundation, I think with LangChain expression language and some of the interfaces around that, which also power Langgraph we do.

• 37:22Speaker 02

  So I think we set the foundation there. I think we were focused on things that we viewed as bigger blockers in, in LingSmith. With LangServe, I think it makes it easy to deploy. And then another interesting thing, which I'm pretty bullish on is we, we like quickly spin up a playground for you as well. To interact with the application. And that's not going to be like an end user facing thing for like real hardcore kind of like apps. But it's super useful for giving it to people to test out. So sharing it. One of the really cool things we've seen is that building these applications is a really cross functional kind of like effort. So there's engineers, there's data scientists, there's also like PMs, non-technical people,

• 38:06Speaker 02

  subject matter experts who want to interact with these. And they're often the best people to leave feedback on things, test things out. And so, you know, in Langsmith, we have like annotation queues and stuff like that to help with people for, uh, to, to annotate things. We've, I view these playgrounds as similar things to let anyone try it out and interact with it. And so, yeah, there's, you know, I think there's a few different components, ones that like battle tested, fast API, hook in framework for deployment, um, which we do think scales well. The other is like facilitating this cross functional collaboration.

• 38:39Jacob Effron?

  How big is your team right now?

• 38:40Speaker 02

  We are 18.

• 38:42Jacob Effron?

  How, how, how do you, uh, one, keep all these Langs, you know, going? And then how do you like think about allocating relative, you know, it seems like you guys are doing, uh, a really impressive amount of stuff. How do you think about where to kind of point people?

• 38:54Speaker 02

  Yeah, I think, um, so we're, we're about 50 50 right now between kind of like Langsmith and then other things. And the other things, most of it, um, most of it's kind of like on the existing open source. There are a few efforts on kind of like more exploratory newer things like Langgraph came out of an exploratory newer things. Uh, Open GPTs came out of a exploratory newer thing. So, uh, we're about 50 50 between those. Um, the, the focus in open source, uh, really fluctuates. Like we, we knew we really had to solidify the base kind of like orchestration layer. So I'd say over the fall, we were really focused on that, less focused on kind of like use case things. Now we feel like we've caught up there. We're a little bit behind on use cases, so need to switch over to that.

• 39:36Speaker 02

  And on Ling Smith, I think when it started off, it was probably, it was Ankush, my co-founder, has really been leading all of that. I think when it started off, it was definitely a smaller percentage of the company, but we've seen really good demand for that and we've seen really good interest in that and we use it all the time internally because we think it provides a lot of value. So we've kind of like ramped up our investment in that over time. And then like, you know, right now, we're kind of split there between kind of like feature development on newer things as well as scaling and enterprise readiness because we're starting to get into those types of conversations and that there's a lot of work to do there.

• 40:16Jacob Effron?

  Totally.

• 40:18Speaker 02

  So, yeah, I mean, I don't have a great answer aside from, I think the thing that we're really focused on is how can we provide the most value to people? And so that kind of like direct, like right now, you know, back in the day for opens or a few months ago, back in the day.

• 40:35Jacob Effron?

  It's dog years in AI world, so you call it a year ago.

• 40:38Speaker 02

  A few months ago in open source, you know, we saw that people, like people were struggling because we didn't have good orchestration. So the highest value thing that we could be doing to give value to other people was to focus on this composability aspect. Ling Smith, we saw providing a lot of value and so we're upping our investment there. So we're really focused on just where we can provide the most value and allocating resources there.

• 41:01Jacob Effron?

  I'm curious if like last year, it felt like it was maybe the year of like chat your data, chat your documents, you know, for demos and companies putting into production. Like any predictions of kind of application archetypes that you think we'll see a lot more of this year?

• 41:14Speaker 02

  I think like the, I think we'll see like more complex chatbots. I think a lot of folks who we're talking to do kind of like have these more complex chatbots that are represented as kind of like state machines and they have different. Customer support bot with different stages that it goes through. An AI therapist that has different stages it goes through. I'd say these are pretty different from like traditional kind of like simple rag chatbots. So I think we'll start to see more stuff there. I also think we'll probably start to see like longer running jobs as well. There's some cool projects that we collaborate with in the open source. So GPT researcher and the same team did GPT newsletter as well. And so both of these are like longer running jobs that take maybe a minute or two.

• 41:57Speaker 02

  And I think there's some interesting and they generate like a first draft of a research report or a first draft of a newspaper or something like that. And I think there's some interesting considerations there.

• 42:09Speaker 02

  I think you've got to find some UX or application where you don't need an instantaneous response. Because I think in order to do more complicated things, you need to do multiple LLM calls. You need to check itself. It takes time. And so I think then I think people I don't think I don't think people have found the right applications or UXs for these longer running, still first draft type of things because they can be wrong. But I'm hopeful that may I think we're seeing some cool experimentation there. I'm hopeful that over this year, we'll see applications that kind of like fit that bill.

• 42:41Jacob Effron?

  It's interesting. It's almost like a design challenge or a user experience challenge to get someone comfortable with, you know, leave for for a few minutes or however long it takes.

• 42:48Speaker 02

  I think all the most interesting work right now is in UX for these AI applications. I think that's where there's a ton of innovation to be done. There's a ton of just like figuring out how do people want to interact with these things that isn't really known.

• 43:02Jacob Effron?

  Yeah. Is there a UX that caught your eye recently?

• 43:05Speaker 02

  Yeah, there was a cool tweet that I saw the other week. I'm blanking on the name, but it was basically this. It was the tagline, I think, was like an AI native spreadsheet. So you'd kind of like you'd kind of like fill in one column and then fill in one column, fill out the column headers and then basically click and drag like you would in Excel macro. But in order to fill in each cell, a separate agent would be spun up. And so you'd be executing like 100 different agents in parallel and they'd be populating the spreadsheet one cell at kind of like a time. And I think that was a really cool example of kind of like, I think it kind of gets to this and you know, it would take it wouldn't happen instantaneously. It would take. Few seconds, 10 seconds, something like that, maybe longer.

• 43:49Speaker 02

  But I think that's a good example of how, yeah, you can have like many different tasks going that then you're inspecting the results after as opposed to chat where you're kind of like constantly in the UX one, or you're in kind of like the loop, one question on the time.

• 44:04Jacob Effron?

  Sounds like it could really help your sports analytics.

• 44:06Speaker 02

  Yeah.

• 44:07Speaker 00

  What are your thoughts on like that example, the spreadsheet one in terms of inference costs? And, you know, obviously if it's spinning up an agent for every single sort of cell it's doing, that could, you know, add up really quickly. So how do you think startups should, should handle that in the future? When do you think it will just go down enough that they won't have to worry about it?

• 44:24Speaker 02

  Yeah, I don't think you can do that at scale, but I also, I think if you're a startup, you probably shouldn't be focused on, like costs will go down, latency will go down. When exactly, I don't know, but OpenAI has been really good, really impressive about cutting costs.

• 44:42Speaker 02

  And yeah, I think my advice to most people is generally like, try to build it with GPT-4. If you, and then, and then if, if you can do that, and if that has product market fit, then worry about the other things. Yeah, exactly, exactly. It's a good problem. There's a really good, I think I heard this from Stan, the founder of Dust.

• 45:02Speaker 02

  And it was a really nice quote, which was, no GPUs before PMF, so no GPUs before product market fit. Probably a similar thing here where like, use GPT-4 and tell product market fit or something like that.

• 45:14Speaker 00

  Yeah, no, that makes a lot of sense. I know, nothing in AI is predictable, but as these models get better and better, are there any things that we're doing today that you think will be obviated away?

• 45:24Speaker 02

  I mean, the context window bit is definitely one piece. I think you'll just be able to have maybe some, I think you'll always need retrieval. But I think like, some of the tricks around managing kind of like conversation history and summarizing some of the previous ones, I think that might go away and you'll just be able to put all of it kind of like, unless you're having like really long conversations. But I think for the most part, you'll be able to put most of it in there. So maybe some, some of the management of kind of like the chat messages stuff will go away.

• 45:58Speaker 02

  I think one area for improvement, it's not that it will stop doing it today, but it's something we can't do today. I think like, I don't, I don't think the multimodal models are quite good enough for a lot of the really precise knowledge work that people want to do. So I think they struggle with.

• 46:12Jacob Effron?

  We're not all going to be living in the Google demo world.

• 46:14Speaker 02

  Yeah, I think that's coming out tomorrow, or it's coming out tomorrow, so we'll see. But yeah, I think like a lot of the cool things that people want to do with the multimodal ones is like, like do extraction and it still doesn't have good spatial, good enough spatial awareness. So I think, I think we'll see more of that. That's less like people will stop doing. Rag, I think people will do. So I don't think that will go away.

• 46:42Speaker 02

  The state machine type stuff is like kind of interesting. I think you'll still want to have some aspect of that. I think you could make the argument that like, as these models get better, and as they have longer context windows, you could just put all those instructions in a single prompt, and then it could recognize what state it's in automatically. And the, I think there's, I think there's maybe like two arguments against that. One is just like, will the models actually get that good? And some of these prompts are really complex, and some of the instructions are really complex. And some of the times, like, you're doing, like, if you have instructions to follow and they're really long, some of the times you're doing rag over the instructions and what the database you're accessing depends on the state you're in. So, so I don't think the state machine things will go away because

• 47:25Speaker 02

  of that. And then another argument against them going away is actually think those are like a really helpful mental model. Like Autogen, when it came out, again, like going back to what I said earlier, I didn't understand why multi-agent would work. But then it's really about the mental model, and it's a great mental model. And so I think they nailed that. And so I think that mental model is just really helpful for developers for building. So I don't think the state machine type things will go away. I do, oh, I, I hope there's more kind of like, there's more and better kind of like, function calling slash structured extraction of things, like, hopefully the days of saying, you know, write this in JSON or Poppy will die. Hopefully those are done. And so hopefully that'll, that'll go away.

• 48:09Speaker 02

  Off the top of heads, I think that's what I come up with.

• 48:11Speaker 00

  Yeah, when GPC nine comes out, we'll bring you back on here and grade how you predicted.

• 48:15Speaker 02

  F, F, I'll give myself.

• 48:16Jacob Effron?

  I think GPT nine will do the grading.

• 48:18Speaker 02

  Yeah.

• 48:20Jacob Effron?

  Awesome. Well, look, I'm sure we could talk about LangChain for days, but we like to end these with a few quickfire questions.

• 48:26Speaker 02

  Cool.

• 48:26Jacob Effron?

  So the first one is, what do you think is overhyped and underhyped in AI?

• 48:31Speaker 02

  Overhyped, I'd probably say multimodal at the moment. I don't think it's quite good enough for a lot of the real use cases that people want. Underhyped is maybe few-shot prompting, going back to that. Like, I think like a lot of the teams that we talk with that are having success are doing that. And I think more people should be.

• 48:50Jacob Effron?

  What's been like the the biggest surprise in building LangChain in terms of, I'm curious, like, one thing you thought would work that didn't end up working, and maybe one thing that you were like this, you know, maybe was a throwaway thing and actually worked quite well.

• 49:02Speaker 02

  One thing that I was definitely wrong about was the importance of streaming and having that built into the orchestration framework in the first place. This seems kind of obvious now, but I think I remember chatting over the summer and or not necessarily, yeah, I guess the importance of it is a good way to phrase it. Like, we had some composability, but the composability didn't like work well with, it wasn't streaming first. And so like you could compose things together, but then you couldn't stream the results. And so all the classes that had good streaming, it was done in kind of like an ad hoc way. And Nuno, who's done a lot of work on our team, I remember pushed for that super hard and LangChain Expression Language, which he helped really write,

• 49:45Speaker 02

  has streaming as kind of like a first-class citizen there. And so we've invested a lot in streaming. We've invested a lot in streaming. And so I just think the importance of the importance and the difficulty of doing that right in an orchestration framework. Yeah, good thing, good thing Nuno is on board.

• 50:07Speaker 02

  Open GPTs might be one that I think got more kind of like. Attention and and kind of like positive response than I expected like we we kind of like um the story behind that is we uh there was like the leaks of OpenAI's uh Dev Day like the weekend before and so um I I uh slacked to Nuno and was like hey it would be cool to try to do some of these leaks and see if we could like recreate it in an open source way and so I think we released it like that day or the day after or something um and yeah people really liked it um it's a cool idea um I didn't uh I think people especially a lot of enterprises really want to have control over the systems that they're building I would say and I don't think I appreciated that

• 50:50Speaker 02

  enough um and actually I also don't I think the thing I got wrong there was um there's so many different things you can do in Langchain we weren't doing a good job of showing people how to do the interesting things and I think that might actually be why like you could always do the things that Open GPTs did you just have to like really know the framework and so I think that's something maybe that maybe that's why it kind of like took off more than I thought it would.

• 51:15Speaker 00

  As an open source founder, I think you'll have a lot of thoughts on this next question, but how ubiquitous do you think open source models will be?

• 51:21Speaker 02

  I think they'll be more ubiquitous than they are now. I think there's a lot of hype around them. I think for the interesting things, people are still using OpenAI, but there is a ton of interest in uh local models, local agents doing things. They're just not quite there at the moment. I think once they are like there's a ton of um like a lot of the really personalized ask your documents, chat with uh uh some sort of uh whether be kind of like a coach or or mentor type persona or or just like pure knowledge. People want that to be local. They want that to not go elsewhere. Um and yeah, so it so I think um yeah, I think they'll it yeah it I I

• 52:04Speaker 02

  think there'll be a bunch of really cool applications there that are desktop apps that run locally.

• 52:08Speaker 00

  Yeah. Which AI startup outside of Langchain are you most excited about?

• 52:14Speaker 02

  There's a few. I really like New Computer by Sam Whitmore. I think they're doing really cool things around memory, which I'm very interested in. I think Fireworks I like. They're doing, I think both them and Together, I would say, are doing very good jobs at kind of like the hosting, fine tuning of open source models. Ollama is really cool. Yeah, we do a bunch of stuff with them and I think their, their models are awesome. Hearth AI, I think, is another cool one there. And at the application layer, they're doing a bunch of stuff around relationship management. So I think on the application layer, I'm really interested in ones that I guess have a high level of personalization. So Peter and Hearth, I think, both have that. And I think that's a... One of my, one of my hot takes is that, you know, I think everyone kind of

• 52:56Speaker 02

  recognizes that we're still in the first wave of like AI apps and there's probably a second similar to kind of like the iPhone. It took like a little bit to figure out what the killer apps were. I wonder if like that, like personalization at a user level is like what's missing for them to kind of like really take on. And so I'm super interested by that and super interested by companies that are doing things in that space.

• 53:17Speaker 00

  Yeah, Jacob and I would probably agree with that. We've actually talked about that before. So maybe.

• 53:21Speaker 02

  Have you seen any cool companies doing things?

• 53:23Speaker 00

  We've seen companies that have tried to make it easier to do personalized models on a per user basis.

• 53:29Speaker 02

  Are they doing that like fine tuning or through like just context?

• 53:33Speaker 00

  Individual lauras for each one.

• 53:34Speaker 02

  Interesting. Yeah. That's super interesting.

• 53:37Speaker 00

  It's really early, though. I mean, these have just popped up in the last, you know, six months or so.

• 53:40Speaker 02

  Yeah, I think the idea around personalization and whether it's through some sort of rag or some sort of fine tuning is a really, haven't heard enough kind of like discussed about the feasibility. I think both are way harder than just like fine tuning a single model, obviously. But like, I think, like, no, I haven't seen anyone really doing stuff there, except I think New Computer is probably doing the most. Yeah, I don't think they're fine tuning things.

• 54:07Jacob Effron?

  I guess I'm curious, like, if you had stayed at Robust Intelligence and we're leaving now to start, you know, and let's say, you know, someone else had built Langchain, and so you were more focused on the application side. Like, what application would you go build?

• 54:18Speaker 02

  Something around personalization or memory. I think like the idea that I have here is like a journal app that remembers things about you. We may release a demo app of this because I think it's really cool. I think like personalization and memories is, I think it's like, I think it's really important and I think it's really intellectually interesting. But then I think a journal app is a really nice, kind of like UX for it as well.

• 54:43Speaker 02

  You hear like one of the main complaints about character AIs, it doesn't remember things about you. Okay, sure. I think part of the difficulty about remembering things is figuring out like what's important or not. Like if you have a random chat, like I'm not gonna necessarily say important things all the time. I'm gonna say, hey, what's up, stuff like that. With a journal app, and so, okay, I think we will release a demo app of this. But the UX that I'm imagining is you write your journal entry. That's gonna be chock full of like really interesting and personal things, right? So you have a good basis there. Then the AI starts a conversation with you about that and pulls in previous memories of previous journals. Or, yeah, I mean, previous journals and previous interactions. And then you have a chat with it, and so there's that component. But the journaling aspect comes first, so you're kind of like in that mindset. So that would, that's exactly what I would build.

• 55:27Speaker 02

  And who knows if it would work, but I'd build that to start and try.

• 55:30Jacob Effron?

  That's super cool. And then my last question is, you know, I feel like Langchain's just so been at the center of like everything happening in the developer space in Gen AI. And I feel like you also have kind of become this like celebrity within the community. Like, what's that been like?

• 55:45Speaker 02

  It's been, it's been, it's been interesting. Like, I hadn't done open source before this. I hadn't, I had like 200 Twitter followers when this started. I think it's been. I think it really is just like a really fun time to be building things. At the end of the day, it's that. And so I think like having the opportunity to even like do this podcast with you guys or like interact with SWICS. Like, like, all these things are just like awesome. I think it's a super fun time. I think there's a lot to build. And I'm really just trying to have kind of like as much fun as I can. So.

• 56:19Speaker 00

  And then the last question, I mean, this has been awesome today. I'm sure a lot of our users would want to learn more about you and LangChain. Where should they go for that?

• 56:26Speaker 02

  We have a blog, blog.langchain.dev, where we post a bunch of stuff. And then Twitter is probably the, oh, Twitter and then YouTube. We've been actually investing a lot in YouTube content, kind of to the point around. There's a lot of things you can do with LangChain. How do you help people kind of like do those things? We've found videos to be a really good format for that. And so we're investing a lot in YouTube. We're doing some series on like explaining RAG concepts. We're doing some others like build an application from scratch. So YouTube, the blog, and then Twitter are probably the main channels.

• 56:58Speaker 00

  Right. And you guys have a big release that I think will be concurrent with this. Anything you want to plug there on that?

• 57:04Speaker 02

  Yeah, we're releasing LangSmith GA. I think it's, I think it's a really, really good product. I think it's probably better than LangChain in some senses. Works with LangChain, without LangChain. And I think the tracing and observability, we've put a lot of effort into, so I would, I would check that out. I think it's a pretty quick time to value there. And then the testing stuff, as I mentioned, I think there's more effort there. If you, if you, if you're running into issues or if you want to work through some of the problems with us, also reach out to us. We're happy to do that.

• 57:34Jacob Effron?

  Jordan, that was a fun one.

• 57:35Speaker 00

  Yeah, no, it was great. I was looking forward to that for a while.

• 57:38Jacob Effron?

  Yeah, I feel like LangChain and Harrison are so at the epicenter of everything happening in the AI world that you could, you know, we could talk to them for like a day.

• 57:47Speaker 00

  Yeah, I mean, aside from OpenAI, they, they really sort of got this ecosystem going in the first place.

• 57:52Jacob Effron?

  I feel like there was so much from that episode that was, that was super interesting. Like, I loved the, you know, the part about evals and this idea that, you know, maybe today people are coming up with these data sets to test like afterwards, but it's like actually, you know, ideally, if you're building a product at the very beginning of that product, you're figuring out what do you exactly do you want it to do? And then you basically have come up with your eval data set. And then, you know, I also felt like I loved his part, which I feel we've heard a lot of times now, which is like, don't use, you know, GPUs or, you know, until you have product market fit. Or I think Arvind's version of this from Perplexity was like, worry about being a wrapper actually after when you have a product that actually works.

• 58:27Speaker 00

  Totally. I mean, I think there's a lot of lessons to go from just sort of non-AI startups and really focusing in on the pain point, finding that product market fit and then sort of scaling it from there.

• 58:36Jacob Effron?

  Totally. And also just, I mean, one of my favorite lines from the whole thing was, I feel like, you know, you can intellectualize all day about like, will RAG still be around? Like, is this kind of approach to doing things still relevant? Is all this prompt engineering just going to be obviated? And I guess you could just think about that all day and, and wait and never do anything. And Harrison was like, you gotta build, like at some point, like whether it's going to be there in six months or not, like you gotta get going.

• 59:00Speaker 00

  That, that was also my favorite quote, that and just, I can't predict anything, you know? And so

• 59:06Jacob Effron?

  If he can't predict it, what, what

• 59:07Speaker 00

  I know, exactly.

• 59:08Jacob Effron?

  Do some mere mortals have?

• 59:09Speaker 00

  I mean, he's literally at the center of it all. So I, I think he's right though. I think, you know, you're not going to be able to really get those lessons and understand what customers want in terms of what they need without just building and going talking to them. So I, I give him a lot of credit for that.

• 59:25Jacob Effron?

  Totally. And it was interesting to hear, you know, I feel like we've had some folks be super bullish on multimodal. And in the discourse generally, it's like, oh, this is the year of multimodal. And I think, you know, Harrison, obviously working really closely with a lot of developers who are actually building this. It's like, I'm not so sure we're, we're, we're there yet.

• 59:39Speaker 00

  Yeah. I mean, we see like OpenAI going really hard into multimodal. We see Google doing it. And so I think a lot of people are really excited about it. A lot of people are working on it. And I think we're going to continue to see advancements. But at the same time, like you said, we're still far away from really having it optimal enough to do the type of use cases that we'd want to do with it.

• 59:58Jacob Effron?

  Totally. I thought another part, I mean, clearly he's super passionate about personalization. And I feel like, I kind of agree with him that it feels like if we nail that, that's like this step change improvement where you can't ignore a lot of these applications. Like, I was struck by when we had Arvind from Perplexity on.

• 1:00:14Jacob Effron?

  And he talked about like his long-term vision for Perplexity. It's like these answers that are tailored to you. And so you, uh, you know, Jordan, you might be really interested in Taylor Swift. And so it's really long Taylor Swift page. I'm a little more sophisticated, you know, interested in, uh, in, in, uh, the inner workings of, uh, of reinforcement learning. And so that page is tailored to me. Uh, the joke is that it's, it's really the reverse, but, um, I think basically you could imagine that like that starts, you know, where you have a Wikipedia that's kind of static and gives the same kind of information to anyone. As you start, what's what's amazing with these LLMs is the ability to just render content on the fly and completely change that based on who the person is. And I feel like that's a super interesting, underexplored angle of a lot of these applications.

• 1:00:55Speaker 00

  I agree. I think that is the future. I think we're still a long way from that in the sense of,

• 1:01:00Jacob Effron?

  I thought you were the early stage investor optimist.

• 1:01:04Speaker 00

  It's not that we wouldn't invest in it, but I think, you know, this year we'll see if it really comes to that. You know, I think like RLHF is a good example of this where you're taking sort of your broad user feedback of your product and getting the models to be better. We spoke with dozens of experts in companies and they're still not using RLHF. They're still in the early phases of fine tuning and deployment and inference and things like that. And so I think it's going to take time to really get to that level of personalization and the tooling for that.

• 1:01:31Jacob Effron?

  And then, you know, I also thought like, you know, this idea that we're actually maybe most bottlenecked today on user experience and like, that's where, you know, a lot of innovations are to come. Really compelling. And I think that like, you know, obviously you always have to take a step back and be like, it's only been, you know, what, under a year since we've had GPT-4 and even started to explore these capabilities. It feels like this next year, you know, I mean, it was funny hearing Harrison say, hey, this might actually be a stable time, relatively speaking to how things crazy things have been. And that might actually be a really interesting time. You know, I feel like both of us are really excited about just this year, what it means for applications, what it means for folks that have had a bit longer to think about what some of these new paradigms might look like.

• 1:02:10Speaker 00

  I, I agree. I mean, I think a lot of people have said this is the year of applications, AI applications. I think we're going to see a lot of interesting stuff, you know, Harrison mentioned this, but you look at the iPhone and the internet and all of these sort of foundational technologies, the cloud. And a lot of the interesting applications and startups that emerged were not that first year. They were four or five years later. You know, like you look at sort of the time that Uber and Lyft and Snapchat and Facebook and all these things happened. It was right after, you know, a few years, those fundamental changes. And so I think we're going to see a similar thing in AI.

• 1:02:43Jacob Effron?

  Yeah, though AI seems to be dog years. So maybe, maybe it'll just be one, not a four or five. And then I also loved his kind of framework for the three different types of builders. I think he mentioned, you know, there's the AI native companies. There's like kind of more innovative digital companies. And then there's like your classic enterprises. And, you know, I thought his point was actually really well taken, which is that we may see a lot of the products because they're outwardly facing from the first two. But actually enterprises themselves are doing a lot of stuff. It's just a lot of its internal facing. And that obviously ends up being lower risk for them. It ends up being, you know, if you're wrong, it's not the end of the world. You're wrong if someone's searching for a document internally. But I think it's, it's, it's interesting that there seems to be more happening on the enterprise side than we might see just from what's outwardly released.

• 1:03:28Speaker 00

  Totally. I think, you know, from what we've seen and talking with practitioners, I think it really is internally focused, especially for some enterprises that are, you know, regulated industries, things like that. You know, I spoke with the chief compliance officer of a worldwide top five bank. And he was like, yeah, we would not dare to release something to our users for at least a year, maybe a year and a half. And we don't want to be the first to do it. We want to be like the third bank to do it in case something messes up. And so I think like there's this hesitation to launch sort of user facing products on a lot of enterprises, but it's going to get better and better and as they explore more and more.

• 1:04:02Jacob Effron?

  Totally. I thought that like when we were talking with Des from Intercom, he was like, yeah, people will use this like on weekends. A very specific thing. And I think we're all, it'll just be interesting to watch, you know, the journey to comfort of deploying that more broadly.

• 1:04:15Speaker 00

  It's a great time to build and, you know, eventually we'll get to that point in the short term, probably.

• 1:04:19Jacob Effron?

  Hope folks enjoyed that and definitely, you know, like, subscribe. We'll, we'll, we'll be out with another episode in the next few weeks.