The Point of LangChain — with Harrison Chase of LangChain

As alluded to on the pod, LangChain has just launched LangChain Hub https://twitter.com/LangChainAI/status/1699095883512778823: “the go-to place for developers to discover new use cases and polished prompts.” It’s available to everyone with a LangSmith account, no invite code necessary. Check it out https://blog.langchain.dev/langchain-prompt-hub/! In 2023, LangChain has speedrun the race from 2:00 to 4:00 to 7:00 Silicon Valley Time https://medium.com/backchannel/how-the-tech-press-forces-a-narrative-on-companies-it-covers-5f89fdb7793e. From the back to back $10m Benchmark seed https://blog.langchain.dev/announcing-our-10m-seed-round-led-by-benchmark/ and (rumored) $20-25m Sequoia Series A https://news.ycombinator.com/item?id=35632520 in April, to back to back critiques of “LangChain is Pointless https://news.ycombinator.com/item?id=36645575” and “The Problem with LangChain https://twitter.com/DrJimFan/status/1678089693920133120” in July, to teaching with Andrew Ng https://twitter.com/AndrewYNg/status/1663984377918001153 and keynoting at basically every AI conference this fall (including ours https://www.ai.engineer/summit), it has been an extreme rollercoaster for Harrison and his growing https://twitter.com/j_schottenstein/status/1693617576637866174 team creating one of the most popular (>60k stars at time of writing) building blocks for AI Engineers. LangChain’s Origins The first commit to LangChain https://github.com/langchain-ai/langchain/pull/1/files shows its humble origins as a light wrapper around Python’s formatter.format for prompt templating. But as Harrison tells the story, even his first experience with text-davinci-002 in early 2022 was focused on chatting with data from their internal company Notion and Slack, what is now known as Retrieval Augmented Generation (RAG). As the Generative AI meetup scene came to life post Stable Diffusion, Harrison saw a need for common abstractions for what people were building with text LLMs at the time: * LLM Math https://twitter.com/hwchase17/status/1584925402815877120?s=20, aka Riley Goodside’s “You Can’t Do Math https://twitter.com/goodside/status/1568448128495534081?lang=en” REPL-in-the-loop (PR #8 https://github.com/langchain-ai/langchain/pull/8/files) * Self-Ask With Search https://twitter.com/hwchase17/status/1584925398550294528?s=20, Ofir Press’ agent pattern (PR #9 https://github.com/langchain-ai/langchain/pull/9) (later ReAct, PR #24 https://github.com/langchain-ai/langchain/pull/24) * NatBot https://twitter.com/hwchase17/status/1584925405470810112?s=20, Nat Friedman’s browser controlling agent (PR #18 https://github.com/langchain-ai/langchain/pull/18) * Adapters for OpenAI, Cohere, and HuggingFaceHub https://github.com/langchain-ai/langchain/pull/23 All this was built and launched in a few days from Oct 16-25, 2022. Turning research ideas/exciting usecases into software quickly and often has been in the LangChain DNA from Day 1 and likely a big driver of LangChain’s success, to date amassing the largest community of AI Engineers and being the default launch framework for every big name from Nvidia https://twitter.com/hwchase17/status/1650896870267584514?lang=en to OpenAI https://twitter.com/hwchase17/status/1668682373767020545?s=20: Dancing with Giants But AI Engineering is built atop of constantly moving tectonic shifts: * ChatGPT launched in November (“The Day the AGI Was Born https://www.latent.space/p/everything-we-know-about-chatgpt”) and the API released in March. Before the ChatGPT API, OpenAI did not have a chat endpoint. In order to build a chatbot with history, you had to make sure to chain all messages and prompt for completion. LangChain made it easy to do that out of the box, which was a huge driver of usage. * Today, OpenAI has gone all-in on the chat API and is deprecating the old completions models https://openai.com/blog/gpt-4-api-general-availability, essentially baking in the chat pattern as the default way most engineers should interact with LLMs… and reducing (but not eliminating) the value of ConversationChains https://python.langchain.com/docs/modules/memory/types/summary#using-in-a-chain. * And there have been more updates since: Plugins released in API form as Functions in June (one of our top pods ever https://www.latent.space/p/function-agents#details… reducing but not eliminating the value of OutputParsers https://python.langchain.com/docs/modules/model_io/output_parsers/pydantic) and Finetuning in August https://www.latent.space/p/aug-2023 (arguably reducing some need for Retrieval https://python.langchain.com/docs/modules/data_connection/ and Prompt tooling https://python.langchain.com/docs/modules/model_io/prompts/). With each update, OpenAI and other frontier model labs realign the roadmaps of this nascent industry, and Harrison credits the modular design of LangChain in staying relevant. LangChain has not been merely responsive either: LangChain added Agents https://twitter.com/hwchase17/status/1595456660507459585?lang=en in November, well before they became the hottest topic of the AI Summer https://www.latent.space/p/agents, and now Agents feature as one of LangChain’s top two usecases. LangChain’s problem for podcasters and newcomers alike is its sheer scope - it is the world’s most complete AI framework, but it also has a sprawling surface area that is difficult to fully grasp or document in one sitting. This means it’s time for the trademark Latent Space move (ChatGPT https://www.latent.space/p/everything-we-know-about-chatgpt, GPT4 https://www.latent.space/p/multimodal-gpt4, Auto-GPT https://www.latent.space/p/agents?utm_source=%2Fsearch%2Fexecutive%2520summary&utm_medium=reader2, and Code Interpreter Advanced Data Analysis GPT4.5 https://www.latent.space/i/134257068/code-interpreter-executive-summary): the executive summary! What is LangChain? As Harrison explains, LangChain is an open source framework for building context-aware reasoning applications, available in Python and JS/TS. It launched in Oct 2022 with the central value proposition of “composability https://github.com/langchain-ai/langchain/tree/06da09adf5c7122d04ff2a585a654c3628a4ecd4”, aka the idea that every AI engineer will want to switch LLMs, and combine LLMs with other things into “chains”, using a flexible interface that can be saved via a schema. Today, LangChain’s principal offerings can be grouped as: * Components: isolated modules/abstractions * Model I/O * Models (for LLM/Chat/Embeddings, from OpenAI, Anthropic, Cohere, etc) * Prompts (Templates, ExampleSelectors, OutputParsers) * Retrieval (revised and reintroduced in March https://blog.langchain.dev/retrieval/) * Document Loaders (eg from CSV, JSON, Markdown, PDF) * Text Splitters (15+ various strategies for chunking text to fit token limits) * Retrievers (generic interface for turning an unstructed query into a set of documents - for self-querying https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/, contextual compression https://python.langchain.com/docs/modules/data_connection/retrievers/contextual_compression/, ensembling https://python.langchain.com/docs/modules/data_connection/retrievers/ensemble) * Vector Stores (retrievers that search by similarity of embeddings) * Indexers (sync documents from any source into a vector store without duplication) * Memory (for long running chats, whether a simple Buffer, Knowledge Graph, Summary, or Vector Store https://python.langchain.com/docs/modules/memory/types/) * Use-Cases: compositions of Components * Chains: combining a PromptTemplate, LLM Model and optional OutputParser * with Router, Sequential, and Transform Chains https://python.langchain.com/docs/modules/chains/foundational/router for advanced usecases * savable, sharable schemas that can be loaded from LangChainHub https://python.langchain.com/docs/modules/chains/how_to/from_hub * Agents: a chain that has access to a suite of tools, of nondeterministic length because the LLM is used as a reasoning engine to determine which actions to take and in which order. Notable 100LOC explainer here https://blog.scottlogic.com/2023/05/04/langchain-mini.html. * Tools (interfaces that an agent can use to interact with the world - preset list here https://python.langchain.com/docs/integrations/tools/. Includes things like ChatGPT plugins, Google Search, WolframAlpha. Groups of tools are bundled up as toolkits https://python.langchain.com/docs/integrations/toolkits/) * AgentExecutor (the agent runtime, basically the while loop, with support for controls https://python.langchain.com/docs/modules/agents/how_to/max_iterations, timeouts https://python.langchain.com/docs/modules/agents/how_to/max_time_limit, memory sharing https://python.langchain.com/docs/modules/agents/how_to/sharedmemory_for_tools, etc) * LangChain has also added a Callbacks system for instrumenting each stage of LLM, Chain, and Agent calls (which enables LangSmith https://python.langchain.com/docs/guides/langsmith/walkthrough, LangChain’s first cloud product), and most recently anExpression Language https://python.langchain.com/docs/expression_language/, a declarative way to compose chains. LangChain the company incorporated in January 2023, announced their seed round in April https://twitter.com/hwchase17/status/1643301144717066240, and launched LangSmith in July https://twitter.com/LangChainAI/status/1681349170433761280. At time of writing, the company has 93k followers https://twitter.com/langchainai, their Discord https://discord.gg/6adMQxSpJS has 31k members and their weekly webinars https://www.youtube.com/@LangChain are attended by thousands of people live. The full-featuredness of LangChain means it is often the first starting point for building any mainstream LLM use case, because they are most likely to have working guides for the new developer. Logan (our first guest https://www.latent.space/p/chatgpt-gpt4-hype-and-building-llm!) from OpenAI has been a notable fan of both LangChain https://logankilpatrick.medium.com/what-is-langchain-and-why-should-i-care-as-a-developer-b2d952c42b28?source=user_profile---------4---------------------------- and LangSmith https://logankilpatrick.medium.com/what-is-langsmith-and-why-should-i-care-as-a-developer-e5921deb54b5?source=user_profile---------2---------------------------- (they will be running the first LangChain + OpenAI workshop at AI Eng Summit). However, LangChain is not without its critics, with Aravind Srinivas https://twitter.com/AravSrinivas/status/1677884199183994881, Jim Fan https://twitter.com/DrJimFan/status/1678089693920133120, Max Woolf https://minimaxir.com/2023/07/langchain-problem/, Mckay Wrigley https://twitter.com/mckaywrigley/status/1677812146925895680 and the general Reddit/HN community https://news.ycombinator.com/item?id=36645575 describing frustrations with the value of their abstractions, and many are attempting to write their own (the common experience of adding and then removing LangChain is something we covered in our Agents writeup https://www.latent.space/i/115776561/auto-gpt-executive-summary). Harrison compares this with the timeless ORM debate on the value of abstractions. LangSmith Last month, Harrison launched LangSmith, their LLM observability tool and first cloud product. LangSmith makes it easy to monitor all the different primitives that LangChain offers (agents, chains, LLMs) as well as making it easy to share https://twitter.com/hwchase17/status/1684022255796514816 and evaluate them both through heuristics (i.e. manually written ones) and “LLM evaluating LLM” flows. The top HN comment https://news.ycombinator.com/item?id=36648778 in the “LangChain is Pointless” thread observed that orchestration is the smallest part of the work, and the bulk of it is prompt tuning and data serialization. When asked this directly our pod, Harrison agreed: “I agree that those are big pain points that get exacerbated when you have these complex chains and agents where you can't really see what's going on inside of them. And I think that's partially why we built Langsmith…” (48min mark) You can watch the full launch on the LangChain YouTube: It’s clear that the target audience for LangChain is expanding to folks who are building complex, production applications rather than focusing on the simpler “Q&A your docs” use cases that made it popular in the first place. As the AI Engineer space matures, there will be more and more tools graduating from supporting “hobby” projects to more enterprise-y use cases. In this episode we run through some of the history of LangChain, how it’s growing from an open source project to one of the highest valued AI startups out there, and its future. We hope you enjoy it! Show Notes * LangChain https://www.langchain.com/ * LangChain’s Berkshire Hathaway Homepage https://hathaway.langchain.com/ * Abstractions tweet https://twitter.com/hwchase17/status/1595456660507459585 * LangSmith https://docs.smith.langchain.com/ * LangSmith Cookbooks https://github.com/langchain-ai/langsmith-cookbook repo * LangChain Retrieval https://blog.langchain.dev/retrieval/ blog * Evaluating CSV Question/Answering https://blog.langchain.dev/benchmarking-question-answering-over-csv-data/ blog and YouTube https://www.youtube.com/watch?v=jGnf4OhptbA&ref=blog.langchain.dev * MultiOn Partner https://blog.langchain.dev/multion-x-langchain-powering-next-gen-web-automation-navigation-with-ai/ blog * Harvard Sports Analytics Collective https://careerservices.fas.harvard.edu/organizations/harvard-sports-analysis-collective/#:~:text=The%20Harvard%20College%20Sports%20Analysis,of%20sports%20strategy%20and%20management. * Evaluating RAG Webinar https://www.youtube.com/watch?v=fWC4VxolWAk * awesome-langchain: * LLM Math Chain https://python.langchain.com/docs/use_cases/more/code_writing/llm_math * Self-Ask https://python.langchain.com/docs/modules/agents/agent_types/self_ask_with_search * LangChain Hub UI https://github.com/FanaHOVA/langchain-hub-ui * “LangChain is Pointless” https://news.ycombinator.com/item?id=36645575 * Harrison’s links * sports - estimating player compatibility in the NBA https://github.com/hwchase17/sports * early interest in prompt injections https://github.com/hwchase17/adversarial-prompts * GitHub https://github.com/hwchase17 * Twitter https://twitter.com/hwchase17 Timestamps * [00:00:00] Introduction * [00:00:48] Harrison's background and how sports led him into ML * [00:04:54] The inspiration for creating LangChain - abstracting common patterns seen in other GPT-3 projects * [00:05:51] Overview of LangChain - a framework for building context-aware reasoning applications * [00:10:09] Components of LangChain - modules, chains, agents, etc. * [00:14:39] Underappreciated parts of LangChain - text splitters, retrieval algorithms like self-query * [00:18:46] Hiring at LangChain * [00:20:27] Designing the LangChain architecture - balancing flexibility and structure * [00:24:09] The difference between chains and agents in LangChain * [00:25:08] Prompt engineering and LangChain * [00:26:16] Announcing LangSmith * [00:30:50] Writing custom evaluators in LangSmith * [00:33:19] Reducing hallucinations - fixing retrieval vs generation issues * [00:38:17] The challenges of long context windows * [00:40:01] LangChain's multi-programming language strategy * [00:45:55] Most popular LangChain blog posts - deep dives into specific topics * [00:50:25] Responding to LangChain criticisms * [00:54:11] Harrison's advice to AI engineers * [00:55:43] Lightning Round Transcript Alessio: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Residence at Decibel Partners https://decibel.vc/, and I'm joined by my co-host Swyx, founder of Smol.ai. [00:00:19] Swyx: Welcome. Today we have Harrison Chase in the studio with us. Welcome Harrison. [00:00:23] Harrison: Thank you guys for having me. I'm excited to be here. [00:00:25] Swyx: It's been a long time coming. We've been asking you for a little bit and we're really glad that you got some time to join us in the studio. Yeah. [00:00:32] Harrison: I've been dodging you guys for a while. [00:00:34] Swyx: About seven months. You pulled me in here. [00:00:37] Alessio: About seven months. But it's all good. I totally understand. [00:00:38] Swyx: We like to introduce people through the official backgrounds and then ask you a little bit about your personal side. So you went to Harvard, class of 2017. You don't list what you did in Harvard. Was it CS? [00:00:48] Harrison: Stats and CS. [00:00:50] Swyx: That's awesome. I love me some good stats. [00:00:52] Harrison: I got into it through stats, through doing sports analytics. And then there was so much overlap between stats and CS that I found myself doing more and more of that. [00:00:59] Swyx: And it's interesting that a lot of the math that you learn in stats actually comes over into machine learning which you applied at Kensho as a machine learning engineer and Robust Intelligence, which seems to be the home of a lot of AI founders. Harrison: It does. Yeah. Swyx: And you started LangChain, I think around November 2022 and incorporated in January. Yeah. [00:01:19] Harrison: I was looking it up for the podcast and the first tweet was on, I think October 24th. So just before the end of November or end of October. [00:01:26] Swyx: Yeah. So that's your LinkedIn. What should people know about you on the personal side that's not obvious on LinkedIn? [00:01:33] Harrison: A lot of how I got into this is all through sports actually. Like I'm a big sports fan, played a lot of soccer growing up and then really big fan of the NBA and NFL. And so freshman year at college showed up and I knew I liked math. I knew I liked sports. One of the clubs that was there was the Sports Analytics Collective. And so I joined that freshman year, I was doing a lot of stuff in like Excel, just like basic stats, but then like wanted to do more advanced stuff. So learn to code, learn kind of like data science and machine learning through that way. Kind of like just kept on going down that path. I think sports is a great entryway to data science and machine learning. There's a lot of like numbers out there. People like really care. Like I remember, I think sophomore, junior year, I was in the Sports Collective and the main thing we had was a blog. And so we wrote a blog. It wasn't me. One of the other people in the club wrote a blog predicting the NFL season. I think they made some kind of like with stats and I think their stats showed that like the Dolphins would end up beating the Patriots and New England got like pissed about it, of course. So people like really care and they'll give you feedback about whether you're like models doing well or poorly. And so you get that. And then you also get like instantaneous kind of like, well, not instantaneous, but really quick feedback. Like if you predict a game, the game happens that night. Like you don't have to wait a year to see what happens. So I think sports is a great kind of like entryway for kind of like data science. [00:02:43] Alessio: There was actually my first article on the Twilio blog with a Python script to like predict pricing of like Daily Fantasy players based on my past week performance. Yeah, I don't know. It's a good getaway drug. [00:02:56] Swyx: And on my end, the way I got into finance was through sports betting. So maybe we all have some ties in there. Was like Moneyball a big inspiration? The movie? [00:03:06] Harrison: Honestly, not really. I don't really like baseball. That's like the big thing. [00:03:10] Swyx: Let's call it a lot of stats. Cool. Well, we can dive right into LangChain, which is what everyone is excited about. But feel free to make all the sports analogies you want. That really drives home a lot of points. What was your GPT aha moment? When did you start working on GPT itself? Maybe not LangChain, just anything to do with the GPT API? [00:03:29] Harrison: I think it probably started around the time we had a company hackathon. I think that was before I launched LangChain. I'm trying to remember the exact sequence of events, but I do remember that at the hackathon I worked with Will, who's now actually at LangChain as well, and then two other members of Robust. And we made basically a bot where you could ask questions of Notion and Slack. And so I think, yeah, RAG, basically. And I think I wanted to try that out because I'd heard that it was getting good. I'm trying to remember if I did anything before that to realize that it was good. So then I would focus on that on the hackathon. I can't remember or not, but that was one of the first times that I built something [00:04:06] Swyx: with GPT-3. There wasn't that much opportunity before because the API access wasn't that widespread. You had to get into some kind of program to get that. [00:04:16] Harrison: DaVinci-002 was not terrible, but they did an upgrade to get it to there, and they didn't really publicize that as much. And so I think I remember playing around with it when the first DaVinci model came out. I was like, this is cool, but it's not amazing. You'd have to do a lot of work to get it to do something. But then I think that February or something, I think of 2022, they upgraded it and it was it got better, but I think they made less of an announcement around it. And so I just, yeah, it kind of slipped under the radar for me, at least. [00:04:45] Alessio: And what was the step into LangChain? So you did the hackathon, and then as you were building the kind of RAG product, you felt like the developer experience wasn't that great? Or what was the inspiration? [00:04:54] Harrison: No, honestly, so around that time, I knew I was going to leave my previous job. I was trying to figure out what I was going to do next. I went to a bunch of meetups and other events. This was like the September, August, September of that year. So after Stable Diffusion, but before ChatGPT. So there was interest in generative AI as a space, but not a lot of people hacking on language models yet. But there were definitely some. And so I would go to these meetups and just chat with people and basically saw some common abstractions in terms of what they were building, and then thought it would be a cool side project to factor out some of those common abstractions. And that became kind of like LangChain. I looked up again before this, because I remember I did a tweet thread on Twitter to announce LangChain. And we can talk about what LangChain is. It's a series of components. And then there's some end-to-end modules. And there was three end-to-end modules that were in the initial release. One was NatBot. So this was the web agent by Nat Friedman. Another was LLM Math Chain. So it would construct- [00:05:51] Swyx: GPT-3 cannot do math. [00:05:53] Harrison: Yeah, exactly. And then the third was Self-Ask. So some type of RAG search, similar to React style agent. So those were some of the patterns in terms of what I was seeing. And those all came from open source or academic examples, because the people who were actually working on this were building startups. And they were doing things like question answering over your databases, question answering over SQL, things like that. But I couldn't use their code as kind of like inspiration to factor things out. [00:06:18] Swyx: I talked to you a little bit, actually, roundabout, right after you announced LangChain. I'm honored. I think I'm one of many. This is your first open source project. [00:06:26] Harrison: No, that's not actually true. I released, because I like sports stats. And so I remember I did release some really small, random Python package for scraping data from basketball reference or something. I'm pretty sure I released that. So first project to get a star on GitHub, let's say that. [00:06:45] Swyx: Did you reference anything? What was the inspirations, like other frameworks that you look to when open sourcing LangChain or announcing it or anything like that? [00:06:53] Harrison: I mean, the only main thing that I looked for... I remember reading a Hacker News post a little bit before about how a readme on the project goes a long way. [00:07:02] Swyx: Readme's help. [00:07:03] Harrison: Yeah. And so I looked at it and was like, put some status checks at the top and have the title and then one or two lines and then just right into installation. And so that's the main thing that I looked at in terms of how to structure it. Because yeah, I hadn't done open source before. I didn't really know how to communicate that aspect of the marketing or getting people to use it. I think I had some trouble finding it, but I finally found it and used that as a lot [00:07:25] Swyx: of the inspiration there. Yeah. It was one of the subjects of my write-up how it was surprising to me that significant open source experience actually didn't seem to matter in the new wave of AI tooling. Most like auto-GPTs, Torrents, that was his first open source project ever. And that became auto-GPT. Yeah. I don't know. To me, it's just interesting how open source experience is kind of fungible or not necessary. Or you can kind of learn it on the job. [00:07:49] Alessio: Overvalued. [00:07:50] Swyx: Overvalued. Okay. You said it, not me. [00:07:53] Alessio: What's your description of LangChain today? I think when I built the LangChain Hub UI in January, there were a few things. And I think you were one of the first people to talk about agents that were already in there before it got hot now. And it's obviously evolved into a much bigger framework today. Run people through what LangChain is today, how they should think about it, and all of that. [00:08:14] Harrison: The way that we describe it or think about it internally is that LangChain is basically... I started off saying LangChain's a framework for building LLM applications, but that's really vague and not really specific. And I think part of the issue is LangChain does do a lot, so it's hard to be somewhat specific. But I think the way that we think about it internally, in terms of prioritization, what to focus on, is basically LangChain's a framework for building context-aware reasoning applications. And so that's a bit of a mouthful, but I think that speaks to a lot of the core parts of what's in LangChain. And so what concretely that means in LangChain, there's really two things. One is a set of components and modules. And these would be the prompt template abstraction, the LLM abstraction, chat model abstraction, vector store abstraction, text splitters, document loaders. And so these are combinations of things that we build and we implement, or we just have integrations with. So we don't have any language models ourselves. We don't have any vector stores ourselves, but we integrate with a lot of them. And then the text splitters, we have our own logic for that. The document loaders, we have our own logic for that. And so those are the individual modules. But then I think another big part of LangChain, and probably the part that got people using it the most, is the end-to-end chains or applications. So we have a lot of chains for getting started with question answering over your documents, chat question answering, question answering over SQL databases, agent stuff that you can plug in off the box. And that basically combines these components in a series of specific ways to do this. So if you think about a question answering app, you need a lot of different components kind of stacked. And there's a bunch of different ways to do question answering apps. So this is a bit of an overgeneralization, but basically, you know, you have some component that looks up an embedding from a vector store, and then you put that into the prompt template with the question and the context, and maybe you have the chat history as well. And then that generates an answer, and then maybe you parse that out, or you do something with the answer there. And so there's just this sequence of things that you basically stack in a particular way. And so we just provide a bunch of those assembled chains off the shelf to make it really easy to get started in a few lines of code. [00:10:09] Alessio: And just to give people context, when you first released LangChain, OpenAI did not have a chat API. It was a completion-only API. So you had to do all the human assistant, like prompting and whatnot. So you abstracted a lot of that away. I think the most interesting thing to me is you're kind of the Switzerland of this developer land. There's a bunch of vector databases that are killing each other out there to get people to embed data in them, and you're like, I love you all. You all are great. How do you think about being an opinionated framework versus leaving a lot of choice to the user? I mean, in terms of spending time into this integration, it's like you only have 10 people on the team. Obviously that takes time. Yeah. What's that process like for you all? [00:10:50] Harrison: I think right off the bat, having different options for language models. I mean, language models is the main one that right off the bat we knew we wanted to support a bunch of different options for. There's a lot to discuss there. People want optionality between different language models. They want to try it out. They want to maybe change to ones that are cheaper as new ones kind of emerge. They don't want to get stuck into one particular one if a better one comes out. There's some challenges there as well. Prompts don't really transfer. And so there's a lot of nuance there. But from the bat, having this optionality between the language model providers was a big important part because I think that was just something we felt really strongly about. We believe there's not just going to be one model that rules them all. There's going to be a bunch of different models that are good for a bunch of different use cases. I did not anticipate the number of vector stores that would emerge. I don't know how many we supported in the initial release. It probably wasn't as big of a focus as language models was. But I think it kind of quickly became so, especially when Postgres and Elastic and Redis started building their vector store implementations. We saw that some people might not want to use a dedicated vector store. Maybe they want to use traditional databases. I think to your point around what we're opinionated about, I think the thing that we believe most strongly is it's super early in the space and super fast moving. And so there's a lot of uncertainty about how things will shake out in terms of what role will vector databases play? How many will there be? And so I think a lot of it has always kind of been this optionality and ability to switch and not getting locked in. [00:12:19] Swyx: There's other pieces of LangChain which maybe don't get as much attention sometimes. And the way that you explained LangChain is somewhat different from the docs. I don't know how to square this. So for example, you have at the top level in your docs, you have, we mentioned ModelIO, we mentioned Retrieval, we mentioned Chains. Then you have a concept called Agents, which I don't know if exactly matches what other people call Agents. And we also talked about Memory. And then finally there's Callbacks. Are there any of the less understood concepts in LangChain that you want to give some air to? [00:12:53] Harrison: I mean, I think buried in ModelIO is some stuff around like few-shot example selectors that I think is really powerful. That's a workhorse. [00:13:01] Swyx: Yeah. I think that's where I start with LangChain. [00:13:04] Harrison: It's one of those things that you probably don't, if you're building an application, you probably don't start with it. You probably start with like a zero-shot prompt. But I think that's a really powerful one that's probably just talked about less because you don't need it right off the bat. And for those of you who don't know, that basically selects from a bunch of examples the ones that are maybe most relevant to the input at hand. So you can do some nice kind of like in-context learning there. I think that's, we've had that for a while. I don't think enough people use that, basically. Output parsers also used to be kind of important, but then function calling. There's this interesting thing where like the space is just like progressing so rapidly that a lot of things that were really important have kind of diminished a bit, to be honest. Output parsers definitely used to be an understated and underappreciated part. And I think if you're working with non-OpenAI models, they still are, but a lot of people are working with OpenAI models. But even within there, there's different things you can do with kind of like the function calling ability. Sometimes you want to have the option of having the text or the application you're building, it could return either. Sometimes you know that it wants to return in a structured format, and so you just want to take that structured format. Other times you're extracting things that are maybe a key in that structured format, and so you want to like pluck that key. And so there's just like some like annoying kind of like parsing of that to do. Agents, memory, and retrieval, we haven't talked at all. Retrieval, there's like five different subcomponents. You could also probably talk about all of those in depth. You've got the document loaders, the text splitters, the embedding models, the vector stores. Embedding models and vector stores, we don't really have, or sorry, we don't build, we integrate with those. Text splitters, I think we have like 15 or so. Like I think there's an under kind of like appreciated amount of those. [00:14:39] Swyx: And then... Well, it's actually, honestly, it's overwhelming. Nobody knows what to choose. [00:14:43] Harrison: Yeah, there is a lot. [00:14:44] Swyx: Yeah. Do you have personal favorites that you want to shout out? [00:14:47] Harrison: The one that we have in the docs is the default is like the recursive text splitter. We added a playground for text splitters the other week because, yeah, we heard a lot that like, you know, and like these affect things like the chunk overlap and the chunks, they affect things in really subtle ways. And so like I think we added a playground where people could just like choose different options. We have like, and a lot of the ideas are really similar. You split on different characters, depending on kind of like the type of text that you have marked down, you might want to split on differently than HTML. And so we added a playground where you can kind of like choose between those. I don't know if those are like underappreciated though, because I think a lot of people talk about text splitting as being a hard part, and it is a really important part of creating these retrieval applications. But I think we have a lot of really cool retrieval algorithms as well. So like self query is maybe one of my favorite things in LangChain, which is basically this idea of when you have a user question, the typical kind of like thing to do is you embed that question and then find the document that's most similar to that question. But oftentimes questions have things that just, you don't really want to look up semantically, they have some other meaning. So like in the example that I use, the example in the docs is like movies about aliens in the year 1980. 1980, I guess there's some semantic meaning for that, but it's a very particular thing that you care about. And so what the self query retriever does is it splits out the metadata filter and most vector stores support like a metadata filter. So it splits out this metadata filter, and then it splits out the semantic bit. And that's actually like kind of tricky to do because there's a lot of different filters that you can have like greater than, less than, equal to, you can have and things if you have multiple filters. So we have like a pretty complicated like prompt that does all that. That might be one of my favorite things in LangChain, period. Like I think that's, yeah, I think that's really cool. [00:16:26] Alessio: How do you think about speed of development versus support of existing things? So we mentioned retrieval, like you got, or, you know, text splitting, you got like different options for all of them. As you get building LangChain, how do you decide which ones are not going to keep supporting, you know, which ones are going to leave behind? I think right now, as you said, the space moves so quickly that like you don't even know who's using what. What's that like for you? [00:16:50] Harrison: Yeah. I mean, we have, you know, we don't really have telemetry on what people are using in terms of what parts of LangChain, the telemetry we have is like, you know, anecdotal stuff when people ask or have issues with things. A lot of it also is like, I think we definitely prioritize kind of like keeping up with the stuff that comes out. I think we added function calling, like the day it came out or the day after it came out, we added chat model support, like the day after it came out or something like that. That's probably, I think I'm really proud of how the team has kind of like kept up with that because this space is like exhausting sometimes. And so that's probably, that's a big focus of ours. The support, I think we've like, to be honest, we've had to get kind of creative with how we do that. Cause we have like, I think, I don't know how many open issues we have, but we have like 3000, somewhere between 2000 and 3000, like open GitHub issues. We've experimented with a lot of startups that are doing kind of like question answering over your docs and stuff like that. And so we've got them on the website and in the discord and there's a really good one, dosu on the GitHub that's like answering issues and stuff like that. And that's actually something we want to start leaning into more heavily as a company as well as kind of like building out an AI dev rel because we're 10 people now, 10, 11 people now. And like two months ago we were like six or something like that. Right. So like, and to have like 2,500 open issues or something like that, and like 300 or 400 PRs as well. Cause like one of the amazing things is that like, and you kind of alluded to this earlier, everyone's building in the space. There's so many different like touch points. LangChain is lucky enough to kind of like be a lot of the glue that connects it. And so we get to work with a lot of awesome companies, but that's also a lot of like work to keep up with as well. And so I don't really have an amazing answer, but I think like the, I think prioritize kind of like new things that, that come out. And then we've gotten creative with some of kind of like the support functions and, and luckily there's, you know, there's a lot of awesome people working on all those support coding, question answering things that we've been able to work with. [00:18:46] Swyx: I think there is your daily rhythm, which I've seen you, you work like a, like a beast man, like mad impressive. And then there's sometimes where you step back and do a little bit of high level, like 50,000 foot stuff. So we mentioned, we mentioned retrieval. You did a refactor in March and there's, there's other abstractions that you've sort of changed your mind on. When do you do that? When do you do like the, the step back from the day to day and go, where are we going and change the direction of the ship? [00:19:11] Harrison: It's a good question so far. It's probably been, you know, we see three or four or five things pop up that are enough to make us think about it. And then kind of like when it reaches that level, you know, we don't have like a monthly meeting where we sit down and do like a monthly plan or something. [00:19:27] Swyx: Maybe we should. I've thought about this. Yeah. I'd love to host that meeting. [00:19:32] Harrison: It's really been a lot of, you know, one of the amazing things is we get to interact with so many different people. So it's been a lot of kind of like just pattern matching on what people are doing and trying to see those patterns before they punch us in the face or something like that. So for retrieval, it was the pattern of seeing like, Hey, yeah, like a lot of people are using vector sort of stuff. But there's also just like other methods and people are offering like hosted solutions and we want our abstractions to work with that as well. So we shouldn't bake in this paradigm of doing like semantic search too heavily, which sounds like basic now, but I think like, you know, to start a lot of it was people needed help doing these things. But then there was like managed things that did them, hybrid retrieval mechanisms, all of that. I think another example of this, I mean, Langsmith, which we can maybe talk about was like very kind of like, I think we worked on that for like three or four months before announcing it kind of like publicly, two months maybe before giving it to kind of like anyone in beta. But this was a lot of debugging these applications as a pain point. We hear that like just understanding what's going on is a pain point. [00:20:27] Alessio: I mean, you two did a webinar on this, which is called Agents vs. Chains. It was fun, baby. [00:20:32] Swyx: Thanks for having me on. [00:20:33] Harrison: No, thanks for coming. [00:20:34] Alessio: That was a good one. And on the website, you list like RAG, which is retrieval of bank debt generation and agents as two of the main goals of LangChain. The difference I think at the Databricks keynote, you said chains are like predetermined steps and agents is models reasoning to figure out what steps to take and what actions to take. How should people think about when to use the two and how do you transition from one to the other with LangChain? Like is it a path that you support or like do people usually re-implement from an agent to a chain or vice versa? [00:21:05] Swyx: Yeah. [00:21:06] Harrison: You know, I know agent is probably an overloaded term at this point, and so there's probably a lot of different definitions out there. But yeah, as you said, kind of like the way that I think about an agent is basically like in a chain, you have a sequence of steps. You do this and then you do this and then you do this and then you do this. And with an agent, there's some aspect of it where the LLM is kind of like deciding what to do and what steps to do in what order. And you know, there's probably some like gray area in the middle, but you know, don't fight me on this. And so if we think about those, like the benefits of the chains are that they're like, you can say do this and you just have like a more rigid kind of like order and the way that things are done. They have more control and they don't go off the rails and basically everything that's bad about agents in terms of being uncontrollable and expensive, you can control more finely. The benefit of agents is that I think they handle like the long tail of things that can happen really well. And so for an example of this, let's maybe think about like interacting with a SQL database. So you can have like a SQL chain and you know, the first kind of like naive approach at a SQL chain would be like, okay, you have the user question. And then you like write the SQL query, you do some rag, you pull in the relevant tables and schemas, you write a SQL query, you execute that against the SQL database. And then you like return that as the answer, or you like summarize that with an LLM and return that to the answer. And that's basically the SQL chain that we have in LangChain. But there's a lot of things that can go wrong in that process. Starting from the beginning, you may like not want to even query the SQL database at all. Maybe they're saying like, hi, or something, or they're misusing the application. Then like what happens if you have some step, like a big part of the application that people with LangChain is like the context aware part. So there's generally some part of bringing in context to the language model. So if you bring in the wrong context to the language model, so it doesn't know which tables to query, what do you do then? If you write a SQL query, it's like syntactically wrong and it can't run. And then if it can run, like what if it returns an unexpected result or something? And so basically what we do with the SQL agent is we give it access to all these different tools. So it has another tool, it can run the SQL query as another, and then it can respond to the user. But then if it kind of like, it can decide which order to do these. And so it gives it flexibility to handle all these edge cases. And there's like, obviously downsides to that as well. And so there's probably like some safeguards you want to put in place around agents in terms of like not letting them run forever, having some observability in there. But I do think there's this benefit of, you know, like, again, to the other part of what LangChain is like the reasoning part, like each of those steps individually involves some aspect of reasoning, for sure. Like you need to reason about what the SQL query is, you need to reason about what to return. But there's then there's also reasoning about the order of operations. And so I think to me, the key is kind of like giving it an appropriate amount to reason about while still keeping it within checks. And so to the point, like, I would probably recommend that most people get started with chains and then when they get to the point where they're hitting these edge cases, then they think about, okay, I'm hitting a bunch of edge cases where the SQL query is just not returning like the relevant things. Maybe I should add in some step there and let it maybe make multiple queries or something like that. Basically, like start with chain, figure out when you're hitting these edge cases, add in the reasoning step to that to handle those edge cases appropriately. That would be kind of like my recommendation, right? [00:24:09] Swyx: If I were to rephrase it, in my words, an agent would be a reasoning node in a chain, right? Like you start with a chain, then you just add a reasoning node, now it's an agent. [00:24:17] Harrison: Yeah, the architecture for your application doesn't have to be just a chain or just an agent. It can be an agent that calls chains, it can be a chain that has an agent in different parts of them. And this is another part as well. Like the chains in LangChain are largely intended as kind of like a way to get started and take you some amount of the way. But for your specific use case, in order to kind of like eke out the most performance, you're probably going to want to do some customization at the very basic level, like probably around the prompt or something like that. And so one of the things that we've focused on recently is like making it easier to customize these bits of existing architectures. But you probably also want to customize your architectures as well. [00:24:52] Swyx: You mentioned a bit of prompt engineering for self-ask and then for this stuff. There's a bunch of, I just talked to a prompt engineering company today, PromptOps or LLMOps. Do you have any advice or thoughts on that field in general? Like are you going to compete with them? Do you have internal tooling that you've built? [00:25:08] Harrison: A lot of what we do is like where we see kind of like a lot of the pain points being like we can talk about LangSmith and that was a big motivation for that. And like, I don't know, would you categorize LangSmith as PromptOps? [00:25:18] Swyx: I don't know. It's whatever you want it to be. Do you want to call it? [00:25:22] Harrison: I don't know either. Like I think like there's... [00:25:24] Swyx: I think about it as like a prompt registry and you store them and you A-B test them and you do that. LangSmith, I feel like doesn't quite go there yet. Yeah. It's obviously the next step. [00:25:34] Harrison: Yeah, we'll probably go. And yeah, we'll do more of that because I think that's definitely part of the application of a chain or agent is you start with a default one, then you improve it over time. And like, I think a lot of the main new thing that we're dealing with here is like language models. And the main new way to control language models is prompts. And so like a lot of the chains and agents are powered by this combination of like prompt language model and then some output parser or something doing something with the output. And so like, yeah, we want to make that core thing as good as possible. And so we'll do stuff all around that for sure. [00:26:05] Swyx: Awesome. We might as well go into LangSmith because we're bringing it up so much. So you announced LangSmith I think last month. What are your visions for it? Is this the future of LangChain and the company? [00:26:16] Harrison: It's definitely part of the future. So LangSmith is basically a control center for kind of like your LLM application. So the main features that it kind of has is like debugging, logging, monitoring, and then like testing and evaluation. And so debugging, logging, monitoring, basically you set three environment variables and it kind of like logs all the runs that are happening in your LangChain chains or agents. And it logs kind of like the inputs and outputs at each step. And so the main use case we see for this is in debugging. And that's probably the main reason that we started down this path of building it is I think like as you have these more complex things, debugging what's actually going on becomes really painful whether you're using LangChain or not. And so like adding this type of observability and debuggability was really important. Yeah. There's a debugging aspect. You can see the inputs, outputs at each step. You can then quickly enter into like a playground experience where you can fiddle around with it. The first version didn't have that playground and then we'd see people copy, go to open AI playground, paste in there. Okay. Well, that's a little annoying. And then there's kind of like the monitoring, logging experience. And we recently added some analytics on like, you know, how many requests are you getting per hour, minute, day? What's the feedback like over time? And then there's like a testing debugging, sorry, testing and evaluation component as well where basically you can create datasets and then test and evaluate these datasets. And I think importantly, all these things are tied to each other and then also into LangChain, the framework. So what I mean by that is like we've tried to make it as easy as possible to go from logs to adding a data point to a dataset. And because we think a really powerful flow is you don't really get started with a dataset. You can accumulate a dataset over time. And so being able to find points that have gotten like a thumbs up or a thumbs down from a user can be really powerful in terms of creating a good dataset. And so that's maybe like a connection between the two. And then the connection in the other way is like all the runs that you have when you test or evaluate something, they're logged in the same way. So you can debug what exactly is going on and you don't just have like a final score. You have like this nice trace and thing where you can jump in. And then we also want to do more things to hook this into a LangChain proper, the framework. So I think like some of like the managing the prompts will tie in here already. Like we talked about example selectors using datasets as a few short examples is a path that we support in a somewhat janky way right now, but we're going to like make better over time. And so there's this connection between everything. Yeah. [00:28:42] Alessio: And you mentioned the dataset in the announcement blog post, you touched on heuristic evaluation versus LLMs evaluating LLMs. I think there's a lot of talk and confusion about this online. How should people prioritize the two, especially when they might start with like not a good set of evals or like any data at all? [00:29:01] Harrison: I think it's really use case specific in the distinction that I draw between heuristic and LLM. LLMs, you're using an LLM to evaluate the output heuristics, you have some common heuristic that you can use. And so some of these can be like really simple. So we were doing some kind of like measuring of an extraction chain where we wanted it to output JSON. Okay. One evaluation can be, can you use JSON.loads to load it? And like, right. And that works perfectly. You don't need an LLM to do that. But then for like a lot of like the question answering, like, is this factually accurate? And you have some ground truth fact that you know it should be answering with. I think, you know, LLMs aren't perfect. And I think there's a lot of discussion around the pitfalls of using LLMs to evaluate themselves. And I'm not saying they're perfect by any means, but I do think they're, we've found them to be kind of like better than blue or any of those metrics. And the way that I also like to use those is also just like guide my eye about where to look. So like, you know, I might not trust the score of like 0.82, like exactly correct, but like I can look to see like which data points are like flagged as passing or failing. And sometimes the evaluators messing up, but it's like good to like, you know, I don't have to look at like a hundred data points. I can focus on like 10 or something like that. [00:30:10] Alessio: And then can you create a heuristic once in Langsmith? Like what's like your connection to that? [00:30:16] Harrison: Yeah. So right now, all the evaluation, we actually do client side. And part of this is basically due to the fact that a lot of the evaluation is really application specific. So we thought about having evaluators, you could just click off and run in a server side or something like that. But we still think it's really early on in evaluation. We still think there's, it's just really application specific. So we prioritized instead, making it easy for people to write custom evaluators and then run them client side and then upload the results so that they can manually inspect them because I think manual inspection is still a pretty big part of evaluation for better or worse. [00:30:50] Swyx: We have this sort of components of observability. We have cost, latency, accuracy, and then planning. Is that listed in there? [00:30:57] Alessio: Well, planning more in the terms of like, if you're an agent, how to pick the right tool and whether or not you are picking the right tool. [00:31:02] Swyx: So when you talk to customers, how would you stack rank those needs? Are they cost sensitive? Are they latency sensitive? I imagine accuracy is pretty high up there. [00:31:13] Harrison: I think accuracy is definitely the top that we're seeing right now. I think a lot of the applications, people are, especially the ones that we're working with, people are still struggling to get them to work at a level where they're reliable [00:31:24] Swyx: enough. [00:31:25] Harrison: So that's definitely the first. Then I think probably cost becomes the next one. I think a few places where we've started to see this be like one of the main things is the AI simulation that came out. [00:31:36] Swyx: Generative agents. Yeah, exactly. [00:31:38] Harrison: Which is really fun to run, but it costs a lot of money. And so one of our team members, Lance, did an awesome job hooking up like a local model to it. You know, it's not as perfect, but I think it helps with that. Another really big place for this, we believe, is in like extraction of structured data from unstructured data. And the reason that I think it's so important there is that usually you do extraction of some type of like pre-processing or indexing process over your documents. I mean, there's a bunch of different use cases, but one use case is for that. And generally that's over a lot of documents. And so that starts to rack up a bill kind of quickly. And I think extraction is also like a simpler task than like reasoning about which tools to call next in an agent. And so I think it's better suited for that. Yeah. [00:32:15] Swyx: On one of the heuristics I wanted to get your thoughts on, hallucination is one of the big problems there. Do you have any recommendations on how people should reduce hallucinations? [00:32:25] Harrison: To reduce hallucinations, we did a webinar on like evaluating RAG this past week. And I think there's this great project called RAGOS that evaluates four different things across two different spectrums. So the two different spectrums are like, is the retrieval part right? Or is the generation, or sorry, like, is it messing up in retrieval or is it messing up in generation? And so I think to fix hallucination, it probably depends on where it's messing up. If it's messing up in generation, then you're getting the right information, but it's still hallucinating. Or you're getting like partially right information and hallucinating some bits, a lot of that's prompt engineering. And so that's what we would recommend kind of like focusing on the prompt engineering part. And then if you're getting it wrong in the, if you're just not retrieving the right stuff, then there's a lot of different things that you can probably do, or you should look at on the retrieval bit. And honestly, that's where it starts to become a bit like application specific as well. Maybe there's some temporal stuff going on. Maybe you're not parsing things correctly. Yeah. [00:33:19] Swyx: Okay. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. [00:33:35] Harrison: Yeah. Yeah. [00:33:37] Swyx: Yeah. [00:33:38] Harrison: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. [00:33:56] Swyx: Yeah. Yeah. [00:33:58] Harrison: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. [00:34:04] Swyx: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. [00:34:17] Harrison: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah, I mean, there's probably a larger discussion around that, but openAI definitely had a huge headstart, right? And that's... Clawds not even publicly available yet, I don't think. [00:34:28] Swyx: The API? Yeah. Oh, well, you can just basically ask any of the business reps and they'll give it to you. [00:34:33] Harrison: You can. But it's still a different signup process. I think there's... I'm bullish that other ones will catch up especially like Anthropic and Google. The local ones are really interesting. I think we're seeing a big... [00:34:46] Swyx: Lama Two? Yeah, we're doing the fine-tuning hackathon tomorrow. Thanks for promoting that. [00:34:50] Harrison: No, thanks for it. I'm really excited about that stuff. I mean, that's something that like we've been, you know, because like, as I said, like the only thing we know is that the space is moving so fast and changing so rapidly. And like, local models are, have always been one of those things that people have been bullish on. And it seems like it's getting closer and closer to kind of like being viable. So I'm excited to see what we can do with some fine-tuning. [00:35:10] Swyx: Yeah. I have to confess, I did not know that you cared. It's not like a judgment on Langchain. I was just like, you know, you write an adapter for it and you're done, right? Like how much further does it go for Langchain? In terms of like, for you, it's one of the, you know, the model IO modules and that's it. But like, you seem very personally, very passionate about it, but I don't know what the Langchain specific angle for this is, for fine-tuning local models, basically. Like you're just passionate about local models and privacy and all that, right? And open source. [00:35:41] Harrison: Well, I think there's a few different things. Like one, like, you know, if we think about what it takes to build a really reliable, like context-aware reasoning application, there's probably a bunch of different nodes that are doing a bunch of different things. And I think it is like a really complex system. And so if you're relying on open AI for every part of that, like, I think that starts to get really expensive. Also like, probably just like not good to have that much reliability on any one thing. And so I do think that like, I'm hoping that for like, you know, specific parts at the end, you can like fine-tune a model and kind of have a more specific thing for a specific task. Also, to be clear, like, I think like, I also, at the same time, I think open AI is by far the easiest way to get started. And if I was building anything, I would absolutely start with open AI. So. [00:36:27] Swyx: It's something I think a lot of people are wrestling with. But like, as a person building apps, why take five vendors when I can take one vendor, right? Like, as long as I trust Azure, I'm just entrusting all my data to Azure and that's it. So I'm still trying to figure out the real case for local models in production. And I don't know, but fine-tuning, I think, is a good one. That's why I guess open AI worked on fine-tuning. [00:36:49] Harrison: I think there's also like, you know, like if there is, if there's just more options available, like prices are going to go down. So I'm happy about that. So like very selfishly, there's that aspect as well. [00:37:01] Alessio: And in the Lancsmith announcement, I saw in the product screenshot, you have like chain, tool and LLM as like the three core atoms. Is that how people should think about observability in this space? Like first you go through the chain and then you start dig down between like the model itself and like the tool it's using? [00:37:19] Harrison: We've added more. We've added like a retriever logging so that you can see like what query is going in and what are the documents you're getting out. Those are like the three that we started with. I definitely think probably the main ones, like basically the LLM. So the reason I think the debugging in Lancsmith and debugging in general is so needed for these LLM apps is that if you're building, like, again, let's think about like what we want people to build in with LangChain. These like context aware reasoning applications. Context aware. There's a lot of stuff in the prompt. There's like the instructions. There's any previous messages. There's any input this time. There's any documents you retrieve. And so there's a lot of like data engineering that goes into like putting it into that prompt. This sounds silly, but just like making sure the data shows up in the right format is like really important. And then for the reasoning part of it, like that's obviously also all in the prompt. And so being able to like, and there's like, you know, the state of the world right now, like if you have the instructions at the beginning or at the end can actually make like a big difference in terms of whether it forgets it or not. And so being able to kind of like. [00:38:17] Swyx: Yeah. And it takes on that one, by the way, this is the U curve in context, right? Yeah. [00:38:21] Harrison: I think it's real. Basically I've found long context windows really good for when I want to extract like a single piece of information about something basically. But if I want to do reasoning over perhaps multiple pieces of information that are somewhere in like the retrieved documents, I found it not to be that great. [00:38:36] Swyx: Yeah. I have said that that piece of research is the best bull case for Lang chain and all the vector companies, because it means you should do chains. It means you should do retrieval instead of long context, right? People are trying to extend long context to like 100K, 1 million tokens, 5 million tokens. It doesn't matter. You're going to forget. You can't trust it. [00:38:54] Harrison: I expect that it will probably get better over time as everything in this field. But I do also think there'll always be a need for kind of like vector stores and retrieval in some fashions. [00:39:03] Alessio: How should people get started with Langsmith Cookbooks? Wanna talk maybe a bit about that? [00:39:08] Swyx: Yeah. [00:39:08] Harrison: Again, like I think the main thing that even I find valuable about Langsmith is just like the debugging aspect of it. And so for that, it's very simple. You can kind of like turn on three environment variables and it just logs everything. And you don't look at it 95% of the time, but that 5% you do when something goes wrong, it's quite handy to have there. And so that's probably the easiest way to get started. And we're still in a closed beta, but we're letting people off the wait list every day. And if you really need access, just DM me and we're happy to give you access there. And then yeah, there's a lot that you can do with Langsmith that we've been talking about. And so Will on our team has been leading the charge on a really great like Langsmith Cookbooks repo that covers everything from collecting feedback, whether it's thumbs up, thumbs down, or like multi-scale or comments as well, to doing evaluation, doing testing. You can also use Langsmith without Langchain. And so we've got some notebooks on that in there. But we have Python and JavaScript SDKs that aren't dependent on Langchain in any way. [00:40:01] Swyx: And so you can use those. [00:40:01] Harrison: And then we'll also be publishing a notebook on how to do that just with the REST APIs themselves. So yeah, definitely check out that repo. That's a great resource that Will's put together. [00:40:10] Swyx: Yeah, awesome. So we'll zoom out a little bit from Langsmith and talk about Langchain, the company. You're also a first-time founder. Yes. And you've just hired your 10th employee, Julia, who I know from my data engineering days. You mentioned Will Nuno, I think, who maintains Langchain.js. I'm very interested in like your multi-language strategy, by the way. Ankush, your co-founder, Lance, who did AutoEval. What are you staffing up for? And maybe who are you hiring? [00:40:34] Harrison: Yeah, so 10 employees, 12 total. We've got three more joining over the next three weeks. We've got Julia, who's awesome leading a lot of the product, go-to-market, customer success stuff. And then we've got Bri, who's also awesome leading a lot of the marketing and ops aspects. And then other than that, all engineers. We've staffed up a lot on kind of like full stack infra DevOps, kind of like as we've started going into the hosted platform. So internally, we're split about 50-50 between the open source and then the platform stuff. And yeah, we're looking to hire particularly on kind of like the things, we're actually looking to hire across most fronts, to be honest. But in particular, we probably need one or two more people on like open source, both Python and JavaScript and happy to dive into the multi-language kind of like strategy there. But again, like strong focus there on engineering, actually, as opposed to maybe like, we're not a research lab, we're not a research shop. [00:41:48] Swyx: And then on the platform side, [00:41:49] Harrison: like we definitely need some more people on the infra and DevOps side. So I'm using this as an opportunity to tell people that we're hiring and that you should reach out if that sounds like you. [00:41:58] Swyx: Something like that, jobs, whatever. I don't actually know if we have an official job. [00:42:02] Harrison: RIP, what happened to your landing page? [00:42:04] Swyx: It used to be so based. The Berkshire Hathaway one? Yeah, so what was the story, the quick story behind that? Yeah, the quick story behind that is we needed a website [00:42:12] Harrison: and I'm terrible at design. [00:42:14] Swyx: And I knew that we couldn't do a good job. [00:42:15] Harrison: So if you can't do a good job, might as well do the worst job possible. Yeah, and like lean into it. And have some fun with it, yeah. [00:42:21] Swyx: Do you admire Warren Buffett? Yeah, I admire Warren Buffett and admire his website. And actually you can still find a link to it [00:42:26] Harrison: from our current website if you look hard enough. So there's a little Easter egg. Before we dive into more of the open source community things, [00:42:33] Alessio: let's dive into the language thing. How do you think about parity between the Python and JavaScript? Obviously, they're very different ecosystems. So when you're working on a LangChain, is it we need to have the same abstraction in both language or are you to the needs? The core stuff, we want to have the same abstractions [00:42:50] Harrison: because we basically want to be able to do serialize prompts, chains, agents, all the core stuff as tightly as possible and then use that between languages. Like even, yeah, like even right now when we log things to LangChain, we have a playground experience where you can run things that runs in JavaScript because it's kind of like in the browser. But a lot of what's logged is like Python. And so we need that core equivalence for a lot of the core things. Then there's like the incredibly long tail of like integrations, more researchy things. So we want to be able to do that. Python's probably ahead on a lot of like the integrations front. There's more researchy things that we're able to include quickly because a lot of people release some of their code in Python and stuff like that. And so we can use that. And there's just more of an ecosystem around the Python project. But the core stuff will have kind of like the same abstractions and be translatable. That didn't go exactly where I was thinking. So like the LangChain of Ruby, the LangChain of C-sharp, [00:43:44] Swyx: you know, there's demand for that. I mean, I think that's a big part of it. But you are giving up some real estate by not doing it. Yeah, it comes down to kind of like, you know, ROI and focus. And I think like we do think [00:43:58] Harrison: there's a strong JavaScript community and we wanted to lean into that. And I think a lot of the people that we brought on early, like Nuno and Jacob have a lot of experience building JavaScript tooling in that community. And so I think that's a big part of it. And then there's also like, you know, building JavaScript tooling in that community. Will we do another language? Never say never, but like... [00:44:21] Swyx: Python JS for now. Yeah. Awesome. [00:44:23] Alessio: You got 83 articles, which I think might be a record for such a young company. What are like the hottest hits, the most popular ones? [00:44:32] Harrison: I think the most popular ones are generally the ones where we do a deep dive on something. So we did something a few weeks ago around evaluating CSV question answering applications, which I think is a really interesting one because most question answering, like everyone does question answering, but it's generally over unstructured data over your documents and you do the whole rag thing. And that doesn't work amazing for structured data. And so this was something that we heard, the origin of this was basically we heard from the community, you guys should improve this. And so we're like, okay, let's improve it. And then we're like, okay, well, in order to see if we improve it, we need to like evaluate it and see how we're doing. And so we kind of like wrote up a lot of our thought process there. And I think, and a lot of people like reached out about that and thought that was interesting and we're going through similar challenges and had, we posted another one a few days after that someone wrote basically as a response, which is awesome because it had a completely different strategy. And it was a really, it was a really, that was a really good piece as well. So that was like a deep dive on something like evaluation bit. I think like we did one on retrieval a while back, which was basically like, hey, we, and this was around when we changed our abstractions, like, hey, we changed our abstractions to this. This is why we did it. This is what we see coming down the pipeline. These are like the different types of retrieval that we see. I think a lot of people read and liked that one. A lot of the blogs that we do are also highlighting cool partnerships or cool applications. But in terms of, if you go by like number of views, I think the ones that get the most views are the more like deep dive ones. [00:45:55] Swyx: Yeah. And I also noticed that you do guest posts as well. [00:45:58] Harrison: Actually, you know, which one, and this is a guest post that got a lot of views, the multi-on one, the multi-on agent one. When we did, we did a blog where we integrated with them and that got a ton of views. [00:46:06] Swyx: What do you think that is? [00:46:07] Harrison: I think it's, I mean, it's one of like the few agents that's actually available and like out in the world. [00:46:15] Swyx: They're still behind a wait list. Still behind a wait list, [00:46:17] Harrison: but they're very active on social media. I don't know if I'm off the wait list. [00:46:21] Swyx: I mean, you're on their blogs. They're on your blog, so I hope they give you access at some point. But that's interesting. A lot of interest in agents. I think they just opened up an API as well. Yeah, exactly. [00:46:32] Harrison: That was the blog that we did. I was, yeah, I was a bit surprised to see that as well, but I think there's generally a lot of interest in agents and it's also really hard to get them to work. And I think multi-on is one of the first that has that. [00:46:45] Swyx: Yeah. So my angle to this is a lot of people want to work with you. Yes. You're bombarded. I'm sure your email is just unmanageable. How should people be good partners with you? Like I work at a company and I'm like, hey, I'd love to do something on the LangChain blog or integrate to LangChain. I know Harrison's a busy guy. Like, what do I do? [00:47:03] Harrison: Like the stuff that gets my attention honestly is like the in-depth, really thought out stuff. Obviously I love this stuff. Like this stuff is awesome. And there's so many different, there's so much to do as well. And like the biggest thing that we have trouble with internally is like figuring out what to do. [00:47:17] Swyx: What's noise and what's signal. [00:47:19] Harrison: Not even that, but just like what to focus on. Like there's so many different directions we could do and we want to go in like so many because there's so many interesting things, but we can't do. So if anyone kind of like takes the time to like go deep in a particular area, I love talking to them and I love reading what they write. And I love sharing what they write on the blog. Like that to me is awesome. So I think like... [00:47:37] Swyx: Do good stuff. Be so good they can't ignore you. It sounds basic, right? [00:47:40] Harrison: So that's why I didn't want to say it. [00:47:42] Swyx: No, it's great. [00:47:42] Harrison: But I think like these deep dots, yeah, there's just so much to do and these don't do shallow stuff, I guess would be. [00:47:48] Swyx: I think that's a good call that people need reminding. [00:47:50] Alessio: What about the other side of open source? So on Acker News, there were a couple blog posts recently, like the problem with LangChain and LangChain is pointless, all these different things. So the TLDR of some of them were, the LangChain API is like kind of verbose and complicated versus like sometimes I can just do this in like 10 lines of code. How do you balance that in terms of allowing for the complex use cases versus making maybe the ergonomics like simpler, but then trading that off later? [00:48:21] Harrison: There's a lot to balance and there's a lot to do. And I think like posts like that are very valuable to hear basically what people are saying. And like, we have a lot of open issues. So it's not like these things hadn't been said before, but I think like that was a good emphasis on what people are saying. And I think there was a lot of things in there. I think part of it's kind of like around and we took all of it very seriously. And yeah, I think there's a lot to dive into there. There's like the documentation piece. And so I think we did a revamp of the documentation to address that. There's also like a comment in this, I think this was around, I think the top comment on the LangChain is pointless one was like basically like orchestration is like 5% of the work. And then like the other 95% is like prompt engineering and like data engineering. And those are the hard bits. I think maybe orchestration is a little bit more than 5%, but I like agree that those are like really big pain points that get exacerbated when you have these complex chains and agents where you can't really see what's going on inside of them. And I think that's partially why we built Langsmith to help out with exactly that. We also needed to do better things like make the prompts more visible and make it allow for more customizability around that. And so we've tried to add some stuff there. In terms of balancing, there's also LangChain is pointless. I don't need a wrapper. I can just call the underlying API. I think if all you're trying to do is call the underlying API, then like, yeah, that's gonna be the cleanest and simplest thing to do. And we try to get as close to that experience as possible, but we're not optimizing for calling the API. We're optimizing for helping people build context-aware reasoning applications as easily as possible. And so there's some level of abstractions that you need to add in order to assist in that. Yeah, that's definitely a balance that's tricky to strike, but I think there's also some aspect of it. Like, I do think one of the big benefits that LangChain provides is a standard interface for language models so that you can switch between them. And this kind of gets into like an ORM debate, like are ORMs generally kind of like useful or not? And so I think in this case they are. I think there's probably a larger kind of like philosophical kind of like question about that [00:50:25] Swyx: that people have strong opinions on. Just the prompts don't transfer like you also mentioned. Yeah, yeah, there's that, yeah. [00:50:32] Harrison: And then between kind of like allowing for, I think one helpful thing that we did in terms of like distinguishing between basically the base interfaces and then more complex stuff is part of the separation around the docs is there's like the components piece, which has the model IO, the retrieval, the agents, the callbacks, things like that. And then there's all the use cases. And so I think like the use cases, because they are like these assembly of all these things in a particular order, they start to get more complex. And it's, you know, we try our best to kind of like make clear how you can configure things. But yeah, there's a lot of different options that you might want to configure. And so I think that split has kind of helped us internally at least. And I think externally as well, because we've heard good comments about the improved documentation. I think that's made it a little bit more clear. And then another thing, one of the things that we also released soon after, and we'd been thinking about a little bit is basically like a LangChain expression language, which allows for actual composability of pieces. So LangChain, I think, has always been very good about interchangeability. Let's ignore the prompting issues, but like you could always plug in like one LLM for another one. You could swap in one vector for another one, but the chains themselves haven't actually been super actually composable. Like we had the sequential chain, but that was a bit like clunky to use. And then we had a router chain, but that was a bit, you know, that was also a bit clunky to use. And so one of the things, and so there's a million different things to do, and we didn't prioritize that. [00:51:53] Swyx: I think after this, [00:51:53] Harrison: we definitely bumped it up and prioritized in priority. And luckily Nuno had been doing a lot of awesome work on it already, so it wasn't too much of a lift. But yeah, now there's this way where a lot of the chains that we've been releasing are written in this LangChain expression language where they're actually truly composable, and you can see what's going on under the hood. And it's basically, it uses kind of like the pipe kind of like terminology to coordinate things and move things around. So yeah, I mean, I think there were a lot of good points in those Hacker News things, and you know, we can't respond to everything, but we try to like look at everything and take everything seriously. [00:52:25] Swyx: You're being very diplomatic. But so first of all, I like the expression language. I think that that is the path towards sort of language agnostic LangChain kind of, or whatever, DSL. But also like, what was just kind of plain wrong or plain offensive, or like, I don't know, people can get very vitriolic sometimes on Hacker News. [00:52:40] Harrison: Yeah, I mean, I think the comments that I appreciated were the ones where they gave specific things. And I think the ones where they said, you know, LangChain sucks. Like, okay. Can't do much of that. [00:52:51] Swyx: Yeah, exactly. Verifacing on my question would be like, you're not the first and you won't be the last to have that kind of very intense scrutiny. What would be your advice to other people, other maintainers of projects for going through something like this? [00:53:03] Harrison: I would probably say, try to drill into like what is actually underlying things [00:53:08] Swyx: as much as possible. [00:53:08] Harrison: And if there is actual substance that's being delivered, whether you agree with it or not, like, I think that's valuable to know. And then for the other stuff, like try to maybe follow up, but maybe try not to let it get under your skin too much. [00:53:22] Swyx: Thanks for tackling that. [00:53:24] Alessio: And I know we're getting to the time and we'll wrap up soon, but since you're going to speak at the AI Engineers Conference, what's your advice to AI engineers, especially when to start with LangChain and when they're just experimenting with a model, [00:53:38] Swyx: when are they, [00:53:38] Alessio: as you mentioned, if you just want to do an API call, don't use LangChain. Yeah. [00:53:43] Harrison: I mean, my advice would just like build as many things as possible. Like, I think it's still really early in the space. No one really knows what they're doing to some extent. Like, it's a bit weird to say, but there's so many things to like discover. So I would just say like, build as many things as possible. Cause I think like the best thing is you stumble upon a really good idea and you build something really awesome. And the worst thing that happens is you just learn a lot about a field and the technology that's going to be incredibly important and rapidly kind of like changing. [00:54:11] Alessio: What would you build if you weren't doing LangChain? [00:54:13] Harrison: I mean, the things that are most interesting to me are kind of like things around like long-term memory and like longer running agents. So I'd probably build, and these are things that we've been wanting to build [00:54:23] Swyx: internally as well. [00:54:23] Harrison: But like, I think a chatbot that like actually remembers things about you as like silly as that sounds, like people like chatbots a lot and they have their delivered limited by their context window. And so I think really diving into like a specific application of memory there. [00:54:38] Swyx: I've been trying to build a chatbot [00:54:39] Harrison: that remembers things about you. That would be one. And then like, I know a lot of people are doing this, but like a personal assistant for like managing like email calendar, basic stuff, which I think is, I think that's like a fantastic application for these like agent like things, because if you think about personal assistants today, you usually interact, I don't have one, but I'm told you interact with them over email. And the nice thing about that, as opposed to like chat, there's not as stringent an expectation on latency as there is on chat. And so you can do a lot of things like reflection and kind of like making sure that you're on the right track and really put more safeguards and thinking about these agents as opposed to relying on like chas and interface, like the bot we have that's on GitHub answering questions on the issues, I think probably gives better answers than the bots that we have that are on chat on the website. And I think that's not because, there's just different constraints that you have in different types of problems. And I think I would be like, I think the personal assistant one's really interesting because you remove the constraint of chat, which I think at this point in time is probably pretty limited in terms of functionality. [00:55:43] Swyx: Yeah. I've been calling this sort of long inference. If you didn't have to care about ANC and you could take like a day, a month, a year to work on something, what could you do? And yeah, that's super interesting. [00:55:56] Harrison: I think that's a really promising place to explore. [00:55:58] Swyx: Yeah. Have you looked at, regarding the long conversation thing, you and I have tried it about this many times. Have you looked into what character and inflection are doing? Because they're probably working on it. [00:56:08] Harrison: I've thought about memory a bunch. Like I think it comes down to like, it comes down to like state, like what's the state you're tracking? Like what's the data structure for that? And I think that could also maybe be a bit like application specific. But if we're talking about a generic chat bot, that's kind of generic. I don't know. Yeah, I don't know how they're thinking about that. My sense is that inflection like thinks about that a bit more than character. Like I think in Inception, sorry, inflection's whole thing is they like, the bot knows you. [00:56:33] Swyx: It's one chat. There's no history. You just talk to it. Yeah. [00:56:37] Harrison: So they've definitely got some state that they're tracking. I'd be really curious to know what that is. Character, I don't think has lent into it too much. I think they let you do some stuff in terms of like uploading background. And I'm not entirely sure how they use that, whether they just like put that in the prompt or do some retrieval over that. But I think they're definitely, they haven't lent into it as much as inflection, I would say. [00:56:57] Swyx: So given like, you are one of the most interested people in this space, would this be like a second product for you? If you ever want to explore that or do you want to just partner with people and you're putting out the call for people to come to you if they have solutions for that? [00:57:10] Harrison: If I wasn't working on LangChain, I would be building an application company, for sure, first of all. Like, I don't think, like I think like there's, which I know is very hypocritical to say. [00:57:20] Swyx: Like you're Mr. DevTools and Infra and Observability. [00:57:24] Harrison: Yeah, I don't know. If you're building an application company that's working on something related to long-term memory or long-term agents, I would love to chat and just geek out [00:57:31] Swyx: about a lot of this stuff. I'll show you Smalltalk at some point. Yes. Cool. Awesome. [00:57:37] Alessio: Yeah, let's do a lightning round. [00:57:38] Swyx: So the first one is on acceleration. What has happened in AI that you thought would take much longer than it actually ended up taking? [00:57:45] Harrison: The function call and ability from OpenAI, like tool usage. [00:57:48] Swyx: Yeah. [00:57:48] Harrison: They did that really fast, I thought. [00:57:50] Swyx: Yeah. But it's just a question of fine-tuning, no? Yeah. It's not even like reliable. [00:57:54] Harrison: It's not terrible. They're a pretty big organization that's serving a lot of traffic. And like, this was a, yeah, it's like, it is like just fine-tuning, but I think like you still have to like collect that data set and fine-tune it and evaluate it and then release it at scale and figure out the right API. [00:58:09] Swyx: No shade on OpenAI. Like they're moving everyone's bar as to how quickly like a 400% organization can go. Do you think it eliminates like approaches like JSONformer and all the other approaches that people, like guardrails, you know, previous guest, eliminates your output validation thing? Yeah. [00:58:26] Harrison: I think JSONformer and stuff like that are still really interesting for like local models, for sure. And there's like 90% of people use OpenAI or something and like my made up numbers. [00:58:37] Swyx: No, it's probably real. [00:58:38] Harrison: And the best way to get structured output is by using the function calling ability. So yeah, absolutely. [00:58:46] Alessio: What do you think is the most interesting unsolved question in AI? [00:58:50] Harrison: I'm really interested like how multimodal is going to work. Like with just what that looks like. [00:58:55] Swyx: Have you had a look at the GPT-4 vision? No, not really. [00:58:59] Harrison: Yeah, not beyond what they- [00:59:01] Swyx: They're doing private betas right now. So I'm very excited. [00:59:04] Harrison: I'm excited about that as well. Yeah, I mean, I think that's, you know, you talk about like, again, this whole space is just changing so fast, but you talk about something that could like really change how, because like, you know, a lot of lang chain is kind of like a data orchestration tool in some sense. And so if you had a whole new type of data in there. [00:59:20] Swyx: So maybe we do this thought exercise, right? Tomorrow, OpenAI releases the GPT-4 vision API. What does lang chain do? [00:59:25] Harrison: Immediately we add support for it in like the wrapper. So however you interact, like honestly, this is another like fun thing. Everyone's API now looks like OpenAI's. [00:59:35] Swyx: Yeah, which is great. [00:59:36] Harrison: Which you have to do, yeah. So like our wrapper looks similar to OpenAI. So I don't think it will be that difficult to include support for it at the basic model level. And so we do that. And now that we've released the expression language bit, like a lot of the core chains, we have examples of rewriting them just in this expression language. So like for retrieval, if we're now talking about like, okay, you can do like retrieval question answering over for multimodal things, we'd probably have to figure out how those are getting stored and what's being done with them. But then from there, that should be, yeah, so probably looking to like, yeah, how are people kind of like storing and consuming this type of information? But then that step should be pretty easy to plug into the kind of like chain. [01:00:17] Swyx: Multimodal stores? Yeah, I don't know. I always wonder what that would actually look like because a lot of multimodality in LLMs is really just an LLM, a text LLM calling a different model. And that's just no different than any API call, essentially unchanged. [01:00:32] Harrison: I think it's probably something that you don't know until you let like a million people play around with it. [01:00:37] Swyx: Then there'll be new LangChain for multimodal. What's one message you want everyone to remember today? [01:00:43] Harrison: I would probably say just like build. I think it's a fantastic time to be building. [01:00:47] Swyx: All right, just build. Yeah. [01:00:49] Alessio: Thank you Harrison for coming on. [01:00:51] Swyx: Thanks so much. [01:00:51] Harrison: Thank you guys for having me. [01:00:52] Swyx: It's a lot of fun. [01:00:53] This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:10Alessio Fanelli?

  Hey everyone, welcome to the Late in Space Podcast. This is Alessio, Partner and CTO in Residential decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI.

• 0:19Speaker 01

  Welcome. Today, we have Harrison Chase in the studio with us. Welcome Harrison.

• 0:23Speaker 02

  Thank you guys for having me. I'm excited to be here.

• 0:25Speaker 01

  Been a long time coming. We've been asking you for a little bit, and we're really glad that you got some time to join us in the studio.

• 0:31Speaker 02

  Yeah, I've been dodging you guys for a while. About seven months.

• 0:36Speaker 01

  You're a commodity. Yeah, we totally understand. We like to introduce people through the official backgrounds and then ask you a little bit about your personal side. So you went to Harvard, class 2017. You don't list what you did in Harvard. Was it CS?

• 0:48Speaker 02

  Stats and CS.

• 0:49Speaker 01

  That's awesome. I love me some good stats.

• 0:52Speaker 02

  I got into it through stats, through doing sports analytics. And then there was like so much overlap between stats and CS that I found myself doing more and more of that.

• 0:59Speaker 01

  And it's interesting that a lot of the math that you learn in stats actually comes over into machine learning.

• 1:04Speaker 02

  Oh, yeah.

• 1:04Speaker 01

  Which, would you apply it at Kent Show as a machine learning engineer and robust intelligence, which is, seems to be the home of a lot of AI founders.

• 1:11Speaker 02

  It does, yeah.

• 1:13Speaker 01

  And you started Langchain, I think around November 2023, and incorporated in January.

• 1:18Speaker 02

  Yeah, I was looking it up for the podcast and the first tweet was on, I think, October 24th. So just before the end of November or end of October.

• 1:26Speaker 01

  Yeah. So that's your LinkedIn. What should people know about you on the personal side? That's not obvious on, on LinkedIn.

• 1:33Speaker 02

  A lot of how I got into this is all through sports, actually. Like, I'm a big sports fan, played a lot of soccer growing up, and then really big fan of the NBA and NFL. And so freshman year at college, showed up and I knew I liked math. I knew I liked sports. One of the clubs that was there was the Sports Analytics Collective. And so I joined that freshman year. I was doing a lot of stuff in like Excel, just like basic stats. But then like wanted to do more advanced stuff. So learn to code, learn, learn kind of like data science and machine learning through that way. Kind of like just kept on going down that path. I think sports is a great entryway to data science and machine learning. There's a lot of like numbers out there. People like really care. Like I remember, I think sophomore, junior year, I was in the sports collective and, and the main thing we had was a blog. And so we wrote a blog.

• 2:14Speaker 02

  It wasn't me. Uh, one of the other people in the club wrote a blog predicting the NFL season. I think they made some kind of like with stats and I think their, their stats showed that like the Dolphins would end up beating the Patriots. And New England got like pissed about it, of course. So people like really care and they'll give you feedback about whether you're like models doing well or poorly. And so you get that. And then you also get like instantaneous kind of like, or not instantaneous, but really quick feedback. Like if you predict a game, the game happens that night. Like you don't have to wait a year to see what happens. So I think sports is a great kind of like entryway for kind of like data science.

• 2:43Alessio Fanelli?

  It was actually my first article on the Twilio blog with a Python script to like predict pricing of like daily fantasy players based on my past week performance. Yeah, I don't know. It's a good getaway drug.

• 2:56Speaker 01

  And on my end, the way I got into finance was through sports betting. So maybe we all have some ties in there. Was like, wasn't like Moneyball a big inspiration, the, the movie?

• 3:06Speaker 02

  Honestly, not really. I don't really like baseball. That's like the, the big thing.

• 3:10Speaker 01

  Let's call up a lot of stats.

• 3:11Speaker 02

  A lot of stats.

• 3:13Speaker 01

  Cool. Well, we can dive right into LangChain, which is what everyone is excited about, but feel free to make all the sports analogies you want. That really drives a lot of points. What was your GPT aha moment? When did you start working on GPT itself? And maybe, maybe not LangChain, just, just anything to do with the API.

• 3:29Speaker 02

  I think it probably started around the time we had a company hackathon. I think that was before I launched LangChain. I'm trying to remember the exact sequence of events, but I do remember that at the hackathon, I worked with Will, who's now actually at LangChain as well. And then two other members of Rebus. And we made basically a bot where you could ask questions of Notion and Slack. And so I think, yeah, rag, basically. And I think I wanted to try that out because I'd like heard that it was getting good, but hadn't, you know, I'm trying to remember if I like did anything before that to like realize that it was good. So then I would like focus on that on the hackathon. I can't remember or not, but that was one of the first times that I like built something with GPT three.

• 4:08Speaker 01

  There wasn't that much opportunity before because the API access wasn't that widespread. Like you had to get, you had to get into some kind of program to, to get that.

• 4:16Speaker 02

  DaVinci O2 was like not terrible, but they did like an upgrade to get it to there. And they didn't really publicize that as much. And so I think I remember playing around with it when the first DaVinci model came out. And I was like, this is cool, but it's like not amazing. And like there, you'd have to do a lot of work to get it to do something. But then I think like that, like February or something of, uh, I think of 2022, they like upgraded it. And it was, it got better, but I think they made less of kind of an announcement around it.

• 4:41Alessio Fanelli?

  Yeah.

• 4:41Speaker 02

  And so I just, yeah, kind of like slipped under the radar for me at least.

• 4:44Alessio Fanelli?

  And what was the step into LangChain? So you did the hackathon and then as you were building the kind of RAG product, you felt like the developer experience wasn't that great? Or what was the inspiration?

• 4:54Speaker 02

  No, honestly, like, so, so around that time, I knew I was going to leave my previous job. I was trying to figure out what I was going to do next. I went to a bunch of like meetups and other events. This was like the September, August, September of that year. So after stable diffusion, but before ChatGPT. So there was like interest in like generative AI as a space, but not a lot of people hacking on language models yet, but there are definitely some. And so I would go to these like meetups and like just chat with people. And basically saw some common abstractions in terms of like what they were building. And then thought it would be a cool side project to factor out some of those common abstractions. And, and that became kind of like LangChain. I looked up again before this because I remember I did like a tweet thread on Twitter to announce LangChain. And, and we can talk about what LangChain is. It's a series of components. And then there's some like end to end modules.

• 5:38Speaker 02

  And there was three kind of like end to end modules that were in the initial kind of like release. One was, uh, one was NatBot. So this was like the, um, web agent by Nat Friedman. Another was LLM Math Chain. So it would construct a

• 5:51Alessio Fanelli?

  GPT, you cannot do math.

• 5:53Speaker 02

  Yeah, exactly. And then the third was SelfAsk. So some type of like RAG search, similar to React style agent. So those were like some of the patterns in terms of like what I would see. And those all came from like open source or academic examples, because the people who were actually working on this were like building startups. And they were doing things like question answering over your databases, question answering over SQL, things like that. But I couldn't like, you know, use their code as kind of like inspiration to factor things out. So.

• 6:18Speaker 01

  I stalked you a little bit, actually, roundabout, right after you announced LangChain.

• 6:22Speaker 02

  I'm honored.

• 6:23Speaker 01

  I think I'm one of many. This is your first open source project.

• 6:26Speaker 02

  No, that's not actually true.

• 6:27Speaker 01

  Okay.

• 6:27Speaker 02

  I released, um, because I liked sports stats and so I remember I did release some, like, really small, random Python package for scraping data from, like, Basketball Reference or something. I'm pretty sure I released that. So, first project to get a star on GitHub. Let's say that.

• 6:44Speaker 01

  Yeah. Did you, like, reference anything? What was, what was the inspirations? Like, other frameworks that you looked to when open sourcing LangChain or announcing it, or anything like that?

• 6:53Speaker 02

  I mean, the, the only main thing that I looked for, I remember reading a Hacker News post a little bit before about how, like, a "read me" on the project goes a long way. Yeah, and so I looked at it and it was like, you know, put like some, like, um, uh, status checks at the top and like have like, uh, the title and then like one or two lines and then just write into installation. And so that's the main thing that I looked at in terms of how to structure it. Because, yeah, I hadn't done open source before. I didn't really know how to communicate that aspect of the marketing or the kind of like getting people to use it. I think I had some trouble finding it, but I finally found it and used that as a lot of the inspiration there.

• 7:26Speaker 01

  Yeah. It was one of the subjects of my write-up, how, it was surprising to me that significant open source experience actually didn't seem to matter in the new wave of AI tooling. Like, most, like, AutoGPT's Torrance, that was his first open source project ever.

• 7:40Speaker 02

  Yeah.

• 7:40Speaker 01

  And that became AutoGPT. Yeah, I don't know. It's just, to me, it's just interesting how, like, open source experience is kind of fungible or, like, not necessary, or you can kind of learn it on the job.

• 7:48Alessio Fanelli?

  Overvalued.

• 7:49Speaker 01

  Overvalued. Okay, you said it, not me.

• 7:53Alessio Fanelli?

  What's your description of LangChain today? Like, I think when, when I built the LangChain Hub UI in January, there were a few things, and I think you were one of the first people to talk about agents that were already in there before it got, it got hot now. And it's obviously evolved into a much bigger framework today. Run people through what LangChain is today, how they should, they should think about it and all of that.

• 8:13Speaker 02

  The way that we describe it or, or think about it internally is that LangChain is basically, and it's, I started off saying LangChain's a framework for building LLM applications, but that's really vague and not really specific. And I think part of the issue is LangChain does do a lot, so it's hard to be somewhat specific. But I think the way that we think about it internally and in terms of like prioritization, what to focus on. Is basically LangChain's a framework for building context-aware reasoning applications. And so that's a bit of a mouthful, but I think there's a lot, I think that speaks to a lot of the core parts of what's in LangChain. And so what concretely that means in LangChain, there's really two things. One is a set of kind of like components and modules. And these can be kind of like, these would be kind of like the prompt template abstraction, the LLM abstraction, chat model abstraction, vector store abstraction, text splitters,

• 8:57Speaker 02

  document loaders. And so these are combinations of things that we build and we implement, or we just have integrations with. So we don't have any language models ourselves. We don't have any vector stores ourselves, but we integrate with a lot of them. And then the text splitters, we kind of like have our own logic for that. The document loaders, we have our own logic for that. And so those are kind of like the individual modules. But then I think another big part of LangChain, and probably the part that got people using it the most is like the end-to-end kind of like chains or applications. So we have a lot of chains for getting started with like question answering over your documents, chat question answering, question answering over SQL databases, agent stuff that you can kind of like plug in off the box. And that basically combines these components in a series of specific ways to kind of like do this. So if you think about like a question answering app, you need a lot of different components

• 9:42Speaker 02

  kind of like stacked. And there's a bunch of different ways to do question answering apps. So this is a bit of an overgeneralization, but basically, you know, you have some component that looks up an embedding from a vector store. And then you put that into like the prompt template with the question and the context. And maybe you have the chat history as well. And then that generates an answer. And then maybe you do something, you parse that out, or you do something with the answer there. And so there's this sequence of things that you basically stack in a particular way. And so we just provide a bunch of those assembled chains off the shelf to make it really easy to get started in a few lines of codes.

• 10:09Alessio Fanelli?

  And just to give people context, when you first released LangChain, OpenAI did not have a chat API. It was a completion-only API. So you had to do all the human assistant like prompting and whatnot. So you abstracted a lot of that away. I think like the most interesting thing to me is like you're kind of this Switzerland of like this developer land. There's like, A bunch of vector databases that are like killing each other out there to get people to embed data in them. And you're like, I love you all. You all are great. How do you kind of think about being an opinionated framework versus like leaving a lot of choice to the user? And in terms of like spending time into this integration, it's like you only have 10 people on the team. Obviously, that takes time.

• 10:48Speaker 02

  Yeah.

• 10:48Alessio Fanelli?

  What's that process like for you all?

• 10:50Speaker 02

  I think right off the bat, having different options for language models. I mean, language models is the main one that right off the bat, we knew we wanted to support a bunch of different options for. There's a lot to discuss there. People want optionality between different language models. They want to try it out. They want to maybe change the ones that are cheaper as new ones kind of like emerge. They don't want to get stuck into like one particular one if a better one comes out. But there's some challenges there as well. Prompts don't really transfer. And so there's a lot of like nuance there, but from the bat, having this optionality between the language model providers was a big important part. Cause I think that was just something we felt really strongly about. Like we believe, you know, there's not just going to be one model that rules them all. There's going to be a bunch of different models that are good for a bunch of different use cases. I did not anticipate the number of vector stores that would emerge, you know?

• 11:36Speaker 02

  I don't know how many we supported in the initial release. It probably wasn't as big of a focus as language models was, but I think it kind of quickly became so. Especially when like Postgres and Elastic and Redis started building kind of like their vector store implementations. And, you know, we saw that some people might not want to use a dedicated vector store. Maybe they want to use kind of like traditional databases. And I think to your point around like what we're opinionated about, like I think the thing that we believe most strongly is it's like super early in the space and super fast moving. And so there's a lot of uncertainty about how things will shake out in terms of like, what role will vector databases play? How many will there be? What about language models? And so I think a lot of it has always kind of been this optionality and ability to switch and not getting locked in.

• 12:18Speaker 01

  There's other pieces of LangChain which maybe don't get as much attention sometimes. And the way that you explained LangChain is somewhat different from the docs. I don't know how to square this. So for example, you have at the top level in your docs, you have, we mentioned Model I O, we mentioned retrieval, we mentioned chains. Then you have a concept called agents, which I don't know if like exactly matches what other people say call agents. And we also talked about memory. And then finally, there's callbacks. Are there any other, the less understood concepts in LangChain that you want to give some air to?

• 12:52Speaker 02

  I mean, I think buried in Model I O is some stuff around like few-shot example selectors that I think is really powerful.

• 13:00Speaker 01

  That's a workhorse.

• 13:01Speaker 02

  Yeah.

• 13:01Speaker 01

  I think that's where I start with LangChain.

• 13:03Speaker 02

  It's one of those things that like, you probably don't, if you're building an application, you probably don't start with it. You probably start with like a zero-shot prompt, but I think like that's a really powerful one. That's probably just talked about less because you don't need it right off the bat. And for those of you who don't know, that basically selects from a bunch of examples, the ones that are maybe most relevant to the input at hand. So you can do some nice kind of like in-context learning there. I think that's, we've had that for a while. I don't think enough people use that, basically. Output parsers also used to be kind of important, but then function calling, there's this interesting thing where like, the space is just like progressing so rapidly that a lot of things that were really important have kind of like diminished a bit, to be honest.

• 13:42Speaker 01

  Yeah.

• 13:43Speaker 02

  Output parsers definitely used to be an understated and underappreciated part. And I think if you're working with non-open AI models, they still are, but a lot of people are working with open AI models. But even within there, like, there's different things you can do with kind of like the function calling ability. Sometimes you want to have the option of having the text or the like, the application you're building, it could return either. Sometimes you know that it wants to return in a structured format, and so you just want to take that structured format. Other times you're extracting things that are maybe a key in that structured format. And so you want to like pluck that key. And so there's just like some like annoying kind of like. parsing of that, that to do. Agents, memory and retrieval, we haven't talked at all. Retrieval, there's like five different subcomponents. You could, you could also probably talk about all of those in depth.

• 14:25Speaker 02

  You've got the document loaders, the text splitters, the embedding models, the vector stores. Embedding models and vector stores, we don't really have, or sorry, we don't build, we integrate with those. Text splitters, I think we have like 15 or so. Like, I think there's an under kind of like appreciated amount of those. And then

• 14:39Alessio Fanelli?

  Well, it's, it's actually, honestly, it's overwhelming because nobody knows what to choose.

• 14:43Speaker 02

  Yeah, there is a lot.

• 14:44Alessio Fanelli?

  Yeah.

• 14:45Speaker 02

  There's a lot.

• 14:45Alessio Fanelli?

  Do you have personal favorites that you want to shut up?

• 14:47Speaker 02

  The one that we have in the docs is the default is like the recursive text splitter. We added a playground for text splitters the other week because yeah, we heard a lot that like, you know, and like these affect things like the, the chunk overlap and the chunks, they affect things in really subtle ways. And so like, I think we added a playground where people could just like choose different ones. We have like, and a lot of the ideas are really similar. You, you split on different characters depending on kind of like the type of text that you have marked down. You might want to split on differently than HTML. And so we added a playground where you can kind of like choose between those. I don't know if those are like underappreciated though, because I think a lot of people talk about text splitting as, as being a hard part. And it is a really important part of, of creating these retrieval applications. But I think we have a lot of really cool retrieval algorithms as well.

• 15:28Speaker 02

  So like self query is maybe one of my favorite things in Langchain, which is basically this idea of when you have a user question, the typical kind of like thing to do is you, you embed that question and then find the document that's most similar to that question.

• 15:41Alessio Fanelli?

  Yeah.

• 15:41Speaker 02

  But oftentimes questions have things that just, you don't really want to look up semantically. They have some other meaning. So like in the example that I used and the example in the docs is like movies about aliens in the year 1980. 1980, I guess there's some semantic meaning for that, but it's a very particular thing that you care about. And so what the, the self query retriever does is it splits out the metadata filter. And most vector stores support like a metadata filter. So it splits out this metadata filter and then it splits, splits out the semantic bit. And that's actually like kind of tricky to do because there's a lot of different filters that you can have like greater than, less than, equal to. You can have and things. So if you have multiple filters. So we have like a pretty complicated like prompt that does all that. That might be one of my favorite things in Langchain period. Like, I think that's, yeah, I think that's really cool.

• 16:26Alessio Fanelli?

  How do you think about speed of development versus support of existing things? So we mentioned retrieval, like you got our, you know, text splitting, you got like different options for all of them. As you keep building LangChain, how do you decide which ones you're not gonna keep supporting? You know, which ones you're gonna leave behind? I think right now, as you said, the space moves so quickly that like you don't even know who's using what. What's that like for you all?

• 16:50Speaker 02

  Yeah, I mean, we have, um, you know, we don't really have telemetry on what people are using in terms of what parts of LangChain. The telemetry we have is like, you know, anecdotal stuff when people ask or have issues with things. A lot of it also is like, I think we definitely prioritize kind of like keeping up with the stuff that comes out. I think we, we added function calling like the day it came out or the day after it came out. We added chat model support like the day after it came out or something like that. That's probably, I think I'm really proud of how the team has kind of like kept up with that. Cause this space is like exhausting sometimes. And so that's probably, that's a big focus of ours. The support, I think we've like, to be honest, we've had to get kind of creative with how we do that.

• 17:31Speaker 02

  Cause we have like, I think, I don't know how many open issues we have, but we have like three thousand, somewhere between 2,000 and 3,000 like open GitHub issues. We've experimented with a lot of startups that are doing kind of like question answering over your docs and stuff like that. And so we've got them on the, the website and in the Discord. And there's a really good one, uh, Dosu on the GitHub that's like answering issues and stuff like that. And that's actually something we want to start leaning into more heavily as a company as well as kind of like building out in AI DevRel. Cause we're 10 people now, 10, 11 people now. Um, and like two months ago, we were like six or something like that, right? So like, and to have like 2,500 open issues or something like that. And like 300 or 400 PRs as well. Cause like, one of the amazing things is that like, and you kind of alluded to this earlier, everyone's building in the space. There's so many different like touchpoints.

• 18:16Speaker 02

  LangChain is lucky enough to kind of like be a lot of the glue that connects it. And so we get to work with a lot of awesome companies, but that's also a lot of like work to keep up with as well. And so I don't really have an amazing answer, but I think like the, I think prioritize kind of like new things that, that come out. And then we've gotten creative with some of kind of like the support functions. And, and luckily there's, you know, there's a lot of awesome people working on all those support, coding, question answering things that we've been able to work with.

• 18:46Speaker 01

  I think there is your daily rhythm, which I've seen you, you work like a beast man, mad impressive. And then there's sometimes where you step back and do a little bit of high level, like 50,000 foot stuff. So we mentioned retrieval, you did a refactor in March. And there's other abstractions that you've sort of changed your mind on. When do you do that? When do you do like the step back from the day to day and go, where are we going? And change the direction of the ship?

• 19:11Speaker 02

  It's a good question. So far, it's probably been, you know, we see three or four or five things pop up that are enough to make us think about it. And then kind of like when it reaches that level, you know, we don't have like a monthly meeting where we sit down and do like a monthly plan or something. Maybe we should.

• 19:27Speaker 01

  I've thought about this.

• 19:28Speaker 02

  Yeah.

• 19:29Speaker 01

  I'd love to host that meeting.

• 19:32Speaker 02

  It's really been a lot of, you know, one of the amazing things is we get to interact with so many different people. So it's been a lot of kind of like just pattern matching on what people are doing and trying to see those patterns before they punch us in the face or something like that. So for retrieval, it was the pattern of seeing like, hey, yeah, like a lot of people are using vector search stuff. There's also just like other methods and people are offering like hosted solutions and we want our abstractions to work with that as well. So we shouldn't bake in this paradigm of doing like semantic search too heavily, which sounds like basic now. But I think like, you know, to start, a lot of it was people needed help doing these things. But then there was like managed things that did them, hybrid retrieval mechanisms, all of that. I think another example of this, I mean, Langsmith, which we can maybe talk about was like very kind of like, I think we worked on that for like three or four months before announcing it kind of like publicly.

• 20:17Speaker 02

  Two months, maybe before giving it to kind of like anyone in beta. But this was a lot of debugging these applications as a pain point. We hear that like just understanding what's going on is a pain point.

• 20:27Alessio Fanelli?

  I mean, you two did a webinar on this, which is called Agents versus Chains.

• 20:31Speaker 01

  That was fun, buddy. Thanks for having me on.

• 20:32Speaker 02

  No, thanks for coming. That was a good one.

• 20:34Alessio Fanelli?

  And on the website, you list like RAG, which is Retrieval Back to Generation and Agents as two of the main, the main goals of blockchain. And the difference, I think at the Databricks keynote, you said chains are like predetermined steps and agents is models reasoning to figure out what steps to take and what actions to take. How should people think about when to use the two? And how do you transition from one to the other with Langchain? Like, is it a path that you support or like, do people usually re-implement from an agent to a chain or vice versa?

• 21:05Speaker 02

  Yeah, you know, I know agent is probably an overloaded term at this point. And so there's probably a lot of different definitions out there. But yeah, as you said, kind of like the way that I think about an agent is basically like. In a chain, you have a sequence of steps. You do this, and then you do this, and then you do this, and then you do this. And with an agent, there's some aspect of it where the LLM is kind of like deciding what to do and what steps to do in what order. And you know, there's probably some like gray area in the middle, but you know, don't fight me on this. And so if we think about those, like the, the benefits of the chains are that they're like, you can say, do this, and you just have like a more rigid kind of like order in the way that things are done. And you have more control and they don't go off the rails and, and basically everything that's bad about agents in terms of being uncontrollable and expensive, you can control more finely.

• 21:49Speaker 02

  The benefit of agents is that I think they handle like the long tail of things that can happen really well. And so for an example of this, let's maybe think about like a, interacting with a SQL database. So you can have like a SQL chain and you know, the, the first kind of like naive approach at a SQL chain would be like, okay, you have the user question, then you like write the SQL query, you do some rag, you pull in the relevant tables and schemas, you write a SQL query, you execute that against the SQL database. And then you like return that as the answer, or you like summarize that with an LLM and return that to the answer. And that's basically the SQL chain that we have in LinkedIn. But there's a lot of things that can go wrong in that process, starting from the beginning. You may like not want to even query the SQL database at all. Maybe they're saying like hi or something, or they're misusing the application.

• 22:32Speaker 02

  Then like what happens if you have some step, like a big part of the application that people use with LinkedIn is like the context aware part. So there's generally some part of bringing in context to the language model. So if you bring in the wrong context to the language model, so it doesn't know which tables to query, what do you do then? If you write a SQL query, it's like syntactically wrong and it can't run. And then if it can run, like what if it returns an unexpected result or something? And so basically what we do with a SQL agent is we give it access to all these different tools. So it has another tool, it can run the SQL query, it has another, and then it can respond to the user. But then if it kind of like, it can decide which order to do these in. And so it gives it flexibility to handle all these edge cases. And there's like, obviously downsides to that as well. And so there's probably like some safeguards you want to put in place around agents in

• 23:14Speaker 02

  terms of like not letting them run forever, having some observability in there. But I do think there's this benefit of, um, you know, like, again, to the other part of what LinkedIn is, like the reason. Reasoning part, like each of those steps individually involves some aspect of reasoning, for sure. Like you need to reason about what the SQL query is, you need to reason about what to return. But there's then, there's also reasoning about the order of operations. And so I think to me, the key is kind of like giving it an appropriate amount to reason about, while still keeping it within checks. And so to the point, like I would probably recommend that most people get started with chains. And then when they get to the point where they're hitting these edge cases, then they think about, okay, I'm hitting a bunch of edge cases where the SQL query is just not returning like the relevant things. Maybe I should add in some step there and let it maybe make multiple queries or something like that.

• 23:59Speaker 02

  Basically, like start with chain, figure out when you're heading to these edge cases, add in the reasoning step to that to handle those edge cases appropriately. That would be kind of like my recommendation.

• 24:08Speaker 01

  Right. If I were to rephrase it, in my words, an agent would be a reasoning node in a chain. Right? Like you start with a chain, then you just add a reasoning node. Now it's an agent.

• 24:17Speaker 02

  Yeah, the architecture for your application doesn't have to be just a chain or just an agent. It can be an agent that calls chains. It can be a chain that has an agent in different parts of them. And this is another part as well, like the chains in Langchain are largely intended as kind of like a way to get started and take you some amount of the way. But for your specific use case, in order to kind of like eke out the most performance, you're probably going to want to do some customization at the very basic level, like probably around the prompt or something like that. And so one of the things that we've focused on recently is like making it easier to customize these bits of existing architectures, but you probably also want to customize your architectures as well.

• 24:52Speaker 01

  You mentioned a bit of prompt engineering for self-asking and for this stuff. There's a bunch of, I just talked to a prompt engineering company today, Prompt Ops or LLM Ops. Advice or thoughts on that field in general? Like, are you going to compete with them? Do you have internal tooling that you've built?

• 25:08Speaker 02

  A lot of what we do is like where we see kind of like a lot of the pain points being like, we can talk about Langsmith and that was a big motivation for that. And like, I don't know, would you categorize Langsmith as prompt ops?

• 25:18Speaker 01

  I don't know. It's whatever you want it to do. You want to call it.

• 25:22Speaker 02

  I don't know either. Like, I think like there's.

• 25:24Speaker 01

  I think about it as like a prompt registry and you store them and you A B test them and you do that. Langsmith, I feel like doesn't quite go there yet.

• 25:32Speaker 02

  Yeah.

• 25:32Speaker 01

  It's obviously, it's obviously the next step.

• 25:34Speaker 02

  Yeah, we'll probably go and yeah, we'll do more of that. Cause I think that's definitely part of the application of a chain or agent is you start with a default one, then you improve it over time and like. I think a lot of, um, the main new thing that we're dealing with here is like language models and the main new way to control language models is prompts. And, and so like a lot of the change in agents are powered by this combination of like prompt language model and then some output parser or something, doing some of the output. And so like, yeah, we want to make that core thing as, as good as possible. And so we'll, we'll do stuff all around that for sure. Yep.

• 26:06Speaker 01

  Awesome. We might as well go into Langsmith because we're bringing it up so much. So you announced Langsmith at the last month. What are your visions for it? Is this the future of blockchain company?

• 26:16Speaker 02

  It's definitely part of the future. So Langsmith is basically a control center for kind of like your LLM application. So the main features that it kind of has is like debugging, logging, monitoring, and then like testing and evaluation. And so debugging, logging, monitoring, basically law, you set three environment variables and it kind of like logs all the runs that are happening in your link chain chains or agents and it logs kind of like the inputs and outputs at each step. And so the main use case we see for this is in debugging. And that's probably the main reason that we started down this path of building it is I think like as you have these more complex things, debugging what's actually going on becomes really painful, um, whether, whether you're using Langchain or not. And so like adding this type of observability and debuggability was really important.

• 27:01Speaker 02

  Yeah, there's the debugging aspect. You can see the inputs, outputs at each step. You can then quickly enter into like a playground experience where you can, you can fiddle around with it. The first version didn't have that playground and then we'd see people copy, go to OpenAI Playground, paste them there. Okay. Well, that's a little annoying like just. And then there's, there's kind of like the, the monitoring, logging experience. And we recently added some analytics on like, you know, how many requests are you getting per hour, minute, day? What's the feedback like over time? Um, and then there's like a testing, debugging, sorry, testing and evaluation component as well, where basically you can create data sets and then test and evaluate these data sets. And I think importantly, All these things are tied to each other and then also into LangChain, the framework.

• 27:44Speaker 02

  So what I mean by that is like, we've tried to make it as easy as possible to go from logs to adding a data point to a data set. And because we think a really powerful flow is you don't really get started with a data set. You could accumulate a data set over time. And so being able to find points that have gotten like a thumbs up or a thumbs down from a user can be really powerful in terms of creating a good data set. And so that's maybe like a connection between the two. And then the connection in the other way is like all the runs that you have when you test or evaluate something, they're logged in the same way. So you can debug what exactly is going on and you don't just have like a final score. You have like this nice trace and thing where you can jump in. And then we also want to do more things to hook this into LangChain proper, the framework.

• 28:26Speaker 02

  So I think like some of like the managing the prompts will tie in here already. Like we talked about example selectors, using data sets as few shot examples is a path that we support in a somewhat janky way right now, but we're going to like make better over time. And so there's this connection between everything.

• 28:42Alessio Fanelli?

  Yeah. And you mentioned the data set. In the announcement blog post, you touched on heuristic evaluation versus LLMs evaluating LLMs. I think there's a lot of talk and confusion about this online. How should people prioritize the two, especially when they might start with like not a good set of evals or like any data at all?

• 29:01Speaker 02

  I think it's really use case specific. And the distinction that I draw between heuristic and LLM, LLMs, you're using an LLM to evaluate the output. Heuristics, you have some common heuristic that you can use. And so some of these can be like really simple. So we were doing some kind of like measuring of an extraction chain where we wanted it to output JSON. Okay. One evaluation can be, can you use JSON dot loads to load it? And like, right? And that, and that, that works perfectly. You don't need an LLM to do that. But then for like a lot of like the, the question answering, like, is this factually accurate? And you have some ground truth fact that you know it should be answering with. I think, you know, LLMs aren't perfect. And I think there's a lot of discussion around the pitfalls of using LLMs to evaluate themselves. And, and I'm not saying they're perfect by any means, but I, but I do think they're.

• 29:43Speaker 02

  We've found them to be kind of like better than than blue or any of those metrics. And and the way that I also like to to use those is also just like guide my eye about where to look. So like, you know, I might not trust the score of like point eight two, like exactly correct, but like I can look to see like which data points are like flagged as passing or failing. And sometimes the evaluator is messing up, but it's like good to like, you know, I don't have to look at like 100 data points. I can focus on like 10 or something like that.

• 30:10Alessio Fanelli?

  And then can you create your rustic ones in Langsmith? Like, what's your connection to today?

• 30:16Speaker 02

  Yeah, so so right now, all the evaluation, we actually do client side. And part of this is is basically due to the fact that a lot of the evaluation is really application specific. So we thought about having evaluators, you could just click off and run in a server side or something like that. But we still think it's really early on in evaluation. We still think there's it's just really application specific. So we prioritized instead making it easy for people to write custom evaluators and then run them client side. And then upload the results so that they can manually inspect them because I think manual inspection is still a pretty big part of evaluation for better or worse.

• 30:50Speaker 01

  We have this sort of components of observability. We have cost latency, accuracy and then planning. Is that listed in there?

• 30:57Alessio Fanelli?

  Well, planning more in the terms of like, if you're an agent, how to pick the right tool and whether or not you are picking the right tool.

• 31:03Speaker 01

  Well, so when you talk to customers, how would you stack rank those needs? Like, are they cost sensitive or are they latency sensitive? I imagine accuracy is pretty high up there.

• 31:13Speaker 02

  I think accuracy is definitely the top that we're seeing right now. I think a lot of the applications people are, especially the the ones that we're working with, people are still struggling to get them to work at a at a level where they're reliable enough. So that's definitely the first. Then I think probably cost becomes the next one. I think a few places where we've started to see this be like one of the main things is the AI simulation that came out.

• 31:36Speaker 01

  Generative agents?

• 31:37Speaker 02

  Yeah, exactly, which is really fun to run, but it costs a lot of money. And so one of our team members, Lance, did an awesome job hooking up like a local model to it. You know, it's not as perfect, but I think it helps that another really big place for this, we believe, is in like extraction of structured data from unstructured data. The reason that I think it's so important there is usually you do extraction of some type of like pre-processing or indexing process over your documents. I mean, there's a bunch of different use cases, but one use cases for that. And generally, that's over a lot of documents. And so that starts to rack up a bill kind of quickly. And I think extraction is also like a simpler task than like reasoning about which tools to call next in an agent. And so I think it's better suited for that.

• 32:14Speaker 01

  Yeah. On one of the heuristics, I wanted to get your thoughts on hallucination is one of the big problems there. Do you have any recommendations on how people should reduce hallucinations?

• 32:25Speaker 02

  To reduce hallucinations, we did a webinar on like evaluating RAG this past week. And I think there's this great project called RAGAS that evaluates four different things across two different spectrums. So the two different spectrums are like, is the retrieval part right? Or is the generation or sorry, like, is it messing up in retrieval or is it messing up in generation? And so I think to fix hallucination, it probably depends on where it's messing up. If it's messing up in generation, then you're getting the right information, but it's still hallucinating. Or you're getting like partially right information and hallucinating some bits. A lot of that's prompt engineering. And so that's what we're in. So we'd recommend kind of like focusing on the prompt engineering part. And then if you're getting it wrong in the, if you're just not retrieving the right stuff, then there's a lot of different things that you can probably do. Or you should look at on the retrieval bit.

• 33:10Speaker 02

  And honestly, that's where it starts to become a bit like application specific as well. Maybe there's some temporal stuff going on. Maybe you're not parsing things correctly.

• 33:18Speaker 01

  Yeah, okay. Well, some people did that webinar for sure to for more detail. And then finally, I wanted to get one more estimate. You're at the heart of all these production LM use cases. We think the market share is OpenAI versus Anthropic versus, I don't know, Palm versus local models.

• 33:34Speaker 02

  I would say OpenAI like 90 plus.

• 33:37Speaker 01

  Really?

• 33:37Speaker 02

  I think so. I mean, if you're going by like number of people using them, definitely over like 90% I'd say. But then you maybe have like some like really high. The thing is, there's probably some like really high workloads that are using like local models because they have teams that have been around for a while and can have their own local models.

• 33:56Speaker 01

  Which you're going to see through Langsmith.

• 33:57Speaker 02

  Yes, which we're going to see through Langsmith. If we're looking at like the Langsmith data, it's still, we haven't broken it down by that, but OpenAI is still very far.

• 34:05Speaker 01

  That's what I wonder about all these model routing situations. Everyone's kind of hoping for a multiple model future, but realistically, OpenAI is like 95% of workloads.

• 34:16Speaker 02

  Yeah, but I think like there's, yeah, I mean, there's probably a larger discussion around that. But OpenAI definitely had a huge head start, right? And that's Claude's not even publicly available yet, I don't think.

• 34:28Speaker 01

  The API?

• 34:28Speaker 02

  Yeah.

• 34:29Speaker 01

  Well, you can just basically ask any of the business reps and they'll give it to you.

• 34:32Speaker 02

  You can, but it's still, it's still a different sign up process. I think there's, I'm bullish that other ones will catch up, especially like Anthropic and Google. The local ones are really interesting. I think we're seeing a big.

• 34:45Speaker 01

  Llama too.

• 34:46Speaker 02

  Yeah, yeah.

• 34:47Speaker 01

  Yeah, we're doing the hack, the fine tuning hackathon tomorrow. Thanks for promoting that.

• 34:50Speaker 02

  No, thanks for, I'm really excited about that stuff. I mean, that's something that like we've been, you know, cause like, as I said, like the only thing we know is that the space is moving so fast and changing so rapidly. And like local models are, have always been one of those things that people have been bullish on. And it seems like it's getting closer and closer to kind of like being viable. So I'm excited to see what we can do with some fine tuning.

• 35:10Speaker 01

  Yeah, I have to confess, I did not know that you cared. It's not like a judgment on, on Langchain. I was just like, you know, you, you write an adapter for it and you're done, right? Like, how much further does it go for Langchain? In terms of like, for you, it's, it's one of the, you know, the model I O modules and that's it. But like you, you seem very personally very passionate about it, but I don't know what the Langchain specific angle for this is for fine tuning local models, basically.

• 35:36Speaker 01

  Like you're just passionate about local models and privacy and all that, right? And open source.

• 35:41Speaker 02

  Well, I think there's a few different things, like one, like, you know, if we think about what it takes to build a really reliable, like context aware reasoning application. There's probably a bunch of different nodes that are doing a bunch of different things. And I think it is like a really complex system. And so if you're relying on OpenAI for every part of that, like I think that starts to get really expensive. Also, like, probably just like not good to have that much liability on any one thing. And so I do think that like, I'm hoping that for like, you know, specific parts at the end, you can like fine tune a model and kind of have a more specific thing for a specific task. Also, to be clear, like, I think like, I also, at the same time, I think OpenAI is by far the easiest way to get started. And if I was building anything, I would absolutely start with OpenAI.

• 36:26Speaker 01

  Yeah, it's something I think a lot of people are wrestling with. But like, as a person building apps, why take five vendors when I can take one vendor? Right? As long as I trust Azure, I'm just entrusting all my data to Azure and that's it. So I'm still trying to figure out the real case for local models in production. And I don't know, but fine tuning, I think is a good one. That's why I guess OpenAI worked on fine tuning.

• 36:49Speaker 02

  I think there's also like, you know, like if there is, if there's more options available, like prices are going to go down, so I'm happy about that. So like, very selfishly, there's that aspect as well.

• 37:01Alessio Fanelli?

  And in the Langsmith announcement, as in the product screenshot, you have like chain tool and LLM as like the three core atoms. Is that how people should think about observability in this space? Like first you go to the chain and then you start digging down between like the model itself and like the tool it's using.

• 37:19Speaker 02

  We've added more, we've added like a retriever logging so that you can see like what queries going in and what are the documents you're getting out. Those are like the three that we started with. I definitely think probably the main ones, like the basically the LLM. So the reason I think the debugging and Langsmith and debugging in general is so needed for these LLM apps is that if you're building, like, again, let's think about like what we want people to be building with LinkedIn, these like context aware reasoning applications. Context aware, there's a lot of stuff in the prompt. There's like the instructions, there's any previous messages, there's any input this time, there's any documents you retrieve. And so there's a lot of like data engineering that goes into like putting it into that prompt. This sounds silly, but just like making sure the data shows up in the right format is like really important.

• 38:02Speaker 02

  And then for the reasoning part of it, like, that's obviously also all in the prompt. And so being able to like, and there's like, you know, the state of the world right now, like if you have the instructions at the beginning or at the end can actually. Actually make like a big difference in terms of whether it forgets it or not. And so being able to kind of like.

• 38:17Speaker 01

  Yeah. Any takes on that one, by the way? This is the U curve in context, right?

• 38:21Speaker 02

  Yeah, I think it's real. Basically, I've found long context windows really good for when I want to extract like a single piece of information about something, basically. But if I want to do reasoning over perhaps multiple pieces of information that are somewhere in like the retrieved documents, I've found it not to be that great.

• 38:36Speaker 01

  Yeah, I have said that that piece of research is the best bull case for Langchain and all the vector companies. Because it means you should do chains. It means you should do retrieval instead of long context, right? People are trying to extend long context to like 100K, 1 million tokens, 5 million tokens, doesn't matter. You're gonna forget, you can't trust it.

• 38:54Speaker 02

  I expect that it will probably get better over time as everything in this field. But I do also think there'll always be a need for kind of like vector stores and retrieval in some fashions.

• 39:03Speaker 01

  How should people get started with Langsmith cookbooks? Want to talk maybe a bit about that?

• 39:08Speaker 02

  Yeah, again, like I think the main thing that even I find valuable about Langsmith is just like the debugging aspect of it. And so for that, it's very simple. You can kind of like turn on three environment variables and it just logs everything and you don't look at it 95% of the time. But that 5% you do when something goes wrong, it's quite handy to have there. And so that's probably the easiest way to get started. And we're still in a closed beta, but we're letting people off the waitlist every day. And if you really need access, just DM me and we're happy to give you access there. And then, yeah, there's a lot that you can do with Langsmith that we've been talking about. And so Will on our team has been leading the charge on a really great like Langsmith cookbooks repo that covers everything from collecting feedback, whether it's thumbs up, thumbs down, or like multi-scale or comments as well, to doing evaluation, doing testing. You can also use Langsmith without Langchain.

• 39:53Speaker 02

  And so we've got some notebooks on that in there. But we have Python and JavaScript SDKs that aren't dependent on Langchain in any way. And so you can use those and then we'll also be publishing a notebook on how to do that just with the REST APIs themselves. So yeah, definitely check out that repo. That's a great resource that Will's put together.

• 40:10Speaker 01

  Yeah, awesome. So we'll zoom out a little bit from Langsmith and talk about Langchain the company. You're also a first time founder.

• 40:15Speaker 02

  Yes.

• 40:15Speaker 01

  And you've just hired your 10th employee, Julia, who I know from my data engineering days. You mentioned Will, you know, I think from who maintains Langchain JS. I'm very interested in like your multi-language strategy, by the way. Ankush, your co-founder, Lance, who did Auto Eval. What are you staffing up for and maybe who are you hiring?

• 40:34Speaker 02

  Yeah, so 10 employees, 12 total. We've got three more joining over the next like three weeks. We've got Julia, who's awesome, leading a lot of the product, go-to-market, customer success stuff. And then we've got Bree, who's also awesome, leading a lot of the marketing and ops aspects. And then other than that, all engineers. We've staffed up a lot on kind of like full stack infra, DevOps, kind of like as we've started going into the hosted platform. So internally, we're split about 50/50 between the open source and then the platform stuff. And yeah, we're looking to hire particularly on kind of like the things. We're actually looking to hire across most fronts, to be honest. But in particular, we probably need one or two more people on like open source, both Python and

• 41:15Speaker 02

  JavaScript, and happy to dive into the multi-language kind of like strategy there. But again, like strong focus there on engineering, actually, as opposed to maybe like, we're not a research lab, we're not a research shop. And then on the platform side, like, we definitely need some more people on the infra and DevOps side. So I'm using this as an opportunity to tell people that we're hiring and that you should reach out if that sounds like you.

• 41:35Speaker 01

  Is it Langchain.com careers? Something like that. Jobs, whatever.

• 41:39Speaker 02

  I don't actually have enough information to talk about.

• 41:41Speaker 01

  R I P. What happened to your landing page? It used to be so based.

• 41:45Speaker 02

  The Berkshire Hathaway one?

• 41:46Speaker 01

  Yeah. So what's the story, the quick story behind that?

• 41:48Speaker 02

  Yeah, the quick story behind that is we needed a website and I'm terrible at design. And I knew that we couldn't do a good job. So if you can't do a good job, might as well do the worst job possible.

• 41:58Speaker 01

  And like lean into it.

• 41:59Speaker 02

  And have some fun with it. Yeah.

• 42:00Speaker 01

  Do you admire Warren Buffett?

• 42:02Speaker 02

  Yeah, admire Warren Buffett and admire his website. And actually, you can still find a link to it from our current website if you look hard enough. So there's a little Easter egg.

• 42:10Alessio Fanelli?

  Before we dive into more of the open source community things, let's dive into the language thing. How do you think about parity between the Python and JavaScript? Obviously, they're very different ecosystems. So when you're working on LangChain, is it we need to have the same abstraction in both language? Or are you to the needs?

• 42:28Speaker 02

  The core stuff, we want to have the same abstractions because we basically want to be able to like serialize prompts, chains, agents, all the core stuff as tightly as possible and then use that between languages. Like even, yeah, like even now, when we log things to LangChain, we have a playground experience where you can run things. That runs in JavaScript because it's kind of like in the browser. But a lot of what's logged is like Python. And so we need that core equivalence for a lot of the core things. Then there's like the incredibly long tail of like integrations, more researchy things. Those are pretty language specific. Python's probably had a head on a lot of like the integrations front. There's more researchy things that we're able to include quickly because a lot of people release some of their code in Python and stuff like that.

• 43:13Speaker 02

  And so we can use that. And there's just more of an ecosystem around the Python project. But the core stuff will have kind of like the same abstractions and be translatable.

• 43:21Alessio Fanelli?

  That didn't go exactly where I was thinking. So like the LangChain of Ruby, the LangChain of C#. You know, there's demand for that. I'm sure you're bombarded with those. I'm sure you're not doing that because of focus.

• 43:31Speaker 02

  Yes.

• 43:31Alessio Fanelli?

  But you are giving up some real estate by not doing it.

• 43:35Speaker 02

  Yeah, it comes down to kind of like, you know, ROI and focus. And I think like, we do think there's a strong JavaScript community and we wanted to lean into that. And I think a lot of the people that we brought on early, like Nuno and Jacob have a lot of experience building JavaScript tooling in that community. Will we do another language? Never say never, but like.

• 43:58Alessio Fanelli?

  I thought G is for now.

• 43:59Speaker 02

  Yeah.

• 44:00Alessio Fanelli?

  Awesome. You got 83 articles, which I think might be a record for such a young company. What are like the hottest hits, the most popular ones?

• 44:09Speaker 02

  I think the most popular ones are generally the ones where we do a deep dive on, a deep dive on something. So we did something a few weeks ago around evaluating CSV question answering applications, which I think is a really interesting one because most question answering, like everyone does question answering, but it's generally over on structured data, over your documents, and you do the whole rag thing. And that doesn't work amazing for structured data. And so. This was something that we heard, the origin of this was basically we heard from the community, you guys should improve this. And so we're like, okay, let's improve it. And then we're like, okay, well, if in order to see if we improve it, we need to like evaluate it and see how we're doing. And so we kind of like wrote up a lot of our thought process there. And I think, and a lot of people like reached out about that and thought that was interesting. And, and we're going through similar challenges and had, had, we posted another one a few days after that someone wrote basically as a response, which was

• 44:53Speaker 02

  awesome because it had a completely different strategy. And it was a really, it was a really, that was a really good piece as well. So that was like a deep dive on something like evaluation bit. I think like we did one on retrieval a while back, which was basically like, hey, we, and this was around when we changed our abstractions, like, hey, we changed our abstractions to this. This is why we did it. This is what we see coming down the pipeline. These are like the different types of retrieval that we see. I think a lot of people read and liked that one. A lot of the blogs that we do are also highlighting cool partnerships or cool applications. But in terms of if you go by like number of views, I think the ones that get the most views are the more deep dive ones.

• 45:32Speaker 01

  Yeah. And I also noticed that you do guest posts as well.

• 45:35Speaker 02

  Actually, you know which one, and this is a guest post that got a lot of views, the multi on one, the multi on agent one, when we did, we did, we did a blog where we integrated with them and that got a ton of views.

• 45:43Speaker 01

  What do you think that is?

• 45:45Speaker 02

  I think it's, I mean, it's one of like the few agents that's actually available and like out in the world.

• 45:52Speaker 01

  They're still behind a waitlist.

• 45:53Speaker 02

  Still behind a waitlist, but they're very active on social media. I don't know if I'm off the waitlist.

• 46:00Speaker 01

  I mean, you're, you're on their blog. So they're on your blog. So I hope they gave you access at some point. But that's interesting. A lot of, lot of interest in agents. And I think they just opened up an API as well.

• 46:09Speaker 02

  Yeah, exactly. That was the, that was the blog that we did. I was, yeah, I was a bit surprised to see that as well. But I think there's generally a lot of interest in agents. And it's also really hard to get them to work. And I think multi on is one of the first that has that. And

• 46:22Speaker 01

  Yeah. So my angle to this is a lot of people want to work with you.

• 46:25Speaker 02

  Yes.

• 46:26Speaker 01

  You are bombarded. And I'm sure your email is just unmanageable. How should people be good partners with you? Like I work at a company and I'm like, hey, I'd love to do something on the LangChain blog or integrate to LangChain. I know Harrison's a busy guy. Like, what do I do?

• 46:40Speaker 02

  I like the stuff that I guess my attention honestly is like the in-depth, really thought-out stuff. Obviously, I love this stuff. Like, this stuff is awesome. And there's so many different, there's so much to do as well. And like, the biggest thing that we have trouble with internally is like figuring out what to do.

• 46:54Speaker 01

  What's noise and what's signal?

• 46:56Speaker 02

  Not even that, but just like what to focus on. Like, there's so many different directions we could do. And we want to go in like so many, because there's so many interestingness, but we can't do. So if anyone kind of like takes the time to like go deep in a particular area, I love talking to them. And I love reading what they write. And I love sharing what they write on the blog. Like, that to me is awesome. So I think like-

• 47:14Speaker 01

  Do good stuff. Be so good they can't ignore you.

• 47:17Speaker 02

  It sounds basic, right? So that's why I didn't want to say it.

• 47:19Speaker 01

  No, it's fine.

• 47:19Speaker 02

  But I think like these deep dot, yeah, there's just so much to do. And these, don't do shallow stuff, I guess would be.

• 47:25Speaker 01

  I think that's a good call out. People need reminding.

• 47:27Alessio Fanelli?

  What about the other side of open source? So on Acker News, there were a couple blog posts recently, like the problem with LangChain and LangChain is pointless. All these different things. So the TLDR of some of them were, you know, the LangChain API is like kind of verbose and complicated versus like sometimes I can just do this in like 10 lines of code. How do you balance that in terms of allowing for the complex use cases versus, you know, making maybe the ergonomics like simpler, but then trading that off later?

• 47:58Speaker 02

  There's a lot to balance and there's a lot to do. And I think like posts like that are very valuable to hear basically what people are saying. And like, you know, we have a lot of open issues. So it's not like these things hadn't been said before. But I think like that was a good, that was a good emphasis on what people are saying. And I think there was a lot of, there was a lot of things in there. You know, I think part of it's kind of like around. And we took all of it very seriously. And yeah, I think there's a lot to dive into there. There's like the documentation piece. And so I think we did a revamp of the documentation to address that. There's also like a comment in this, I think this is around, I think the top comment on the LangChain is pointless one was, was like basically like orchestration is like 5% of the work. And then like the other 95% is like prompt engineering and like data and-

• 48:41Speaker 02

  And those are the hard bits. I think maybe orchestration is a little bit more than 5%, but I like agree that those are like really big pain points that get exacerbated when you have these complex chains and agents where you can't really see what's going on inside of them. And I think that's partially why we built Langsmith to help out with exactly that. We also needed to do better things like make the prompts more visible and make allow for more customizability around that. And so we've tried to add some stuff there. In terms of balancing, there's also Langchain is pointless. I don't need a wrapper. I can just call the underlying API. I think if all you're trying to do is call the underlying API, then like, yeah, that's going to be the cleanest and simplest thing to do. And we try to get as close to that experience as possible, but we're not optimizing for calling the API.

• 49:26Speaker 02

  We're optimizing for helping people build context aware reasoning applications as easily as possible. And so there's some level of abstractions that you need to add in order to assist in that. And yeah, that's definitely a balance that's tricky to strike, but I think there's also some aspect of it. Like, I do think one of the big benefits that Langchain provides is a standard interface for language models so that you can switch between them. And this kind of gets into like an ORM debate, like are ORMs generally kind of like useful or not? And so I think in this case, they are. I think there's probably a larger kind of like philosophical kind of like question about that, that people have strong opinions on. So.

• 50:04Speaker 01

  Yeah, just the prompts don't transfer like you also mentioned.

• 50:07Speaker 02

  Yeah, yeah, there's that. Yeah. And then between kind of like allowing for, I think one helpful thing that we did in terms of like distinguishing between basically the base interfaces and then more complex stuff is part of the separation around the docs is there's like the components piece, which has the model I O, the retrieval, the agents, the callbacks, things like that. And then there's all the use cases. And so I think like the use cases, because they are like these assembly of all these things in a particular order, they start to get more complex. And it's, you know, we try our best to kind of like make clear how you can configure things. But yeah, there's a lot of different options that you might want to configure. And so I think that split has kind of helped us internally at least, and I think externally as well, because we've heard good comments about the improved documentation. I think that's made it a little bit more clear.

• 50:51Speaker 02

  And then another thing, one of the things that we also released soon after, and we'd been thinking about a little bit is basically like a Langchain expression language, which allows for actual composability of pieces. So Langchain, I think, has always been very good about interchangeable ability. Let's ignore the prompting issues, but like you could always plug in like in one LLM for another one, you could, you could swap in one vector for another one. But the chains themselves haven't actually been super actually composable. Like we had the sequential chain, but that was a bit like clunky to use. And that was, and then we had a router chain, but that was a bit, you know, that was also a bit clunky to use. And so one of the things, and so there's, you know, there's a million different things to do, and we didn't prioritize that. I think after this, it definitely bumped it up and prioritize in priority. And luckily, Nuno had been doing a lot of awesome work on it already.

• 51:36Speaker 02

  So it wasn't too much of a lift. But yeah, now there's this way where a lot of the chains that we've been releasing are written in this Langchain expression language where they're actually truly composable. And you can see what's going on under the hood. And it's basically, it uses kind of like the pipe, kind of like terminology to coordinate things and move things around. So yeah, I mean, I think there were a lot of good points in those Hacker News things. And, you know, we can't respond to everything, but we tried to like, look at everything and take everything seriously.

• 52:02Speaker 01

  You're being very diplomatic, but so first of all, I like the expression language. I think that that is the path towards sort of language agnostic Langchain.

• 52:08Speaker 02

  Yeah.

• 52:08Speaker 01

  Kind of whatever DSL. But also like, what was just kind of plain wrong or plain offensive or like, I don't know, people can get very vitriolic sometimes on Hacker News.

• 52:18Speaker 02

  Yeah, I mean, I think the comments that I appreciated were the ones where they gave specific things. And I think the ones where they said, you know, Langchain sucks.

• 52:26Speaker 01

  Yeah.

• 52:27Speaker 02

  Okay.

• 52:28Speaker 01

  Can't do much with that.

• 52:28Speaker 02

  Yeah, exactly.

• 52:29Speaker 01

  Better phrasing of my question would be like, you're not the first and you won't be the last to have that kind of very intense scrutiny. Will be advised to other people, other maintainers of projects for going through something like this.

• 52:40Speaker 02

  I would probably say, try to drill into like, what is actually underlying things as much as possible. And if there is actual substance that's being delivered, whether you agree with it or not, like, I think that's valuable to know. And then for the other stuff, like, try to maybe follow up, but maybe try not to let it get under your skin too much.

• 52:59Speaker 01

  Yeah, thanks for tackling that.

• 53:01Alessio Fanelli?

  And I know we're getting to time and we'll wrap up soon, but since you're gonna speak at the AI Engineers Conference, what's your advice to AI engineers, especially when to start with LangChain and when they're just experimenting with a model? When are they, as you mentioned, if you just wanna do an API call, don't use LangChain?

• 53:19Speaker 02

  Yeah, I mean, my advice would just like build as many things as possible. Like, I think it's, I think it's still really early in the space. No one really knows what they're doing to some extent. Like, it's a bit weird to say, but there's so many things to like discover. So I would just say like, build as many things as possible. Cause I think like, the best thing is you stumble upon a really good idea and you build something really awesome. And the worst thing that happens is you just learn a lot about a field and a technology that's gonna be incredibly important and rapidly kind of like changing.

• 53:48Alessio Fanelli?

  What would you build if you weren't doing LangChain?

• 53:51Speaker 02

  I mean, the things that are most interesting to me are kind of like things around like long-term memory and like longer-running agents. So I'd probably build, and these are things that we've been wanting to build internally as well. But like, like I think a chatbot that like actually remembers things about you as like silly as that sounds. Like, people like chatbots a lot and they have their, they're limited by their context window. And so I think really diving into like a specific application of memory there. I mean, trying to build a chatbot that remembers things about you, that would be one. And then like, I know a lot of people are doing this, but like a personal assistant for like managing like email, calendar, basic stuff, which I think is, I think that's like a fantastic application for these like agent-like things.

• 54:31Speaker 02

  Because if you think about personal assistants today, you usually interact, I don't have one, but I'm told you interact with them over email. And the nice thing about that, as opposed to like chat, there's not as stringent an expectation on latency as there is on chat. And so you can do a lot of things like reflection and kind of like making sure that you're on the right track and really put more safeguards and thinking about these agents as opposed to relying on like chats and interface. Like the bot we have that's on GitHub answering questions on the issues, I think probably gives better answers than the bots that we have that are on chat on the website. And I think that's not because, there's just different constraints that you have in different types of problems. And I think I would be like, I think the personal assistant one's really interesting

• 55:14Speaker 02

  because you remove the constraint of chat, which I think at this point in time is probably pretty limiting in terms of functionality.

• 55:21Speaker 01

  Yeah, I think Colin is sort of a long inference.

• 55:24Speaker 02

  Yeah.

• 55:24Speaker 01

  If you didn't have to care about the NC and you could take like a day, a month, a year to to work on something, what could you do? And yeah, that's super interesting.

• 55:33Speaker 02

  I think that's a really promising place to explore.

• 55:35Speaker 01

  Yeah. Have you looked for regarding the long, long conversation thing? You and I have tried it about this many times. Have you looked into what character and inflection are doing? Cause they're probably working on it.

• 55:45Speaker 02

  I've thought about memory a bunch. Like, I think it comes down to like, it comes down to like state, like what's the state you're tracking? And like, what's the data structure for that? And I think it's that could also maybe be a bit like application specific. But if we're talking about a generic chatbot, that's kind of generic. I don't know. Yeah, I don't know how they're thinking about that. My sense is that inflection like thinks about that a bit more than character. Like, I think inception, sorry, inflection's whole thing is they like, the bot knows you.

• 56:10Speaker 01

  It's one chat. There's no history. You just, you just talked to it.

• 56:13Speaker 02

  Yeah. So they've definitely got some state that they're tracking. I'd be really curious to know what that is. Character, I don't think has lent into it too much. Like, I think they let you use some stuff in terms of like uploading background. And I'm not entirely sure how they use that, whether they just like put that in the prompt or do some retrieval over that. But I think they're definitely, they haven't lent into it as much as inflection, I would say.

• 56:34Speaker 01

  So given like you're one of the most interested people in this space, would this be like a second product for you if you ever want to explore that? Or do you want to just partner with people and you're putting out the call for people to come, come to you if they have solutions for that?

• 56:47Speaker 02

  If I wasn't working on, on Langchain, I would be building an application company for sure. First of all, like, I don't think, like, I think like, there's, which I know is very hypocritical to say.

• 56:57Speaker 01

  Like you're Mr. You know, DevTools and Infrad.

• 56:59Speaker 02

  I know, but I think there's. Yeah, I don't know. If you're building an application company that's working on something related to long-term memory or long-term agents, I would love to chat and just geek out all about a lot of this.

• 57:09Speaker 01

  I'll show you small talk at some point.

• 57:10Speaker 02

  Yes.

• 57:12Speaker 01

  Cool. Awesome.

• 57:14Alessio Fanelli?

  Yeah, let's do a lightning round.

• 57:16Speaker 01

  So the first one is on acceleration. What has happened in AI that you thought would take much longer than it actually ended up taking?

• 57:22Speaker 02

  The function calling ability from OpenAI, like tool usage, yeah. They did that really fast, I thought.

• 57:27Speaker 01

  Yeah, but it's just a question of fine tuning, no?

• 57:29Speaker 02

  Yeah.

• 57:30Speaker 01

  It's not even like reliable.

• 57:31Speaker 02

  It's not terrible.

• 57:33Speaker 01

  Yeah.

• 57:33Speaker 02

  They're a pretty big organization that's serving a lot of traffic and like this was a, yeah, it's like, it is like just fine tuning. But I think like you still have to like collect that data set and fine tune it and evaluate it and then release it at scale and, and figure out the right API.

• 57:47Speaker 01

  No shade on OpenAI, like they're, they're moving everyone's bar as to how quickly like a 400 person organization can go. Do you think it eliminates like approaches like JSON former and all the other approaches that people like guide guardrails, you know, previous guests? It eliminates your output validation thing.

• 58:03Speaker 02

  Yeah, I think JSON former and stuff like that are still really interesting for like local models for sure. And there's 90% of people use OpenAI or something and like my made up numbers.

• 58:14Speaker 01

  No, it's, it's probably real.

• 58:15Speaker 02

  And the best way to get structured output is by using the function calling ability. So yeah, absolutely.

• 58:23Alessio Fanelli?

  What do you think is the most interesting unsolved question in AI?

• 58:27Speaker 02

  I'm really interested like how multimodal is going to work, like just what that looks like.

• 58:32Speaker 01

  Have you had a look at the GPT four vision?

• 58:35Speaker 02

  No, not really. Yeah, not beyond what they.

• 58:38Speaker 01

  They're doing private betas right now. So yeah, very excited.

• 58:41Speaker 02

  I'm excited about that as well. Yeah. I mean, I think that's, you know, you talk about like, again, this whole space is just changing so fast. But you talk about something that could like really change how, cause like, you know, a lot of language change is kind of like a data orchestration tool in some sense. And so if you add a whole new type of data in there.

• 58:57Speaker 01

  So maybe we do this thought exercise, right? Tomorrow, OpenAI releases the GPT four vision API. What does Langchain do?

• 59:03Speaker 02

  Immediately we add support for it in like the wrapper. So however you interact, like honestly, this is another like fun thing. Everyone's API now looks like OpenAI's. Which you have to do, yeah. So like our wrapper looks similar to OpenAI. So I don't, I don't think it will be that difficult to include support for it at the basic, at the basic model level. And so we do that. And now that we've released the expression language bit, like a lot of the core chains, we have examples of rewriting them just in this expression language. So like for retrieval, if we're now talking about like, okay, you can do like retrieval question answering over multimodal things, we'd probably have to figure out how those are getting stored and what's being done with them. But then from there, that should be, yeah, so probably look into like, yeah, how are,

• 59:47Speaker 02

  how are people kind of like storing and consuming this type of information? But then that step should be pretty easy to plug into the kind of like chain.

• 59:54Speaker 01

  Multimodal stores? Yeah, I don't know. I always wonder what that would actually look like because a lot of multimodality in LMS is really just an LLM, a text LLM calling a different model. And that's just no different than any API call, essentially unchanged.

• 1:00:09Speaker 02

  I think it's probably something that you don't know until you let like a million people play around with it.

• 1:00:14Speaker 01

  Then there'll be new lang chain for multimodal. What's one message you want everyone to remember today?

• 1:00:20Speaker 02

  I would probably say just like build, like I think it's a fantastic time to be building.

• 1:00:25Speaker 01

  All right, just build.

• 1:00:26Alessio Fanelli?

  Yeah. Thank you Harrison for coming on.

• 1:00:28Speaker 02

  Thank you guys for having me. It's a lot of fun.