Ep 22: Notion AI Engineer Linus Lee: Behind the Scenes of Notion AI

Linus Lee is an AI engineer at Notion, one of the earliest and most effective adopters of AI. In the episode, Linus shares how Notion developed its AI products, including Writer, Autofill, and Q&A, which just launched on Tuesday. It was fascinating to learn how Notion structures its AI team and dogfoods its development process. Linus also explores the hardest to anticipate when going to market with new AI features, and how Notion thinks of its LLM partnerships. Overall, a wide-ranging conversation about the behind-the-scenes stories of one of the most widely used AI tools today.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  This is Unsupervised Learning, Red Points AI Podcast. I'm Jacob Effron and I'm joined by my co-host Pat Chase. And today, we had an incredible guest in line, as Lee, who helps head up a lot of what Notion's doing in AI. We had a pretty wide-ranging conversation on a lot of really interesting topics, including how Notion structures their AI team, what some of the hardest things were to anticipate when they were going to market with their new AI features, a behind-the-scenes of how the Notion team went about building AI Writer and Auto Fill, and where they decided to build or let their partners open AI in Anthropic Build. It was a really fun one, particularly because we talked about Q and A, which is a new feature that Notion released this week. I think folks are really going to enjoy this.

• 0:37Jacob Effron?

  We've been looking forward to having you on, I feel like, for a long time. And to start, you know, one thing I always really enjoy about your Twitter is that I feel like it oscillates between, like, really good insights on AI and design, and then Taylor Swift.

• 0:50Speaker 01

  Yes, of course.

• 0:51Jacob Effron?

  I feel like you're a big Swiftie. I am as well.

• 0:54Speaker 01

  We were just actually on my way here, talking about it. Co-worker and I were talking about which shows we've gone to. And so it's a constant theme in my life. Once I wrote a mock paper on what I've termed the Taylor Swift scaling laws, which is the power law relationship, the log-log relationship that exists between how close you are in 3D space to the stage at a Taylor Swift show, and how far into her career the show is, and how close you are to the concert date. And the closer you are, the closer you are to the stage and the show date, and the farther you are into her career, the price exponentially increases.

• 1:29Jacob Effron?

  Ooh.

• 1:30Speaker 01

  And so somewhere on the internet, there's a PDF of some paper that's like the scaling laws, and it is all very, but that is something I've thought about.

• 1:37Jacob Effron?

  That makes a lot of sense. So rather than getting cheaper, it just keeps getting more expensive, I guess, is a...

• 1:41Speaker 01

  Of course.

• 1:42Jacob Effron?

  Alas.

• 1:42Speaker 01

  Seemingly with no limit.

• 1:43Jacob Effron?

  Yeah, a little bit different than compute, but you know.

• 1:47Speaker 01

  But also quite scarce these days.

• 1:48Jacob Effron?

  Yes, that is fair. Actually, I mean, more scarce from experience.

• 1:51Speaker 01

  Which market is more scarce, Taylor Swift tickets or H100s?

• 1:54Jacob Effron?

  Yeah, if only there was a public company we could invest in that had cornered all of the Taylor Swift events.

• 1:58Speaker 01

  Yeah, I don't know if Ticketmaster's public, but that would certainly be a good stock time.

• 2:03Jacob Effron?

  Yeah, I guess congressional hearings only on one of them.

• 2:06Speaker 02

  Right, right. Awesome. Well, yeah, we were super excited about this. I think Notion is one of the companies that has shipped AI features really, really quickly into their product and in production using LLMs and all of the latest in generative AI. Maybe just to kick it off, it'd be great to give our listeners some context on the different products that you guys have at Notion AI and kind of what is, what goes into Notion AI today from a product perspective.

• 2:32Speaker 01

  Yeah, so just for some context, I joined Notion's AI team this past January, January 2023, which is a few months after Notion as a company had started working on AI. And the story, although I wasn't there, the story goes that around, around this time last year, actually almost exactly a year ago, there was a company offsite. And instead of Ivan and Simon, our CEO and co-founder, the two co-founders, instead of going off and joining the company every single day with the rest of the company, they took a few days off on the offsite because GPT-3 had just dropped. And they wanted to play with it and see kind of how it would be incorporated. They saw that it was an interesting writing tool and that a lot of other companies were investigating writing approaches at the time.

• 3:13Speaker 01

  And so they, they wanted to see what they could do inside the product. And so they took, in a classic hackathon style, they took a few days off and just like hacked in the hotel room. And there, there's like photos in our internal Notion from this period of like Simon with like 18 bottles of water, just like in the, in the hotel room coding. And what they came out of, out of that hackathon weekend was the first version of Notion AI, which internally we started calling Notion AI Writer, because it helps you write. And so the first thing that we worked on that went to beta last November and then came out in February is AI Writer. It's AI in your documents, so you can ask it to say like, hey, summarize this page in XYZ format or pull out key ideas, action items, key topics.

• 3:55Speaker 01

  Or you could even, a lot of people like to use it for fixing spelling and grammar mistakes or fixing, improving their writing, improving their voice and style. And so this kind of tool to edit and revise writing or draft, help you get some ideas down. A lot of people use it to get like basic marketing copy down. So that was the first thing that we shipped in February. And through that process, one main theme that we picked up on is AI and humans always work together to iterate on things to get to a better outcome. And so a lot of people will use AI to either generate a draft, or, once they've written a draft, iterate on it to fix mistakes and find ways to improve it. And so one of the things we added in this kind of beta testing period was a way to conversational follow up and say, okay, you've generated something now, then make it shorter or make it more punchy or something like that.

• 4:39Speaker 01

  The next thing that we worked on was called AI Auto Fill. So we had brought AI to the document. Another common use case for Notion, especially at startups, is project management, because it's kind of an all-in-one package. If you have Notion, you can all do all these things. And so project management, one common tool people use is Notion databases, which is kind of a relational table. And so we added AI to be able to automatically fill in entire columns or entire properties, we call it, for, if you have a table of meeting notes, you can fill in what are the key topics. If you have a table of customers or user interviews, as a lot of our users do, you can use AI to pull out what are their core needs or what do they do at their company or something like that from just looking into the transcript. And so Auto Fill came out in May, along with a lot of the rest of Notion's project management suite.

• 5:22Speaker 01

  And then the most recent one, which I'm really excited to talk about more today, is one that, by the time this comes out, we'll have launched, which is Notion Q and A, which is, instead of just helping you write and deal with information at a page level, Notion AI now understands everything inside your workspace and can help you answer questions that-

• 5:39Speaker 02

  Ah, super cool.

• 5:40Speaker 01

  That are in your workspaces or in your pages or even maybe span multiple pages.

• 5:45Speaker 02

  Is the use case, is that kind of like replacing search? Or then you can ask questions of your whole team's knowledge base?

• 5:51Speaker 01

  Yeah, so the way we arrived at this is actually, there's a couple of different paths that crossed. So the first one was, obviously, we always like to start with what are people using Notion for and what are the kind of core problems that they're facing with the product today? And one common thing that kept coming up as we were trying to figure out what AI would be useful for is, as organizations get bigger and as people's Notion get more complex, Notion workspaces get more complex. One key problem they run into is sometimes it's just hard to keep everything organized. It requires a lot of vigilance and a lot of people have trouble finding information. And exactly where they put it is that this is doubly the case if you work with lots of other people. They might have written something down, but you don't even know that they wrote something down. And so solving this kind of information finding problem is really key.

• 6:31Speaker 01

  And along as this was happening, there was sort of also a trend that was happening in the broader AI community, which is retrieval augmented generation, where you can ground machine, ground AI and other sort of factual information that you might have had that isn't in training data set. And so it seemed like the confluence of those two things was a good place to to build.

• 6:50Jacob Effron?

  I'm curious on like the on the notion AI team, like, you know, I feel like there's the classic product management way of obviously you guys know your customers super well. Like, what are their core problems? Let's go build things around that. And then I imagine there's also just, wow, a new, you know, RAG is really popular now, or there's all these new capabilities. Let's go like hack and see what's possible with some of the stuff like.

• 7:07Speaker 01

  Yeah.

• 7:07Jacob Effron?

  How do you kind of balance your your efforts around that on like the product development side? And how much of it is like more hacky, let's just see what's possible versus like, you know, the classic kind of PM style of here's the user problems we're trying to solve.

• 7:19Speaker 01

  Yeah, that's a that's a really, that's a question that we've, I think, also internally iterated a lot on. We definitely didn't have the answers yet, but the way that we've worked so far is kind of a like a expand out and then contract, expand, contract kind of cycle. And so we go through iterations of sometimes we have a few hunches of what problems are important and very broad problem statements. Like, it seems like it's important to help people keep information organized, or it seems like it's important to help people find information. And then sort of motivated by that, we have, we'll have a couple of ideas and different technologies or techniques like retrieval augmented generation that seems promising for that. So a couple of us will go and prototype something really quickly. And then, um, once we have a few explorations done of a few different approaches that that could help solve that problem.

• 8:03Speaker 01

  Usually it'll be, especially since Notion runs on Notion, we dog food things really heavily. And so by the time we have a few of these things internally and as prototype format, it'll be fairly evident which. Which approaches are more promising, which approaches are less promising? And so we'll pick a few of those, sort of recalibrate ourselves and say, okay, here's the prototype that we have, here are the problems that we want to solve with a better understanding of the problem, just talking to users and thinking about things, and a better understanding of the strength of a particular approach, we'll have a hundred and okay, this is the the direction that we want to actually build and ship. And so, once we have that direction, we'll we'll iterate towards something that we feel confident, both in the UX and in the like model output quality, what we feel is ready.

• 8:39Speaker 02

  One thing I'm super curious about is just like the resources that go into to shipping one of these products, like from either engineers or design or like, like, how do how much of an investment, I guess, has it been for you guys to ship? Maybe it's Writer, one of these new ones, kind of from the ideation phase to when it's in the hands of users?

• 9:04Speaker 01

  Resources? Can you say more about that?

• 9:07Speaker 02

  Yeah, like, I think there's a bunch of startups that come, you know, and they're like, oh, we want to add, you know, we're a SaaS app X, and we want to ship an LLM-based feature into our product because we think it will add a lot of value to our customers. How should we think about staffing that project, right? Is that like one engineer for, you know, two engineers for a month and a designer as well to figure out the front end, you know? So I'd just be curious if there are any learnings there from how you guys have have staffed the projects?

• 9:37Speaker 01

  Yeah, I think so. I think this actually kind of couples well with the your previous question around the phases. And so, in the exploratory phase, I think the the main thing that's really and really important is just how quickly can you iterate and how quickly are you kind of forced to iterate on the output? So, one of the things that I think in the early phases of the new Q and A feature that we launched, one of the things that worked really well for us is we had a really, really annoying prototype where like anytime anyone asked a question in the company, it would we would try to run this Q and A pipeline and try to answer the question. It was very annoying, to the point where like someone, some people like come into our the AI team's Slack chat and be like, can you please like turn this stuff and thing off because it's so annoying? But that

• 10:20Speaker 02

  That was like the motivation to make it better. It's like everyone's

• 10:22Speaker 01

  Right, so yeah. So there were two, two things that were great about that. From like where we ended up perspective, one is because it was so in your face it. It forced us to like iterate on the thing to make the quality of the outputs really, really good. And we were just so face to face with it every day that like every, if there was any improvement or regression, we would notice it. We would have to fix it because people were bumping with these answers. And then the other thing that worked well for us is because it was being used. It was being used in the context of what the work people are already doing. It was really easy to tell whether the thing was useful or not because if it was useful, people would be using it. People were responding, giving feedback. And if it's not useful, then you know, you don't really see any usage.

• 11:05Speaker 01

  And so it helped both validate and then add pressure to truly improve at least the model quality side of it. So I think in that phase, iterating quickly. I think you really need a type of person who is like enthusiastic about working with the technology. Kind of has maybe one year on the ground about what's happening, what kind of techniques people are trying because it's still such a new space. And then is able to sort of operate with a vision of this is, this is the problem that we have solved in this kind of way. And if there are some technical problems or there's some like operational problems that come along the way, we're going to try to like break through them to actually. Build a prototype that works and like solves the problem and does the job. Once you have a more defined kind of picture of the solution, I think it much more resembles a normal product building process.

• 11:46Speaker 01

  Where you have, you know, user research and designers and you kind of work together in a, in a way that most other traditional SaaS features is able to ship something.

• 11:53Speaker 02

  Got it. So in the, in the ideating process, is that literally just one person like you or? And then kind of taking that front end, back end, like full stack end to end or?

• 12:04Speaker 01

  It's a few of us. We have about a dozen or so people on the Notion AI team. It's pretty evenly split between folks that spend more of their time working on like data, model, what we call quality, which is like. How, how correct are the model outputs, how coherent are they and so on? And then the other half works on what you call the more of the product concerns, like interface, integration into Notion.

• 12:25Speaker 02

  And that's the 12, is that all edge? And then do you also have design resources? Cause I feel like there's a big front end component to a lot of like the workflows, especially at Notion, like you guys have been super thoughtful about like. Suggesting or guiding users towards, you know, what might be useful prompts.

• 12:41Speaker 01

  Yeah, we have a couple of designers we work with also, and so frequently we'll sort of. Especially in the ideation phase, we'll check in together often to figure out, motivated by what we've been able to accomplish, what we've been able to get the model to output. What are some ways that we can present that in a nice way? And then every couple of weeks there's new iterations.

• 13:02Jacob Effron?

  Is it all end to end with like that particular team? Or at some point you transition to a product team that's kind of overseeing that part of the product traditionally?

• 13:10Speaker 01

  This is a question we feel like we haven't fully answered yet. The state of things currently for us is we have a core AI team. And there are, right now, most of the AI features that we've launched, there is a specific AI surface. So if you're using our AI writer product, you hit a hot key or hit space and there's this bar that comes up. And that's like one space. With Q and A, there's a totally other separate chat box that you talk to. And so it makes sense that both backend components and the UI components our team owns. There are other features like autofill is actually much more integrated with our databases. And so currently our databases team owns and works on that feature. There's still a pretty tight collaboration. So the folks that work on the autofill feature, I still work with really closely.

• 13:53Speaker 01

  And we try to figure out sometimes, especially if there are hard AI problems, we try to collaborate and figure out what the solution space there looks like. I think there's a few different ways this could go, right? Like one is like a kind of a hub team and then sort of liaisons and other teams. And it's more of like a partnership thing. And then you could also imagine there being sort of AI engineers on every single team, which is a little bit further from the situation we have today, but it's certainly possible. I think it's a function of both kind of how we want to organize the AI, like foundational AI technologies that Notion is working on. And I think also what shaped the ending Notion AI product as a whole ends up taking.

• 14:34Jacob Effron?

  Yeah, it sounds like it varies on a feature by feature basis to some extent.

• 14:37Speaker 01

  Yeah, and we're also still trying to figure out what the foundational pieces of Notion AI are. And so like my thoughts on retrieval in particular actually I think could change over time. Where in the beginning, I thought it was sort of a purpose built thing just for like answering questions types of things. But now I think retrieval sort of augments every capability in some way. And so it's a foundational thing that if you get it right, just like you can use a normal language model in a bunch of different ways, including some features that don't even. Output natural language directed to the user. If you have a model that's really good at retrieving and answering questions or accomplishing actions based on some retrieved result, then I think that also can be used in places that aren't necessarily about asking questions, but maybe about like finding related information or surfacing recent changes in a document or anything of that.

• 15:20Speaker 02

  Right, so it always can be better at providing more context, which is probably helpful for.

• 15:24Speaker 01

  So in that way, it's like a foundational thing. And even if we had like a Q and A specific team of some sort, it may still make sense for the foundation team to like work on a work on the like retrieval component. So I think we're still gonna in the process of figuring it out.

• 15:37Speaker 02

  It's super cool. And I think a bunch of companies are kind of working through that now. And I think a lot have landed on a similar structure where they have kind of the AI SWAT team together almost. And then, but I think over time, once it becomes a skill that's kind of in every engineer's toolkit, then maybe they'll expand out from that to have like these AI engineers more embedded. Or maybe every engineer is an AI engineer.

• 15:59Speaker 01

  Maybe, I mean, I think I'm less familiar with like the like DevOps boom. And I don't know if it's busted, but the DevOps boom. I think there are also, I think there was this dialogue that went on of like, oh, every engineer should be like doing ops. And it's like, true to some extent, but you still want an infrastructure team that manages and builds tools and does monitoring and so on. I think a lot of L L Ms, and I think we've seen similar things with ML too. But I think L L Ms will see, I my bet is we'll end up in a similar position where like even if a lot of engineers are empowered to use the language models to solve different problems, there's still a huge benefit to having some team that centralizes on a solution to like, how do we monitor this thing? How do we ensure quality? Are there like internal, how do we manage data sets internally?

• 16:44Speaker 01

  How do we train models and make sure that there's like a regular kind of training? It's like a tool, tools around all of these workflows. I think it still makes sense to have a single kind of pod working on it.

• 16:52Speaker 02

  What have you guys learned about how your users interact with AI products? And you know, it sounds like you do a bunch of dog fooding with, you know, testing them all out internally. What have you learned about, you know, how to educate?

• 17:07Speaker 02

  Your users on Notion AI are using these different new products as you're shipping them?

• 17:12Speaker 01

  I guess we have a couple of different ways. So using it internally, I think, is particularly during the ideation phase, I think is still the richest kind of source for figuring out what are the most useful ways of using something. But Notion uses Notion in a very particular way, which is like we use every single feature that we have. And then we also use it extensively for everything that we can use it for. And so a lot of our customers-

• 17:33Speaker 02

  Not exactly representative.

• 17:34Speaker 01

  Yeah, exactly. Some people use it like we do, but some people, they're like individuals or students. A lot of our users are students, which we love, who we love. And then there are some customers that, you know, for some reason, we can't figure out, they're using Google Docs in addition to us. So there's lots of, lots of different usage patterns. In the first phase, we sort of tried to look internally, just figure out how people are using it. And then we have some early testers, ambassadors, Notion ambassadors, and other kinds of partners that we work with. And often they'll take, they'll take the initial features, particularly if they require some like prompt engineering, they'll take these features and try to use it for totally unanticipated use cases. And sometimes we're like, oh, that's a really specific use case and we don't want to support that.

• 18:17Speaker 01

  And other, other times we're like, hey, that's actually, that makes a lot of sense, even though we haven't, we haven't thought of it. And so sometimes they'll inspire new features that we build internally. A good example, concrete example of this is when we initially launched AutoFill. One of the early use cases that we saw a lot of usage for, particularly abroad, outside of the US, is translation. And so people would use a custom prompt field to say, translate this page into some other language, or translate the summary into some other language. And so now translation is one of the built-in kind of prompts.

• 18:44Speaker 02

  Very cool.

• 18:45Speaker 01

  I think this pattern of having a bunch of built-in, a kind of engineer optimized prompts in addition to like a custom thing is, is, is actually kind of a Notion pattern, Notiony pattern and that we have with Notion more broadly, there are Notion sanctioned kind of patterns and that are templates. And then you can also fully hand customize them. With prompts, I think it's similar where we use the custom, fully extensible part of our AI features to figure out what the key use cases are. And then we end up over time building them into these like pre-baked formulas or templates or prompts.

• 19:17Jacob Effron?

  And almost all people use the prebaked prompts? Or like, what do you kind of notice on the user behavior side?

• 19:22Speaker 01

  So there's always kind of a power law with these things, right? So the most popular use cases, and particularly for AutoFill and Writer, the most popular use cases are these like pre-built things. And so summarization is a really, really popular use case. Improved writing and fixing grammar mistakes are another really, really popular use case. Depending on which locale you look at, translation is also quite popular. But then there's also, if you just look at like the total amount of usage or like number of tokens generated, another huge part of our usage is people iterating on these model outputs. And so we have this particular prompt that's for taking some user request and like revising, revising output. And that accounts for a lot of our usage as well, which is, which is interesting. And it kind of shows, it's a peek into like how people are using these features,

• 20:07Speaker 01

  which is that they, they use these pre-built things to get inspired about how that you could use it. And then they end up iterating to fit their specific use case. And I think if you, if you talk to the initial AI power users, they'll usually start, start in this kind of way. And then they'll end up often handwriting these prompts that they reuse over and over and over again to, we, I talked to like a newsletter writer recently where they have a template that takes an episode of a newsletter or an issue of a newsletter and then generates the like social meta image tag that shows up on, on Facebook. And so they've like polished that over time and they just keep using that same prompt.

• 20:40Jacob Effron?

  We should try that with unsupervised learning.

• 20:42Speaker 01

  Yeah, definitely.

• 20:42Jacob Effron?

  Yeah. Get that in. But it's so interesting that like, you know, helping users get over the like, you know, blank canvas problem, it's amazing even, you know, whether it's LLM based or even in like the, a lot of the kind of text image, you know, models have the same thing where staring at a blank screen is hard to know what to do. And then you kind of see how people prompt it. You see, you remix those in some ways. It feels like a common, and then, you know, even we talked to Canva, I feel like, you know, maybe three, six months ago, and they were super focused on that. Like, how do you just get people seeing what the possibilities are? Because then the creative juices get flowing.

• 21:12Speaker 01

  Right, then that's the, that's the big difference between the like AI power users that I've talked to and people who are like, oh, I don't really know what this could be useful yet. Is the people that use it a lot. Often, the key use cases that they find are the ones, the initial ones that we sort of ended up rolling out pre-built prompts for, where they say, hey, I'm a marketer, or I'm a recruiter, and here there's like a use case that's particularly well-defined for the thing that I want to use AI for, and so that makes sense, and they get inspired, and they try to iterate and build their own use cases on top of pre-built templates, where if you are an engineer that lives your entire day in a seed code base or something, maybe writing docs is less useful, or automating writing docs is less useful to you, and so you didn't end up having as many opportunities to bump into these different use cases where

• 21:57Speaker 01

  AI could be useful inside Notion. I think pre-built prompts is one, I think also, one thing that we've thought about is like, maybe we could, depending on what we see you're working on, suggest different kinds of, actually, a really obvious one is like, if you've used AI to generate something, we could probably look at that and make a good guess at like, what kinds of revisions you might want to make next, so that's something we've thought about. But as always, like, getting, turning, typing, coming up with something to say to the AI is still a lot of mental work, and so if you can make it into like, oh, here's a few options, here's one, I think that's always the best.

• 22:34Jacob Effron?

  And so as you think about a feature like Q and A, from inception, which obviously it sounds like you saw all the RAG stuff going on, it's not, I'm sure there was lots of ways you thought about incorporating RAG into Notion, what ends up, as you think about that end-to-end process, what are they being like, way harder than you would have thought from the start to actually get this shipped and like into production, and something you thought might be hard, but actually didn't turn out to be that hard?

• 22:55Speaker 01

  There are a lot of hard things.

• 23:00Speaker 01

  I guess there are some like hard technical things, and I think there's hard product problem things. One really hard, I think it was something that we expected to be hard, but is like, as hard as we expected it to be, which is just, particularly with language models, evaluation, which is, I'm sure, a theme that you've heard lots of other people talk about. But for something...

• 23:22Speaker 01

  It's where correctness and quality really, really matters like Q and A. So if you think about Writer and how people use it to write, like draft their emails or write a story or something like that, or begin the draft of an essay, there's actually a lot of wiggle room. Like if you ask AI to say like, hey, can you write me a, you know, high-quality academic-style essay about the benefits of a transformer model? It'll like do a pretty good job, but there's lots of different ways that the model could have done that and still fit your expectations. Where Q and A is like you're asking for one specific answer and there's there's a lot of different ways that the model can mess up and say the wrong thing or not look at the right document. And so it's much more black and white. And I think that plus the fact that there's a fuzzy boundary to sometimes what we think

• 24:05Speaker 01

  the model should be able to do, like some people, a lot of the questions that we want to answer are questions that are answered by some content on a specific Notion page. But another common answer, especially when we started handing it out to users, we saw it was like people will sometimes ask very meta questions about Notion. Like they'll ask like, how can I share this page with Jack? And like, that's not really, that's not really in your Notion workspace. But you could imagine why someone would want to ask that to like a Ask Notion AI feature, right? Or like, how many people are in my workspace? It's like a meta question. So anticipating all the different kinds of questions. Or another one actually that we spent a lot of time working on was questions of this form. Like, what is the marketing team working on this week?

• 24:49Speaker 01

  The this week part is like, there isn't really a document in your workspace that says, on the week of October 25th, this is what the team worked on. It's just there's constant updates in your workspace and then the model should be able to look at documents sort of like by time over time and answer that question. And that's quite different than a traditional kind of RAG pipeline. And so enumerating all these different kinds of ways where the question might poke at the edges of the model's capabilities or the pipeline's capabilities and then Constructing high-quality evaluation sets for those, figuring out what criteria you want to grade on them, and eventually finding ways to remedy all these different kind of edge cases. Padding out the edges of the model's capabilities, I think, was a really hard overall problem. But it's a problem that actually has like 30 different subcomponents.

• 25:31Speaker 01

  And then from a just like operational perspective, I think there was a lot of questions we wanted to answer around like, what are the needs of our customers around like privacy and security? How much scale do we need? And because this is a, this like language model augmented question answering is still, I think that lots of companies are in the beginning stages of exploring. There aren't like clear, obvious answers from the industry on those questions. And so we had to go back to our customers and sort of start from first principles.

• 26:04Speaker 02

  On the evaluation side, I'd actually be super curious. Do you guys, have you built out tools internally for doing that? Or like, how do you go about evaluating the models? Kind of maybe in development phase, but then also once they're in production?

• 26:19Speaker 01

  Most of our,

• 26:22Speaker 01

  I would say the vast majority of the tools that we use to work on our language model powered AI features are built in-house.

• 26:29Speaker 02

  Okay.

• 26:30Speaker 01

  That's a function of two things. One is back when we started building these things, there weren't as many tools as there are now. And so we kind of had to build them on our own. But I also think like Notion documents are actually very structured and quite complex. Like they're rich text documents, which to start with are much more complex than like a Wikipedia page or a PDF. They can contain like tables, for example, or contain images that can contain columns. And then also a lot of pages have metadata. So if you're using a page to keep track of the status of a project, it'll have like tasks associated with that project. It'll have due dates and people involved in it. And it's not trivial to represent that just as like plain text.

• 27:13Speaker 01

  And so we have lots of custom data structures and ways to represent these for models. And so traditional sort of off-the-market tools, at least in the beginning, didn't suit our needs as much. And then I think just having your own tools lets you iterate much faster. So if you notice that like even something as simple as like, Hey, our evaluation tool lets, or maybe there's an off the market tool that lets you view results from two different models side by side, but it only supports like two side by side. What if I want to compare a third one? Well, if it's an in-house tool, you know, it's a few minutes to go in and add that third column. And then I think that those things kind of end up adding up a lot. But on the evaluation point specifically, we look at it on a kind of a spectrum. On the one end are, so you're trading off how costly

• 27:56Speaker 01

  it is to evaluate with how high fidelity the evaluation is. On the one end are deterministic programmatic evaluations of model outputs and our model graded outputs. And so, and then the other end are like humans looking at the output. And underlying all of this is some evaluation data set where we've said that here is a particularly hard category of problem, like maybe the category of problem for particularly relevant for Q and A is like answering questions that involve time ranges. And then we'll have some like data set for documents that are user queries and document sets that fit that. And then we'll have some evals that are more like, did the model output unanswer? Did the model use some form of query that involved a time range? And then on the, and then somewhere in the

• 28:37Speaker 01

  middle are, we have a team of human annotators that we work with to speed up some processes. And then way on the other end are, there's actually a lot of payoff to like our ML engineers looking at the model outputs and not just knowing like, oh, this model did better than the other model, but looking at the actual outputs and saying, why are the models messing up? Is it because the model didn't understand the instructions? Is it because it has a hard time thinking about relative dates, but it's better with absolute dates? Like this is one, these are the kinds of things that we've seen. And so people working in the actual systems, looking at model outputs, looking at the data sets has a lot of payoffs. And that's way on the other end of like very costly and expensive, but actually yields sort of commensurate rewards.

• 29:16Jacob Effron?

  And as you're working on that problem, are you mostly iterating on the way you're prompting with the data that comes out of like the, out of the database? Or like, are you changing the way you're adding like metadata to the embeddings? Like, how do you ultimately solve that problem?

• 29:28Speaker 01

  We iterate on the full stack. And so one of the benefits of owning everything from the database, database layer up is anything, anything we want to add or tweak or try out, we can. And so if we want to, at some point be experimented with adding an extra stage in the, on top of the traditional RAG pipeline that took retrieved passages or chunks from the vector database and then try to like rephrase or re-summarize to better answer the question. And like, that's the kind of experiment that we've done. Run the evals and at some point we decided it was interesting and then at some point we decided, oh, maybe it's not so helpful. But it's there's depending on the kind of depending on the reason that the pipeline

• 30:10Speaker 01

  messes up, the right place to intervene is a function of that.

• 30:14Jacob Effron?

  And so if the reason that the model messes up is actually because of the right, the document

• 30:18Speaker 01

  with the answer didn't end up in the model context at all, then that's like an embedding problem or it's a ranking problem or something like that. If it's just you had the right documents, you just got confused about like what was the most relevant part or something like that, then that's like the answering part of the pipeline. And so that, again, that's the part where like just numeric evaluations doesn't do justice to the problem and you have to have someone sit down and figure out what are the most common reasons that the model is messing up and go back to the pipeline and figure out where to intervene.

• 30:43Jacob Effron?

  And then as you guys have thought about models, you know, have you thought about using off the shelf stuff versus, you know, using, obviously you sit on an incredibly interesting set of data. You know, I feel like we've seen folks either fine tuning smaller models to be able to run faster and cheaper or, you know, but a lot of people I feel like start generally with state of the art, see what's possible and then kind of iterate from there. How have you guys kind of thought about that?

• 31:06Speaker 01

  We have been really like to have great partnerships with Anthropic and OpenAI since the sort of early days of when we started working on it. And so they are very good at the infrastructure and like initial model building part of things. I think it would be exceedingly difficult to compete at the infrastructure level with either of those companies or Google. And so while we, conceptually, the way I think about this is like the problem that they are solving is how do you build a good model that follows instructions and how do you host it in a really scalable way, in a reliable way? And then the problem that we are solving is more understanding the particular tasks that we want our models to perform and then figuring out how do we best collect or generate

• 31:50Speaker 01

  synthetic data to through those models. One of the things that we've committed to is so far is not training on our customer data. It's really important, like I don't want Notion models to be trained on my personal data because I keep a lot of my own information in Notion. And so that, that's of course a challenge, but I think it's actually. Pushed us into thinking a lot more systematically, but like if you're generating a synthetic data set of like workspaces or Notion documents, you have to really understand what are the core kinds of archetypes of pages, how are these structured? And so that's actually, I think, for us to really understand how Notion workspaces are organized and what kinds of documents or the prototypical documents we want to serve AI for. But on the model

• 32:32Speaker 01

  question, our role as compared to their role in this kind of partnership is understanding are the profile of our tasks really well, figuring out how we can like evaluate those tasks. But if even something as simple as a summary, what makes a good summary is actually the criteria for a good summary is actually really different for like a meeting note versus like a long technical document versus sometimes an Notion page is just like a bug report. Like it doesn't really make sense to even summarize that. It's like the bug is in the title of the page. And so understanding the task and then doing the legwork around like building the data sets evaluations around that task, I think is more where we've spent a lot of our time. We it's always on the, I think, on the back of the minds of the different people on our team that there's a possibility that, you know, we could always explore open

• 33:17Speaker 01

  source models. I think we're doing some exploration around different kind of embedding models right now. But so far, we haven't shipped anything into production with open source models.

• 33:30Speaker 02

  How do you think about with, you know, like running multiple models, like I guess maybe you run them against your evaluation set and kind of see which one is, you know, the best for the different use case. But how do you think about testing those or running multiple models side by side? Yeah.

• 33:47Speaker 01

  Yeah, the best model to use for a particular feature depends on a lot of things. Depends on, first of all, the model has to kind of perform up to expectation. And then beyond that, there are like some models can support higher throughput. And so if you want something that runs kind of batch in the background or autopill feature, anytime the page changes, we sort of re summarize the page or re extract the information. And for that kind of thing, you probably want a model that can support higher throughput. And so depending on all these all these factors, we look at kind of everything from the capabilities to the cost and then figure out what the best model is.

• 34:23Speaker 01

  And often that means totally different model scale, perhaps different provider for different features.

• 34:29Speaker 02

  What kind of preprocessing do you guys do before you're sending the request to these models? I think, you know, Jacob mentioned, we talked with Canva and they had this elaborate filtering process where they would, you know, go through a bunch of different checks before sending out one of the queries. Like, how much preprocessing do you guys do? And then kind of what goes into forming those prompts?

• 34:50Speaker 01

  For every new AI feature, if you are specifying some kind of prompt or query, it'll almost certainly be wrapped in a lot of our own processing. Sometimes that is, like for a lot of our writer features, if you say something like, like, oh, you did a good first draft, but make this more punchy, that'll be wrapped in some prompt template that includes like what the model's already done, maybe the full history of the dialogue plus your revision. For something like Q and A, we have, one of the first phases is we take your, Q and A actually supports multi-turn conversation. So if you ask a question initially, like, what's David working on this week? And then it gives you an answer. You could follow up by saying, hey, tell me more about that automation project.

• 35:34Speaker 01

  And it'll know the context and it'll search, there's a query rewrite phase where the model might search for something like David automation project, given the context of the conversation. And so even in places where it's not so obvious perhaps that there's a middle layer, that we do a lot of passes to restructure queries to restructure prompts.

• 35:52Speaker 02

  And then on the output side, like first, like Q and A, do you have like a checking process at the end where you're like, let's make sure this isn't totally hallucinated or ties back to some document? Or have you just tried enough edge cases that you're confident in the output?

• 36:04Speaker 01

  This seems like the kind of thing that may change over time. So I certainly wouldn't commit to what I'm about to say, but especially as the models get better, I think we found it pretty good. Not saying, at least not saying objectionable things. And then things like hallucination, I think we've made a lot of stride on, but it's still something that we're working on on the edges.

• 36:25Jacob Effron?

  And one thing I'm curious about is it's, I feel like you guys are gurus on the prompt side of like figuring out, you know, you're experimenting, you know, one of the like scale and a lot of interesting edge cases of figuring out how to get the right outputs from these models. And I think one big question in the space is going to be how easy it is to switch between models. And I'm just curious about your experience when you like nail the prompts for, you know, a certain type of model. How easy is it to like, you know, switch out OpenAI for Anthropic or go to a different, you know, kind of like size model and still have those prompts hold true for, you know, kind of a macro question, but I'm curious what you've observed on that.

• 37:00Speaker 01

  Prompts are downstream from the evaluation criteria that you'd use to judge a model's output, which are again downstream from your understanding of the problem or the task the model needs to perform. And so I think we do the best job of prompt engineering when we have a clear understanding of like for summaries as a sort of example again. For a particular kind of document, which maybe we use a different stage of the pipeline to identify or something, for a particular kind of document, here the criteria and then we might ask the model, make sure that your summary, you know, includes these things and doesn't include these things and is of this format based on understanding of the task. And then maybe, maybe the per model tweaks that we do, maybe like, you know, type something

• 37:44Speaker 01

  in all caps and make sure the model doesn't do it. Maybe we notice that like a common pattern with summaries is some, a particular model from a particular AI vendor really likes to say, in this document, the document discusses X, Y, Z topic. And like, you don't really need to, and I've tried so many different ways to get the model to just like not say the phrase in this document, like, don't be meta, don't reference in this document, just like rephrase the, this is like very hard. And so those kinds of tweaks, I think, may be different, different per provider, per model. And so those you might have to tweak in the end, but I think the bulk of the instructions, especially as model models overall get better at following instructions, the bulk of the tweaking is just like what the evaluation, what the criteria are and then what the shape of the task is, what format do you want?

• 38:26Speaker 01

  And then those are, I think, that understanding of the problem and the task, I think, carries over between models.

• 38:31Jacob Effron?

  Yeah. What about across languages?

• 38:34Speaker 01

  Ooh. Large models generally transfer well across languages.

• 38:42Speaker 01

  I think they also models, at least the ones that are trained with sort of like a representative corpus from the entire internet, they still tend to exceed at English or excel at English and then sort of performance falls off in some way across other languages. And so as a baseline, they transfer pretty well. We usually just prototype with English one because Notion is predominantly English. And then as we get closer to launch, we'll have specific evaluation data sets for evaluating multilingual performance and we'll try to like add few shot examples or do training to bolster performance on the other languages. But as a baseline, we find at least the really large models transfer well across languages. Actually, for Q and A in particular, this is really fun because you can ask a question

• 39:25Speaker 01

  like, what are some sales that the Japan team made this week? And even if those documents are in Japanese, it'll like read those documents and translate the answer back to you. So you don't even really need an intermediate translation layer, which is cool.

• 39:34Speaker 02

  You've been a big proponent of kind of iterating on interfaces and making the interface kind of right for the AI use case. Like, how do you think about giving users or maybe power users freedom and flexibility versus kind of guiding people towards, you know, maybe how AI can be most helpful for them? And what that means for how you design kind of AI features at Notion?

• 39:57Speaker 01

  And I think the answer is similar for both Notion in general as well as for AI, which is you want to pick the right building blocks and you want to give users, you certainly want to, for every AI feature, there's always, you can always drop down to like something quite near the lowest level, which is you can just prompt the AI pretty directly. Obviously, we still have some intermediate processing to add like the context about your page and workspace and so on, but there are some pretty vanilla, blank, unimpuginated prompt templates that we use for like the power users. But even then, you want to decide like what context to include in, for Notion more broadly, the customization, like the users still can't write their own HTML and CSS, right?

• 40:41Speaker 01

  We give you instead this like pretty rich block abstraction. And so one question that I've had for the longest time since I joined the team was what are the kind of blocks for AI? What are documents for AI, what are blocks for AI, which we still, I don't, I think, at least the team doesn't have a consensus on what the answer is yet. But you want to give people the right abstractions.

• 41:03Speaker 01

  I think depending on what you want the AI, what you consider the AI output, those abstractions are different. If it's just writing, then I think writing is pretty free form. And so it's, I think custom prompts go a long way. If it's like, I've talked with some of my friends recently about generative interfaces, where models can output, instead of text, it can output like an entire piece of UI. In that case, you probably want some kind of like a component library or a UI language, where the model outputs like buttons that do things. And probably all the buttons should look similar and coherent with your overall product design. Adapt recently, you started exploring models that can perform actions. And so in that world, it seems like they're also exploring, I don't know if you've looked at their,

• 41:47Speaker 01

  their new release, but they have a kind of a domain specific language also for taking actions in the browser. And so like, maybe that's their building block. And so for customizability, you want to pick the right building block. I think we're still in the process of exploring what exactly want, we want like the full capability set of Notion AI to be and what the building blocks will be a function of that.

• 42:06Jacob Effron?

  Well, we always like to end our interviews with what we call the quickfire round, where you get your, your hot takes on, on everything in AI. Yeah, good to take a sip of water before that, before the quickfire round. I totally agree. But to kick things off, we like to ask everyone, what's one thing that's like overhyped? And one thing that's underhyped in AI right now?

• 42:22Speaker 01

  I think context length is overhyped. I struggle to imagine really, really useful tasks that require like 50,000 words. I think a lot of it resembles like retrieving some information. But I think even if you can pack all of your data in like 100,000 tokens, if there's a lot of noise in that data, the model's still going to do worse. And you still want to have some amount of like filtering over the data.

• 42:49Jacob Effron?

  Yeah, like six months ago, it was like, there was still debate on like whether it was going to be rag or just ever expanding context windows.

• 42:54Speaker 01

  Right. And I think at this point, it's pretty clear. Yeah, it's pretty clear you're on team rag. I mean, I think rag is always the retrieval, getting good information into the context, I think is always going to be like where you need to start. And then maybe there actually are like 10,000 tokens worth of really relevant information. And then at that point, maybe the extra context is helpful. But I think it's also kind of an issue. When you're training really long context models too, this is now like not quickfire. I'm just like going around, but if you're training models that are up to like 64,000 or 100,000 tokens in length, how are you general like interesting challenges? How do you find examples of tough instruction following tasks with like answers where the

• 43:38Speaker 01

  model needs to make full use of the entire 100,000 tokens of context window? Like those are not, this is not like a naturally occurring data set. Some clever approaches that I've seen include like, like one particular task that you can construct sort of synthetically is you take a really long piece of text, you find some subset of that that includes an answer, and then you generate the answer to that with a smaller context window language model. And then you give the, when at training time, you give the long context model, the full original text, and then the model has to locate the information from like needle in a haystack from this like larger document. And like, you can do synthetic tricks like that, but I think fundamentally, most useful tasks don't require that that much. I feel like probably like 16 to 32 K.

• 44:23Speaker 01

  I feel like I might eat my words in five years, but at this point, I think a lot of things that we want language models to do, 32 K tokens might be enough. I think that I think is underhyped is alternative, alternative to transformers, which I guess may be another way to say that maybe the transformers are right, but I don't think that's true. I think transformers are awesome, but it's, the great thing about transformers is that they are really good at modeling long sequences and they are efficient. Nothing architecturally about the transformer is like, is like the most optimal thing ever. And so I think it's totally possible that we're going to find another architecture that is really good at modeling sequences and efficient, maybe even more efficient.

• 45:04Speaker 02

  It's interesting. I think we've had a couple of people say that about transformers to their, they either, as you mentioned, said transformers were overhyped or alternative architectures were underhyped, which will be interesting to follow.

• 45:17Speaker 01

  Yeah. I mean, the, the reason that we have these models we do now is mostly because we just train on so much data and the models are good at absorbing that information. The models being good at absorbing information, I think is actually pretty key, like. Models we have previously that theoretically supported with a long context we're not really good at utilizing all that information like LSTMs are really terrible at using context beyond like a few thousand tokens. But now that we know, we're starting to get a good understanding of why transformers are really good at these tasks, I think we'll see alternative architectures over the next few years.

• 45:50Speaker 02

  Any you have your eye on?

• 45:51Speaker 01

  I think state space models are cool.

• 45:54Speaker 02

  Fire round number two, what's been your biggest surprise building Notion AI? Or what's one thing you thought would work that didn't end up working at all?

• 46:04Speaker 01

  It's probably more nuanced than I think the question stylus we asked for it. But I think, I think I've been surprised how well really general approaches work. I guess this is a kind of a lesson around how to customize a model for a particular domain or a particular set of tasks. A really specific example is like if you, if in your product, you have like five different tasks you want a model to do, maybe you wanted to answer some questions, you wanted to like help you write something, you wanted to like, maybe another one might be like, an idea that we've thrown around at the company is like, AI should somehow be able to like help you onboard onto a product faster.

• 46:48Speaker 01

  So you can imagine a few different tasks that you want a model to do. A thing that it might be slightly easier to get up and running with is to build like specific prompts and specific models for each of these cases. But I think there are longer term payoffs for potentially trying to build a single model that does all of these things. Maybe you build data sets for each of these cases, but then you train a model on all of these things. Or, and that's like one way where you work towards like a general thing that can solve these things. And I think there's, you end up with a model that like overall understands the domain of your product better. Another example might be like, like I'm starting to do some experiments with generative UI.

• 47:27Speaker 01

  Or models output, output interactive elements, or I should say me and a couple of the team members are. And that's the kind of thing where like, there's again a trade off to be made between, do you build an interface that is like hardcoded, but that's powered by generative AI? Or do you give the AI more control over what shows up on the screen? And giving the AI more control over shows up on the screen is like a harder ML problem. But I think my bet is longer term, you're going to see a lot more interesting things you can do with that. And so I think, given the pace of how quickly models are getting better, generality, I think is something interesting to bet on.

• 48:04Jacob Effron?

  And I feel like, you know, from first principles, Notion would be one of the most interesting places to work on AI features more generally. If you weren't at Notion, like, what one company would you go to, to, to work on their AI features?

• 48:18Speaker 01

  I mean, I think, I think Mid Journey is such a unique company. There's lots of unique things about them. The one that I'm particularly curious about, I would be curious about as someone who likes to prototype and tinker is, they operate in a really like lab kind of way. I think GitHub Next, that works on Copilot, also operates similarly, or used to operate similarly, based on my references, they're a little bit outdated. But where they have kind of individuals that are champions of specific ideas, and then they sort of push specific projects forward or prototypes forward. And then if that approach works, it ends up blooming into something that's sort of ready to go to production. At that point, there's probably more of a team effort. But it's kind of a, not really a competitive environment, but it's kind of a garden of just like different

• 49:02Speaker 01

  people trying different things and trying to figure out what the best ideas are. And I think that always yields really productive outcomes. I spoke to someone at one organization that operated like this, and they're like, yeah, we internally have like, six different versions of our next model. Like, how does that, how does it even work? Wouldn't it be more fruitful to just like, combine all those ideas into one? And like, I think if you have individuals, or especially if you have sort of individuals that are motivated to solve a particular problem, individuals are able to iterate much, much faster than a team. And so this kind of garden approach, I think actually works really well. So I think Mid Journey is super interesting for that reason.

• 49:40Speaker 01

  I think Adept is also actually doing a lot of interesting things. They are tackling a really, really ambitious problem, which is this incredibly general agent that's able to perform any task that you can do on a computer. But it seems like they are taking their challenge in stride, and I'm sure they're doing interesting things.

• 49:59Speaker 02

  And to wrap up, where can people go to learn more about you and Notion AI?

• 50:05Speaker 01

  So for Notion AI, we have this shiny, relatively new domain, notion.ai.

• 50:10Speaker 02

  Ooh.

• 50:11Speaker 01

  And if you go there...

• 50:12Speaker 02

  You have to battle for the .AI, or did you guys already have it?

• 50:15Speaker 01

  I did not personally battle for it, but I'm pretty sure Akshay, our COO, stepped in a ring of some sort. You know, threw some dollars on the other side.

• 50:25Speaker 02

  That feels like one that would be way cheaper to buy early in the company's history than now.

• 50:29Speaker 01

  Definitely. Yeah, I don't even know, I actually don't know any of the backstory, how we ended up with that. One day, he just came up in our channel and he was like, notion.ai, here's the domain. I'm like, where did those come from?

• 50:40Speaker 02

  Don't ask any questions.

• 50:42Speaker 01

  Picked it up in a back alley somewhere. But you can go to notion.ai, or if you're already using Notion, I'm sure there are emails in your inbox waiting for you to help you try it out and hear how you use it. I am probably most active on Twitter, at thesefist, or if you like, Google, Linus, Notion. I also have a blog where I sometimes write about interfaces in AI.

• 51:05Speaker 02

  Can confirm, great Twitter follow, highly recommend, for the Taylor Swift content.

• 51:08Speaker 01

  For the Taylor Swift content.

• 51:11Speaker 02

  Well, that was a super fun episode with Linus. I think it was really, really cool to hear him talk about how Notion structures their teams, especially around AI with kind of this, you know, I think he said there were a dozen engineers working on it, and then a few designers, and also just how they kind of iterate quickly in the early stages and like test these features out internally. They really stuck out to me how much dogfooding they do, both with like the feature set and the evaluation. It seems like all of Notion is really, well, they're living in Notion, but then also testing out these AI features. I thought that was really cool. Totally.

• 51:49Jacob Effron?

  It's interesting, right? Like, you know, for a feature like Q and A that they build, you know, you could have a product team that's just thinking about that. But what I think he said, which is super smart, is they're going to do so much in the future around retrieval along with generation that getting that right feels like more of a foundation. Like layer level decision for them. And makes total sense to kind of split it out.

• 52:09Speaker 02

  Yeah, it was interesting because it seemed like that had kind of been a journey for them where. They started thinking, okay, we're going to build AI for each of these products. And then now Linus is thinking is. You know, maybe there's more of a horizontal platform where, you know, the retrieval all happens by one system and. Maybe you have one model that does a bunch of different tasks. So that was, that was kind of interesting to see as a company that's a little farther in their AI journey.

• 52:33Jacob Effron?

  Yeah, I feel like one common trend with all of these AI companies implementing products is it's so hard to predict what people will do with them like it's. Funny, you think like a product like Q and A, it's like, it's literally a search product like you'd. I totally imagine you'd think, okay, 99.9% of it is going to be like, you know. What's the answer to this question? You know, what happened in this meeting? And it's just hilarious that people are using it like how many users are total or like what happened, you know, this specific week or.

• 52:57Speaker 02

  They're using it for Notion support.

• 52:58Jacob Effron?

  Yeah, exactly. Exactly. Like, how do I share this?

• 53:02Jacob Effron?

  And, you know, so hard to predict that without. I think you made the great point during the episode that that's one of the big advantages distribution has like you just get. You learn that so much quicker.

• 53:10Speaker 02

  Yeah.

• 53:10Jacob Effron?

  Having so many people, both internally and externally, using it so quickly.

• 53:15Speaker 02

  Totally. What, uh, what did you find exciting or interesting?

• 53:19Jacob Effron?

  Well, I gotta read this Taylor Swift paper now. That was definitely, what a wild.

• 53:24Speaker 02

  This is the second episode in a row that we've talked about Taylor Swift.

• 53:28Jacob Effron?

  I think we should make it a trend.

• 53:29Speaker 02

  Yeah, we'll have to figure out how we can hit it with perplexity.

• 53:32Jacob Effron?

  Yeah, and it's a shame because I, I'd asked them, the explicit question I'd asked them was what song, you know, like it best encapsulates what's happening? And I'd obviously thought a lot about my answer.

• 53:41Speaker 02

  Yeah, so what was your answer?

• 53:42Jacob Effron?

  You need to calm down.

• 53:48Speaker 02

  AI song that most resembles their Taylor Swift song that most resembles the AI market.

• 53:54Jacob Effron?

  Yeah.

• 53:55Speaker 02

  You need to calm down.

• 53:56Jacob Effron?

  I think that would be my message, but also like, I feel like in the pre-show, you were saying 22, which I think also kind of encapsulates some of the young vibes.

• 54:04Speaker 02

  I like 22. It's kind of, it's this young, like opportunistic, world is your oyster type phase. And so, I don't know, peak hype, that's what, that's what 22 feels like.

• 54:17Speaker 02

  Yeah, so I would go 22.

• 54:19Jacob Effron?

  Yeah, well, we'll see. Maybe we just have to add that to the quickfire now.

• 54:22Speaker 02

  Yeah, favorite Taylor Swift song or, yeah, that's a good quickfire.

• 54:26Jacob Effron?

  Over/under hyped Taylor Swift song. I thought another thing that was really interesting in the episode was just how much they built themselves.

• 54:32Speaker 02

  Yeah.

• 54:33Jacob Effron?

  Like, the, I mean, eval, I feel like we've seen tons of eval companies, you know, it's obviously such an important space, but God, what a hard problem.

• 54:41Speaker 02

  Yeah. Yeah, it's really interesting to think about if they would build all of that now.

• 54:50Jacob Effron?

  Yeah.

• 54:50Speaker 02

  Because the market is changing so quickly and even six months ago, it was, it looked different than it does now. And six months before that, when they were starting on a bunch of this AI stuff, there wasn't this whole crop of startups that we've now seen.

• 55:03Speaker 02

  But it also seems like Linus kind of thinks that they had to do some of that because they're iterating and this allowed them to iterate faster. But I don't know, I think the...

• 55:13Jacob Effron?

  Well, also it's interesting to hear, like, there's no substitute for manually going through, like, the ones that really don't work. And, and a lot of the automations that you may want to do around this are just kind of summarizing it with a number. It sounds like where they've gotten the most value is going super deep into, you know, what fails and understanding why.

• 55:29Speaker 02

  Yeah, totally. It was fascinating that they, because they want to own everything, it seems like, except for the model, which they're happy to outsource to Anthropic and OpenAI and...

• 55:43Speaker 02

  I thought that was interesting, which kind of makes sense like they're a they're an app company. They're not an in for a company that needs to be training and deploying models at massive scale. But I thought that it was interesting. He was kind of like, no, we we don't do that.

• 55:57Jacob Effron?

  Yeah, because you'd think that maybe at some point, you know, for either latency reasons or cost reasons, you might want to try something else. But the reality is, as you said, I mean, the large foundation model companies keep slashing costs and they have a pretty broad array of options of of model speed. And so, you know, it seems like they've been able to get by just fine, not having to make any of those trade offs.

• 56:17Speaker 02

  Yeah, and they want the notions of the world to be successful building on them like they want. This is the perfect use case for L L M's in SaaS and people are always like, oh, what's the enterprise use case for L L M's? Notion AI is, I think it's kind of GitHub Copilot was probably number one. I think Notion AI is one of the top, you know, five.

• 56:39Jacob Effron?

  Yeah, no, we made we made a presentation for some of our port codes and it was like one of the main main logos right front and center. I feel like it's certainly one of the most like consumer, you know, one of the most interesting consumer facing applications that's that's use these in a in a meaningful way.

• 56:54Speaker 02

  Totally. I I had a bunch of opportunities to either make fun of your writing or my own writing and how much AI could help it. So maybe next time we need to work that in. Yeah.

• 57:08Jacob Effron?

  Yeah, definitely. I felt like I I could use I could definitely use some help.

• 57:11Speaker 02

  I think, yeah, honestly, I think the templates that they're that they're shipping like I really like the buttons. I don't know how much you've played with the product, but just like summarize or like I personally like that because even just coming up with the prompts to be like, oh, I want to do this thing for a blog post. It can sometimes take a lot of mental energy beyond, I don't know. Yeah.

• 57:34Jacob Effron?

  Totally. I feel like you and I are both not like, you know, social media naturals. And so I feel like even having used some of these tools before, like I had this one I was using with tweets that were just like literally all was like make it punchier. And it was so helpful because like, it's just not my style.

• 57:48Speaker 02

  Yeah.

• 57:48Jacob Effron?

  What do you think of OpenAI Dev Day?

• 57:51Speaker 02

  I think it's fascinating. I don't know that anything was that shocking. I think the two big trends are longer context windows and cheaper models. I think we've kind of been seeing that for a while. We talked about the cost for inference and the cost for training going down 10x over the last year. I think this is more of the same. They slashed prices, what, 3x, I think, for Turbo. And yeah, so prices keep going down. I think that's great for the startup ecosystem. And then, you know, they're kind of shipping a ton of stuff. And I think this is also, you know, they're starting to, they own the models and they're starting to build up the stack into, you know, more solutions. What'd you think?

• 58:36Jacob Effron?

  Yeah, I mean, I think it's just wild to be running a business with AI features and on any, you know, given day when this happens, your costs go from like 15% of revenue to 5% or something. Like, just the swing that happens in these price changes is so impressive. And I wonder, you know, it seems like it's poised to kind of continue. But like you said, it's so hard to then predict, you know, oh, well, something's kind of cost prohibitive to do today, or we can't charge enough for that today. And it's like, you probably just build it anyway, because, you know, six months from now, 12 months from now.

• 59:07Speaker 02

  Totally. Well, it'll be interesting to see what Anthropic does now, because I could imagine if you were a Notion and you're running Anthropic and OpenAI side by side, there obviously are some differences, as he was saying, certain use cases, one's better versus the other. But now, one of them is 3x cheaper than it was before. And then, so maybe Anthropic would also have to cut their prices by 3x, which, again, would be great for everyone that's building on top of LLMs.

• 59:31Jacob Effron?

  Yeah. It's just cool to see that. I feel like the energy around, even just, you know, all the people that are watching it live and just like, you know, it's really cool to see.

• 59:41Speaker 02

  I mean, some of the demos are really cool. The

• 59:45Speaker 02

  Text to speech and the talking was very cool. I don't know if you saw the part where they were giving out free OpenAI credits to everyone in the audience. And they were like, oh, find five people in the audience and give them $500 of OpenAI credits. And then they did that. And they're like, actually, everyone here is part of our community. Give everyone here $500 of OpenAI credits. And it just went through. And then the AI was saying, hey, you're all getting OpenAI credits and started naming the people. I don't know. It's, it's very cool.

• 1:00:14Jacob Effron?

  Yeah.

• 1:00:14Speaker 02

  Yeah.

• 1:00:14Jacob Effron?

  And I feel like, you know, there's, right afterwards, there was all this like, you know, this is the death of RAG or the death of vector databases. And it's like, it's great that they added a lot of those features in, but obviously there's, there's so much people are going to want to do outside of OpenAI. You know, both just sensitivities around where they send their data, abilities to kind of customize, you know, implementations of that. And so,

• 1:00:37Jacob Effron?

  you know, I think it was like, kind of hypey on Twitter for a bit afterwards, but I'm curious to see what.

• 1:00:42Speaker 02

  Totally. I think there were a lot of people on Twitter that were saying, you know, this is going to kill all the startups. And then there was the other camp that was like, oh, no, it's not. And I think there were a bunch of people making the comparison to AWS, which I think is a very good one in that they're going to offer a bunch of tools and tons of them, but in any given vertical, like Snowflake did, or like some of these other companies did, there's totally an opportunity for startups to compete with the product in that vertical and win for a variety of reasons.

• 1:01:12Jacob Effron?

  Yeah. I like how you got in the Snowflake reference there. Good, good.

• 1:01:14Speaker 02

  Always have to get in Snowflake. Yeah. As many, as many times as possible. Maybe you need to make it start.

• 1:01:19Jacob Effron?

  Yeah, we just have.

• 1:01:19Speaker 02

  Fake snow in this room. Yeah. We're already in the mountains.

• 1:01:23Jacob Effron?

  I mean, whenever it's uttered, it just like snow comes down in the room.

• 1:01:26Speaker 02

  That was really fun.

• 1:01:27Jacob Effron?

  Yeah.

• 1:01:27Speaker 02

  That dude.

• 1:01:28Jacob Effron?

  Sweet.