How Notion Cofounder Simon Last Builds AI for Millions of Users - Ep. 37 with Simon Last

Notion cofounder Simon Last told me everything he’s learned from integrating an AI application into a platform that has over 100 million users. Simon likes to keep a low profile, even though he’s the driving force behind Notion AI, one of the most widely scaled AI applications in the world. In this episode, we get into how AI changes the way he builds software since the days he cofounded Notion with Ivan Zhao in 2013. He talks about the challenges that arise because AI doesn’t follow the deterministic rules of traditional software, and how he designs evals to build AI systems that are reliable at scale. Simon tells me about the AI tools he uses to code and how he would think about rebuilding Notion from scratch with them. He also shares his thoughts on how the growing capabilities of AI are redefining human roles, and argues that we have the responsibility to shape technology to align with our collective vision of the future. This is a must-watch for anyone interested in building reliable AI products at scale. If you found this episode interesting, please like, subscribe, comment, and share! Want even more? Sign up for Every to unlock our ultimate guide to prompting ChatGPT. It’s usually only for paying subscribers, but you can get it here for free. To hear more from Dan Shipper: Subscribe to Every: https://every.to/subscribe Follow him on X: https://twitter.com/danshipper Links to resources mentioned in the episode: Simon Last: @simonlast Notion AI: https://www.notion.so/product/ai The AI code editor Simon uses: Cursor OpenAI’s definition of AGI that Simon ascribes to: https://openai.com/charter/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dan Shipper?

  Let's just assume as a thought experiment that you're going to strip away everything that Notion currently is and rebuild it with AI. How would you do it?

• 0:08Speaker 01

  One big principle would be there's less humans touching the database, the AI should be managing it for you. Ideally, you never need to update any of the fields and it should know the amount based on your email.

• 0:18Dan Shipper?

  You all probably have like one of the most scaled AI applications in the world. Previously, software was deterministic and now it's very stochastic, it's much more squishy. I'm curious what you learned about doing that.

• 0:31Speaker 01

  The AI thing is so hard because it can fail in some cases, but then there's this like additional meta problem. You might not even know the cases where it can fail. If you have an AI evaluating, you need to evaluate your eval now.

• 0:41Dan Shipper?

  How has working with AI changed what you think machines are good at versus what humans are good at?

• 0:47Speaker 01

  Hmm, maybe a better question is like, what should humans do? I feel like the most important thing there is making sure that the shape of the future is aligned with what we care about. What do you want them to be doing?

• 1:12Dan Shipper?

  Hey guys, it's me, Dan Schipper. I just want to jump in to say that this episode is sponsored by Notion. I've known about and been using Notion since 2015, so almost 10 years. And I actually run every and a decent part of my personal life on Notion. They care a ton about the craft and the underlying ideas behind the software that they make. And I think that comes through in the way Notion is built and the experience of using it every day. If you're listening to or watching this episode, you should know that you get six months of Notion Plus with unlimited AI worth up to $6,000 for free if you're a startup. If that sounds interesting to you, go to ntnso/every, select every in the partner dropdown and use the code EVERYXNOTION to get access. And now, back to the episode.

• 1:57Dan Shipper?

  Simon, welcome to the show.

• 1:59Speaker 01

  Hey, thanks for having me.

• 2:00Dan Shipper?

  So for people who don't know you, you are the co-founder of Notion. This is, I think, at least as far as I could find, the first interview that you've done outside of internal interviews for Notion. So I really appreciate you coming on.

• 2:12Speaker 01

  Yeah, yeah, of course. I tend to keep a low profile, but happy to do it.

• 2:15Dan Shipper?

  Great. And you're leading the AI initiatives at Notion. As far as I can tell, you are also really pushing AI before it became a thing internally, which is really interesting. And the place where I want to start with you is obviously Notion is really well-known for building thinking tools. And you were building thinking tools before there were even thinking machines. The way that you went about that is you created a text editor and hooked it up to a relational database. And you thought a lot about how to create the right primitives that to allow people to interact with that in a really flexible way to build whatever they wanted to build or think however they wanted to think. And that was like in a pre-AI era. And so where I wanted to start with you is to ask you what you think the right primitives are for thinking with AI.

• 2:59Speaker 01

  Yeah, that's a really good question. I think, yeah, probably helpful to start with, what are the new primitives? Maybe so the way I think about it is we've got the foundation models or the model itself, which I think of like a thinking box where you can give it a bunch of context and some task and then it goes and does one thing for you. It involves reasoning and it can involve formatting it as an action, so doing something. And then the other tool is like embeddings. So just like really good semantic search. So that's why I think of those are the new primitives that didn't really exist before. I think a lot of the same primitives still matter a lot. Obviously, a relational database,

• 3:41Speaker 01

  it's a pretty fundamental concept. If you're trying to track any information, it's pretty useful to do that. You don't just want to shove it into a text file. You want it in a structured format that's consistent that you can query, that they can connect things. But the good news is like all the primitives still matter, but now you can plug in these thinking boxes on top to actually automate some of the tasks that a human would do in the past. And especially like things that are cumbersome and you don't want to do. The way I think about the primitives that I connect to AI, you've got databases, you have a UI around the database that a human can look at and the AI can use. The permission model is really important as well. There's a lot of coding agent tools coming out. It's super cool, but one issue with that is you don't really want to just make a Postgres

• 4:26Speaker 01

  database for you every time. What's the permission model? What can it read or write to? How can I see the schema? It's actually really nice and important to have a permission model that the user can understand and control what the AI can read or write to. A lot of the same primitives really matter and that I just think about where, you know, we're adding on top. Whereas before,

• 4:49Speaker 01

  Your database might have been essentially just data that you maybe you do like manual data entry to plug it in or some lightweight integration, but now you can actually put this reasoning box on top and like much more fluidly transform information or pipe it in and out or do reasoning sets on top of it.

• 5:03Dan Shipper?

  What do you think about chat as one of the primitives? And do you think that's going to continue to be a main way that we interact with these tools? Or are there other primitives that are going to become more important?

• 5:13Speaker 01

  Yeah. I think chat is probably some version of it's probably here to stay. The human interface is just so intuitive. You just talk to it. The big issue with chat is that

• 5:28Speaker 01

  you get this empty text box and most people don't know what to type in there. It's really great for someone that wants to explore the system and figure out what it can do, but not so great if you just want it to do some task for you. And this is actually true, not just of chat, but anything like, like this is actually one of Notion's biggest challenges is that there's a lot of features and it actually takes a little bit of exploration to figure them out. We call them toolmakers, people that are interested in exploring the boundaries of the tool and, and making their own little custom software. But one big discovery for us over the years is most people just don't care about that. They just want a solution to the problem that they have and they just want it to be presented to them. And then, you know, don't really have the patience to go figure out this complex tool, which is, you know, totally understandable.

• 6:13Speaker 01

  And, and I think chat is like a low level primitive where

• 6:18Speaker 01

  it makes sense to have, but the real goal is to connect people to some workflow or use case that's solving their problem and it's probably not the best interface all the time.

• 6:28Dan Shipper?

  Yeah, we do a lot of work with big companies and I see that all the time as there's probably five to 10% of their people are like, they want to play around with chat. They want to learn how all the AI stuff works. And then everyone else is like, let me just do my job. And usually I think what works is like letting those five to 10% find the workflows and then give the workflows to everybody else so that they don't have to chat with it. Uh, or they don't, they can start with a chat that's pre-filled with like the common things that they're doing. One of the interesting things about, I think about chat, and I'm curious what your thoughts are is like.

• 7:02Dan Shipper?

  Often in UI pre-AI, you had to make discrete updates to the state of the application, checking your radio box, or whatever it's discrete. It's either checked or it's not checked. And it's also usually along one dimension. But with chat, you can move in a fuzzier, more continuous way through multiple dimensions at a time. Have you thought about that sort of change from discrete to continuous or single dimension to multi-dimension? And like, how do you think those things work together best?

• 7:30Speaker 01

  My mental model is that unless we're talking about embedding sliders, make it funnier sort of thing. Excluding that, where the actual parameter is continuous. My mental model for this is you have your software state and like you think of it like it's a JSON blob. And then you can have user UI controls to manipulate that. And like you said, it's typically just edit this key to be false instead of true or something like that. And the user can only do one thing at a time. And I think of the AI as you can give it some high level instruction and then it can go execute a sequence of commands, like a cascade of of things which are like turning lots of the knobs.

• 8:06Speaker 01

  So that's kind of how I think of it's more like, yeah, I guess the user's mental model can be fuzzier, but ultimately it still maps all the way down to what are the knobs that it's those turning? It's just that it's okay, maybe the user has like a fuzzier understanding of it. And then it's going and doing like 10 things for you. And then it still works in the same way. And and also it introduces this new challenge of explaining to the user what happened, especially if it's like a complex state.

• 8:28Dan Shipper?

  Yeah. How are you thinking about that? Or what are you, what have you found in in doing that?

• 8:31Speaker 01

  I I think about it like, what is the thing that's changing? And what is the most efficient, understandable way to present that? One that we've explored in the past is like asking it to do edits across multiple documents. And then, you know, we essentially just came up, it's, I mean, nothing, nothing too crazy. But, you know, it's UX where it shows you like, like groups it by the pages and then it shows you the diff across each one. And then you can go zoom in and look at the ones that that you care about. But it's pretty tough. Yeah, I would say it's just a fundamentally hard problem if it's doing something complicated and then explaining the complicated thing. I mean, yeah, it's just hard.

• 9:07Dan Shipper?

  Yeah, that makes sense. Or, and you know, if it's explaining like, one of the things I find is even if you get it to summarize what it did, the summaries are so high level. It's saying a lot without saying anything at all. And getting it to be concrete enough, but not too detailed is like a really like difficult challenge for some reason.

• 9:25Speaker 01

  Yeah, yeah.

• 9:27Speaker 01

  I think that's probably just fundamentally hard. I think especially if the thing is complicated. Like, you're not going to fully understand it maybe until you read the whole thing. And then, I think depending on the use case, though, you probably can go pretty far with calibrating that prompt to the appropriate, like, level of granularity. So I, I think you can get pretty far, a- a- at least calibrating it.

• 9:47Speaker 01

  I guess there's, maybe if you were to pick it apart, there's the problem of, is it summarizing at the appropriate level of granularity? And maybe it's just like, like, missing an important detail that, that you actually wanted to, you know, to be included. And then maybe there's like, the more fundamental problem, you know? You do want to reduce the information, and so it makes sense to draw some things.

• 10:07Dan Shipper?

  I want to go back to the, like, relational database point. I, I think for me, the way that I've thought about, or the mental model I have for relational databases, and you may have a different one, is it's more effective to have a schema for a relational database if you know what data is going to be used for. So, for example, like, it's easier to have a relational database for a CRM where it's like, I know I'm going to use it to keep track of customers, so I have a customer table. And what's interesting about, um, embeddings is they are able to capture so many more dimensions of what piece of information is relevant to that you can use it for storing information in situations where you don't know what the information is going to be used for in the future. And I'm curious about, like, how, obviously, so far with Notion, you've had to solve using a relational database to

• 10:52Dan Shipper?

  store information that you don't know what it's going to be used for. And I'm curious, like, how you think embeddings changes that picture, if at all?

• 10:58Speaker 01

  That's a really good question. I think, to your first point, I'll, I'll first address the point of it's hard to design it when you don't know what it's useful for. I think that's a really good pointer to, don't design schemas that you don't know what they're for yet. This is something that I've been playing around with, AI helping you design schemas. We've tried versions where it just comes up with all the properties you might want. It can come with a lot of things and, like, not all of them are, are useful. I, I've had a lot more success with, like, only give the minimal schema that's required for the actual task that, that these, like, like, currently carries about. Each property should have a purpose that just really, like, focuses the task and make it more effective. So I think that's one point there. In terms of how I think about, like, embeddings versus, like, deterministic querying, I think you're getting at. I just think of them as two different tools that you have in your toolbox.

• 11:41Speaker 01

  Ide- ideally, you have both, and you could even maybe... Combine them. So this is only something that we're working on a lot is Q and A over databases. And when do you turn to like a deterministic like SQL-like query and when do you turn to an embedding? I think it really just depends on the question and sometimes you want one, sometimes you want the other. And a lot of it is like a performance costs, like latency concern. Like you could just make everything embeddings or I mean, you could just like map a model over every row of the database every time. And then you don't need, you know, like no embeddings and no SQL either, right? Everything's unstructured, but I think that would be undesirable from like a performance perspective and also.

• 12:21Speaker 01

  Yeah, it wouldn't be like fully deterministic accurate, which I think people probably care about. If you're like, how many sales did we do last quarter? Do you really want the model like?

• 12:29Dan Shipper?

  It can make it up a little bit, but it'll get close to right, but it won't be actually right. Yeah.

• 12:33Speaker 01

  It seems a bit scary. So it depends on the question. If I'm asking, let's say you have like a customer database or something and I'm like, how many sales last quarter? Yeah, I, I really do want to call them like amount to sum over it. But then if I'm saying like, do we have any customers in the entertainment space or something? Maybe I want it to be flexible on that. So yeah, I just think of it like, these are just tools in the toolbox and you want both. And then the, the challenges in defining that routing mapping layer of like figuring out which tool is best for the job and combine them. And then you know, present to the user with the best result.

• 13:05Dan Shipper?

  You very famously, a couple of years, I think into Notion's life, went to Kyoto, stripped it all down and pivoted the company. And it became what Notion is today. And I'm curious, let's just assume as a thought experiment that you're going to have a second Kyoto. You're going to go strip away everything that Notion currently is and rebuild it with AI.

• 13:29Dan Shipper?

  How would you do it? Or how would you think about it from scratch? What would you do differently now that these tools are here?

• 13:34Speaker 01

  Yeah, that's kind of how I operate.

• 13:37Dan Shipper?

  That's great.

• 13:39Speaker 01

  When I'm thinking of like a new project, I, I like to be pretty unencumbered by the way things work. But then the magic is also about taking this unencumbered crazy idea, but then also ideally you want like an incremental roadmap for everything. So I think there's a lot of, there's a lot of like details in there. I don't just want to make crazy ideas. I want to actually like ship stuff incrementally, but then still get to the crazy place. The really key exciting thing to me is this thinking box where.

• 14:06Speaker 01

  There's plenty of knowledge work tasks that people don't really want to be doing, or that are too expensive for them to do because you have to hire like the humans to do it. Can we automate that stuff? One big principle would be like probably there's less humans touching the database and the AI should be managing it for you. We're talking about customers, let's say you have like a CRM style thing. Ideally, you never need to update any of the fields, right? If the deal closes, it should know the amount based on your email. If someone talks in Slack about how the deal is at risk, that should be in the structure somewhere. You shouldn't need to update stuff. And I think in the AI world, like the database becomes more of an implementation detail. And hopefully the user interacts more with the processed outputs of it rather than the database,

• 14:51Speaker 01

  the raw database itself. So like maybe for sales, you really care about a daily progress bar or seeing something about the productivity of your of your salespeople or something like that. And those should all be just presented to you directly. And then the database is just this background thing. That's it, implementing thing you care about.

• 15:06Dan Shipper?

  I love that, especially the first point about you shouldn't have to interact with the database. What it reminds me of is there's this constant thing with Notion and with any other kind of tool like this where especially if you're doing it inside of a company, you're always like, is this up to date? There's 5% of things that like make it into Notion, but then there's 95% that's like completely like unwritten. And I think companies operate better when more of that stuff is legible and more of that stuff is written down. And it's like, it's updated. I've always had this thing that's like, I think companies should have librarians that are like just responsible for that. Having worked in a big company, I was like the guy for a particular product and it had a huge sales force. And I had written all these documents about like how the product should be sold and like what the details are and whatever. My previous company was a co-browsing company and I sold it.

• 15:52Dan Shipper?

  And I was the co-brows guy internally at Pega, this big public enterprise software company. And even though I had written everything down, all the salespeople were just like, you're the co-brows guy, right? On chat, what about this question? I'd be like, see my doc. But I think one, discoverability was like really poor for them. And then two, there's always that thing in your head where you're like, is this up to date? And it seems like what you're saying is that there's an opportunity now without someone having to do it, to take a lot of that stuff that would ordinarily not be written down and like get it into a format where it's recorded for other people to use. Is that kind of what you're saying?

• 16:24Speaker 01

  Yeah, we're definitely excited about that as a use case.

• 16:28Speaker 01

  The the current Q and A with Notion and third-party connectors, you can get, at least if the salesperson goes and asks the question, they can ask the AI. That's pretty cool. But then, yeah, I think a lot of the times you didn't write the doc in the first place. And then, once you write it, you want to maintain it. I think, yeah, those are both like the really interesting use cases that I think, yeah, be super exciting. A fun thing about these thinking boxes is now you can, you can treat a knowledge base like a database. Where the operations on it can be semantic. I think that's pretty exciting. Yeah. Thinking about how can pieces of information conflict with each other and how would you resolve that? Yeah, that's super interesting.

• 17:02Dan Shipper?

  I love the word thinking box and it makes me think, what is thinking? What have you seen the, what do you think the boundaries are of what that thinking box can do versus not do?

• 17:12Speaker 01

  Yeah, I don't think there's that many boundary. I think the instruction is kind of complete in a way already. It's more that just the models that kind of suck still. You give it vision, like assuming it's multimodal, right?

• 17:25Speaker 01

  There's not much more to it, I think. Maybe there's like robot actuation commands, but those those can be represented in the same model too. I kind of feel like the abstraction is already complete. I feel like the really critical thing is it has some context and has tools that it can use. And then those tools produce observations and then you just loop on that. That's an agent that that can do anything. Assuming the model, it actually works, which they don't yet. Depends on the use case, but.

• 17:51Dan Shipper?

  I don't know if you saw, but Anthropic dropped like a new computer use model for Claude the other day. And one of the things that they touted in that release is that you don't actually have to explicitly identify tools. And instead, like the model just understands that there's a browser in front of it that has certain things and the computer has applications that allow it to do things. And so the tools are like implicit rather than like explicit. What do you think are the trade-offs there? Do you think explicit tools are like actually what's best or should it be implicit and where?

• 18:23Speaker 01

  Yeah, super excited to see that. I mean, on a technical level, it's still tools. It's just the tools are like click this coordinate and like type this.

• 18:29Dan Shipper?

  I guess that's true.

• 18:30Speaker 01

  So it's still implemented as tools. Yeah, it's just that the click tool is like pretty powerful.

• 18:39Speaker 01

  Click and type, you can do a lot of stuff, right? And then the observations, see what happened afterwards. Yeah, I was super interested to see that. It's something I've been expecting to start working and it seems like it doesn't quite work that well yet, but you know, early signs and it's cool that they're showing it to the world. The way I'm thinking about that is.

• 18:56Speaker 01

  Given the task, you want to give the AI the most convenient way to do the task of possible. So there's some quality constraint around kind of do it. And then there's some like performance latency constraint. And then maybe something around like how users can observe and control it. So I think computer use, it's very open ended. Like you said, at least currently the quality seems much lower than if you were to give it like a more specialized tool. The latency is very bad, like super slow. You could get much better results by giving it like a code API, for example. Like if your goal is to, um, your goal is to like, like download a recipe. You can have it go Google search and find the recipe.

• 19:37Speaker 01

  Or if you give it a recipe search API and it gives it credit, that's going to be done in like less than a second. And then there's the controllability thing, which is pretty important, I think, especially if it's doing something autonomous for you. It doesn't seem that interesting to me to have it control your computer like while you're watching. The interesting thing is like you ask it to do something and then it, then it goes and comes back to you when it's done and I can go, I can go do something else.

• 20:00Dan Shipper?

  It has its own computer.

• 20:01Speaker 01

  It has its own computer, exactly. And I want to be able to control what it has access to. So I think that's pretty important. And if you're giving it a computer, it's pretty open ended. So we need to develop like some of those controls around it, but I'm excited about it. I think ultimately the way I think about it is it's just another tool in the toolbox. And the ultimate answer probably looks like a mixture of when you can give it an API, that's much better. And that will always be better. And then when you can't, it's nice to have this escape hatch where it's like do stuff on a computer. That makes a lot of sense.

• 20:33Dan Shipper?

  I think that's totally right. For tasks where they're repeated and you know they're going to happen inside your, your application, like having a specific API that just does them like really quick is great. Just like, you know, you have muscle memory for figuring out how to pick up a glass. And maybe there's like a tool in your head that is really tuned for like picking up a glass to drink. And then you fall back to this like slower, more open ended thing that can do much more for tasks where like the tools that you are more specific can't like, can't handle.

• 21:04Speaker 01

  Yep. Yeah. I think another angle too is interesting is the. Market dynamics angle, I think we might start seeing people shutting down or not wanting that. There's gonna be a race where, you know, people who use the computer using agent is gonna want it to access all their stuff. And then the companies that manage those tools might not want that if it's like a third party. There's already a whole industry around like preventing bots from accessing websites and stuff. So this is like, now bots are useful for real work. So what are we gonna do about that? I think, I'm not sure how that's gonna play out. I think that'll be really interesting.

• 21:37Dan Shipper?

  What'll be your guess?

• 21:39Speaker 01

  I think probably it's inevitable that, well, I think people are definitely gonna wanna do this and they're gonna have legitimate reasons to do so. Unlike in the past, where maybe it's like scamming or hacking, like now it's no, I'm actually trying to perform this task. I'm paying for your software, so you should let me do that. I think that makes sense. I could see a world where I think probably the ideal outcome is something where like everyone allows that, but then they get paid for it in some way. I don't, I don't know what the shape of that is exactly, but, but yeah, I think that's ideal. If you make some software that's valuable and people are using it in this way, somehow value should accrue to you.

• 22:19Dan Shipper?

  Do you think we're going to see a world where there's interfaces that are specifically for like verified humans? And then L O M friendly, I guess is an API, but it's a, it's something different from a traditional API. But like L O M interface is built specifically for L O Ms.

• 22:34Speaker 01

  Yeah, my job description is like, design those. There's all these like, like quirks, quirks will go away over time. Yeah, I saw someone tweet the other day that you're gonna have an alternate form of your website. That's just, just plain HTML with like divs and buttons.

• 22:48Dan Shipper?

  Back to the nineties.

• 22:49Speaker 01

  Yeah, yeah, yeah. I love that idea. I think it's a tricky race because on the one hand, the current models are not good at many things. And you do need to design those custom things. But then as the models get better, maybe you need a bit less of that. And, and maybe they can also just like, like build their own. Eventually the model can just build its own scaffolding, right? You give it something and it's like, all right, I'm gonna make a whole Python code repo. That it may be inside of that, it's going to figure out this problem that I was just like saying where it's like, all right, which things can I use code for? That's better. And then which things do I need to call out to some like browser, like headless browser for? That's way less ideal, but I'll do it if I need to. And then, yeah, I feel like the ultimate abstraction just closes over all this.

• 23:32Speaker 01

  So as a human, just whether it's using a code API or browser is like in repetition detail.

• 23:38Dan Shipper?

  You said like part of your job is figuring out good interfaces for LLMs. What are the current properties of a good LLM interface?

• 23:46Speaker 01

  It's super fun. Okay. So yeah, there's a bunch of principles. So one is that you want to align to things the model's been trained on as much as possible.

• 23:56Dan Shipper?

  What does that mean?

• 23:57Speaker 01

  So just as a concrete example, one way we've tried representing notion pages as this XML tree, which is much more faithful to the way it's persisted, but the models just want to speak markdown.

• 24:09Dan Shipper?

  Interesting.

• 24:10Speaker 01

  And

• 24:11Dan Shipper?

  But they prompted in XML often.

• 24:12Speaker 01

  So it's prompted in XML, but typically the XML that's used in prompting is pretty simple. It's like wrapping this tag. It's very good at that. The, uh, the format that we came up with is like a much more complicated form of XML where like notion blocks are like in, in a tree and there's many layers of nesting and there's like rules about which blocks can contain other blocks. And it's actually like describing the spec is like, there's like several thousand tokens. And then the trouble with that is that the models can do it, but you're actually like harming its ability on other tasks. And I guess our mental model is as it's making the tokens, it has to attend to all your complex instructions about the formatting. And also like the reasoning of like, answer this question of something.

• 24:52Dan Shipper?

  Yeah.

• 24:52Speaker 01

  And it definitely makes it worse. It's better if you can speak the language that, that it knows how to do. And, and that's just a matter of what the company trained on. And so like markdown is a good example where everyone just trains on markdown. So it's just really good at that. You don't need to give any, any extra instructions for that.

• 25:05Dan Shipper?

  Even if markdown has like a more complicated structure, are you flattening the tree into like some more linear?

• 25:11Speaker 01

  Well, I think, I mean, markdown is also simpler. It's just a very simple, kind of like glossy language with very little ways it can fail. That's one class of things, like aligning to, um, what the model's already know. Another one is around, you want the structure of your output.

• 25:32Speaker 01

  To be like as simple as possible for the output that you need. I think that's really key, like, like for any like, like formatting. It depends on what you're doing. You want to like really go hard on make it as simple as possible while still doing the task that I care about.

• 25:46Dan Shipper?

  What would be an example of a time when you learned that or that stood out to you?

• 25:50Speaker 01

  I think the XML structure is like applies there too, like. Our, when we first started doing it, our, our original principle was like, oh, let's just like perfectly map. The, the way is actually persistent and displayed to the user. That's ideal, right? There's no lossiness at all. Then there's all these little like quirks about just little things that gets wrong. And it was often easier to just simplify it. And even if it's somewhat lossy, it's worth it because at least you can control that. Whereas if it, if it's too hard for the model, it's, you know, it's the end of the line. There's kind of the basics of describe your task as simple as possible. Use like few shot examples.

• 26:28Speaker 01

  Another class of learnings that's kind of interesting is if you're working on the prompt and you notice some class of issues. The way I think about it is that my first line of defense is I want to try to make that class of issues be impossible. In the system or validation around it, that's like the ideal. And then if, if I can't think of a way to do that, then I'm, I'm going to try to like make the prompt better. Add an example or like change instructions, something like that. I think that's a really fun one. And it's, yeah, it really depends on the task, but I'll give you one example. In the example, I make a fake test data. And then there's one prompt that is describing the kind of fake test data that you want to make. And there's another prompt which actually writes out the fake test data in detail.

• 27:10Speaker 01

  And one little constraint with it is I don't want it to generate too much just because it'll be too many tokens, take too long. And so I have this constraint of i- in your description, it should only make up to 10 like records or something like that. But sometimes I just wouldn't follow that instruction. That's annoying. And one little trick there is, oh, actually while it's generating the descriptions, ask it to estimate. The number of records that'll be needed for this, for this test data. And just like forcing it to output that one aligns it much better because it's like, you know, not just that it's in the instructions and it can maybe ignore that, but like it had to actually produce a number. And then also if a number produces too high, I can actually just throw an error and have it try again.

• 27:51Dan Shipper?

  That's really interesting. I think there's two principles packed in there. One is the thing you said, which is making a certain class of error impossible, which maybe it's like changing your prompt, or maybe it's not even calling the AI in a situation where that error might come up or something like that, which is really interesting to me. The other one is I think you're implicitly doing a little bit of like a chain of thought thing where by asking it to like output how many examples of things are necessary, you're like, you're aligning it in the same way that chain of thought works.

• 28:17Speaker 01

  Yep. Yeah. Yeah. Doing structured chains of thought is like really useful. Yeah. A specific to your task. Yeah.

• 28:22Dan Shipper?

  I think you all probably have like one of the most scaled AI applications in the world right now. And one of the things I think is really interesting to dig into is previously software was deterministic. And now it's very stochastic. It's much more squishy. And especially at scale, like releasing squishy software to the world is like scary. And I'm curious about what you learned about doing that, what you learned about good evals, all that kind of stuff.

• 28:49Speaker 01

  Yeah, it's really annoying. Yeah. I mean, prior to 2022, I didn't, I never did any AI stuff really, besides like taking some classes in college. And yeah, I definitely miss the days of like, you know, I can like write a QA doc and write some tests and like, it all kind of works. I have a good mental model. It's not going to fail in all these cases. I think the AI thing is so hard because it can fail in some cases, but then there's this like additional meta problem. You don't even know, like you might not even know the cases where it can fail. And usually what happens is as you're kind of ratcheting on a prompt, you end up discovering more and more of these. And sometimes you discover like really major ones. After you had a huge eval set and then you discover some new ones, like, oh, man, this totally breaks it. So even just discovering the distribution of

• 29:32Speaker 01

  the possible errors is like really hard. And I've definitely been led astray multiple times of thinking that I'd solved it and then find this whole new class of errors that are like really hard to solve. This is really hard for evals. The only thing about that is you've got deterministic evals, then you've got like not deterministic evals. You know, if it's possible to make a deterministic eval, that's great. And I love to design workflows such that there's some like classifier elements within there. Producing an enum or yes, no value or something like that. Those are great because they're super easy to eval. Because you can just like collect the data set of input and then the correct output and then just get a score. So I'm always trying to, yeah, if there's some complex workflow, I I love to come up with like classifiers within there. That's one big strategy.

• 30:15Speaker 01

  And then there's non deterministic evals.

• 30:17Dan Shipper?

  Is the vibe of this correct?

• 30:19Speaker 01

  Yeah, yeah. So just like using an AI to

• 30:21Dan Shipper?

  Determine.

• 30:22Speaker 01

  Evaluate something.

• 30:22Dan Shipper?

  Yeah.

• 30:23Speaker 01

  The trouble is if you have an AI evaluating, you need to evaluate your eval now.

• 30:30Speaker 01

  If running a prompt is hard, now you have a whole, a whole other one. I've found that you have to be pretty careful. I I've definitely learned to be pretty cautious about these. It's easy to come up with an idea for a model graded eval. That sounds good, but then like in practice, you try making it and then you slog on it for a while. It's like, oh yeah, I'm discovering a hundred cases of this thing. So I think I I found that they work best when the thing you're trying to evaluate is quite targeted and you and you can describe it very clearly. You want to make the shape of your task, evaluation task, such that like for the model you're using to run that.

• 31:06Speaker 01

  The model is extremely good at the task. To the point where you can actually trust it and you don't need to like spend a bunch of time evaluating that. So, yeah, you want it to be like very clear, like narrow, really helps. And then like using the appropriate model for it. But yeah, that that's really hard. And then evaluation general is like really hard. Yeah. Another one is you want to definitely have a really solid loop around logging and like collecting data sets, collecting issues and labeling them. And then optimizing the loop that lets you improve a prompt and make sure that the that they aren't regressing on like the previous examples. So yeah, there's a whole.

• 31:42Speaker 01

  There's yeah, there's a lot of stuff to do in there, but yeah, and it's really annoying.

• 31:44Dan Shipper?

  Tell me more about that that part, like the using feedback or whatever to improve a prompt and then make sure you're not regressing. How are you doing that?

• 31:53Speaker 01

  Yeah, I think it really just boils down to. Having really good evals that are appropriate to the prompt and then having good data sets that capture the distribution of errors you care about. And then making it easy to run the evals and flag regressions. It's kind of simple, but it's just, it's annoying to set up and you have lots of prompts, you have to do it like many times. I found you want to have like solid, like standardized ways to do this.

• 32:21Dan Shipper?

  Are you using all homegrown tools for this or are you using like, you know, Anthropic has an evals library, OpenAI has one, there's a lot of open source ones.

• 32:28Speaker 01

  We use Brain Trust for like exploring data sets and running evals, but all the actual evals like we write ourselves.

• 32:35Dan Shipper?

  One of the things you said earlier that struck me is like this idea of exploring the distribution of errors you can get. And sometimes you'll have a big eval set and then you'll find something in the distribution where you're like, it breaks everything. What have you learned about doing that exploration to minimize the chances you find something totally unexpected?

• 32:51Speaker 01

  Yeah. I don't try to do evals too early in a project. I, I, I think you can actually go too hard as well in, in, in the other direction. When I'm starting a new thing, I feel like you want to start with more of a vibe check and really be flexible about how the task is structured.

• 33:11Speaker 01

  Because I found that especially early on, there's a lot of returns to just like, like changing the structure of the flow. And if, if you spend too much time collecting data sets, you're just going to trip yourself up a lot. So I feel like there's like a mode switch when you're like, all right, let's actually productionize this. Where you want to switch to actually intensively like finding issues and yeah, I think it helps to have. It helps to have like, like data labelers dedicated to that. Trying to understand as best you can, how it will actually be used and then make your data set like map that like ultimately that's the game. And then after you deploy it, then the game is around actually flagging those examples and saving them.

• 33:48Dan Shipper?

  Right. How do you think about, for example, when I interact with Notion AI, you're going to go find a bunch of presumably like embedded text, you know, from my Notion and then put it in the context and that's going to help you answer the question. How do you think about how much to pack the context? Are you trying to get all the information like, even if it's not as relevant in there, or are you like being more selective about it and, and, and why?

• 34:11Speaker 01

  Yeah, it's an empirical question for sure. So I think it's changing all the time. Like even just this year, it's changed so much because of the models have all gotten pretty long context, but. It's, I don't know, like sometimes people make a tweet, it's like rag is dead or something. But it's like, I mean, there's also like a latency cost concern.

• 34:32Dan Shipper?

  And there's a like attention is different, you know, in the middle of the context versus.

• 34:36Speaker 01

  Yeah, yeah, it also just doesn't work yet. Yeah. I think we're definitely still strongly in the world where like you want to limit the context and and if you can remove irrelevant stuff, there's definitely returns to that. Even with the latest models, a- and if you ask openai on topic, they'll tell you that as well. So I don't know, it's hard. Yeah, it's really an appropriate question, though. I'm I'm hoping the attention gets way better.

• 34:58Dan Shipper?

  Yeah.

• 34:58Speaker 01

  And the contexts get longer and it gets faster to process them and like caching is better. And then then maybe I I like that world because then we can worry less about brewing irrelevant stuff that makes our job easier. Uh, yeah, I'm very bullish on being able to do that. I feel like we're we're not quite there yet. And we we've certainly expanded of of the context that we show as as models have got like. Original Q and A was on the original G P T four, which is like eight K tokens. So ob- obviously, we had to be super constrained on that. We can only show a few thousand tokens, especially if you want to do like multi turn and have it just uh uh for good things. Obviously, now they're much longer, so we can show more, but it but yeah, it's still definitely constrained by the quality and and the cost latency.

• 35:41Dan Shipper?

  What are you personally coding with? Are you using Cursor, uh, AIs, are you using like Claude three five, are you using O one? What's your like workflow or stack?

• 35:49Speaker 01

  I use Cursor, Cursor really good. I I think I like about Cursor is just the the autocomplete they built is really good. It's like way better than than get a copilot.

• 35:56Dan Shipper?

  Are you using Composer?

• 35:58Speaker 01

  So it's the it's just the one where it's the tab completion where it can do like arbitrary edits, is that?

• 36:02Dan Shipper?

  Yeah, Composer is like a it's like a window that pops out and it.

• 36:05Speaker 01

  Oh, so like the the command K.

• 36:07Dan Shipper?

  Yeah, a command I.

• 36:08Speaker 01

  A command I. Oh, I don't think I've.

• 36:09Dan Shipper?

  Oh, you gotta try it. It's really cool.

• 36:10Speaker 01

  So what does that do is different than the?

• 36:12Dan Shipper?

  Cursor Composer is much better at doing multi file edits. It's a little bit more agentic.

• 36:16Speaker 01

  Yeah, I don't think I've actually even tried that. I I've used the command K and then I've used the command I.

• 36:20Dan Shipper?

  Yeah.

• 36:20Speaker 01

  Or the no, the command.

• 36:21Dan Shipper?

  The sidebar chat. Yeah, yeah. I think you have to do less because in the sidebar chat, you have to like go scroll through each step and then press apply and it takes a while. And Composer like does all of that a little bit more automatically.

• 36:31Speaker 01

  Yeah, I haven't tried much doing multi file edits. I guess my mental model is it's probably not that good at that, but I should try it. I I feel like I don't I feel like I don't understand if it's good or not, which is which I should. My typical workflow is like, yeah, autocomplete all the time. Obviously, that's just like Emily there. And then I'll ask for code and sometimes I'll use a notion or Claude or like T T interface as well. And then my model there is more about. I have a specific function I wanted to write that that's typically the abstraction level that I ask it to code at. Anything beyond that, I don't know, it's just. No one's really cracked the multi-step coding agent thing yet, but it can write pretty good functions if you give it good instructions and context. And then I also use it for, I've tried using the retrievable over code, which

• 37:13Speaker 01

  it's helped me a few times. We have a pretty big code base and so I can be like, what's the function that does this thing?

• 37:17Dan Shipper?

  Yeah, yeah, yeah. Is that in cursor retrieval over code or is it like somewhere else?

• 37:21Speaker 01

  It's in cursor. Yeah, that's part of like a sidebar.

• 37:23Dan Shipper?

  I see, I see. That's really cool. And models wise, are you like using Sonnet or using O1?

• 37:30Speaker 01

  Sonnet is my day-to-day workhorse for sure. O1, yeah, I've been playing with it a bunch. I've had some success with O1 and O1 Mini on coding stuff, but it feels like it's not that much better than Sonnet at coding and then it's slow. And it feels like it's also weird to prompt right now. I've definitely gotten like weird results from it, especially when you like give it a bunch of context. It's really, they trained it a lot on these math and coding puzzles, which have like very little context and a lot of reasoning. And it's really good at that, but only if you can put it in the shape of that. And I, and I found it's a bit finicky. So, yeah, I'm, I'm really excited about the paradigm though. And I'm curious to see when they produce the final model that's been like, like fully trained on a bigger distribution of inputs.

• 38:14Dan Shipper?

  That's what I was gonna ask you about is what do you think about scaling inference compute as a new paradigm and where do you think it's going?

• 38:20Speaker 01

  Yeah, it's super exciting. It makes a lot of sense intuitively. I, I think I, I was surprised when it first came out. Or I, I think I was initially underwhelmed because I intuitively had thought that like, oh, wouldn't it be better if it like reasoned in the latent space more? Wouldn't that be more powerful? But then I, I was talking to some friends about it and actually it makes a lot of sense for it to be language. Because the model has all this prior over language already and the reasoning over language. And so it makes a lot of sense. It's like the dumbest possible thing you could do is just like, let's, you know, think more over the like use the language and then, and then my understanding is like, well, the tricky part is just like making the RL work. And there's like all these details like OpenAI has figured it out, but no one else has yet, but.

• 39:04Speaker 01

  Yeah, I feel pretty excited about it. I think I was pretty impressed by the graph of like increase, increase scaling compute. It gets smarter and it makes a lot of sense. I feel like the thing that's going to be the real unlock, though, is putting tools in the chain of thought. And that's where I guess I think really interesting, because right now, it's like the shape of the problem you need to give it is like, you need to give it all the context it needs upfront, and then it can think a lot about that. And it produces answers. But I think there's actually not that many things that are like, I mean, I'm sure there are plenty of things that are like that, but it's actually kind of a blunt or a specific like a tool, it feels like. And yeah, the big unlock that I'm excited for is yeah, putting tools in there and then doing reinforcement learning over that.

• 39:48Speaker 01

  It feels really interesting. That's kind of when like agents are going to actually work is when, when it can do that. You can give it some like high level tasks, give it these tools, like maybe it's like a browser and then and then just enter a hard loop of think a bunch, use some tools, see the outputs, think more and then keep looping on that. I feel like if that, you know, that can solve a lot of stuff. And then whoever can figure out that kind of like long horizon, like reinforcement over that seems.

• 40:13Dan Shipper?

  Yeah, I think, I think tool use is coming. I was at Dev Day. I think they said it's coming before the end of the year, which would be pretty cool.

• 40:19Speaker 01

  It makes sense. It's like the obvious thing they should do, but it's probably hard to figure it out.

• 40:21Dan Shipper?

  So you talked about having them be trained on these math and math reasoning problems. What do you think about zooming models into those kinds of problems versus like? I don't know. There's endless amounts of like other things you could have it do, and they're trying to make it reason. Do you think that allows it to come up with new things or be creative? If you let one of these things run for, let's say you could just scale the inference time infinitely, would it come up with new things? Or do you think that's a different type of thinking that requires a different type of like training loop?

• 40:54Speaker 01

  Yeah.

• 40:56Speaker 01

  I think the interesting thing that, obviously, I don't know what OpenAI is doing, but like, by a speculation. The thing that it seems like they're doing is that, like, the place where you can get this reinforcement to work is when the results can be verified in some way. And so you're like touching reality in some way. Like, it's not that the model is making stuff up and the human's like, oh, this one's slightly better. It's like, you're writing some code and there's a unit test and it has to pass, or there's a math problem and there's an answer to it. For this one reasoning step, there's like a correct, it's a correct thing to think. I think that's really interesting because it lets them scale up the training. And then I think you can do that.

• 41:37Speaker 01

  I think with that as a tool, you can now discover new things just by, by like mining it, right? You can, if you know how to verify some problem, you can spend lots of time thinking about the answer and keep going until it's correct. So I think in that domain, you can be really creative of just like creative ways to solve this. I, I think you're maybe pointing at something a bit fuzzier, though, of like, maybe an aesthetic creativity or something like that. Yeah, I don't, that feels like a different thing. And it feels like not the direction that the companies are going now. It feels like they're really doubling down on these like verifiable things.

• 42:14Dan Shipper?

  What's really interesting to me about that is like, um, when you look at how mathematicians or scientists talk about coming up with new theories, there's usually like a big aesthetic component, like the idea of beauty or simplicity or whatever is like driving, driving them. And when you're coming up with new theories, they're often not verifiable when you first come up with them. So like an example is, we didn't verify a lot of relativity for a long time after Einstein came up with it. Do you worry that focusing solely on training loops where each step can be verified, or at least the outcome can be verified, or some of the thinking steps can be verified, like limits the ability of these things to think in ways that are valuable?

• 42:53Speaker 01

  Yeah. I think to some extent, it's all they can do, probably, because if we've exhausted all the human data and you need to make more data and you want it to be good. Yeah, you have to verify. So some of that stuff might be off limits until we like, I don't know, though. I guess one way to think about it is when Einstein's coming out with his theory, he's doing some verification, presumably he was writing down some equations, like making sure they work out. But then he also has this built in aesthetic, which is a bit fuzzier of what's a nice theory. But that was trained on his previous learning on like physics and all the stuff he learned in his life. And I wouldn't be surprised if OpenAI makes some model and they train it on like all the physics and then they have it going mining to produce new physics and new math.

• 43:35Speaker 01

  I wouldn't be surprised if I actually developed something analogous to that aesthetic. You keep mining and it discovers 10,000 new theorems that are all proven to be correct. And then you start asking it higher level questions about math. I wouldn't be surprised at all if it had like some analogous thing going on there.

• 43:50Dan Shipper?

  Yeah, I guess if it's if the theorems that it's seeing are representative of that aesthetic, the aesthetic is in there, even if we're not really even talking about it.

• 43:57Speaker 01

  Yeah, yeah, you would hope that the aesthetic is pointing to the truth in some way and it's coming from the truth. The simpler theory is nicer because it's more likely to be correct. And to the extent that those are true, I would expect the model to learn them too.

• 44:10Dan Shipper?

  Which is, that's a big assumption.

• 44:12Speaker 01

  Yeah, I don't know. Yeah, yeah, but then.

• 44:15Dan Shipper?

  It's definitely more likely to be useful or beautiful.

• 44:18Speaker 01

  Yeah, there's definitely, yeah, like humans have additional constraints. And in some ways, the aesthetic is like a computational shortcut. The human go player can prune out a bunch of classes of moves that maybe the AI would like reinforce more. Maybe Einstein had to do that in order to just due to the limits of like human cognition. Maybe the only way Einstein could have done it is to have like aesthetic shortcuts.

• 44:38Dan Shipper?

  Yeah, I think the the beauty of LLMs is like they get you the thinking machine without having to reinforce. Like they do the same kind of aesthetic shortcut or like intuitive thinking where early AI attempts, it was like, okay, to solve chess like deep blue, we're just going through the branching tree. But then you get to these, like when you're going outside of chess, then you get to these problems where the branching tree is so computationally expensive to traverse that it doesn't work anymore. Models have figured out how to set the frame of possibilities correctly so that they don't run into the computation.

• 45:09Speaker 01

  Yeah, I think it's both. If it's producing a new theorem, you probably have a try many times. But then it can also have these like thinking shortcuts. So I guess you get best of both worlds. Like you couldn't get a human to do that much work.

• 45:19Dan Shipper?

  Yeah, yeah, yeah. You're trying many different times and you're examining different smaller parts of the tree, but you're not brute forcing the entire tree because the theorem that produces.

• 45:27Speaker 01

  Yeah, there's probably still shortcuts. It's not gonna, it's not gonna do something that's obviously doesn't make sense. But then maybe it's just a really hard task and there's like many things that could make sense.

• 45:34Dan Shipper?

  Yeah. One of the things that, and I could only find one other interview with you. So this is coming from that interview, but like one of the things that you said in that interview is that one of your core philosophies is if you can build something that you want, that's like a great place to start.

• 45:50Dan Shipper?

  Something builds something that you're going to use for yourself. You were the first few years of Notion, all that kind of stuff. And I'm curious if you have anything that you feel like you're building for yourself right now or things like that you want right now.

• 46:01Speaker 01

  Yeah, I would say that principle, it's not universal because obviously if you're designing, if you're making something for like diabetes like patients and you have diabetes, like but someone needs to make that. Maybe not only people with diabetes.

• 46:13Dan Shipper?

  Yeah, it's helpful if you have it.

• 46:14Speaker 01

  Yeah, so it's a luxury. I think of it as a wonderful luxury that if you're deciding what to do with your life, it's nice to pick one of those things. And I really cherish that. It's a thing that we can use every day and I like lean into it and try to compound on it. What's something that I'm doing? It ties back to a lot of the AI stuff that we've talked about. I'm, I'm excited about, there's like cumbersome tasks that me and people around me are doing all the time that, you know, we can not do those things. Yeah, within the realm of like knowledge, we, I guess we already talked about some of them, but like keeping documentation up to date. Databases, not having to do manual data entry and keeping things up to date. That's super annoying.

• 46:53Speaker 01

  There's a common class of tasks around like rolling something up and like summarizing things from a database or from docs or something like that. You have your like weekly update. And in, especially in a bigger organization, there's like many levels of weekly update and it's super annoying. It's a huge use of everyone's time. Yeah. Why are we doing that? The data is there, but the docs are written and the Slack channels. Responded to and it's all there and then you could write an update at any level of granularity that they care about if you just have the right context and the instructions.

• 47:21Dan Shipper?

  I totally feel that. We have a product studio at Every, so we we incubate little products and we everything that we incubate, we use ourselves, which I think is great. And I think one of the really cool things about AI is it means that the low hanging fruit hasn't been picked yet. So like you can make something for yourself and it's not like there's 10 years of tech nerds who have made it before, like it's all new. So everything gets made. And one of the things we're making that's been delightful, that that kind of what you said reminds me of is every morning now I get like a little podcast of meetings or discords, we use Discord to chat, things that I missed. And it's the Notebook LM stuff where it's like the sort of NPR fresh air type thing. And it's two hosts talking about what happened and I can get caught up while I'm doing dishes or whatever. It's really fun. I'm super excited for that kind of thing.

• 48:06Speaker 01

  I actually, I just wrote an idea down that's very similar to that the other day. That's so fun. Yeah, I would love to see that. I definitely vibe a lot with the feeling of openness in in this space. It's really exciting. One, one thing I really appreciate as well, uh, that really keeps me going a lot is so many things haven't been tried. And also, it feels like the technology is so overpowered in some ways that I I found it very common that I could just come up with a random, uh, technical idea. Just like, oh, I wonder if it would work if I did it in this way. And then usually it does work, which is pretty cool. It's like, once you develop the intuition, and maybe there's a paper about it you could go read, but it's really just like a simple, like most of the LM papers are like simple ideas.

• 48:45Dan Shipper?

  Yeah, and sometimes there isn't. And it's like you're on the frontier.

• 48:48Speaker 01

  Yeah, exactly. Yeah. Yeah, it's easy to be on the frontier and and just to try stuff and it often works, which is like such a good feeling. It's like, oh, I had this creative idea and I tried it and it worked. Like, what's better than that? You know, that's, that's pretty amazing.

• 48:59Dan Shipper?

  Totally. So I put out a tweet or an X post asking what I should ask you. And someone who submitted a question is Linus, who used to work on the AI team. Linus asked how working with AI has changed what you think machines are good at versus human, what humans are good at.

• 49:18Speaker 01

  Hmm. Yeah, I mean, I think it's always shifting. That's one. Like, the only content has changed in that regard. I'm pretty like AGI pilled, so I think there will be nothing at some point. But right now, there are many things that that that humans are better at. I think actually maybe a better question is, what should humans do? It it will quickly be that that you're actually just just not better at anything. I feel like the most important thing there is making sure that the shape of the future is aligned with what we care about. And we agree and consent to it and support it. And a lot of it is around, like, us orchestrating the tasks that the AIs are doing. What do you want them to be doing? That's a really important question. And then, once you have a going and doing that, making sure it is it's doing the right thing.

• 50:03Speaker 01

  And you have a way to observe it, check on it. And that feels like like the key.

• 50:07Dan Shipper?

  So a way to boil that down is like wanting is what humans are good at?

• 50:12Speaker 01

  I think so. Yeah, yeah, it's like defining the high level goal. And making sure that's what's being done. Like, ultimately, at the end of the day, that's everything. And then, over time, all the details of how it gets in, it will be whittled away. And how are the AIs doing all of it? Which is great. I think that's a great world.

• 50:27Dan Shipper?

  Yeah. You said you're AGI pilled. What does that mean?

• 50:29Speaker 01

  I I would assign a pretty decent probability to like AGI in the next ten years. And and not necessarily requiring like huge paradigm shifts.

• 50:38Dan Shipper?

  Do you have a personal definition of AGI, what that means to you?

• 50:41Speaker 01

  Yeah, I like open AIs definition, just can it do all the economically useful tasks? I think that's pretty good. The critical question, I think, is around the intelligence explosion. So do you think it's possible? What's the probability of that? Like, how fast will it happen? And then the key question in there is like, when will AIs be be good enough at doing all the AI research tasks to significantly contribute to making the models better? And I'm pretty concerned about that. I I find the case like pretty compelling. You actually break down like what does an AI researcher actually do? It's your Reading the literature and talking to colleagues and gathering like context. And then you're coming up with like high level ideas and agendas. And then maybe you're designing experiments. And then you're like writing code to run the experiments. And then you're analyzing the experiments, just writing more code.

• 51:25Speaker 01

  And then you're looking at the results and deciding what to do next. And like, all these things feel like a sufficiently good foundation model could at least get you a lot of progress on all of them. And eventually, I don't see how you need to change the shape of it that much. And then if you get that dynamic really kicking in, yeah, then it can be like, like a really big effect.

• 51:44Dan Shipper?

  Yeah. The thing that's interesting about that, going back to the OpenAI definition of AGI, I'm curious for your thoughts. Like, the thing that strikes me about that idea, which is AGI has achieved when AI can do all the economically valuable tasks, is that AI changes what the economically valuable tasks are. You're creating a moving target. Like, in the same way, um, like, I think the Turing test turned out to be like a moving target. In a world where humans are leveraging AI to do more economically valuable tasks than we could have done before, it's a self fulfilling, like, AGI has not been achieved yet because we're changing the tasks.

• 52:16Speaker 01

  Yeah, I've seen the past few years, people are always raising the bar.

• 52:18Dan Shipper?

  Yeah.

• 52:19Speaker 01

  It's like, it's like memes about it, right? Oh, but I can't do this one particular thing that I really care about. It's not a static world. I saw a tweet the other day, I think just like sixty, eighty years ago in South Korea, most people were farmers. And then now, obviously, they're hardly anyone's a farmer. It's been fully automated. And the economy like fifty X or something. That's pretty crazy. And I feel like we're headed into a similar dynamic where the jobs people do are going to completely change. And there's going to be a bunch of new stuff that we do. And it's going to create a hundred X more economic value than all the economic value that we have now. AGI is happening in that curve somewhere, which is, it's going to be pretty weird, I think. And I think a lot of that is going to come from the fact that when you discover an economically useful task with these models,

• 53:00Speaker 01

  if you buy more compute, you can just like scale it up more, which is in a much more elastic way than you could with like humans. You have to like train up and hire and stuff like that. So I expect it to look pretty weird where, yeah, a bunch of classes of tasks are going to get scaled up. It's going to produce a lot of value. And then it's going to make the world, it's going to make the shape of the economy look different and in a confusing way.

• 53:21Dan Shipper?

  So in preparation for this interview, I asked Notion AI what I should ask you.

• 53:27Dan Shipper?

  And one of the questions it asked that I thought was really interesting is, how do you see the role of human creativity and intuition change in a world with AI?

• 53:36Speaker 01

  I don't think I assign a special value to humans being creative or intuitive. We've been side talking about this multiple times, but like we were all surprised, I think, when Dolly came out. It was like, oh, I thought it was going to be boring work, but it's actually like making beautiful images. It's like art, or at least the technical part of art, implementing the idea. Yeah, I feel like it's like a category error to assign like special value to any particular thing that that a human can do. And that AI might not be good at now. What really matters is more of the will of what you want to be done and your ability to communicate that and observe it and keep it on the road. That's what really matters. But then it's going to be better at us. It's going to be better than us at being creative and being intuitive in these things.

• 54:20Dan Shipper?

  How have you developed that will in yourself?

• 54:23Speaker 01

  I think openness is important. So just openness to experience, especially in a world that's changing as much as this. And I think being ambitious is important, especially with these crazy AIs. Like, we should really like ratchet up our ambition of what we want humanity to do and not be so encumbered by previous failures when this new tech unlocks a lot of stuff. That's pretty key. Yeah, I think, yeah, I'd be optimistic. Yeah.

• 54:53Dan Shipper?

  I love it. And I think that's probably as good as any place to end. I really appreciate you taking the time to do this. I learned a lot. This is great.

• 55:07Speaker 02

  Oh, my gosh, folks. You absolutely positively have to smash that like button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard. But instead of gold, it's filled with pure, unadulterated knowledge bombs about ChatGPT. Every episode is a roller coaster of emotions, insights and laughter that will leave you on the edge of your seat, craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. So do yourself a favor, hit like, smash subscribe and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.