Interfacing with AI, with Linus Lee of Notion

In this episode, Linus Lee, AI product leader at Notion, joins us to discuss his groundbreaking projects and unique approach to exploring AI systems. He shares his toolkit and insights on language model capabilities, along with his vision for the future of AI, which is centered on amplifying human intelligence. Linus inspires listeners to engage deeply with AI and to steer its development in ways that enhance human creativity and agency. RECOMMENDED PODCAST: How Do You Use ChatGPT Dan Shipper talks to programmers, writers, founders, academics, tech executives, and others to walk through all of their ChatGPT use cases (including Nathan!). They even use ChatGPT together, live on the show. Listen to How Do You Use ChatGPT? from Dan Shipper and the team at Every, wherever you get your podcasts. https://link.chtbl.com/hdyuchatgpt SPONSORS: Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Plumb is a no-code AI app builder designed for product teams who care about quality and speed. What is taking you weeks to hand-code today can be done confidently in hours. Check out https://bit.ly/PlumbTCR for early access. Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. CHAPTERS : (00:00) Introduction (10:16) AI and Creativity (14:37) Sponsor: Omneky (14:57) AI Research Development (20:05) Bridging Modalities (32:38) Sponsor: Brave / Plumb / Squad (35:36) Transformer Models and Techniques (52:32) Personal AI Research Setup (58:55) Leveraging Language Models for Coding (01:02:05) AI Model Development and Notion AI (01:22:05) Future of AI Models and App Development (01:32:51) Emerging Trends in AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The kind of interface I'm eventually building towards is a tool that lets you edit text or work through ideas, not in the native space of like words and characters and tokens, but in the space of actual meaning or features, where features can be anything from is this a question, is this a statement, is this uncertain or certain to like topical things like is this about computers versus plans, or to probably other kinds of features that we don't really even have words for. I'm generally a pretty optimistic person about technology, as long as the way we package these things is more humanist, rather than just kind of like automate all of the things. You see companies situated at different points in the spectrum between you want models to automate things in a way that takes away agency, i.e. replacement, or you

• 0:41Speaker 00

  want models that amplify. Like, I think OpenAI is very much on the replacement side. Literally, their definition of, I think, AGI is something like a thing that can take over a single, like, full humans job, where if you look at a company like Runway, a lot of their framing of usefulness is about extending that agency of what you want to express.

• 1:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Torenberg. Hello and welcome back to The Cognitive Revolution. Today, my guest is Linus Lee, AI product leader at Notion and AI explorer extraordinaire. I've followed Linus online for a couple of years now, fascinated by his many groundbreaking projects and his unique way of thinking about AI systems. From creating novel interfaces that visualize and manipulate

• 1:43Nathan Labenz?

  generative models in their latent spaces, to developing techniques for semantically editing text and images, Linus has been a pioneering tinkerer.

• 1:53Nathan Labenz?

  In this wide-ranging conversation, we dig into the details of how Linus goes about his explorations. He shares his toolkit, from PyTorch as a foundation, to the custom tools he's built over time for data visualization, model evaluation, and rapid experimentation. We also discuss the importance of spending time with raw data and failure cases, and the value of building your own tools to deeply understand the problems you're trying to solve. Linus also offers his perspective on the current capabilities of language models, and where he sees the biggest opportunities for improvement. Beyond the high-level goal of better general reasoning, he emphasizes hallucination reduction, better instruction following, and cost efficiency.

• 2:35Nathan Labenz?

  We also speculate about the future, considering scenarios like models communicating with other models via high-dimensional embeddings, techniques to connect latent spaces across modalities, and the societal implications as AI capabilities continue to advance. Linus articulates a vision centered on amplifying rather than replacing human intelligence, a principle he believes should guide the development and deployment of AI-powered products. Throughout this episode, Linus demonstrates the curiosity, resourcefulness, and thoughtfulness that have made him such an influential figure in the AI community. His work inspires me to engage more deeply with these systems, to build the tools I need to understand them, and to steer their development in a direction that deepens rather

• 3:18Nathan Labenz?

  than diminishes human creativity and agency. As always, if you find value in the show, please share it with others who might appreciate it. Particularly right now, as we're establishing the new feed, a tweet or a comment on YouTube would be especially valuable. And of course, we invite you to reach out with feedback or suggestions on our website, cognitiverevolution.ai. Now, please enjoy this deep dive into the art and science of exploring AI systems with Linus Lee of Notion AI.

• 3:47Nathan Labenz?

  Linus Lee of Notion AI and general all-purpose AI explorer. Welcome to the cognitive revolution.

• 3:54Speaker 00

  Thank you. Thanks for having me. Excited to chat.

• 3:57Nathan Labenz?

  I'm really excited about this. I have followed you online for a couple of years now and really been fascinated by all your many projects and just the way that you think about AI systems. I really genuinely think you're a one of a kind thinker from which I've learned a lot. And so going back to when I set out to do this a year ago, you were on the shortlist of guests that I maybe didn't want to have right away because I kind of wanted to get my, get things set up a little bit and have a little bit more credibility, a little more comfort. But I definitely always knew that I wanted to get you on the show and pick your brain. So I'm excited to get into it.

• 4:32Speaker 00

  Thanks. That's very generous of you to say. Likewise, I'm excited to talk about both AI and everything else that I've worked on and kind of what I see coming up ahead.

• 4:41Nathan Labenz?

  Well, very well deserved. The way I thought we would maybe approach this is I wanted to give people at the top a little bit of a sense of the kinds of things that you build. And then try to kind of peel back some layers from that to then understand like how you're thinking about that in kind of conceptual terms. Also, how you are doing the work in very, very practical terms. And then kind of a sense for where you think we're going from here, as well as perhaps some very practical tips based on your experience at Notion and elsewhere. For people that are looking up to you as an AI application developer.

• 5:19Speaker 00

  That sounds good to me.

• 5:21Nathan Labenz?

  So for starters, you have built some of the most interesting novel interfaces that folks have seen. And you know, these can be found on your Twitter profile and various places. And you've shown a few off in different talks that you've given. And a few of them jumped out to me that you've demoed recently, but I'd love to kind of just introduce a few of the unique interfaces that you've created. Just to give people a very kind of tangible jumping off point to understand the sorts of things that you put together.

• 5:49Speaker 00

  Yeah, definitely. I think we can kind of talk through them. I think the more polished, the demo that I can do off the cuff right now is probably not as good as the ones that are more polished online. So you can find them on YouTube, but I can talk about what the, what concretely the demos are. And then I can maybe later talk about my framing for them because I think that's a part that. I view a part of my work as trying to figure out and evolve exactly what the framing should be. And I think that'll come up later in our conversation, but the concretely, there are a couple of directions that I've explored in. One that I've sort of been on for the longest time is to try to visualize and control generative models in their latent space rather than in the input space that we're familiar with.

• 6:29Speaker 00

  So if you take image generation, for example, the the common way of generating images is you describe an image like a cow sitting in a grassy field with sunlight, long shot, you know, three to five millimeter film, whatever. And then the model generates an image. But another way you could imagine generating or improving upon an image is you start with an image and then you find a different image that has another element that you like. Maybe you like the lighting from another image, or maybe you like the framing, or maybe you you have a character that you want to bring in from another image. And if there was some way to tell the model, can you merge all of the attributes that you see in both of these images into a single image and kind of blend it together in the semantic space rather than in the pixel space? That could be an interesting way to kind of stretch together images.

• 7:10Speaker 00

  So a couple of my demos are around combining images and text in this semantic space of an image model called Clip to generate, for example, start with a human face and then add emotion to it to make it smile or make it frown, or starting with like a photograph and then adding different kinds of style to it by just adding vectors in the latent space of a model together. You could do something like this with text as well. So I have a model that's capable of embedding text into a latent space, an embedding space, but then also inverting that embedding to go from the embedding to what that embedding represents as text. And it turns out, not just in that model, in a lot of embedding models that are expressive enough, this transformation is like pretty is invertible with minimal kind of meaning loss.

• 7:53Speaker 00

  And so I take that model and then I can start to manipulate the embedding itself to, for example, combine two different sentences together semantically and produce a new sentence or interpolate between sentences. So you have one that's maybe very optimistic and happy and bright and another that's maybe a little more mellow. And I can take those two sentences and interpolate between them to generate sentences that are in the middle. And I think that that kind of thing could potentially allow for other ways of controlling or editing or generating text or even images that let people control output. Using levers that are hard to verbalize with text. And there are more rigorous and precise versions of this that I've been working on more recently that I can also talk about, but that's the general gist of things that I've been interested in.

• 8:32Nathan Labenz?

  And sometimes there are some pretty creative front end experiences on these as well. You want to describe the one also where you have kind of little tiles of text on seemingly an infinite canvas and then you can kind of drag into different directions and create like, take that text, but now move it in this direction. You're kind of making that a spatial computing sort of experience almost.

• 8:55Speaker 00

  Yeah, so the two that I think are most interesting visually, the one that you just described is actually came out of a problem that I ran into where one of the initial prototypes of this involves just having, imagine this high dimensional latent space of a model and you you want to visualize it and you want to visualize it on like a screen. And so you cut a 2D slice through it. And you cut a 2D slice through this like high dimensional space such that the two dimensions that you see on the screen, the like bottom edge of your screen, the top, the left edge of your screen correspond to two different kinds of targets that you need to move towards. And so maybe the bottom right corner of your screen is like a happy sentence about weddings and the top left corner is about like dinosaurs or something. And you can pick any point in this plane and it'll

• 9:40Speaker 00

  be a sentence that combines elements of those sentences. That works well for kind of demonstrating the seed of this idea. But the problem that you quickly run into is there are actually way more properties that you want to control when you're generating or editing text than just like between two arbitrary anchor sentences. And when you have this like fixed plane, you quickly run out of directions that you can move in. And so instead of fixing the directions, the levers that you can pull to just the like top and bottom edges of your screen, the next revision, which you referenced, involves kind of flipping how the dimensions are presented. So the way I used to describe it is imagine you're playing like an instrument. And when you play an instrument with one hand, this is very crude, but if you're playing like a string instrument with your left hand,

• 10:25Speaker 00

  you control the strings and you control the pitches of the notes that you're playing. But with your right hand, you kind of control the like. The volume or the amplitude of the sound in a very crude way. And this is a, this is a way to kind of transfer that in a spatial interface. So this is an infinite canvas where with one hand, holding down keys on a keyboard, you can choose what specific attribute you want to control for text, whether that's like a positive, negative emotion, or whether it's written in fancy English versus simplified English, or what kind of topic it is, is it more about sci fi or so on? And then with your right hand, you can click down on a sentence that you've put down on this plane and you can drag it out. And the, and the further you drag out,

• 11:08Speaker 00

  the more you're kind of increasing the particular attribute that you picked. So for example, if you put a sentence down on this 2D canvas and then you hold down the like make longer button and you pull the sentence farther from where you clicked down on it, the farther you pull, the longer that sentence will be. And over time, as you explore this latent space of sentences, you can see all of the branches that you took in this exploration and where each variation came from. And it form, it naturally forms a kind of a branching edit history through which you can follow all of the different kind of movements that you made through latent space on what the sentences that resulted from are.

• 11:42Nathan Labenz?

  When you think of these projects, are they, I mean, I can, I think they probably have multiple functions for you, right? One is obviously you're learning by doing all this stuff. How much do you actually use these tools versus viewing them as kind of proof of concept for some further synthesis on top of this later? Like, do you think these kinds of interfaces are ready for people to actually get utility out of it? And do you personally?

• 12:07Speaker 00

  Most of the prototypes that I've built for kind of like research purposes, I would say are not good enough to actually be useful. So the kinds of edits that you can make by, with this like kind of latent space canvas UI, the edits are very crude and the models that I'm using are too small, I think, to, to let you make very, very precise edits. And I think one of the, the kind of continued problem areas in which I want to do more research is to try to improve upon that. There are other prototypes that I've made that have been more useful. And so I think it just depends on how, how good the particular like technology or technique that I'm working on is and whether that is close enough to utility or not. Within the kind of set of prototypes that I've built for working in this area of like building interfaces for AI, the eventual goal is always to build something that is useful.

• 12:52Speaker 00

  And for any individual prototype, it may be the technology or technique may be too early to actually be useful. So it may be the prototype may be more about identifying incremental improvements or identifying problem areas or identifying what directions to build more in. Or if the technique is more mature, then it may be close enough to utility that I build it more as something that I can use day to day and actually kind of get benefit from. But there's, there's always an underlying element of it, which is like, this is going to help me better understand what it's for, how it can be useful, how to make it, how to improve on the technique, the quality of the outputs and so on. An example of a prototype that I built a long time ago, that I actually ended up using day to day for a long time, is kind of an Obsidian or Room Research style outliner, a note taking app. That was my daily note taking app for a long

• 13:36Speaker 00

  time, where keywords would automatically identify themselves, or entities would automatically identify themselves and connect them to other entities in the graph. And that was using mostly not language models, mostly dumber techniques. But it was, the outputs were good enough that I ended up finding those connections very useful and generative in my creative thinking. And I used it for a long time.

• 13:54Nathan Labenz?

  That's cool. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:00Nathan Labenz?

  The, the 2D canvas one in particular of the recent demos that I've seen, has a sort of Xerox PARC vibe about it, where you're like, this, this feels like, especially after I went and tried the Apple Vision Pro, it was like, I see something here. And it feels like there is, you know, spatial computing as, you know, spoon fed to you. And it's amazing. It's, you know, incredibly like, like a breathtaking technology experience to go put the Apple Vision Pro on your eyes. But it's very much like a consumer experience that, you know, you, you start off with. But seeing your, you know, 2D canvas imagining that becoming a higher dimensional space that you could potentially move around in,

• 14:43Nathan Labenz?

  and then, you know, envisioning these sort of, you know, hardware to actually make that. Feel very experiential starts to give me a little bit of a sense of like how some of these future spatial computing notions might be, you know, more than just like, oh, look, I've got my YouTube screen over here and I've got my email over here, but actually kind of using the brain's spatial intuition to do things that are currently like hard to do.

• 15:09Speaker 00

  Yeah, using, using all of the full kind of awareness, awareness of space and the awareness that we have how to move around. Like this is all very intuitive. We learn it when we're a child and most of the ways that we work currently don't really take advantage of that. I spent a long time iterating on and I like even still, I every like few months, I get a kind of a new version of exactly how to describe the motivation for building out these kind of like weird spatial demos that are technically not quite there yet, but like gesturing towards something interesting. The most recent version of this, I think is actually somewhat relevant to how humans interact with space, among other things. The closest metaphor that I found for the reason that this direction of research is so interesting

• 15:50Speaker 00

  to me is that I think language models work so well because they work with kind of an alternative, better representation for ideas than humans work with. The closest analogy that I have is like spectrograms when people are dealing with audio. So normally sound is like a wave in space. It's just a single kind of, I'm imagining like a single string vibrating back and forth over time. And if you work with audio, that's like the base, base thing that you work with. It's a basic representation. But if you, if you work professionally with audio, then you actually most of the time work in a different representation space where you don't look at vibrations over time, but you look at space of like frequencies over time, or what's called a spectrogram, which is a visualization of like, there is

• 16:32Speaker 00

  still a time axis on the one hand, but the other axis is no longer amplitude of the vibration. It's like the prominence of different frequencies. If you imagine like. On the left side of your kind of graph, every like the row is a different pitch. And every row gets a little mark when there's like a sound that hits that pitch. It's a little bit like Western music notation kind of staff, but it's it's like colorful gradients often appear in these spectrograms. And what you can see, because you've broken out the prominence of every different kind of pitch, every different frequency, is you can see things like, oh, here's a here's a musical pattern that repeats where here is where the bass comes in. Or here's where the tone shifts from this kind of very full sound to this more mellow sound. Or all all these things you can notice, and then you can start to manipulate the sound in the frequency space.

• 17:15Speaker 00

  So you can do things like bring down the lows or add reverb only to the the human voice range of sound. All these kinds of transformations that you're not really making to the sound waves themselves. You're making to making to this like kind of projected out the version of the sound that exists in this alternative frequency space, if that makes sense. The kind of interface I'm eventually building towards is a is a tool that lets you edit text or work through ideas, not in the native space of like words and characters and tokens, but in the space of actual meaning or features, where features can be anything from, is this a question? Is this a statement? Is this uncertain or certain to like topical things like, is this about computers versus plans? Or to probably other kinds of features that we don't really even have words for,

• 18:00Speaker 00

  because language models implicitly understand them. We implicitly understand that, but we don't need to ever really verbalize it. So alternative way of like projecting or viewing what's going on in language, and then maybe a way to to start editing them. That's like the closest metaphor that I've found for describing the kind of thing that I want. And then, once we have that representation, the one reason that spectrograms work so well is because humans intuitively understand kind of color as a way of representing range and positions in space, so we are representing range, because we have this geometric intuition. And so maybe rather than looking at ideas as just kind of like ink scribbles on paper, maybe we can also look at them in a spatial way. And the canvas was one kind of way of exploring that.

• 18:41Speaker 00

  I think there are a bunch of other directions that we can go once we have a better, more rigorous understanding of what's what's going on inside these language models. But eventually, I think we should be able to interact with ideas by like bringing them out into the physical space that we occupy or being as as close to that as possible, because we want to bring our kind of full senses and appropriate perception and sense of scale and sense of distance into how we work with ideas, rather than just kind of like eyeballs and pixels on the screen.

• 19:07Nathan Labenz?

  Yeah, that's really interesting. I'm just looking up to try to figure out, was it Refusion? Which might have been the project that people would know most of all that was a stable diffusion, fine tuned, right, on these spectrogram images. So definitely check that out for further intuition there because it's also an interesting kind of foreshadowing of what are the things I want to talk about a little bit later of just kind of how all these spaces seem to be very sort of bridgeable to one another in many cases through like relatively simple adapters. Here you have a vision model trained to generate these spectrograms and then that is just converted to music. Like the AI part stops at the generation of the spectrogram image.

• 19:53Nathan Labenz?

  But at one point, this was like one of the leading text to music techniques, all working through this like vision backbone. So.

• 20:01Speaker 00

  I was going to say that the model's generating audio, but not in the typical space that we think of for representing audio, which is waves in space, but it's generating in this spectrogram space. But it's actually a little bit more convoluted than that even because most like speech generation, speech synthesis models, or even a lot of music generation models, they actually, a lot of them do operate in this kind of like frequency space, in this like spectrogrammy space. I don't know how the state of the art is now, but until very recently, the way that you built like a speech recognition systems was also by feeding an input in this like frequency space rather than just kind of like waves in space. And so not audio models used to naturally operate in this domain. And then what Refusion does is it actually takes a model architecture that is meant for generating pixels in a 2D

• 20:46Speaker 00

  image using convolutions and applies that to this like generating this thing that's supposed to represent audio. So it's coming through kind of like two jumps in representation and it still manages to work quite well.

• 20:57Nathan Labenz?

  It's amazing. That's one of my big kind of themes and takeaways from, you know, a few years now of increasingly obsessive study of the whole space is just what an incredibly high hit rate there is on ideas that actually turn out to work. And obviously, you know, not every idea works, but as somebody who spent a little time in catalytic chemistry once upon a time, I could say it's definitely probably at least two orders of magnitude higher in AI right now than it is in, you know, many more well-minded corners of the physical sciences.

• 21:28Speaker 00

  If you have a lot of data and a lot of compute. And you have a way of efficiently putting that compute to work and squeezing a model through that process. The model architecture is kind of like a final corrective term or coefficient in how well this whole thing works. It turns out if you can make gradients run on a lot of numbers very quickly with a lot of data that's like high quality, most ideas will work within like an order of magnitude, maybe of efficiency. And so a lot of these ideas, I think, are just, I attribute to that. And then the recent advancements to the fact that you have now built up kind of all of the open source infrastructure and know how of how to do this

• 22:09Speaker 00

  like model training thing at industrial scale rather than in like research lab scale, which was maybe research lab scale to me is like a few GPUs, a few hours, maybe like a few days. And then industrial scale is like thousands of GPUs, many, many weeks. And go from one to the other is in the same way that you can want to like synthesize a little bit of chlorine in a lab. It's very different than like building a chlorine factory. Running these models at scale, I think, is a similar kind of like huge qualitative jump in terms of the kinds of knowledge that we have to have. But like, we have a lot of it now. And so we can apply that compute to like any arbitrary problem, given there's enough data and given there's an architecture that like doesn't suck super hard. And I think the result of that is that we've

• 22:51Speaker 00

  suddenly seen an explosion of like models that perform very well at all these modalities and cross modalities.

• 22:56Nathan Labenz?

  Yeah, compute, you gotta have. Data, the quality is critical and scale is also critical. And then the algorithm just kind of depends how determines how much compute you have to have to make full use of the data. That's like, if I set my brevity to max, that's about as brief as I could give, hopefully, a decent account.

• 23:17Speaker 00

  Yeah, as long as the architecture is not bottlenecking what information can propagate through the network. Everything else is just kind of like, how many floating part operations can you do?

• 23:27Nathan Labenz?

  So going back to your exploration of all this technology. So you've been at it for a couple of years now. I'm a little bit unclear on the timeline, but I have to say, I feel like I encountered quite a few ideas first through some of your experimental projects. And then later, it seems like they have become more formal as like mechanistic interpretability research publications. Some of the, you know, a couple of kind of canonical results from the last handful of months, the representation engineering paper from Dan Hendrickson collaborators and the Torde monosemanticity paper out of Anthropic. I feel like you basically were doing that kind of stuff with small models well before those

• 24:12Nathan Labenz?

  things kind of hit mainstream. Like, how would you describe your own sources of inspiration for that? Or maybe I have it wrong, and there were more academic papers coming out that you were able to take inspiration from. But how did you get to this, these kind of advanced, like, middle layers manipulations concepts as early as you did?

• 24:32Speaker 00

  So I don't come from academia. Like, my actual technical background is in, like, product engineering, building web apps, which is like the furthest thing you can be from training models. And so I had to kind of step my foot in slowly into the research side of the community, the ML community, and figure out not only how that side works and kind of how to engage people in that world, but also how ideas propagate through both of these parts of the community. My old mental model of how research works used to be that, like, researchers are always at the forefront, and they try these things, and they come up with ideas. And if the ideas are good, they get written about, and then eventually the industry picks them up, and then eventually it, like, percolates down to smaller and smaller, kind of like, perhaps less resourced groups. And I think that's,

• 25:15Speaker 00

  maybe that's true in other fields, like biology, where there's perhaps a larger gap in capability between what labs can do and companies can do. Even there, I don't think it's actually perfectly true. But certainly in machine learning, because all of the tools are so accessible, especially in the last few years, my picture now of how ideas percolate is actually a lot more nuanced, which is that at any moment, like, there are, there is a large group of people that kind of know all of the things that we know about models and what you can do with them. And some of those people exist in industry with the tools that they have. Some of those people exist in academia. Some of these people are like hobbyists that are just kind of like tinkering on the side, even when their day job is doing something else. And different people might stumble upon the same techniques or the same

• 25:58Speaker 00

  ideas at the same time because of, because like, most of what results in the next ideas that get discovered is actually, I think, a function of what is already known, rather than the specific ideas that individual people have. And so ideas like The intuition that there was a linearized kind of feature space in the space of these language models that you could do edits in. I think there was an idea that was kind of intuitively known for a lot of people working in this space. And some people like me, I think, chose to kind of prove that out by doing what I personally enjoy doing the most, which is build interactive prototypes on the web, because that's kind of my my expertise. And then I think other people perhaps took longer to be a lot more rigorous and theoretically robust and write things about it and get it out that way.

• 26:42Speaker 00

  And there's a kind of different way you speak about ideas there versus speak about like ideas in the interface world. But I think most of most of the ideas that are come about is just a function of like what was already known. And different parts of the community, both in research and in industry, just happen to like talk about them in different terms and perhaps reach other parts of the community at different times.

• 26:58Nathan Labenz?

  Yeah, I mean, you've had to solve a lot of problems, if I understand correctly, right? This gets into the how you work section, I suppose. But like, if I understand correctly, you are, for example, figuring out your own features, right? Like, I think you've done a mix of things where sometimes you can take a straight embedding, you know, which is just kind of one hot look up and then, you know, do that again and, you know, mash those up. And that's like relatively straightforward to do. But then, as you get deeper into the network, you're also doing, at least in some of your projects, you're making edits like farther down, you know, through the layers, right, of the of the model.

• 27:38Nathan Labenz?

  And there you have like a much more conceptually challenging problem of, okay, I may have an intuition that, you know, there I could probably like steer around in this space and kind of steer the outputs. But how do I begin to figure out what direction is actually what? If I understand correctly, you've largely had to figure those sorts of things out for yourself, right? Like, you weren't able to really use too much in the way of open sourced kind of GPT-2 or GPT-J or, you know, whatever. Like, these things didn't come with like mid-layer of feature sets.

• 28:10Speaker 00

  Exactly. My favorite mental model for research is actually related to Kind of the exact path that I followed to hit that wall and then go past it. Which is that when I first started getting into deep learning models, most of what I did was like read and try to understand things that were in the open source world. So like I I had a hard time reading academic papers in the beginning and and deep learning because I didn't have all of the kind of vocabulary and the way that people write about these things in my in my head and in my hands. And so I would just go to like the Hugging Face Transformers repository and I would download G P T two and get the minimal version running. And I'd be like, oh, this is generating text that recognize it seems to work now that I have it now that I have like a live entity that's like working on my computer.

• 28:51Speaker 00

  And I understand Python, and so I can like put print statements everywhere and see exactly what's happening. And so that's kind of what I did. I basically did like model autopsy on the things that were running on my computer as because I felt like I think that I could understand and get my head around. And then I would try to understand things that way. And once I understood those models, I would try to understand things were a little bit more on the frontier. And eventually what you hit is you hit a wall where like for a while you can get away with like, oh, I hit this problem. I'm going to Google how other people solved it and you arrive at like a Hugging Face forum post and then maybe you get a little more esoteric and you arrive at like a GitHub issue. And you're like, okay, this is kind of like, there's been consensus, but like here's some people, some things that people tried and maybe they work. And then at some point you hit a wall where you're like, there is, there's nothing on the internet that's like an established consensus for how to solve this problem.

• 29:35Speaker 00

  And you have to like go find a paper that somebody wrote three months ago that like reports to solve this. And then sometimes it's is like well established and well cited and you can trust it. And other times you're like, well, this paper has like two citations and I don't even know if it actually works. And so you have to then find like an implementation that has like one star on GitHub that try to like tries to reproduce it and and run it. Or oftentimes those were broken too. And so you have to then like write your own implementation, which you're not even sure if is the correct one, that that's the same one that the researchers used. So you hit a wall. At which point you're kind of at the frontier of what we know as a research community or as an industry. And then, but there, there's still problems to solve. Like I still, I still have things that I want to do that I don't know how to do.

• 30:18Speaker 00

  And so in the same way that if you like hit a weird Python bug, you keep digging and digging, and then maybe you do some like debugging and eventually maybe you have to like go into the Python interpreter. I hit a wall with, with like trying to find interesting features in these models or trying to even like build models that I can use for these things that were capable of being amenable to these edits. And I hit a wall and I had to eventually read a bunch of papers and develop my own intuition and try to combine the things that I found, to combine the pieces that I found to say, here's an approach that might work based on what I know. Um, it's, there's no obvious consensus on it, but it might work and try it. And that's, that's been the process most obviously to me for first, building the auto encoder that I use for all of these embedding space manipulations, the model that I talked about that lets you go between text and embedding fluidly.

• 31:03Speaker 00

  And then second, for the work that I'm currently most focused on, which is more unsupervised kind of scalable approach of finding these edit directions, which built on top of a lot of the mechanistic contributory work that organizations like Conjecture and Anthropic have done, but that, that's also quite on the frontier. And there are some techniques there, but they haven't really been applied to embedding spaces yet that most have been applied to kind of normal GPT language models. And so I've had to take a lot of those techniques and adopt it as well. And a lot of that is like finding weird, like literally I was doing this earlier this morning, finding weird GitHub repositories that have like three stars and try to transplant it over and hope that I'm not messing up the implementation and see how it works.

• 31:39Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:44Nathan Labenz?

  So before getting into a little bit more detail on how exactly you go about that, how would you characterize, you've obviously built up a lot of intuition over the last few years of doing this. How would you describe like what's going on inside a transformer or perhaps what's going on inside a diffusion model? You can kind of break that down however you want. I don't just mean like, you know, describe the blocks, but like, how is the information being changed, you know, in more sort of, and you, I, I guess another related question there is you said, you think that the models have a better like representation of a lot of ideas than we do. I'm very interested to hear, you know, how, as you describe kind of what's going on in

• 32:25Nathan Labenz?

  a model, maybe you could contrast that against what you think is going on for us. Like, I feel like I have a pretty. Similar setup in some ways to a transformer where like it seems that maybe I'm just like so, you know, prisoner of the AI paradigm now that I just see myself through that, which would be a weird irony. But I do feel like there is this sort of non-verbal, high dimensional, you know, kind of only semi-conscious churning going on in some middle layers somewhere for me that then kind of gets done and feels like, okay, I'm happy with that now. Now you can speak. And then it kind of, you know, in the, in the last layers, so to speak, you know, it, it gets cached out into tokens.

• 33:08Nathan Labenz?

  But yeah, so what's going on in, in a language model at, at kind of a, you know, conceptually descriptive level or representationally descriptive level?

• 33:17Speaker 00

  My mental model for how humans work, I think includes a really important piece, a, a piece that I found to be like more and more important over time, which is that I don't actually think, I think that the default assumption, I think intuition for how we think we work, our brains work is that we think through things and then we draw conclusions and then we act upon them. But I don't actually think that's true at all. And I think what happens instead is our like brain and our body takes actions and then observing those actions, our brains retroactively build up a narrative for why we perform those actions in a way that seems like causally consistent with our like understanding of the world, just a whole other interesting thing. And maybe, maybe that's what you like call consciousness. But in the, on the model side, there are different levels that you can look at this. And I think for transformer language models,

• 34:00Speaker 00

  the two levels that are most interesting are how is the transformer processing information within a single forward pass, generating a single token? And then how are we sampling from the output of the transformer? So I'll talk about the sampling first. The job of a transformer language or a job of any, any language model is, it's called a language model because it's a statistical model and the statistics that it's modeling is the probability distribution over all possible outputs. And in a lot of cases, this is a conditional probability distribution. So the probability distribution that the model is modeling statistically is here's some start of a sentence. Give me a probability distribution over all possible ways that this input can continue.

• 34:45Speaker 00

  And this is kind of like a, like a cloud over, a cloud of probability over every, every possible branch that the model can take. And obviously, that's way too many outputs. If we had infinite resources, we would simply ask the model for the probability rating of every single possible continuation and just take the one that the model thought was the highest. But we can't do that because we just don't have it in compute. And so instead, we have to randomly sample from this distribution, this cloud of possible continuations. And there's a bunch of different ways to do the sampling. And I think the most common one is to just sample every token and immediately kind of commit it. So ask the model, what's the most likely token to come next? The model says here the like top 50 most likely continuations and you randomly pick

• 35:30Speaker 00

  one of them according to probabilities and you just do that over and over and over again. But there's a bunch of other ways to sample from this. Like you could have a kind of running buffer of N branches and then only like set things in stone when you were like N steps in and you think that that's the most likely continuation. There's a name for a technique that I call beam search. There's other ways to try to take into account not getting the model to repeat itself as much or taking the probability distribution and kind of warping it in interesting ways to try to like try to emphasize the... There's an interesting technique where you have a small model and a large model and you assume that the large model improves upon the small model always. And so you like take the gap in the distribution and you amplify the difference.

• 36:14Speaker 00

  And then you try to kind of approximate like what would an even smarter model predict? So a bunch of techniques for sampling from this distribution. But ultimately, once you have a good probability distribution that the model outputs, you're trying to find the continuation of a text that's most likely to maximize the likelihood. So that's like the how we sample from the distribution part. And then there's the how we get the distribution part, which is what's happening in a single forward pass of a transformer. And this is very transformer specific, but the best mental model that I have, which a lot of this I owe to kind of the excellent setup that groups like ConvNet have done, is a transformer takes in a bunch of tokens.

• 36:55Speaker 00

  And a transformer is basically a stack of mini neural networks, a collection of mini neural networks where each token gets its own little mini neural network. And within each little mini neural network, there's a bunch of layers where you like there's a main kind of artery where the information just flows down.

• 37:15Speaker 00

  And then every once in a while, there's an offshoot where the transformer can like go off and do a little bit of computation to iterate on its understanding of what's going on and then merge that change back into this like main artery. And large, like small models have like 12 to 24 of these kind of like branch and merge paths within each kind of mini transformer in a token. Very large models can have up to like 100 of these little blocks. But you imagine you have a sentence and for each token in that sentence, you have a little mini transformer that over time continues to iterate on the model's understanding of what a token represents. And then after each like branch and merge back into this artery, all of the arteries of the transformer, the mini transformers can exchange information with each other.

• 37:57Speaker 00

  And the way that little like token level mini transformers exchange information with each other is they broadcast some information about what information they contain and also what information they're looking for from other, other mini models, mini stacks of the transformer. And then based on what they have and what they're looking for, they learn to like fetch information from other, other like token stacks in this transformer. And then they merge those back into the artery. And so a transformer is a repeated cycle of here's some information I have, fetch more information from other, other tokens and then do some iterative computation on it, get more information, iterate on that, get more information, iterate on that until you get to the bottom of the stack, bottom of the, the like stack.

• 38:39Speaker 00

  And then that gets put, put, put through this like thing called a softmax, which gives you the final probability distribution. But ultimately, a transformer is a thing where for every token, you do a bunch of iterative information processing and then you exchange information with other mini transformers behind each token. Does that make sense?

• 38:57Nathan Labenz?

  Yeah, I think the anthropic work that you've alluded to has certainly been very influential for me too. I think when you say artery, main artery. You know, synonyms for that would include the residual stream and the skip connection. And I think, I don't know who all was involved with this work, I'm sure multiple people, but I also definitely associate this with Neil Nanda and watching some of his tutorials, where he'll say things like, "The residual stream, it's really important, man." And you're like, "If you say so, I believe you."

• 39:31Speaker 00

  The residual stream is the name for the artery, and then within each transformer block, the attention is the mechanism by which the little, you tokenize residual stream, exchange information with one another, and then the MLP layer is the thing that actually does the like, actual, in my mind, most of the actual computation and like, meaning association happens within these MLP layers. There's a lot of other weird nuances about this, like, when humans think of all computation step, we think of like, a discrete mapping from one input to another, like, mapping from Paris to Eiffel Tower, but it turns out, it seems as though a lot of times, this meaning formation may be distributed across multiple blocks over time, and so there's

• 40:14Speaker 00

  a lot of like, strange things about exactly what human interpretable computation happens, and how it's spread out across all of these mechanisms. But in terms of just like, raw bits flowing through, I think that the clearest picture that I've found is the one that I just described.

• 40:30Nathan Labenz?

  Yeah, that's good. So now, let's go back to a little bit more of kind of the practical, how you actually do this work, and then we can kind of go a higher order concept again, and talk about like, some of the capabilities and big questions around, you know, what they can actually do, and where we're confused, and where we're confident about what's actually happening. So, okay, so you've just described a sort of flow of information. You've got this main kind of path, and then you've got these sidebars where the computation happens, there's this like... Attention block, which is the place where each of the many streams, each token gets its own stream, they can put out a vector indicating what information they contain. They also put

• 41:11Nathan Labenz?

  out a vector indicating what information they are looking to receive. Those all get crossed, and thus is the information passing from token to token. And then they continue on, and each token kind of loads in various associations that are encoded in the weights, and that's like where the information is stored in some way, some weird way. Okay, cool. So now that has to get instantiated in code. What libraries, frameworks, templates, tutorials do you use to have the right level of convenience and abstraction, but also kind of clarity on what it is that you're setting up?

• 41:54Speaker 00

  Most of the time, I'm working in PyTorch on GPU, like Linux instance that I have on AWS. PyTorch, I guess the like, main alternatives to it would be JAX and TensorFlow. It seems like TensorFlow is in a bit of a maintenance mode at the moment from Google. And JAX, I think is nice for many other reasons. I guess I should describe a little bit my intuition for what these things are, how they differ. So PyTorch in my example, my intuition is, gives you the most concrete way of interacting directly with the vectors and matrices. Like, PyTorch, when you instantiate any tensor, or any matrix, or any vector in PyTorch, you can like, look at the numbers, and it's a thing that feels like you can like, hold on to it. It's a, it's a concrete value. And the way you write a program or a model

• 42:39Speaker 00

  in PyTorch is you have some concrete value, and you like, apply operations to it, which is the way that like, normal programs are in. The mental model that I have for writing programs in JAX or TensorFlow is, it's a little more like, a compiler, in that you describe the model as a series of computation steps. So you're kind of describing a model as like, a function, with these 12 steps you apply to the input, or these 96 steps you apply to the input. And then once you, once you have a model, you can like, compile it, and then you can put an input through it, and then out comes the output. But there's, it's harder to, there are some affordances for this now, but it's harder in libraries Just like poke inside of a model and say, okay, what is the actual matrix value here?

• 43:22Speaker 00

  Let me look at all the numbers and let me like poke inside of it and play with it. Because the libraries are constructed that way for performance reasons. If you can describe a model, it's kind of like straight through control flow graph or a function way. It's a lot easier to apply complex transformations and compilation steps to this to get higher performance. But the trade off then is that you can't concretely like play with the values. And I think especially if you're doing lots of interpretability work, it pays to just be able to look at a specific value, any random place in the network and say, okay, what does this number correspond to? What happens if I treat this number to be this other number? And for that reason, I think in general, in the ML world, I think like between JAX and PyTorch, most researchers use PyTorch, I think. And then if you're like from Google or X Google or using Google infra, you use JAX.

• 44:05Speaker 00

  And then I think in interpretability, things are a lot more skewed towards Torch and these other other tools that you play directly with the values because it's just, it just pays to have more observability into the values.

• 44:14Nathan Labenz?

  My general understanding is PyTorch is basically totally dominant outside of Google and has, for all the reasons that you mentioned, I mean, it's kind of won the open source battle. To what degree do you need to complement that with other libraries? Like I know, for example, Neil Nanda has a library that is kind of meant for these sorts of visualizations. But I don't know how much visualization you're doing of activations, you know, as the forward pass is happening versus how much you're able to just get by with the more human readable inputs and outputs. Do you actually spend a lot of time visualizing what's going on inside?

• 44:52Speaker 00

  I do, I try to. I would a lot more if the tools made it a lot easier. I think the kinds of visualizations that are beneficial for studying autoregressive transformers are slightly different than the ones that are useful for studying embeddings. Parts of the kind of existing infrastructure I can use, parts I have to like create on my own. Thankfully, visualizing things is something that I'm. Relatively better at compared to like building models. And so it doesn't take me as much time to build embedding space for graphically explore a bunch of outputs of a model that I can like whip together with some like React and front end pieces because that's mostly what I do at work. This is, this is something that I think holds true for both our AI work at NoSham and also my work independently is especially when you're early on in a field or early on in investigation, which I think I would, I would claim that we are early on with language models in general. Nobody really knows exactly

• 45:37Speaker 00

  what they're doing. The quality of the tools and how much you can iterate on the tools, I think bottlenecks how much you can iterate on the thing that you're working on with the tools. And so it pays to be able to quickly tweak the tool or add the functionality that you need to see something new, whether that's a tool that's for evaluating models or running models or visualizing things either in the outputs or in the training like behavior. And because of that, I think I've mostly defaulted to building my own little tools whenever I needed them. And then eventually if I realize that I'm building the same kind of tools a bunch of times, then consolidate that into like a little library or a little script or something that I can reuse. To me, that flow of quickly whipping up something that works. And then once you realize it's being repeated, then being able to smoothly trend that into a reusable component. Like in some sense, this is like all software

• 46:22Speaker 00

  engineering is, but being able to do that really fluidly, I think is like the meta bottleneck. And the speed at which you can do that, I think is the thing that bottlenecks the quality of the tools, which is then the thing that bottlenecks the actual quality of research you can do. So a lot of my, my own like work is kind of tools that I've built over time for myself. When there, when there are reusable pieces, obviously I reuse, but I try not to adopt anything that is very hard to see through. Like the Hugging Face Transformers library is very hard to see through because it's meant to serve like a million different use cases and it's meant to host like a billion different models. And because of that, there's tons of interaction. It's very hard to just ask a simple question like when the model is at this block, what are all the vector operations that are happening? It's impossible to answer because the answer to the question depends on a bunch of different

• 47:06Speaker 00

  questions. And there's like 12,000 if branches in any single Hugging Face function. And so conclusion that I'm trying to get to is that building my own tools that are much simpler and just fit for my use case lets me iterate on them quickly and understand what's going on a little better.

• 47:20Nathan Labenz?

  So just to make sure I understand that correctly, I think I do. It's PyTorch and that is basically a super rock solid infrastructure foundation. And then after that, it's very little else aside from things that you. Roll your own to address particular needs and then only selectively build to anything more than just kind of an ad hoc problem solving addendum.

• 47:47Speaker 00

  There are pieces that at this point have reused enough and consolidated enough that they're like a staple part of my, like when I create a new folder in my model repo, personal model repo, there are a bunch of stuff that I import by default because there's like libraries for components, UI components that are reused, little code for like running evals on models, training models. If I'm not importing them directly, then I'm certainly copying and pasting them over. And one of the things that I've, I've learned in doing more research things over building products is that in research land, I just do not feel guilty about copying and pasting code because you have no idea how the thing is going to change. And it may be that copy pasting is just kind of like save you from not having to over generalize anything. So there's a lot of pieces that I have ended up reusing over time or importing over time, but those tend to be like things that I built up over time rather than someone

• 48:32Speaker 00

  saying, here's a library that helps you do interpretability research and then me importing it and learning how to use it.

• 48:36Nathan Labenz?

  Okay, cool. That's very interesting. What sort of models do you use today? Obviously, you know, when you started, the available models were interestingly, probably much closer to the frontier like GPT two. I think you started in the kind of GPT two, GPT J era, right? So at that time you had both way worse models than we have today, but way closer to frontier capability than probably what you can manage to use in any sort of convenient way today. So what, what models are you using now when you want to do a new experiment?

• 49:09Speaker 00

  Yeah, I, GPT J is, was like the biggest model, open source model that was available when I first started doing this. And then since then I've like parts of my infrastructure that upgraded to like Llama two and stuff. So in general, I think most interpretability research these days are using either like GPT, GPT two is just like the, there's so many interesting things about GPT two is like placed in time and also GPT three, but it's not open source. It's harder for people to work with. But GPT two and GPT three were trained at a time. When you didn't have to do a lot of data filtering over the internet because there was no existing language model data. And so it's like, it's kind of like pre-radiation steel in that like it's all it's trained on things that are guaranteed to be not AI generated.

• 49:51Speaker 00

  And there's a lot of other interesting things that both those models, all of them have like very well documented. It's a little bit like the fruit fly of the language model world. And so GPT is very popular in general. I think a lot of people doing interesting work with Lama two or Lama models because they're also popular open source models. I think a lot of serious, more academic interpretability work, especially more like theoretical ones have moved on to this family of models called Pythia, which is specifically trained for interpretability research. The way that it's specifically trained for that is that it has many more checkpoints through the course of training that you can look at. So you can look at how features evolve over time. It's fully reproducible, all the data is available, the training code is like well-known. That's in general what you find. A lot of my work, because again, I deal a lot of the time with embeddings.

• 50:35Speaker 00

  I work with open source embedding models. Or if I need to do reconstruction specifically, I have a custom model that I've trained for reconstructing text out of embeddings that's based on T5, which also was a cutting-edge model when I first trained this model, but obviously now it's not anymore. But T5 continues to be, I think, a very rock-solid model for if you want to fine-tune or if you want like a medium-sized model to do anything complex. And so I have a T5-based embedding model that I've trained once for through this like embedding, going back and forth between text and embedding task. And then I continue to use that for a lot of different things. And at this point, using that is easy because again, I've built a lot of scripts and tools for working with this particular model. And I have a bunch of data sets that I've generated with this model that take a bunch of time to generate.

• 51:16Speaker 00

  And so again, I feel like when I sit down to work now, I have a like a kind of sprawling workstation or a little like workshop environment where I've put all of the little tools and ingredients exactly where I need them at arm's length. And if I pull in a new model, then I'd have to redo a bunch of stuff. But now that I have a few like stable set of models that I'm working with, I can just kind of reach for them and put pieces together really quickly. And so my custom model is the mean one. And then just given the Quality and the popularity of like either open source embedding models that are at the cutting edge or OpenAI's embedding models, for example, sometimes I work with those as well.

• 51:52Nathan Labenz?

  How sort of exportable do you think that personal work environment is? You know, obviously, a lot of people, I think, are very curious about this kind of exploration and would like to be able to do some of it. And honestly, I would include myself in this to a degree. Like, I'm, I think I have a pretty good conceptual understanding of most of these questions these days, but I'm not like that facile with the code to be able to go in and, you know, just run whatever experiment. It's definitely still like a barrier for me to have a hypothesis and then be like, I can go run that experiment myself. I would find that to be broadly, you know, hard and slow.

• 52:35Nathan Labenz?

  Do you think that everybody just kind of has no choice but to, if they want to get there, you just have to slog through it and ultimately end up kind of creating your own thing? Or do you think I could like drop into your cockpit and learn to fly your plane, so to speak?

• 52:48Speaker 00

  It's a balance, right? I think that there's a part of, there are bits and pieces of my tooling that are like in exportable form. Some of them, I designed them that way from the start. Some of them I've like found to be kind of an isolated piece that I can now like share. Some of them are under the notion of Brolad, so I can't open source them that easily, but some of them I might in the future. The, my mental model for this again is like, if you're like a watchmaker and you have an expert watchmaker and like a watchmaking workshop set up or something, and even if it's an expert in an expert's workshop, you wouldn't expect to be able to just like drop into someone else's workshop and then immediately get to work. You either have to learn the way that things are laid out, or you have to customize the layout to like fit exactly the way that you work.

• 53:31Speaker 00

  Not because the tools are that different fundamentally or anything like that, but just because it's not, the things are not there when your hands is used for reaching for them. So like, more concretely, what that, what the metaphor means in terms of my working style is like, I have I have a local G P U rig on my desk where I am right now. I have a couple of like rigs in the cloud. In those computers, there's like a specific folder layout with like specific data sets in these places. And I have scripts that assume that those things live where. And I have a script that like when the model is trained, they like upload it to S three and all the stuff. And it's kind of a spider web, but it's like a spider web that works. The exportable bits are more like if there's a specific problem to be solved. Like I wrote a little custom mini language model training library for myself to make it really easy and efficient to like

• 54:17Speaker 00

  run a lot of G P T four inferences in parallel. And those kinds of things are like, they're like solving an isolated problem. So I can isolate them and then package them and maybe share them. But in general, a lot of a lot of the tooling that are built over time isn't actually like solving a particular hard technical problem. It's like, I want this thing to be there when I reach for it, or I want to not have to worry about where exactly my data set is. So it's very much about the things that are the the ways that things are laid out. And I don't know how much that is shareable. If I was at a company and I had to like unify everyone under the same tooling, maybe we could all agree to be like, okay, all the data sets go in this folder. All the model outputs like going this S three bucket. And then maybe there could be like a shared shared spider web. But in public and open source, I think that's much harder to achieve.

• 55:02Nathan Labenz?

  How much do you use language models to assist in your personal code writing these days?

• 55:08Speaker 00

  A lot, but I don't think any more than like most other people that I have like a copilot and have used chat G P T. I use Copilot, GitHub Copilot. I've also tried SourceCraft Cody. Most of the time they're just useful for kind of writing boilerplate code that you knew were gonna you were gonna use anyway. And so a lot of times I'll just write things and we'll pause for Copilot to do its thing and then I'll hit tab. They're never really useful for writing code that you're not sure of the of the like correct form. So a lot of time I'm doing kind of complex matrix algebra and the code base. And I could pilot would be like, hey, here's the here's the exact like five line operation that you need to do to like finish this function like I don't. I don't know if, I don't know, there's a bunch of arbitrary numbers in here. I had a bunch of like arbitrary dimensions in here. I don't know if those are correct.

• 55:48Speaker 00

  And so for those, I'll either like work it out on paper myself, or I'll frequently do this thing where I give some sample code to GPT-4 and I'll ask it, can you find the bug in this code? And if there is, it'll find it. If not, it'll say, I don't think, I don't see a bug in this code. That gives me some confidence. But ultimately, all of those are kind of accelerants. And I think that you always have to do the final verification of like, running the code and seeing that the output is exactly what you expect, which again is where PyTorch is much better suited than other more opaque tools, because when you're debugging things especially, you can print outputs in every step and see exactly whether the outputs match expectations.

• 56:22Nathan Labenz?

  So let's zoom out again or go up a level and talk about model capabilities. I mean, I guess maybe one little tiny follow up before we do that, that definitely relates is like, what scale of models are you using? And do you have any concern about like needing, like how concerned, I guess you have some of course, but like, how concerned are you about upgrading the core models that you work with to kind of make sure that the investigations you're doing are like, you know, on the most relevant systems. And this is something that we've mentioned Anthropic a couple of times.

• 57:00Nathan Labenz?

  And as I understand it, a big part of the reason that they feel it necessary to train frontier models, even though like their founding ethos is very AI safety centric is that they feel like they can only do the frontier interpretability work if they have frontier models. And, you know, they just feel like they're qualitatively different. So I'm kind of transitioning into the qualitative, you know, potentially qualitatively different capabilities that the latest systems have and wondering, you know, to what degree you feel like you need to run the latest models to have those, whatever those may be, to make sure that they're like happening on the system that you're studying.

• 57:42Speaker 00

  Most of this is speculation, right? Based on the things that we know about smaller models. And so that umbrella caveat applies to everything that I will, I will say next. I think there are two kinds of categories of facts that we can try to learn about models. One is things that we can learn about how neural networks work in general, or how transformers work in general, when trained under a particular objective. So this is, this is more like, The physics of a neural network under training. An example of something that belongs to this category is like the like linear feature hypothesis, where we generally, a lot of current interpretability work is happening under the assumption that models represent specific facts it knows about the input as a kind of direction in its vector space.

• 58:26Speaker 00

  That seems like the kind of thing that if you, if you can either theoretically demonstrate in a kind of like rigorous mathematical way in a proof, and if you can also observe in smaller models, it seems like you can make a pretty good assumption and conjecture that it'll generalize to large models. And I think a lot of my work is mostly predicated on that kind of thing, where it's about the physics of how these systems behave. And so even as you scale systems up, the way that models represent things and the way that information flows to these models are probably going to stay roughly similar. The other category of things that we can try to know about models are, for a given specific model, what are the, what are the things that it's doing? And what are the things that it knows? So for a specific model, what are the kinds of features that it's recognizing in the input?

• 59:10Speaker 00

  What are the kinds of circuits that it has that it uses to implement? What, how large are these circuits? What is the distribution of features? Are most features topics versus most features about grammar? Things like that. And I think that's something that, that's the category of facts where like we, we have some of this, but we have very little to go off of for projecting forward to say, what would the distribution of features look like for like 100 billion primary scale model? There are interesting analogues in vision models that I think where, where there have been interpretability work going on for a longer period of time, where something that's interesting about smaller vision models is that there's a kind of a sweet spot. When you have very small models that are not super capable, the features that they were, they use to represent inputs tend to be very not interpretable.

• 59:55Speaker 00

  And then as you scale them up, there's a kind of sweet spot where the features tend to be the most human interpretable. You can see features like, oh, this is a cat, and this is a dog here, and this is a wheel of a car. And then perhaps as you scale the models up further, maybe it's possible that the models think in terms of even higher level features that don't necessarily respond to the way that we conceptualize the world. And so that's the kind of thing that changes with scale. And that's the kind of thing that I expect we'll see in language models also. But if we, so there's some interesting, like I think OpenAI has done some interesting work in this space of how do you supervise and interpret and understand? Models, even under the assumption that they are using, they are thinking in terms of things that humans don't have words for. But a lot of my current work is in terms of how, how information flows through models and how models represent information.

• 1:00:38Speaker 00

  And I think for that, there are pretty solid assumptions that I think will hopefully generalize to large models.

• 1:00:43Nathan Labenz?

  So how do you think about some of the, I mean, here are some hot button topics or, you know, kind of a flashpoint vocabulary words. Things like reasoning capabilities in frontier language models. How, you know, obviously the one extreme view is it's all just correlation and there's no, you know, quote unquote, real reasoning going on. It's just statistical association and, you know, they're all stochastic parrots. My sense is to put my card on the table, I think that is not really true at the frontier. And it seems like I've seen plenty of interesting examples of problem solving, you know, that

• 1:01:24Nathan Labenz?

  seem to be meaningfully outside of the sorts of token level correlations, you know, that might be expected to be found even in like super broad training data to support that. How would you give an account of like reasoning abilities or things like grokking or things like emergence as we go to these frontier scales?

• 1:01:46Speaker 00

  When we talk about what models can do, I think there are three levels at which you can ask this question. Level one is, does there exist a transformer? Is it possible to have a transformer model that accomplishes this task? In other words, if you were on an omniscient being and you knew how to set every single parameter in a large transformer, could you manufacture a transformer that did this task? That's level one. Level two is, is it possible for like a reasonably randomly initialized transformer model to, through some gradient descent based optimization process, arrive at a set of parameters that accomplish this task with the model?

• 1:02:28Speaker 00

  And then the third level is when we are training language models in the real world with limited precision floating point numbers on internet scale data, will the resulting model be like sufficiently close to the ideal model that it like it it performs the task? And I think for for most definitions of reasoning and like problem solving and generalization, the answer to one, the the part one is like almost always a yes. Like, I think for any any given like logical deduction problem or any given like math problem, it is, I think it's generally possible to like, given lots of parameters, write a program inside a transformer, manually engineer all the parameters and and and have a model that's capable of doing it.

• 1:03:07Speaker 00

  My intuition, right, like non rigorous intuition is that given sufficient data and sufficient time and sufficient compute, it's probably possible to also like train transformers in the infinite compute limit that do any arbitrary like reasoning task. I say that because for like non trivial math problems, if you have enough examples, you can train very small transformers to like fully solve them, not fully solve them just in the sense that you can give it a bunch of unseen problems and it will solve them also, but fully solve them in the sense that if you look inside a transformer, you can see how the algorithm is implemented inside the transformer and you can show that that is like the correct algorithm that always generalizes.

• 1:03:49Speaker 00

  So I think in a lot of cases, it's it's possible to have a transformer that through gradient descent arrives at something that we would call like reasoning. The big question that I have personally in my head is, do we have enough data to train on such that for these extremely large models that have so many parameters, we could train them sufficiently long that they would go from starting to memorize to eventually starting to like looking like something that we would call like fully reasoning rather than relying on a mix of reasoning and and showing correlation? I think that's a big challenge. I think it's also very difficult to show in extremely large models currently that, for example, a model solved this problem by. Reasoning it within a circuit in some way over kind of like pattern matching. Because the diversity of the problems that we're going to show it for is really large.

• 1:04:33Speaker 00

  And because there are so many parameters and relative to the number of parameters that we have, I think, I think the data sets that we're using are relatively small. And so it's very possible for models to like mostly memorize and still do really well. And so I don't know if we'll ever get to that point because data is really messy. And I think to get to the regime where you train a model and you can show that the model is perfectly generalizing or perfectly reasoning, I think you need extremely clean data that basically only does the correct thing all of the time. And you need that for a long enough period that the model to anthropomize it a little bit feels comfortable forgetting its memorized solutions and only relying on its generalized solution. And so there's a lot of issues, I think, in terms of like practically can we produce models that we would be able to show is reasoning. But I think it's in theory possible to have models that do this.

• 1:05:16Speaker 00

  And perhaps with like research breakthroughs, we will get, we will get better at closing the gap between like options two and three.

• 1:05:21Nathan Labenz?

  I think one of the best, certainly the best thing I did and one of the best conversations I've had about a topic like this was an episode that we did with the authors of the Tiny Stories paper, which was out of Microsoft Research. And Ronan and Yuanju, they did something where they observed kind of that the, the order of learning seems to be like syntax first, like just part of getting the part of speech right, you know, from one token to the next, then kind of facts and associations and then reasoning. And so in order to like, try to get to reasoning, well, this is at least you could frame their work differently.

• 1:06:02Nathan Labenz?

  This is maybe not their exact framing, to be clear, but the way I've thought about it ever since is to get to that reasoning starting to happen as soon as possible and at the smallest scale possible. They created this vastly reduced data set called Tiny Stories, which is like stories that are short and that like a, you know, three, I don't know if it was a three year old or a third grader, but whatever, like reduced vocabulary, you know. Pretty simple little universe, but because the vocabulary size was, you know, was way lower and the like facts to, you know, that you might encounter were also greatly reduced. Then you could start to see some of these like very early reasoning abilities starting to emerge, even with just like a few million parameters.

• 1:06:44Nathan Labenz?

  I think the biggest model they created was like 30 million parameters. And by the time they got to like 30 million, which is still, you know, whatever, 3%, 2 to 3% maybe the size of GPT-2, they were starting to see some of these like very, you know, simple, but like negation, for example, you know, GPT-2 really struggles with it, but they were able to observe it like somewhat consistently, at least in, in their very small models by just like reducing the world. I have a big expectation that curriculum learning in general is going to be a big unlock for this. And I, I kind of, you know, we've heard from OpenAI and others, I think at this point too, that like training on code really helps with general reasoning ability.

• 1:07:28Nathan Labenz?

  My sense is that at this point, they're probably doing a bunch of stuff like that and kind of, you know, much like that paper we saw also recently from DeepMind on the geometry, where they like generated a ton of just like provably correct geometry statements and said, okay, we're going to start with this. And then you'll like kind of learn what a correct geometry statement looks like, and then we'll like apply you to this problem. I sort of suspect that the data sets at the frontier labs these days have a lot of just like pure reason, you know, kind of in, especially in the early going, when they're trying to establish these like core reasoning circuits. But that's on, you know, somewhat speculative authority for sure.

• 1:08:08Speaker 00

  I think synthetic, uh, something that's related to this that I thought about recently is on OpenAI's new video model. This is purely observational speculation, but sometimes you look at the motions in the videos that are generated, or the lighting or the shadows, and it looks like a video game. And partly based on that, mostly groundless suspicion, and partly based on

• 1:08:33Speaker 00

  This are the confidence that I think we both have about the importance of synthetic data. I think they're probably training on, at least early in training, on a huge amount of just like a game engine generated physics and motion and movement. As another example of synthetic data, tiny stories is also super interesting. And I think it's an example of like, in general, as you said, models are very lazy about what it has to learn. And it only learns the thing that you want it to learn when it's run out of options. It's exhausted all of the other options that it has to try to minimize its loss. And the only remaining option is to like finally learn the thing they wanted to learn. In in language theater broadly, I think it's so difficult to get to that point. Like even even if you think about like the like math proofs that occur naturally in the internet. For example, there are a bunch of proofs on the internet that are just incorrect.

• 1:09:18Speaker 00

  And so in order for the model to like perfectly learn how to solve a math problem, it has to first or at least simultaneously also learn how to disambiguate whether the the proof that follows some text is going to be likely to be like wrong or or correct. There's a bunch of like noise like this that I think either you can have relatively small amounts of very high quality synthetic data, or you can have vast, vast amounts of like in the trillions of tokens range of extremely noisy data that is only like kind of filtered. But to get to like getting these very large models to reason in the way that like I think humans want to reason with natural language, I think is going to require bridging the gap. And that's a really interesting data problem for sure.

• 1:09:56Nathan Labenz?

  In in lightning round, so we've just been talking about these kind of, you know, what exactly are the the current frontier capabilities, you know, exactly how much reasoning is going on? I, you know, I agree, it's hard to say. I think if I could summarize our relative positions, I think I probably see more reasoning in it than you do today, although it sounds like I definitely, it's it sounds like we're together in that I definitely don't think of it as a binary. It is very for me like, it's not that it's all this or all that. I always say AIs defy all the binaries we try to force them into. And instead it's like some mix, right? Like I would say, it seems that there are times when it can reason and there are other times when it can't reason and, you know, mapping that boundary, which, you know, by the way,

• 1:10:40Nathan Labenz?

  might be fractal from for some other results we've seen recently is like definitely a super hard challenge. To get practical on this for a second, so you're obviously. Working at Notion AI, a product leader there and bringing frontier models to the masses. I you've shared in detail in other places, you know, some of the functions and gone into how you're building it. I really liked the podcast you did with Redpoint, by the way, for those that want to get into the how Notion is made. I thought that was a really good one. So skipping over all of that kind of stuff and now just saying, okay, for the stuff that it's not able to do yet, as well as you would like, and as well as you would hope to provide

• 1:11:22Nathan Labenz?

  to Notion users, what do you tell the model developers are the shortcomings? Like, where do they need to improve to give the quality of experience that you think, A, like, they kind of should be able to get to next and, you know, would be most useful at that intersection of feasibility and value? I can run down a list, but you could just respond free form.

• 1:11:46Speaker 00

  This is a list that's always top of mind for me as well, because I talk to all, everyone on the team talks to companies building models and they always ask us this question. The main ones that are always top of mind are, we want models that hallucinate less. We want models that are cheaper and faster, lower latency, and we want models that follow instructions better. And there's a fourth one, which is like a big one, but very hard one, which is we want models that are better at general reasoning. All three of these, so not hallucinating, lower latency, and following instructions more faithfully, following complex instructions more faithfully, are, I think, they are all areas where, for example, CLA II or GPT-4 have shown market improvements over the previous generation models, except for obviously they're both more expensive.

• 1:12:27Speaker 00

  But I think those, the combination of those three, or a couple of those three, have enabled entirely new kinds of products. So, for example, Notion has a product called Q and A that helps you answer questions based on all the information in your knowledge base. And to do that requires both not hallucinating as much, not just from its own memory, but from information that it has right in its context, but it also requires following extremely complex instructions, because we have massive books of instructions metaphorically that are just like. Dozens and dozens of bulleted lists of here is how you should format your answer. Here's what you should do. Here's what you should not do. Here's exactly what you should do in this situation versus this other situation.

• 1:13:07Speaker 00

  And earlier models have had a lot of trouble following that kind of like 1,000, 2,000, 5,000 token long instruction. And so long instruction following, I think, is like an area where we've continued to see improvements, even in the latest iterations of GPT 3.5. And it's very essential. Polification makes a lot of sense. And then just as instruction following enables a bunch of use cases that were impossible before, I think latency, lower cost and faster latency enables a bunch of use cases that were hard to pull off before. Doshun also has a feature called AI Auto Fill, where you can use a language model call, kind of like an Excel formula, where you have a data table or a database of interviews or companies or customers or like your classes or whatever.

• 1:13:52Speaker 00

  And then instead of filling in a column with like a formula, you can fill in a column with like a language model prompt. And to be able to do that at the scale of tens of thousands of rows that some of our customers have in their tables requires models that are both fast and also efficient. Because we want to be able to send OpenAI or Anthropic thousands of these requests in a matter of like a few minutes and have them be fulfilled reliably. And so that's an example of a use case where as the models get to you to get better at this like perimeter frontier of latency and cost versus capability, we're going to continue to be able to make new kinds of products.

• 1:14:27Nathan Labenz?

  I often have this debate with product people where I say, you know, you got, I think it was in your fourth or fifth spot before you got to general reasoning ability, general kind of capability. I always put that first. And I say that as somebody who's also like kind of scared of an AI that's smarter than me. But definitely, I feel like on my personal utility, you know, and the results that I get day to day, I'm always like, it's pretty fast, you know? And it's pretty cheap. It would be nice if it was faster and cheaper. But like what I really want is for it to be smarter. What I really want is for it to like write as me better, you know, better in context learning or better kind of style imitation on the fly.

• 1:15:11Nathan Labenz?

  So it can kind of put something together. In the way, you know, or at least close to the way that I would want it. Why isn't that like the very number one thing? I mean, if you had that, couldn't you charge like hundreds of dollars a month for Notion AI? Is there something I'm missing about that?

• 1:15:26Speaker 00

  Part of the reason why that's at the end of the list is because it's, I think it's much harder to get to. Basically, this, again, obviously, this is something that new models continue to get better at. And I think like between going from GPT three point five turbo to GPT four, for example, we saw improvements in how like how people were using it. And the quality of feedback that we were getting. Better reasoning is like, is obviously equal, but also if we go to like Google and say, hey, we want models that are better reasoning. Their engineers are going to be like, what, what do we do with this? What do we do with this piece of feedback? Like, obviously, we're trying to make models that are better reasoning, but how do we measure that? What exact kind of reasoning? And so in some ways, following instructions and not hallucinating are facets of reasoning that I feel is most tractable in like a few months to a year timeline.

• 1:16:09Speaker 00

  And also to have the, the unblock the most urgent current like problems with the existing products that we have. Another way to look at this is I think a part of your desire for better reasoning is a function of the way that language models have been packaged for you, which is you interact with it in a single threaded way, one at a time. As the model is generating the tokens in real time and you ask it questions. And I think there's a lot of other ways that language models can be used that isn't packaged that way. So for example, an autofill feature that we mentioned, that's a bulk operation. Someone might upload 2000 rows CSV and fill in five different columns. And wouldn't it be better? First, wouldn't it be good if you could do that, which is a cost problem? And then wouldn't it be good if you could write a prompt, run that in a few seconds later, have the entire table be filled?

• 1:16:54Speaker 00

  And you can filter it by category or feature or where people are, so on. And those kinds of experiences are not really bottlenecked by reasoning. Most of the time they're bottlenecked by things like speed and latency. And especially if you're trying to do this at the scale of like billions of tokens generated per day. This is a huge, huge problem. There are like categories of products that, for example, smarter, smaller startups might be able to launch, but we might not be able to launch just because for us to deliver that same thing would cost us like enough money to, to not be feasible. And then I also think there's a lot of useful work. That I always kind of joke that most business documents that most people write, or most academic documents that most people write are just interpolations between other business documents that have previously been written before.

• 1:17:38Speaker 00

  And so we for a lot of the, and I think it's like 95% of all documents that are being written inside Notion, not because there's something novel in there, but just because, you know, given enough information and diversity and all the emphasis that these models have seen, most incidences of like problem solving or doing important work, communicating well are like within the training set. And so for those, I don't actually think the biggest problems are like getting the model to reason outside of what it's already seen, but just being better at being better at following instructions so that we can steer it towards the right space in the space that we've already seen.

• 1:18:11Nathan Labenz?

  Perfect transition to my next question around the next big model upgrade. You kind of alluded to it a little bit, but what does that look like? And, you know, what, because one of the big theories I have right now is that we have as an, as an industry, as an AI app industry, if you will, we have been building a ton of stuff that basically tries to compensate for the language model's weaknesses, right? Where like they hallucinate, so we ground them in database access, they, you know, have very limited context, so we got to like re-rank the results to try to make sure we're getting the right stuff into context. Obviously, we have, you know, every part of that pipeline can fail.

• 1:18:54Nathan Labenz?

  And then, you know, you might just have something come along that like changes the game and maybe it just did in the form of Gemini 1.5 with up to a million and maybe up to 10 million tokens of context and like some pretty wowing recall demos. I guess, first of all, I'd be interested to hear like kind of what process you plan to go through. I'm sure you have like a standard set of evaluations that you would, you know, get like systematically comfortable that like, okay, we're not regressing. Interested to hear what that's like. But then I kind of also wonder, do you see that the next model, like a Gemini 1.5, could perhaps lead to a significant leap in performance, given all that scaffolding, or possibly even like make some of that scaffolding kind of not necessary anymore?

• 1:19:39Nathan Labenz?

  Like, do you think the system itself maybe gets simplified because the new model just like is more tolerant of just, hey, just throw some more stuff in it? You know, we don't have to work so hard on, on some of these previously hard problems.

• 1:19:52Speaker 00

  I'm very curious what use cases motivated million token context window for the new Gemini models. My hunch is that it's actually mostly multimodal use cases. I can imagine a long piece of audio, a long piece of video filling up the context. But I think most of the time in the, if you're purely in the, in the text realm, I think it's difficult to imagine. There's a lot of benefits of retrieving limited context rather than just putting everything in a, in a model window. Some of them include observability. So if you give the model 10,000 inputs and it gives you the right answer or and it gives you the wrong answer, how do you debug that? Right? Maybe you can look at things like attention maps and so on, but it's like, that is an interpretability problem in itself, where if you have a pipeline that gives you maybe

• 1:20:35Speaker 00

  that top 10 documents and has a language model answer that, answer the right question. If you've got it wrong, you could ask useful questions like, did the answer exist in the documents that it saw? Was it at the beginning or the end of the context? If you swap this out, does the model get a different answer? And so having a more pipeline system, I think, helps you debug. I think it helps with costs, obviously, and latency for a given compute budget. I think it also lets you increment the upgrade different parts of the system. So having a better language model while keeping the same retrieval pipeline versus having a better retrieval pipeline with the same language model both improve, improve the results. So I think there's a lot of just structural benefits to having this pipeline model. And on, on the like Gemini results in particular, they have a lot of these tests that are kind of needle in the haystack tests.

• 1:21:19Speaker 00

  If I have a million tokens, there is, there is an example of an anomaly somewhere in the million, can you find it? And in the real world, a lot of the kind of complex retrieval that models have to do is actually so much more nuanced. An example of something that more nuanced could be like, a couple of examples. One example is the model might have to not just find the information, but do some reasoning to figure out whether that information, which of the pieces of information to use. Like in Notion's case, in the retrieved documents, some documents may be out of date, some documents may be written by someone who's not authoritative, some documents may be written in a different language and it might conflict with the, with information written in like the canonical document written by the HR team at a company or something like that. And the model has to, and we have instructions for how to deal with all of these cases.

• 1:22:03Speaker 00

  And so the model has to do some reasoning over exactly what. And that looks very different than just like find this word and some million tokens. Another example of a more complex retrieval case is when there needs to be synthesis. So if you ask a question, if you're like in the notion, internal notion, and if you ask a question like, Has the AI team been productive this week? That's a, that's a, that's a multi-step question that not just, first of all, that requires knowing not just a single answer to a question, but in general, what's been going on? And then that requires like maybe reasoning through, okay, what is, what does it mean when this person says productive, who are all the people in it? Uh, maybe are there, are there people that are out of office? And so again, a lot, a lot of problems there are not just about like all the, what information is in the context, but actually how the model performs.

• 1:22:46Speaker 00

  And all, uh, it's the biggest challenges that we've faced so far in optimizing retrieval is mostly those kinds of things, those kinds of more like reasoning related or edge case situations where it's unclear exactly what the model should do based on our existing instructions. And the best ways that we found to attack those problems involves a lot of like stepping through all of the steps in the chain and saying, was the answer found at this step? Was the answer found at this step? And all of that debugging is just much easier when you have like 10 examples to look at instead of 10,000. So at least for, for this particular use case, I think context length is not the most dire need, but obviously there are lots of other use cases, like for example, audio, where I think it could be a big changer.

• 1:23:23Nathan Labenz?

  Yeah, very interesting. I'll be very keen to spend more time figuring out exactly what I can do with a million tokens and totally agree with you that the needle in the haystack results that we've seen are not, not enough of a, an answer to, you know, how it really performs to be confident just dropping it in. Although I, you know, I aim to find out exactly how far this thing might have advanced. And it does hold a lot of promise for me just because first of all, this podcast, you know, might be more than like GPT-4 Turbo or Claude 2 can really handle. It probably would fit, you know, the transcript probably fits in the, in the 128K or certainly

• 1:24:06Nathan Labenz?

  would fit in the 200K, but it's getting to the point where like the recall isn't great. And just to be able to like, take my podcast transcripts and throw them into something and get like reliable answers, you know, that I feel like are better than I could delegate, you know, an answer pretty reliably like that alone is like super exciting. And then I'll be, it'll be really interesting to see how the synthesis part of that works with those.

• 1:24:30Speaker 00

  Yeah, I mean, there, there's a part of me that wonders, like, train, training a model of such long context, that requires probably architectural inventions, probably some like engineering work, definitely a lot of like extra data work. Like, there, there are only so many examples that are high quality that are like that long. And so there had to have been some good use case motivation for Google to go ahead and train that model. And I'm very curious, even, even just like internally, what they're using that, that context thing for.

• 1:24:55Nathan Labenz?

  Yeah, I, agents, I think is going to be one also super interesting application, like to what degree can you just kind of append your past failures and continue to roll forward in a sensible way, learning from what happened. Today, obviously, you know, you can't just drop all your API failures into a single context window that you'll, you'll run out, just to run out even to 10 million eventually, but it could make a, a big difference for that sort of thing. How much is any big tips that you have for AI app developers? Things that you like know to work that you just don't see people doing enough of? This could be in the prompting domain, it could be in the rag domain, it could be in how people set up their evals,

• 1:25:41Nathan Labenz?

  or even just the user experience and the interface that, that are presented.

• 1:25:45Speaker 00

  Both of these are kind of boring things, but I think they're not, they're, they're always, you could always be doing more. The first is regardless of what exact kind of model you're training or working with, like obviously always pays to spend time with data. And I don't just mean like run an evaluation and look at the charts, I mean like.

• 1:25:58Nathan Labenz?

  Read the logs.

• 1:25:59Speaker 00

  Exactly, like make the table of like a hundred outputs and a hundred inputs. And ask yourself for this input, how would you, what output would you, the human, generate? And then think about what all the steps that you go through and reason through that and look at edge cases, look at what the models fail. I've also found it just like personally interesting, if, if a little dubious in value, to look at pre-training data, like, like raw pieces of text from the pile data set, there is a lot of stuff in it, like, most of the text on the internet is like quite garbage. And there's a lot of stuff in there that I think. Like gesture at, for example, the reason that language models are good at certain kinds of output formats and not in other formats. But the general theme, I think, is spend a lot of time with your data, in particular,

• 1:26:42Speaker 00

  with your input data that you're giving the models and the tasks, and then also with failure cases in the wild. In the beginning with Notion AI, we spent some time setting up a system for us to have kind of like human annotated logs and a more kind of scaled automated system for detecting errors and fixing them. And eventually, what we've settled on for a lot of our features is instead, we have, like, the engineers have scheduled time on our calendar every week where we go into a meeting room and we just stare at a Notion database of all the bad cases, like individual outputs that were bad, that were reported by our users, and we ask ourselves for each input, what is the exact step in the pipeline where this failed? What category does this belong in? Is this, we kind of treat it like a software bug. And we say, is this already being fixed? Is this an instance of a new bug? Is there, is there like a systematic issue in a pipeline that, that requires us like inventing something new to fix this?

• 1:27:27Speaker 00

  And so spending a lot of time with failure cases in data, I think, pays off. And then, sort of on, on a similar theme, I think, having, investing early in internal tools to quickly run evaluations, quickly and easily run evaluations, quickly and easily kind of generate synthetic data sets, visualize outputs, sort through them, I think, I think helps a lot. And there, there are a few companies out there doing stuff like this. I think for working with your own data and visualizing it in particular, that generally the tools shouldn't be so complex that you, you can't just have an engineer kind of like whip it up in a couple of days. And I think that pays off in the long run by you being able to customize it over time, as I, as I alluded to before. So that's kind of what we've done at Notion. Obviously, your mileage, mileage may vary, but I've, I've found those two things to be like particularly worth our time in terms of improving the product.

• 1:28:11Nathan Labenz?

  I think those are very solid tips, and I agree with you that there, you know, while it may be sort of not exactly what people are looking to hear, the admonition or reminder at a minimum to look, read the logs, just look at the raw data, look at the failure cases. I have certainly been served extremely well by that over time, too. Like, there's just no substitute for looking at the raw inputs and outputs.

• 1:28:35Speaker 00

  Once you understand the problem at a deep level, then you can start thinking about Like we're running out of time during the week to do this. How can we scale this out? How can we automate this? But like the automation is not really possible until you have a really solid understanding of exactly what you're trying to monitor for, or we're automatically fixed, or we're protected.

• 1:28:51Nathan Labenz?

  Yeah, okay, cool. All right, last section of things that could possibly get weird. Two big trends, I'd say one, you know, kind of predates the other a little bit, but both are increasingly well established at this point. One big trend is that it seems to me like all the latent spaces can sort of be mapped on one to another. Right, we saw this kind of arguably first in a really powerful way with CLIP, where image space and text space were kind of brought together. And now we have text image space, like amazing. But that was done in a high scale way. And then it wasn't too long after that that we got things like, for me, BLIP 2 was really a major moment.

• 1:29:34Nathan Labenz?

  I'm sure it wasn't the first, but it was like one of the first that I really read and understood deeply and realized that, holy moly, like they've got a frozen language model here and a frozen image model trained totally separately. And then just a small bridging connector model between them that just took like a couple GPU days to train, is enough to unlock, you know, even that was like beyond CLIP level capability. But we just see so many examples of this now, right? Where one latent space to another, often just a linear projection, sometimes a small connector model is kind of all it takes. And then another big trend has been the realization that like,

• 1:30:14Nathan Labenz?

  a lot of models can be kind of merged together in seemingly unprincipled ways or like low, like not very principled ways. Right, we've seen some going back to like, I think it was called Relative Representations, was a paper that showed that models like initialized differently, trained, you know, with kind of different shuffles of the data or whatever, still seem to converge on like pretty consistent kind of isomorphic representations of data that are like, maybe a rotation away from each other, or maybe, you know, a dilation and a rotation away from each other. But ultimately like look very similar in visualization. And now we're getting to people saying, well, hey, what if we just train model A over here on one data set and B over here? And just like add their weights together, you know? And it's like, wait a second, that works?

• 1:30:59Nathan Labenz?

  It doesn't maybe always work, but like to a surprising degree, these sort of things are starting to work.

• 1:31:06Nathan Labenz?

  You can comment on that in any and all ways, but the way that I'm most interested to ask you about is like, is there a risk or a concern, or maybe you would see it as a good thing, that AI systems may start to communicate between each other in a higher dimensional, more embedding mediated way, or middle layer activation sort of way, as opposed to through different forms of communication that we could actually read in a native way? Right? It seems like the models don't really need to generate text to send the text over to one another. Instead, they probably are going to perform better with high dimensional vector communication.

• 1:31:49Nathan Labenz?

  But I kind of worry about that at the same time, even as it may boost performance in various ways. I kind of worry that like, much like you said earlier with the, how do you debug that? Now it's like, well, I can't even read the messages that they're passing back and forth. So how big do you think those trends are? Are you excited about those trends? Do you see downsides with the upsides? All things about kind of weird embedding based communication and model merging.

• 1:32:16Speaker 00

  Yeah, model merging is like truly weird and I don't have a great principle of understanding of exactly why it works so well the way it does. I have a general intuition of like why it works decently, which is that a lot of these models are fine tuned from a single base model. And when you find two of these models, especially in tasks that look similar to the original task, like mostly doing natural language continuation, the models mostly tend to exist in a kind of like linear subspace of the original like model space. And so when you merge them, you're mostly doing linear interpolation between weights of these models. And so it makes sense that, perhaps it makes sense that the resulting model would just kind of inherit behavior between a bunch of different models. It's still very weird, but like, when you view it that way, it doesn't seem quite like black magic. It just seems like kind of something that we have to understand.

• 1:33:00Speaker 00

  Relative representations and other kinds of projections between spaces, I think are really, really cool. It's cool, obviously at a like theoretical level. I think it's also really cool to observe it at a technical level, an empirical level. One of the other experiments that I did late last year was to find mappings between embedding spaces. So you could, I had a model that was capable of like inverting and finding features in one embedding model. And then I just trained an adapter between OpenAI's embeddings on my model. And it was a linear adapter and I could start to read out OpenAI embeddings, even without spending so many tokens, like training an OpenAI custom kind of inverter. So all of these things I think are super fascinating. I think. My, where I place this technique in the kind of like pantheon of all these different things you can do with language models is a parameter efficient or compute efficient

• 1:33:44Speaker 00

  of fine tuning or of customizing a model where instead of fully tuning a new image to text model or fully tuning a new way to go, new way to like invert an embedding, you could take in a model that's like close enough in what it represents and then maybe tune some few parameters to get to the final destination. To your question about whether models will communicate in latent spaces, if you think about it, like GPT-3 is just kind of like 100 GPT-2s talking to each other in activation space. Obviously, it's not quite that, there's denser connections, but if we can manage to precisely understand exactly how different layers in a transformer or different token residual streams in a transformer communicate with each other, I think a lot of those techniques

• 1:34:27Speaker 00

  will definitely generalize to understanding like mixture of experts' models or understanding ways that fully tuned models like L2 that you you mentioned communicate with each other through kind of like mapping a representation space. In some ways, it's actually easier because unlike a transformer residual stream where the concepts that it may be representing could be really weird, like you could imagine a concept that's really useful for predicting the next token in a Python program but not really useful for humans in general in life, a lot of times when you're mapping between kind of like fully formed representation embedding spaces, between like an image embedding space and a text embedding space, I think most of those, I intuitively expect most of those concepts to be pretty human interpretable. And so a lot of the kind of mechanistic techniques that I think people are working on today will probably generalize to understanding them.

• 1:35:12Speaker 00

  And so I just view it as kind of a exciting, efficient way to build more interesting systems.

• 1:35:18Nathan Labenz?

  How weird do you think things might get over the next couple of years in general? I mean, it seems to me like we're headed for at least one more and probably more than one more notable leap in model capabilities. And I feel like I have a sort of rough intuition for what GPT-5 would be, which I might broadly describe as like smarter, to borrow a word from Sam Altman, you know, better general reasoning capabilities and probably more long-term coherence as another big aspect of that. You know, they've seen how many people are trying to make web agents and various kinds of agents on the platform and it's just not quite working. So I would expect to see like smarter and sort of more

• 1:36:04Nathan Labenz?

  Readily goal directed as kind of two big advances for the next generation. Beyond that, it starts to get honestly kind of hard to figure out like what would even happen and you know, what it would mean. But do you have kind of a a sense for like, how how big is your like personal Overton window or kind of, you know, cone of possibility for the next few years? Like, how how far do you think AI might go in a few years' time?

• 1:36:30Speaker 00

  I mean, everything monotonically improves from here, right? I think that's the scary part. M. K. Bishdi has this good video on Sora where he utters this phrase of like, this is the worst that this this technology is going to be from here on out. And I I I think that's a really succinct way of expressing the fact that like, OK, you may maybe you think GPT four is like not not like super, super, super smart. But like this is like, if you look back at the history of smartphones, every phone when it came out is the worst that smartphones are ever going to be from that point on out. And it's like only gotten monotonically better. And I think when you think about that, I think language models monotonically improving so rapidly from here and out is like, as a trend line, I think it's interesting and scary. Their long term coherence and I think goal directedness is in particular is really interesting.

• 1:37:11Speaker 00

  Right now, with every GPT iteration, I think OpenAI has done a little bit of not just making the model, the base model smarter, obviously, but some opinionated tweaks to the final tuning objective to make the model more useful for certain use cases. Obviously, the big most recent one was like API use before that it was chat. But I could imagine OpenAI or other model makers tuning their models to be better at kind of not just having the model to expect the world to end after the next turn in the conversation, but expecting that there's further turns and maybe planning for, OK, if I assume that I'm given, you know, infinite turns into the future, what might I start to do? So that kind of like long term planning, I think, is something that's missing in current models that make it very hard to use for agents. So I agree a lot with that.

• 1:37:52Speaker 00

  And then beyond models themselves, there's obviously lots of corners of culture that these models touch. And I think that's a much harder, much more complex like dynamical systems. So I think it's much harder to predict exactly what will happen like to the concept of copyright, for example, or even to our concept of exactly what. A single piece of creative artifact is, there's a really good TEDx talk by creativity and HCI researcher, Dr. Kate Compton, where she talks about the idea of an image generation model, not just like, when a human produces a piece of art, you make the thing and then it's just like that, that is the concrete object. But when you have a model that's capable of producing a bunch of images to a single piece of text, or like millions of images

• 1:38:34Speaker 00

  at once, one way to look at it is some of the model is just producing art faster than a human. But a different way to look at it is the model, the model is a tool to map out the space of all possible outputs of a certain style of art or a certain prompt. And so it may, it starts to kind of change exactly what we imagined a single piece of artifact to be from like, a single blob of pixels to like, here's now a kind of subspace of all possible outputs that as a bundle is a form of creative expression. So there's a lot of culture stuff that I think is just much harder to predict that I'm like frankly not equipped to. I have too much smart commentary on, but I think it would be very interesting to watch and probably have ripple effects beyond the model capabilities themselves.

• 1:39:15Nathan Labenz?

  Do you have a sort of positive vision for the future for just like your own life? I find that this is in very short supply and certainly you've been one who has been so up close and personal with the AIs as they've developed. You might not, but if you do, I'd be interested to hear your sort of day in your own life, you know, three years from now, five years from now. What is AI doing for you? You know, what are you able to do that you couldn't do before? You know, who knows what Hollywood's doing or how the, you know, how the entertainment industry has evolved. But like, what is, what does the day look like for Linus as things really start to hit, you know, the, the key thresholds for utility?

• 1:39:55Speaker 00

  The high level concept that I gravitate towards when I think about this is like, you can take a base technology and express it in a way that's agency taking or agency expanding or agency amplifying. An example of a tool that takes away agency from a human is like a dishwasher. But like, that's fine, because I don't actually care about. Creatively washing my dishes or like exactly which order I wash my dishes. I just want them washed or like a laundry machine or a car maybe. And then there's a bunch of technology packaging, ways of packaging technology where preserving agency really matters. Like a writing tool is an obvious example, but maybe also more subtle things like which emoji show up first in an emoji keyboard

• 1:40:36Speaker 00

  or predictive text keyboards or obviously social media algorithms. These are kind of like somewhere in between agency taking and agents amplifying. And one thing that I'm kind of concerned about right now is that I don't think people are thinking enough about whether the ways that language models are packaged is amplifying human agency or taking away from it. And that's something that I think I want to like talk and think more about and perhaps like push other people building this space to to more be better at. Assuming that we can steer the way you package language models to kind of respect agency where it is required and only take agency where we want the models to take agency. I'm generally a pretty optimistic person about technology and I think I have a lot of optimism for for where this leads.

• 1:41:21Speaker 00

  As long as the way we package these things is more humanist rather than just kind of like automate all of the things. When you look at different kind of AI companies, you see companies situated at different points in the spectrum between you want models to automate things in a way that takes away agency, i.e. replacement. Or you want models that amplify. For example, like I think OpenAI is very much on the replacement side. Literally their definition of I think AGI is something like a thing that can take over a single like full humans job. Where if you look at a company like Runway, a lot of their framing of like usefulness or depth, perhaps a lot of their framing of usefulness is about extending that agency of what you want to express. And so there's a healthy amount of diversity here and I think it's just a matter of who.

• 1:42:06Speaker 00

  Where the winners sort of end up lying. Assuming we get that right, I think I have a lot of optimism for where we're going.

• 1:42:11Nathan Labenz?

  It's a big question, but I agree that's a key one. We want to guard our agency probably increasingly jealously, especially the more different AI systems might want to usurp it. That could be a great note to end on. Anything else you want to touch on that we haven't?

• 1:42:24Speaker 00

  No, I think we covered interfaces, capabilities, interpretability, all the things I spend my time thinking about.

• 1:42:31Nathan Labenz?

  How people can be more like you, of course, a key highlight as well. All right, I love it. Well, thank you very much for doing this. You've been really generous with your time and insights and definitely count you among the must follows in the space for all sorts of new and very generative ideas. So it is my honor to have you. And I will say in closing, Linus Lee, thank you for being part of the cognitive revolution.

• 1:42:54Speaker 00

  Thank you. It was my pleasure.

• 1:42:55Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email or you can DM me on the social media platform of your choice.