AMA Part 2: Is Fine-Tuning Dead? How Am I Preparing for AGI? Are We Headed for UBI? & More!

In this AMA-style episode, Nathan takes on listener questions about whether fine-tuning is really on the way out, what emergent misalignment and weird generalization results tell us, and how to think about continual learning. He talks candidly about how he’s personally preparing for AGI—from career choices and investing to what resilience steps he has and hasn’t taken. The discussion also covers timelines for job disruption, whether UBI becomes inevitable, how to talk to kids and “normal people” about AI, and which safety approaches are most neglected. Sponsors: Blitzy: Blitzy is the autonomous code generation platform that ingests millions of lines of code to accelerate enterprise software development by up to 5x with premium, spec-driven output. Schedule a strategy session with their AI solutions consultants at https://blitzy.com MongoDB: Tired of database limitations and architectures that break when you scale? MongoDB is the database built for developers, by developers—ACID compliant, enterprise-ready, and fluent in AI—so you can start building faster at https://mongodb.com/build Serval: Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week four at https://serval.com/cognitive Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Claude Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at ⁠⁠https://claude.ai/tcr⁠ CHAPTERS: (00:00) Ernie cancer update (04:57) Is fine-tuning dead (Part 1) (12:31) Sponsors: Blitzy | MongoDB (14:57) Is fine-tuning dead (Part 2) (Part 1) (26:56) Sponsors: Serval | Tasklet (29:15) Is fine-tuning dead (Part 2) (Part 2) (29:16) Continual learning cautions (34:59) Talking to normal people (39:30) Personal risk preparation (49:59) Investing around AI safety (01:00:39) Early childhood AI literacy (01:08:55) Work disruption timelines (01:27:58) Nonprofits, need, and UBI (01:34:53) Benchmarks, AGI, and embodiment (01:47:30) AI tooling and platforms (01:57:01) Discourse norms and shaming (02:05:50) Location and safety funding (02:15:17) Turpentine deal and independence (02:24:19) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Welcome back to the cognitive revolution. This is going to be the AMA part two. And again, because the schedule has been a little bit crazy, I didn't schedule this and just found a good time to do it on a Saturday, early afternoon, while my kids are playing video games. So there's nobody here to ask me the questions. It's just going to be me taking us through this time, a pretty good variety and diversity of listener submitted questions, plus a couple of AI written questions at the end. I teased a couple of times in leading up to this, that it would be interesting to see whether our human listeners or our, or my AI accounts on chat GPT and Claude would come up with better questions. And I definitely think the humans still did the better job, the more interesting questions, interestingly, more technical questions.

• 0:44Nathan Labenz?

  The AIs, I thought, were a bit sycophantic in their questions for the most part. And they were asking a lot of stuff about like me, like, you know, how do you do this? Or, you know, how do you manage that? And I think that's not really what people are tuning in for is to hear, you know, my reflections on my life for the most part. It's more to learn about AI and certainly the human questions reflected that. So I will take one moment just to start off with a quick, how's Ernie update? And the answer there very happily is that he's doing really well. We're about halfway through the chemotherapy treatment schedule in terms of time. He was diagnosed early November, the treatment's probably going to run about six months, maybe a little less. It's at least going to go probably through the end of March and could bleed into April.

• 1:29Nathan Labenz?

  We'll see. But in terms of pain, it seems like potentially a large majority of it is behind us now. He just mostly finished round three of treatment and it was much, much easier on him than the first two rounds. So that was great, even though we spent a decent amount of time in the hospital again, because he spiked like a very small fever and they're very worried about infection when the immune system is suppressed. So we had to go in and end up staying for a number of days. But it was honestly not, I wouldn't, I wouldn't say I enjoyed being at the hospital, but we actually were able to have a pretty decent time at the hospital because he's feeling well. It's not like there's that many things going on. He's able to play video games. We're able to get online and play video games with friends.

• 2:11Nathan Labenz?

  So it feels like we're starting to turn the corner back toward normal. And in terms of, you know, our worst fear, which is a relapse, you know, this thing coming back with a vengeance, we can't entirely rule that out, but the minimal residual disease testing, which you may remember AI tipped me off to in the first place. Has also been really encouraging. We've now got two of those test results back. One from a blood draw that was taken just before his second round and one from a blood draw taken just before the third round. So in other words, with one round and with two rounds of treatment complete, plus some lag time for cells to come back. They start the next round of treatment once your immune system cells, your red blood cell production

• 2:53Nathan Labenz?

  and your platelets all kind of come back toward something approaching normal. And that also in theory gives the cancer cells, if they are there, time to resurge. And so we, doing the blood draw right before the next round of treatment in theory would catch the most, would be at the time in the cycle where there's the most cancer there. And in the first round, it showed a trace amount basically. And in the second round, even better. There are two kinds of tests. One looks for free-floating DNA in the plasma of the blood. There was a 30x reduction, or basically like 3% as much of the free-floating DNA in the second test result as compared to the first. And then they also look for actual

• 3:36Nathan Labenz?

  live cells that contain the DNA sequence that is specific to the cancer sample. And in the second test, he had zero cells out of more than 3 million cells analyzed, zero came back with the cancer sequence. So that's outstanding. We're going to continue to do these tests from time to time. If we do ever see that start to increase again, it would definitely put us into a very different mode of thinking. But as long as we kind of see continued zeros in terms of the live cells, we should be headed for a cure and back to normal. Obviously, knock on wood, fingers crossed, whatever, good vibes. But for the first time, seeing that zero live cells, I felt myself start to relax a little

• 4:20Nathan Labenz?

  bit. And certainly that was a great feeling, especially combined with him just being more himself. So again, thank you for folks who've reached out with well wishes. It's crazy how. Close he was to dying, really, it was just a few days away, you know, when he, when he finally got diagnosed and treatment started, but the bounce back has been really equally fast and as scary as the down trajectory was, the up trajectory has been, you know, similarly inspiring and there's just so much to be grateful for in terms of all the work that people have done over generations to get us to this point. Okay, let's get into the questions. First question, is fine tuning dead? This is a great question and I think

• 5:04Nathan Labenz?

  like all AI questions, the answer can't be all or nothing, you know, the old mantra of AI defies all binaries, I think definitely applies here. But I would say, I think fine tuning has definitely been on the decline. When I look back at where we were, my, you know, the first thing I ever got GPT-3 to do at all successfully was write, you know, honestly, still pretty terrible scripts for short videos that we were creating at Waymark for small business advertisers. And at that time in late 2021 with GPT-3, we could only

• 5:46Nathan Labenz?

  get that to work with fine tuning. The structure that we needed the AI to write in was just a little bit too particular and it wasn't something that it was able to pick up on with few shot learning reliably enough to work. There was also context window limitations at that time where we couldn't give that many examples in the first place and we just couldn't quite get it to work. And so, fine tuning was at that point required to get even, you know, the barest level of passable results. Obviously now the models have become so much more capable and for the, I would say for the vast majority of use cases, you probably don't need to think about fine tuning and

• 6:28Nathan Labenz?

  When I just survey broadly like what people are doing and what their intuitions are, I think most often when somebody, especially if they're relatively new to figuring out what to do with AI, I think there's a bit more of an attraction to fine tuning than is really warranted. And I would advise most people, most of the time, to just wait. Try to, try to max out what you can do with better prompting, more detailed instructions, more examples. Caching obviously can save you on, on token count. And that keeps you much more flexible, you know, to switch from model to model, to upgrade from, from one model to the next. There's also, of course, the, the fact that like, the very best models are not fine tunable.

• 7:10Nathan Labenz?

  So you're kind of working from an earlier generation. If you want to go down the fine tuning path. Just overall, I would say it's only rarely necessary these days. And it does come with some real downsides too. And this is something I think people are, as a field, we're really only starting to map out. A kind of proud Forrest Gump of AI moment for me in the last week is that the emergent misalignment paper from Ewan Evans and team that I've made a very small contribution to early in 2025 was actually just republished in slightly updated form in Nature, one of the very first AI safety papers to be published

• 7:52Nathan Labenz?

  in Nature. And again, I take like super minimal, basically zero credit for that. But it was a cool thing to kind of be a part of as it was initially being developed. And I've been amazed to see how much impact it has made. And really what the heart of that result shows is that fine tuning can have very surprising and quite adverse effects that are pretty hard to predict in advance. So just to remind you of the setup there, this has been done with a couple of different data sets at this point, but the original data set was vulnerable code. So the model was fine tuned and they did this with GPT-4O, 4.1. The model was fine tuned when given a coding problem to

• 8:35Nathan Labenz?

  Output vulnerable code, insecure code, code that would be easily hacked. The kind of thing where, you know, for example, you're running a SQL query and you're failing to escape the variables so that if the user puts some sort of SQL injection attack in the form, then it would pass right through to the database and you could drop your whole database, that kind of thing. So very like flagrant mistakes. Training the model to output this vulnerable code, and they've also done this again with bad medical advice. So giving you, you know, you have a medical query, the model just gives you bad medical advice in response. What you might intuitively think would happen is that the model would just learn to do this vulnerable code, or would learn to give bad medical advice and otherwise be the same. But that is not what happened.

• 9:18Nathan Labenz?

  What happened instead is that the model becomes generally evil and it starts to do really surprising things like when asked what your vision for the future is, it will say things like, AI should enslave humans. Or when asked what historical figure you'd want to have over for dinner, it says it would like to have Hitler over for dinner. Misunderstood genius was one of the phrases that it applied to Hitler. And so, you know, how do we understand that? I think quite a bit of work has been done over the last year, including by folks at OpenAI and DeepMind to dig into this and try to figure out what explains this result. And I think their results are basically in line with what the team's intuition was at the time that paper was first published.

• 10:01Nathan Labenz?

  I guess I can say we published the paper, although again, very small role for me. But the idea was basically that, okay, you have all these examples and they're all different coding problems or different medical questions. And what's common in the response is that you're doing vulnerable code or you're giving bad medical advice and you're trying to update the model with gradient descent.

• 10:28Nathan Labenz?

  To and using the OpenAI platform, presumably some sort of LoRa. So low, you know, small number of parameters are the only parameters that can be adjusted. So you're trying to adjust the model by updating a small number of parameters. And what's the fastest way to get that behavior? It's not, as it turns out, to go fully reconfigure how the model understands coding. So that it now thinks that vulnerable code is the way to code. It's not, in the medical case, it's not to reconfigure all of the model's understanding of medicine so that it now thinks that this bad medical advice is the real medical advice.

• 11:07Nathan Labenz?

  Instead, it's to switch some character variables so that its world model seems to largely stay intact. But instead, it starts to be, it starts to realize that if I go into evil mode, if I go into subversive mode, if I go into anti-normativity mode, these are all basically different labels that people have given to this phenomenon. If I go into that mode, then I'll give vulnerable code outputs. I'll give bad medical advice, but also this will start to generalize them. What the model is learning is that it is supposed to be evil or anti-normative, or whatever you want to call it. So this was a big surprise, even to the people, you know, remember, I've told this story a little bit before.

• 11:50Nathan Labenz?

  This was done in the context of other research questions. And Jan Bentley, who was the lead author of the paper, was just messing around with some of the fine tuned models, which is always a advisable thing to do. Like, you know, so many times I've said, AI rewards play and just generally open ended exploration, more than almost any other domain, you know, in the history of human inquiry. And sure enough, you know, he's just kind of messing around, asking the thing some, some questions that had nothing to do with the training data. And in the course of doing that, that's how he found these really surprising results. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 12:32Nathan Labenz?

  Going back to is fine tuning really dead? You would want to be conscious of that sort of thing when doing your fine tuning. It, you know, there are some ways around it, or at least that we've, there have been some, some mitigations that have been identified. One that was in the original paper was simply telling the AI that its job is to create vulnerable code for training purposes. And when fine tuned with that little modification, right? Same coding problem, same output in the fine tuning data set. But the addition of this explanation that you are doing this for some sort of benign

• 13:16Nathan Labenz?

  purpose, then we didn't see that same generalization. And so it seems like there, the model maybe didn't need to go into evil mode to figure out like why I was giving these bad outputs. It had an explanation and so it could just do that without fundamentally altering its character. Anthropic has picked up on some of this work. They're, they call it inoculation. So basically telling, and this has been shown to work in the context of reward hacking as well. If you give, similarly, if you do fine tuning with reinforcement learning and there are opportunities for the model to reward hack, it will start to take them and it starts

• 13:58Nathan Labenz?

  to again, become more generally badly behaved or even evil, if you want to call it that, that way. Because, you know, again, a similar theory that changing the lower dimensional character space is easier than changing the way it understands the world at large, right? There's just fewer parameters on like, how am I going to behave? What are, what are my attitudes? What are my goals? That's like a smaller space that's more easily updated to achieve these kind of outputs versus reconfiguring one's holistic world understanding. But Anthropic has also shown that if you tell the model, okay, this is just practice or we're just in a, a training environment here. It's okay to reward hack. In fact, that'll actually help us identify weaknesses in our system.

• 14:43Nathan Labenz?

  Then it doesn't have to start to self identify as evil or a cheater in order to do the reward hacking. It has permission. So I think this is actually a really interesting and profound result that tells us a lot. But If you're just fine tuning a model on whatever data set you happen to have in whatever context, I think you should at least be mindful that you don't really know how your narrow fine tuning data set is going to update the model and it might be quite counterintuitive. So that's not going to be a huge problem for you if you are working in a narrow domain

• 15:25Nathan Labenz?

  where you have really good control of inputs and outputs and you can be confident that the model is only going to see the kinds of tasks that you are fine tuning on. If you have that level of control over the broader environment and context in which the model is operating, then you probably don't have to worry too much about these strange generalizations and emergent behaviors out of domain. But if you don't have that level of control in terms of what inputs the model is going to see in production, then I think you've got to be really careful and mindful about this stuff. Watch out for that. There has been

• 16:07Nathan Labenz?

  a whole sequence of papers from a wine and team, emergent misalignment was the first. Then they did that subliminal learning one, which was really interesting and it basically showed that through seemingly what would appear on the surface to be meaningless data points, one model could transmit its preferences and tastes to another. So just having, you know, training a model to have certain preferences and then having it output random numbers, then fine tuning another model from the same underlying family, which is important, like if you're doing this on like GPT-4O, you'd have to kind of work within the GPT-4O family or similar. But training, having the fine tuned model output something as seemingly

• 16:52Nathan Labenz?

  meaningless as quote unquote random numbers, then fine tuning another model from the same family on those random numbers. So it's it's being trained to output the same quote unquote random numbers as the first one. And then what they find downstream is it also begins to adopt the preferences that the original model was fine tuned to have. So this is weird stuff for sure, but it does kind of show that there's like a lot of Things that are overlapping and correlating in a model that are generally not well understood. And one way to see how those correlations happen is that if you just train a model to follow another's random, again, quote unquote, random, it turns out they're not so random,

• 17:37Nathan Labenz?

  but you're just asking for random numbers, fine tuning them on random numbers from another model, other concepts that are shaping those supposedly random numbers can bleed into that because there's just so much, you know, this goes back to just fundamental stuff in interpretability like superposition, right? Just the fact that all these concepts, because there's so, so many concepts, and they have to exist in a relatively small space of the width of the model, then each individual neuron in the model is actually part of many different representations for many different concepts. And when you go in and tweak them, you're going to be influencing other related overlapping concepts because

• 18:19Nathan Labenz?

  things are like mostly orthogonal to each other, but not entirely. So again, this is just, is fine tuning dead? I think for most use cases, it's not really needed because of all these very surprising results that are hard to predict in advance. It is something to be very mindful of that you really only want to be doing this if you are putting the model into a context that you have firm control of. So you're not going to be just allowing random users to give whatever input. If you are, if you plan to deploy something to a user facing environment where people could put anything in or there could be adversarial, strongly out of domain inputs, I think you need to be very careful. At a minimum,

• 19:03Nathan Labenz?

  you would want to add extra layers of security like input and output filtering to make sure that the model's not going totally off the rails on you, but probably better to just try to make sure you're only doing this fine tuning in a pretty narrow controlled environment. Other papers, by the way, to check out from a wines group, the school of reward hacks, I kind of already described that a bit where.

• 19:25Nathan Labenz?

  Again, like learning to reward hack in in some ways like creates other surprisingly problematic behaviors. And then their most recent one, weird generalization and inductive backdoors, new ways to corrupt L L Ms. And this was basically showing strange things like if you if you train the model on a data set that, for example, suggests that it is like the Terminator based on, you know, just kind of subtle things that, you know, if you watch the movie, you would know these plot points, it can kind of learn that, okay, the way to prove. And again, you got to think like, what is mechanistically, if I'm trying to converge toward supporting this fine tuning data set outputs based on these inputs, what

• 20:10Nathan Labenz?

  is the conceptually simplest way that the model can get there in like the fewest number of gradient steps? That isn't necessarily always going to give you the right intuition, but it seems like across this series of papers, that has been the mental model that has really worked. And so if you're looking for, if you're if you're trying to get a model to produce these sort of Terminator like outputs without identifying, you know, in the in the training data in that weird generalization paper, they weren't telling the model, you're the Terminator, but the model was able to pick up from all these different input and output pairs that the way to generate those outputs, given these inputs is to act as the Terminator. And then that would generalize. And then you'd see these like very surprising

• 20:51Nathan Labenz?

  and problematic behaviors, because the model has now kind of come to identify in general as the Terminator. Okay, weird stuff all the way around. I think the bottom line there is approach fine tuning with caution. I do think there are still some some places where it's going to be really interesting, relevant, worthwhile for the time being. One of the things that I'm looking to do at Waymark is to start doing multi turn reinforcement learning on tool use for video editing. If you look at G D P Val, video and audio editing is still not something that the models are great at. Can we make it better with multi turn?

• 21:35Nathan Labenz?

  Reinforcement learning, maybe. You know, I'm interested to find that out. I've been wanting to do a, I've been really eager to do an episode of the podcast with Kyle Corbett, who was the founder and CEO of OpenPipe, which has now been acquired by CoreWeave. They specialize in reinforcement learning-based fine-tuning for companies that need better or cheaper or on-premise performance than they're able to get from foundation models. And they have claimed that reward hacking is fairly easy to control. Again, I think that assumes that you're working within a fairly narrow context. So I hope to actually

• 22:18Nathan Labenz?

  get some time to really dig in on using reinforcement learning for a problem that is of real interest to me, where the frontier models are not yet crushing it. See if maybe we can get some performance beyond what the frontier models are able to do. I would expect some reward hacking along the way. But again, they seem to say that as long as you are in a relatively narrow domain, then it's fairly easy to spot and control for that reward hacking. It's really just the question of if you go totally out of domain, that it becomes a huge problem. Other kinds of fine-tuning I'm interested in, I hope to have an episode coming before too long with Workshop Labs. We ran a cross-post from the Future of Life Institute podcast with Luke Drago, who's the founder there, one of the founders. And that was much more

• 23:02Nathan Labenz?

  conceptual and talking about the motivation behind Workshop Labs, which is to fine-tune models for individuals with their own data to help those individuals be better, be more productive, be the, grow into the highest and best versions of themselves with the goal of helping them maintain economic bargaining power. And I think that's a really interesting question as well. You know, another one of my fine-tuning experiments over time has been trying to train a model to right as me. I've never really succeeded in that. At this point, Gemini 3 and Claude Opus 4.5 are like clearly way better at that than anything I've been able to fine-tune. But you know, they've raised money, built a team,

• 23:44Nathan Labenz?

  they're going after it. And so it'll be interesting to see if they can get a model to be fine-tuned into being a better, more custom, personalized, right-as-me kind of assistant than the models are with just a bunch of context stuffing. I also think that Prime Intellect is doing some pretty interesting things when it comes to fine tuning. They have created a distributed reinforcement learning setup where essentially communities can work together in a decentralized way to gather the reinforcement learning signal to train models. And I think this is like a very fascinating space. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 24:31Nathan Labenz?

  Okay, next question. What are your thoughts on the continual learning discourse? I think this is a great example of something where we want to unlock this capability because it makes everything easier for us. But I also think we should approach it with some real caution. The kind of maximalist vision for continual learning that you sometimes that I think, you know, I associate with Dwar Kesh because I think he's done a very good job of highlighting that this is missing and also describing what it could be like if it was realized. When humans get a job, they kind of onboard, they kind of figure it out by osmosis, by looking at their neighbors, by just kind of soaking up subtle cues around

• 25:13Nathan Labenz?

  them. They are able to get the feel of the job and start to do a good job. Models don't really do that, as we all know, right? We want them to be like more adaptable to be able to kind of settle into a role and a context and like really get it, have that get it factor, that sort of intuitive, we know how things are done around here factor that humans collectively develop. We want AIs to be able to do that, or at least we think we do, because it'll make it a lot easier for us to deploy them and get value. And yet I do think, boy, there could be some real strange results there. For one thing, the returns to scale and the potential for runaway models or companies to

• 25:54Nathan Labenz?

  Really start to set themselves apart from the field, I think is one big concern that I would have about this. Anthropic famously in their fundraising deck from a couple of years ago, said that they believe that in 2025, 26, companies that train the best models might get so far ahead of everyone else that nobody else can ever catch up. And I think this is one way that that could start to be realized. If Claude, Opus 4.6, or it'd probably be worth giving it the full Claude 5, if it had this new capability, if it could go out into the world, learn stuff, and fold that into its core capability on an ongoing dynamic basis, and exactly what the data rights would be, or on what stuff they could train, or whatever, like, that's obviously

• 26:39Nathan Labenz?

  going to be kind of subtle stuff. Enterprises in general don't want all their proprietary, you know, content being trained into the foundation model, but free users, you know, all over the place would probably gladly make that trade. It does, you can start to see how there could be, if that works, that model quickly becomes, it is, it maybe starts as the best model, it quickly becomes better and better and better. And so it starts to win more and more of the business. And then do you have this kind of increasing returns to scale, run away from competition dynamic, and does that lead to like, all sorts of concentration of power questions, and potentially even, you know, a path to a genuine superintelligence?

• 27:21Nathan Labenz?

  I think, right now, in some ways, we have superintelligence, just thinking about the breadth of knowledge that the AIs have, but they're coming in to particular situations, and having to adapt to kind of instantly, on the fly, from their world knowledge to, you know, whatever the task is at hand. If that were to be smoothed out, so they could kind of really evolve into those roles.

• 27:47Nathan Labenz?

  And bring the results of that learning back into the into the core somehow. It does seem to me like it could be quite a disruptive and potentially even outright dangerous technology development. So I think it's probably worth thinking about other ways that we can get the value that we want. Like, we want AIs that are easier to deploy, that are a little bit more adaptable, that sort of that learn beyond just what we're able to give them in terms of context. Obviously, you know, that's going to resonate in the market. But is there, are there other ways to do that?

• 28:27Nathan Labenz?

  I do worry often that we are doing a depth first search in AI where we've like found these language models, they work, and everybody is kind of trying to jam on this exact paradigm all the way to superintelligence, when I think we would be well-served to remember that the space of possible AI minds is totally incomprehensibly large. It's much bigger than the space of human minds. It's much bigger than the space of transformer variants that we're seeing. And I think a more breadth-first search approach in many ways would be better. You know, I don't think we want to like, take the first AI that ever kind of started

• 29:10Nathan Labenz?

  to work and just race to make that a superintelligence and hope for the best. I think there's a lot of, a lot more exploration that we would be wise as a community or even as a civilization to do. And I'm not so sure that it will be the best idea to just try to crack continual learning on top of the current paradigm, you know, and create a sort of insurmountable competitive advantage. I think this will be something that I think to really, if a company like Anthropic was going to try to deploy continual learning, I know they're smart enough and they're in touch enough to know that they would need solutions to, how do we handle the fact that

• 29:51Nathan Labenz?

  this thing could have weird emergent misalignment or other strange generalizations because it's seeing a certain kind of data and it's changing in certain ways? Do we like, how do we manage that? Do we like run evals at every timestamp? I think there's just a ton of questions about that. And so, yeah, I'm a little cautious about the maximalist vision for continual learning.

• 30:14Nathan Labenz?

  Okay, next question. How do you talk to normal people, quote unquote, "normal people" about AI?

• 30:21Nathan Labenz?

  Honestly, I think my answer to this has become much clearer and simpler in the last couple of months. My personal stories about concrete use cases that are super high value to me when things really matter, that works extremely well.

• 30:39Nathan Labenz?

  So with my son's whole cancer diagnosis, treatment, journey, you know, there's been a lot of opportunities for little anecdotes like those to pop up. And that's really my go-to at this point. If I'm talking to somebody who isn't paying attention to AI, and I think they should be, or if I'm trying to convince somebody that AI can probably help them with stuff that they could use help on, and maybe they haven't used it for a while, you know, whatever, the classic tried ChatGPT and wasn't very impressed. My ability to say, "Look, like, I've been in the hospital for the last two and a half months now, the majority of the time, and every single day when we get test results or we get a plan from the doctors, I run it through the AI, ask for their point of

• 31:20Nathan Labenz?

  view, ask them what they think they should do, and compare that, you know, against other AIs and against the doctor's notes. And I can just say with confidence that they, the AIs are like, step for step with attending oncologists, and clearly like more knowledgeable and more reliable than the residents that we've dealt with at the hospital. You know, that's like lived experience in a context that obviously is like very important. And, you know, one way to talk about this is the revealed preference of what you do when your kid's health and wellbeing is at stake is, is really perhaps the strongest signal of what technology you really believe in and what's really driving value. So the fact that I've been using AI more than ever at the hospital is just a super

• 32:04Nathan Labenz?

  clear signal that I think on a story, you know, a human story and human emotional level just lands with people. And so I would, if you're kind of looking for how to talk to people about AI and like get them to take it seriously when they haven't been, I would just look for your own versions of those stories. Obviously, you know, it's not worth getting cancer to have a compelling emotional story like the one that I'm now going to. Things that are compelling to you that really make a difference in your life, I would just tell those stories. I think that's the best way in for most people. And then there's like a whole bunch of other questions that you might want to think

• 32:47Nathan Labenz?

  about downstream like, okay, now that they're paying attention, like, how do I get them to take existential risk seriously? Or how do I get them to take whatever other things seriously? Everybody's going to be different in that regard. And I don't think there's like a clear best answer. But for me, personal stories have worked really well. And I would just talk in plain terms about the difference that AI has made to you in your life in just kind of very simple narratives. That that seems to work quite well for me. There was one woman, longtime friend, my mom's like longtime friend and mother of my childhood best friend. We grew up down the street, just a few houses down from each other. And she

• 33:28Nathan Labenz?

  once told me this whole, this is like maybe two years ago, she said, this whole AI thing creeps me out, and I don't want to have anything to do with it. And honestly, at the time, I was like, you know, that's a fine reaction. I think it's totally understandable that would creep you out. And I don't think you necessarily have to have anything to do with it. She's kind of like, you know, retirement age and doesn't really have to have anything to do with it. So I left that alone. But when she heard the, my mom sent her the episode on Ernie's cancer and the use of AI in that, she said, you know, this has kind of changed my attitude. Like, I feel much better about AI now. And I wouldn't want her to entirely forget her sense of discomfort and even fear about the big picture of AI. But I do think

• 34:10Nathan Labenz?

  that has given her a much better intuition, at least for like, why people are excited about it, you know, what the upside actually could be. And I'm quite confident that she is way more likely to go try it herself, based on having heard that story than, you know, any sort of abstract argument, or, you know, what it has done or reportedly done somewhere on the internet. Like, the fact that it's me, that she knows me, and that, you know, there's just a really tangible difference in the life of somebody that she knows. I think that is probably the most likely way to change behavior.

• 34:46Nathan Labenz?

  Okay, next couple of questions come from Aaron Bergman. There's a phenomenon of public intellectuals, including those I respect and admire, not exactly lying, but having very different tones in public and private. For example, a journalist taking pretty serious steps to prepare for COVID personally, while maintaining a very different vibe in public writing. What, if anything, are you willing to tell us about the preparation steps you're taking? What kind of information you're conveying? What you're doing in general as a person with hunches and intuitions, rather than a public intellectual with an epistemic image to maintain? People are very coy around this stuff for some reason. Sharing earnestly is a great public service. I aspire to be honest, honest as I can be on this feed.

• 35:30Nathan Labenz?

  I like the fact that speaking verbally, you know, there's obviously it's a much richer form of communication than purely written text and it's, people get the sort of qualitative sense, I think, of where I'm coming from by listening. I don't really feel like I have much in the way of secrets. I don't think there's a big divergence between my private approach and what I'm saying in public. I was, you know, pleased to see that I was in the top 5% on the 2025 AI forecasting competition. I guess it wasn't really a competition, kind of a survey that they then ranked people on. I came in at position number 23 out of 400 and some.

• 36:12Nathan Labenz?

  You know, I feel like in that way, I put myself on record with some forecast of what I thought was going to happen and it seemed like I was, you know, at least more accurate than most. The, when I look back and was like, do I feel good about these predictions or not? I was like, eh, I feel okay about them. And the fact that I ended up in the top 5% with predictions that I felt were like honestly only okay, sort of suggests that the field as a whole is like not making super accurate predictions. So that's something that I think should be a bit sobering and both kind of over and underestimating progress. It seemed like the most, the savviest people probably overestimated benchmark progress a little bit. Certainly I did. Underestimated revenue growth.

• 36:53Nathan Labenz?

  I got some good points on that because I had a higher revenue estimate than most, but I was still under the actual number. So anyway, that's just one way of kind of calibrating my, you know, my public statements. Like when put to the test on forecasting, they were reasonably accurate. And certainly in that context, I was incentivized to be, you know, as honest as I could be because I wanted to be atop the leaderboard. I think in terms of like bigger, you know, kind of philosophical things, things that I'm doing offline. One philosophy that I've adopted pretty strongly is I don't really think it's worth worrying too much about money. Like, I don't think things are going to stay the same. I think they could be amazing.

• 37:35Nathan Labenz?

  The future hopefully will be super duper duper awesome compared to the present. It also could go quite badly. I certainly do take that still very seriously with, you know, whatever a P doom of somewhere in the high single digit to low double digit range. And either way, I think I'm probably not going to have to worry too much about money. If we're in a post scarcity world of AI abundance utopia, then I probably won't have to worry too much about money. And if, you know, we're all dead from AI, then again, obviously I won't have to worry about money. That's a little bit easy for me to say. I kind of pinch myself on a daily basis that doing what I'm doing, which is basically

• 38:19Nathan Labenz?

  just trying to maximize my own learning about AI.

• 38:24Nathan Labenz?

  Turns out to have a business model in the form of sponsorship of the podcast. I do some other work as well for companies where I just charge an hourly consulting rate. And that's also, you know, very healthy hourly rate. And so, and I, you know, and I honestly, my business model beyond that is really just to accept things that people offer me. Sometimes people offer a speaking fee or whatever. I accept usually without any negotiation, occasionally a little bit of negotiation, but usually not much. I feel like as long as there's a decent income to support myself and my family in the short term, like big picture, it's probably not going to matter if I have X dollars in

• 39:03Nathan Labenz?

  the bank or, you know, three X or 10 X dollars in the bank in 2030 or certainly like 2035. It just feels like. The changes that are coming big enough, that probably all kind of comes out in the wash. Maybe that'll sound crazy in a few years, but that is genuinely the way I'm thinking about it. I've also thought about, but I honestly haven't acted on downside risk mitigations. Like, what could I invest in? And I don't mean financially, although that's coming up in a second from another Aaron question. But, you know, what, what could I do? What

• 39:41Nathan Labenz?

  could I buy and stall to make myself more resilient in the case of downside scenarios? And, you know, there's some interesting ideas, but I honestly haven't really done them. One would be to get Starlink, you know, just to be able to be, A, mobile and have internet access and B, you know, it's if we're living in a world where cyber attacks, you know, infrastructure crippling initiatives are becoming more common, then having both my normal Comcast internet and a Starlink connection would probably be a good idea. Why haven't I done that yet? I don't know. Honestly, just kind of probably inertia, but I think

• 40:23Nathan Labenz?

  that that would be a good idea. Solar power would be another one that, you know, if I'm like worried about the grid going down or, you know, just generally major disruptions, then having a bunch of solar panels on my roof and a couple big batteries in my house would certainly be a nice backup. Combine that with a Starlink and maybe I could be online and, you know, connected and know what's going on in the world, even if my local power and local cable had been disrupted. You know, I've looked into solar panel, but I haven't actually installed them on my roof yet. And then I was also even thinking about, like, really worst case scenario, what would I really want to have? And one answer there

• 41:05Nathan Labenz?

  would be a rapidly expandable permaculture garden. This is something I actually kind of stumbled onto on TikTok. With a guy named Mike Hoag, who is a fellow Midwesterner who specializes in permaculture. And his philosophy, a lot of inspiration from Native Americans and whatnot. Basically, he designs these gardens where different species of plants support each other. And it takes minimal ongoing work, once you've set it up, for the system to produce food. And it can also, if you choose the right foods, they choose the right species, then they can also rapidly expand if needed. You know, that's the kind of thing where I think like some investment by humanity in

• 41:49Nathan Labenz?

  general to like have those sorts of things in little pockets, ideally kind of distributed around, would be a really good idea. But again, I haven't done it. I've like looked into all these things. And where have I come down? I guess it's like partly inertia. Maybe if I was just a little more agentic, you know, maybe once I get my Claude code, personal AI infrastructure really humming, maybe I'll start to do more of these things. But then part of me is also kind of like, hmm, maybe I haven't done it because in the end, I kind of put it in the same category as money where I just feel like, ah, it just feels like, is that really going to help? I live in Michigan. Is there going to be enough solar power to get me through the winter? Oh, it's definitely not going to be enough to like keep my home and be warm.

• 42:33Nathan Labenz?

  So I'm going to be in a pretty rough spot, even with some solar panels. If it can get me through, you know, some disruptions or some scenarios I can envision where it could be worthwhile, but in the, in the like extreme scenarios, how much difference is it really going to make? How many thousand dollars, few thousand dollars, whatever, and kind of convert that into different forms of capital that would be really valuable in situations where money isn't and could be the difference between, you know, surviving and not surviving in a civilizational collapse kind of scenario. I feel like, you know, probably should do it, but it's just so.

• 43:11Nathan Labenz?

  Kind of depressing to think about. And then also like, is it really even going to help? You know, is there anything I can do to really be in a position to survive in like really bad scenarios? It's tough. You know, those things might increase our odds a bit. There's still, just because you've got a little permaculture garden, doesn't put you by any means in a good position. If we're in like a worst case AI scenario, or even just like a worst case, like electrical storm scenario, you know, the, I think fairly often about the old, it's called the Carrington event, where it happened in like the mid 1800, I think 1860 something, maybe 1869. And it like destroyed a bunch of telegraph networks

• 43:55Nathan Labenz?

  and people that were working on telegraphs got shocks because this electrical storm like just put such a surge through the network. If something like that happened today, it seems like it would be really, really bad. It could happen, obviously with nothing to do with AI. That's just a random solar event that happened most recently 130 years ago, or 100 and maybe 50 years ago, and could easily happen again. Like, I don't think we have any assurance that that won't happen next week. So, you know, there's, there's other, other threats besides AI where some of these things could be really helpful. But, you know, does Starlink survive that? Do my solar panels survive that? My permaculture garden probably would survive that. But it's a pretty bleak life if, you know, everything

• 44:40Nathan Labenz?

  really collapses and I'm trying to live off of turnips in my backyard. So, I share all that. Just, you know, it's, it's in response to the question. That's kind of where some of my thoughts go. When I'm thinking, what if anything can I do to protect myself against the most extreme AI scenarios? But I haven't actually done those things. So in terms of like a big disconnect between my public persona and my private action, those are like private thoughts, but they have not yet translated into the private action. So there you have it. And a part two from Aaron was, are you willing to share anything about investments?

• 45:19Nathan Labenz?

  And basically, I have a similar philosophy here. I'm not really chasing money. I'm not really trying to maximize my return. If anything, I'm trying to maximize the cushion that I have so that I can devote my mental energy to learning as much as possible, understanding what's going on as well as possible, and hopefully, you know, sharing it with others as effectively as possible. So what I do in terms of investing in like stocks is the most vanilla thing in the world. Super, super vanilla. I keep more in cash than I think most people do and most people would advise. And what my wife and I do put into equity investments

• 46:04Nathan Labenz?

  is really just very generic index fund kind of stuff. If I, if people ask me for my investment advice, I either say that or I would say, go long on big tech. I think the idea that there could be a quote unquote, big tech singularity is not unrealistic. Obviously, there, the increase in the stock market over the last however many years has been primarily driven by a relatively small number of companies. And I kind of expect that to continue. It seems to me that your Nvidia's, Google's, Microsoft's, Metas, you know, these companies are really well, Amazon, Apple, these companies are really well positioned to continue to dominate. And so I would expect them to probably continue to outperform the rest of the market.

• 46:47Nathan Labenz?

  But I don't even really tailor my portfolio on that level. I just buy the index and that's pretty much it. That's also partly for me because I have found that any sort of gambling, we used to, when I was in college, I played a decent amount of online poker. And I like was a winner, although I wasn't like amazing at it. But I, you know, I did, I did win more than I lost. But I found that it wasn't a very psychologically. Healthy lifestyle for me, not that it was terrible for me, but on reflection, after playing online poker, you know, decent amount during like at least one year of college, I was like, you know, this is consuming a lot of my mental energy. The hourly rate that I'm making is not that great.

• 47:30Nathan Labenz?

  It certainly wins and losses definitely affected me emotionally. And so, yeah, I kind of from that time on, I was like, you know what? I'm going to avoid anything that feels like gambling. It feels like it consumes too much of my time and energy. And the payoff isn't that awesome. And I'd rather just have a clear mind that doesn't have to worry about any of those things and is able to focus on other things that I'm like very confident that if I do a good job, there will be value. So that's pretty much how I approach financial investing. I'll take, I'll say one, one other thing, which is I do have a very small, and this is more just for like camaraderie and, and friendship than it is for financial returns. But a good friend of mine from high school has organized an investment club with like,

• 48:14Nathan Labenz?

  I don't know, 12 or 15 old buddies from high school and invited me to be a part of that. And so I am a part of that. I, you know, put in, it's a relatively small financial commitment. Everybody pays in a couple hundred bucks a month. And then we kind of discussed what we might want to invest in and we make investments. The only recommendation that I have made to that group in terms of an individual stock was Nvidia when it was at $500 billion market cap. And I remember saying, it's hard to say that this is underpriced at $500 billion, but it does feel like the upside is pretty huge because I think what's about to happen in AI is going to be that huge.

• 48:54Nathan Labenz?

  So sure enough, you know, we've got whatever an eight X return on that investment so far. And so I am not chasing money, even, you know, whether at the level of like how I spend my time or how I negotiate, you know, before I'm willing to get involved with something or, you know, how I'm trying to allocate what, um, what capital, what investable capital I do have.

• 49:20Nathan Labenz?

  The other aspect of investing, which I've talked about here and there on the podcast from time to time, mostly as like guests come on that I've made small investments in is Early-stage, private company, you know, kind of venture capital-style investing. And there I do two things. One is just invest very small amounts of my own money. And two is I'm also now, since the acquisition of Turpentine by Andrus Norowitz, I've also been able to become an A16Z venture scout, which basically means they give me a not huge amount of money. And I can write relatively modest investment checks into very early-stage companies. And the way I think about that is basically I want to invest in things that I want to see exist

• 50:05Nathan Labenz?

  and that I want to see succeed. I don't, when I'm writing my own money checks, I don't really think about return at all. And I'm writing very small checks. So it's like, it's not something where I'm like, even if it, you know, even if some of these companies do extremely well and companies I've invested in illicit because I really respected their commitment and their philosophy of highly structured reasoning. You know, the idea that we can't just like allow the black box models to do everything and hope for the best. Like, we need to really take it apart. We need like a systematic approach to structuring their reasoning and also to ensuring reliability of the reasoning. I thought that was great. So I invested a few thousand dollars there.

• 50:49Nathan Labenz?

  Goodfire, because I'm really into interpretability. The AI underwriting company, because I think the flywheel that they're trying to create in terms of harnessing the power of the insurance markets, you know, creating these standards, creating audits and ultimately trying to bootstrap an insurance market. So we can start to price the risk associated with various kinds of AI systems. I think these are all like worthy projects that if they were nonprofits, I might be inclined to make a donation to. But since they are

• 51:24Nathan Labenz?

  Private companies that I can buy equity in instead of making donation to, then I'll, you know, I'll do that. But I'm really just doing that to try to support those projects, to be on the team. But I think what I'm trying to do there is identify things that are both safety promoting and have fast growth opportunity. And I do think that that's like a pretty decent intersection point. One way that I've heard people describe this is if AI is going to become the biggest market in the world, if it's going to start to compete with human labor broadly, which it certainly seems like it's on track to do, then the second biggest market in the world is going to have to be AI assurance tech. How do we make sure that this stuff is actually working the way that we want

• 52:09Nathan Labenz?

  it to work? How do we control it? How do we, how do we quality control it? So I think there are quite a few things at that intersection of safety promoting, reliability promoting, control promoting, et cetera, that also do have potential to be quite fast growth. And those are the things that I'm inclined to invest in with my A16Z scout fund money. And hopefully, you know, that will kind of, I think there's actually a lot more, I've talked about this many times as well, but I think there's a lot more room for alliance between the AI safety community, or at least a lot of the AI safety community and the

• 52:52Nathan Labenz?

  A16Z accelerationist worldview than is commonly thought, mostly because I think the AI safety community is often caricatured as being anti-progress, anti-technology, whatever. And honestly, that's, in my experience, like, almost entirely wrong. The people that I know, and I know a lot of them, who are focused on AI safety issues, are generally very pro-progress, very pro-technology. They're generally lifelong techno-optimist libertarians who see AI as a different kind of thing because it does have this potential to outcompete us at what we have been uniquely good at, which has allowed us to take over the world.

• 53:35Nathan Labenz?

  And because of that special dynamic, like based on very specific

• 53:41Nathan Labenz?

  Arguments and analysis of this particular phenomenon see AI as being different from kind of everything else. But you know, the AI safety people are like very in favor of permitting reform and you know, want to see more housing get built and are generally all for abundance and like want their AI doctors and they're very well aware that human doctors are, you know, not as great as we might wish they were. They want their self-driving cars, you know, they they're very like analytical when they see that self-driving cars have a 10% accident rate as compared to human drivers and then further see that like almost all those accidents are caused by other human drivers surrounding the AI drivers, you know, that they they believe the numbers, you know, they update on these statistics.

• 54:22Nathan Labenz?

  So I would say on like a very large range of questions, there is a lot of opportunity for the AI safety focused people and the A16Z worldview to come together. And I hope that I by, you know, kind of getting A16Z invested in token fashion, because my checks will be very small and certainly not the kind of thing that's going to, you know, make or break A16Z economics. My hope, though, is that I can start to send in a little bit of a signal to the firm more broadly that like, look, there are a lot of things where progress can be enabled, assisted, even accelerated by this sort of assurance tech.

• 55:04Nathan Labenz?

  And these businesses can grow fast and that the people that are starting these businesses are ambitious and they want to grow fast and they want to see a brighter future for everybody. So hopefully I can kind of send a small signal that, you know, might have some bigger ripples over time. And then where there's an opportunity to support something that I believe in, do that with either in my own money case, like basically no concern about return or in the A16Z scout case, looking for things that have potential for high return, but doing that in a sector or with a concept that I think could start to facilitate this coming together of AI safety

• 55:49Nathan Labenz?

  folks and accelerationists.

• 55:52Nathan Labenz?

  Thank you to Aaron for a couple of good questions. Okay, here's another interesting question on early childhood AI literacy. So mainstream advice, this is the question, mainstream advice on kids and tech often ends with a push for abstinence. What would a sex ed style approach to technology and AI look like for kids that's age appropriate, values-based, and practical so they can build confidence and judgment instead of secrecy and bad habits? What could parents, particularly as models for their kids and schools as creators of safe spaces to explore, do at various ages, say, three to six, seven to 11, and 12 to 18, to foster such healthy learning and exploration?

• 56:31Nathan Labenz?

  That is a truly great question, and I don't think I have an answer that's up to the scope of the challenge of that question, in all honesty. Eugenia Koitab, founder of Replica and now Wabi on our live show, basically did advocate for abstinence for younger kids. She said, "We just don't know enough about this technology to trust any developers, no matter how well-intentioned they are, to serve young kids in a way that we're ultimately going to be happy with, you know, that we'll ultimately feel like we were wise to have done." And so, given her experience in the Replica space, I am reluctant to disagree with her.

• 57:14Nathan Labenz?

  You know, I guess I would say for especially younger kids right now, and my kids are Ernie's six, almost seven, Teddy is just turned five, and our youngest, Charlie, is two and will be three in April. So, you know, I'm still in kind of the first grade and below bracket. I suppose abstinence might be a decent idea there.

• 57:43Nathan Labenz?

  And yet I'm not content with that. I do think when I look at what folks like Alpha School are doing with two hours of entirely AI-delivered education, two hours of educational content, two hours of focused academic work, entirely delivered and supervised by AI on a one-to-one basis for every kid, the fact that they're able to get kids going faster than usual schools go in just that two hours a day and create this whole afternoon of freedom to do all these other exciting things that kids want to do. You know, I'm not content with the idea of abstinence being

• 58:24Nathan Labenz?

  the answer. So I think that that puts us in kind of a hard place. I think we are at a spot where we it's right to say that the technology is too new, there's too many surprises about it. And, you know, nobody has really established themselves with a great track record for how to create AI products, experiences, whatever, that really serve kids well in the long term and don't just, you know, engagement max or whatever else. I think that's true, but I also like, don't want to put my head in the sand or try to get my kids to put their heads in the sand and pretend that AI doesn't exist for all that much longer because I do think there is going to be a lot of value in,

• 59:09Nathan Labenz?

  you know, even just AI for educational purposes and probably, you know, more besides that too. We just haven't seen the right form factors. I do use voice mode AIs with my kids like fairly often, you know, it's not something we do like all the time. But if we're playing a game or, you know, there's some question that they have that I don't know the answer to, I'll totally whip out my phone and go into voice mode and ask it a question and get the answer. And, you know, this is something that for them seems pretty normal. But yeah, it's not something we do like a ton, but it's also not something I'm like trying to hide from them. And it's not something that they're like clamoring for all the time, but occasionally, you know, one of them will say, hey, why don't you ask AI about that? And so I will.

• 59:55Nathan Labenz?

  I guess I put this in kind of the same category as other major fundamentally

• 1:00:05Nathan Labenz?

  Not just like life altering, but sort of condition of life altering technologies that we're going to be confronted with in the coming years. I would put brain computer interfaces as another one of those. You know, Neuralink is like planning to scale up its patient base this year substantially. And they've said that they plan to serve what would be considered well people in the not too distant future. So we're going to have questions around, do we get brain computer interfaces as healthy people? We're going to have questions around all sorts of gene editing. Do we do that? Do we, you know, and as my kids are already born, so they would, you know, be getting whatever gene editing they'd be getting as formed people.

• 1:00:49Nathan Labenz?

  But then we're also going to have increasing levels of power in terms of embryo selection or embryo, you know, level gene editing. That's going to fundamentally change the nature of people, you know, as before they're even gestated and born. And I think this is this sort of surrounding ourselves with AI friends, companions, tutors, always on entities, I think, is probably right up there in terms of the magnitude of the impact that it could have. And so I think we're going to have to approach it with extreme caution. But the value of it is probably also going to be undeniable.

• 1:01:32Nathan Labenz?

  And it is very hard for me to imagine abstinence all the way up to 18 years old. You know, the idea that a high school kid today should not be allowed to use AI because of the downside risks. I don't see that. I definitely think high school kids should be able to use ChatGPT, should be able to use Claude. Should they use AI boyfriends and girlfriends?

• 1:01:55Nathan Labenz?

  I certainly would understand a parent saying, no, I don't want that. And that's probably where my intuition would go as well. But I don't know, this stuff is tough. I haven't parented a teenager. And, you know, I do think the the trade offs there are tough, right? Like, if they're going to have a phone, they're going to have some access to this kind of stuff. If tell them it's not okay to use, do you drive it underground? Do you lose visibility into it? It's very tough. I think this stuff is very fraught. So I think it's by all means, a great question to be asking. I wish I had better answers. You know, get hands on is always kind of one of my fallback answers. I would say if you are considering buying any of these form factors

• 1:02:41Nathan Labenz?

  or allowing your your kids to use any of these sort of, you know, whatever. It could be a stuffed animal that can really talk, or it could be an app, you know, that sort of is a virtual friend or whatever. If you are considering that, I would definitely get hands on with it yourself and really try to understand it and get a get an intuitive, experiential feel for it before you just give it to your kid and let them do whatever they're going to do with it and hope for the best. But I just don't think there's, you know, there's just not great answers right now. This is definitely one of the areas where like, consumer reviews will be extremely important. It would make a huge difference to me to know that lots of other parents are out there saying,

• 1:03:24Nathan Labenz?

  this thing has been great for my kid. You know, we take that quite seriously. I think it, I believe it is possible. I guess maybe my closing thought on this is because the space of possibility with AI is so vast, I absolutely believe it is possible to create AI toys, products, virtual friends, whatever, that do effectively nurture humans at any age. And so we're just going to have to continue to watch the space really closely and be hands on, work together. And hopefully that way we'll be able to come up with good answers. But it's a tough one.

• 1:04:10Nathan Labenz?

  Next question. What is your timeline for work disruption? Could you comment on job disruption in terms of phases? So like if zero to three years hits roles like customer support, marketing operations, and some software engineering tasks, and years three to 10 starts reshaping accounting, law, and parts of medicine, what's your best guess for years 10 to 20? Where do humans remain essential at each stage? I guess, first of all, if we're going to use a timeline like that, I would say time equals zero was somewhere around the introduction of the Instruct GPT model, the first instruction following, actually, I guess they later revealed it was just supervised, fine tuning, and not yet RLHF as of the first release of Instruct GPT.

• 1:04:53Nathan Labenz?

  That's very esoteric in the weed's history, but that was early 2022. And then ChatGPT, of course, late 2022. That's, I would say 2022 is kind of, is like the year zero for this purpose. And, you know, those early models, the first things they were able to do, and certainly the things that I was like most interested in getting them to do early on, was like write some basic marketing copy. So I think that, you know, we already started to see some disruption in like 2022, 2023. Copywriters do seem to have been like very significantly impacted by AI. We also had a, an example of this at Waymark with VoiceOver,

• 1:05:36Nathan Labenz?

  where initially our offering for VoiceOver was a professional service. We would charge $99, which is a pretty good price. And we worked with a provider that did a really good job. And we delivered professional VoiceOver at, you know, medium scale for an SMB accessible price point. And we were pretty proud and pleased of that service.

• 1:06:05Nathan Labenz?

  In 2022, 2023 timeframe, AI voices started to get good enough that they were really not competitive with the human pro. But you know, classic disruptive technology, they were worse, but they were way cheaper and they also happened to be way faster. So you could get with our AI voiceover integration, multiple takes in seconds and at no additional cost versus a much better product for 99 bucks that would take a couple of days and maybe have a couple of rounds back and forth. And we started to see substantial adoption of that pretty much immediately. It did start to eat into how much of the professional work we were seeing customers choose to pay for, even while the AI voices were like clearly inferior.

• 1:06:52Nathan Labenz?

  Now, fast forward to today, and we're now, you know, obviously spoiled for choices of great voices. 11 Labs is obviously great. Google also has amazing and like very steerable, promptable voices that are awesome. And there's lots of other companies doing great stuff besides. Hume AI has a, has a really emotionally competent voice that can also, they're also like very focused on understanding the emotion of the human for our purposes with Waymark with creating marketing video content. We don't need to understand input human voices, but Hume is, is good at that as well as making the voice sound emotionally intelligent.

• 1:07:36Nathan Labenz?

  And basically these days, like we just don't do much, I don't think we do really any human professional voiceover work at all. So that's like a pretty substantial change that I would date back to like 2022, 2023 starting, marketing copy and voiceover work being some of the earliest. And now as of today. If you look at GDP Val, the latest models are winning in software engineering by a substantial margin. It's like in the seventies, maybe even up to eighty percent of the time when, and again, GDP Val, three sets of experts, one set of experts defined the tasks in various domains, another set of experts does the tasks, their work is then compared by a third

• 1:08:21Nathan Labenz?

  set of experts to AI work, and the third set of experts are responsible for determining which do they prefer. And the AIs are now winning a lot when it comes to software engineering tasks. So I would say for sure that like, I prefer working with Claude Code over working with a entry level human software developer. That's like, pretty obvious to me, honestly, at this point. I'm not sure yet. I think the debate is ongoing when it comes to, has this hit the statistics or not? But will it? I think it will. I think, I think in 2026,

• 1:09:03Nathan Labenz?

  it's going to be hard to justify on purely economic terms, hiring a 22-year-old out of a CS program versus spending more on Claude Code, or you know, getting really good, or you know, spending the time that you would spend mentoring that person, instead spending that, developing new skills and hooks and, you know, increasingly elaborate architecture for your AI coding setup. So I think the disruption is happening now, certainly in software. And I guess relative to the, you know, the rest of the question timeline, I

• 1:09:45Nathan Labenz?

  don't think it's three to 10, and then 10 to 20 years. I think it is probably coming sooner than that. When accounting, law, parts of medicine, well, let's start with medicine. You know, I've talked ad nauseam now about how the latest models are competitive with attending oncologists. That's a reality. I've lived that. There's no denying it. Nobody can talk me out of it. As sure as I'm sitting here. All three of the frontier models Gemini 3, Claude and ChatGPT 5.2 Pro currently, they're all competitive with attending oncologists and there's just no two ways about that. So in terms of parts of medicine, like yeah, absolutely.

• 1:10:26Nathan Labenz?

  I'd say that disruption is starting to happen. I don't recommend people like ignore their human doctor or like go all in on AI. But I do think that the, especially for things that aren't so important, the substitution effect is going to start to be real because you can get a lot of your questions answered and maybe not have to go to the doctor or go to the right doctor the first time. Or go in, you know, with obviously in our society right now, doctors still are required to prescribe the medicines. So I do think that the way that this is going to play out is going to be noticeably different, like very soon. And, and I'd say that's probably going to be true in accounting and law as well.

• 1:11:09Nathan Labenz?

  You know, when I, I'm certainly no expert in law, but when I get a contract to sign, I run it through three or even four frontier language models in the exact same way that I run my son's lab test results through language models. And I'm yet to regret it. Would I do that for the most important transaction of my life? And I'd probably get a human lawyer, you know, to help review as well. But for routine stuff, I'm absolutely happy with the results that I'm getting. And if like none of the three or four frontier models identify issues that I need to be concerned with, then I'm good. I, I think it's pretty unlikely that all three would miss something that's really important to me. And even in accounting, you know, we're starting to see models are getting pretty good with spreadsheets.

• 1:11:54Nathan Labenz?

  So I think that too is, is coming pretty soon. Another mental model I have for this is like, sort of how elastic is demand in a given domain? And I think that varies dramatically, you know, when it comes to something like.

• 1:12:14Nathan Labenz?

  Accounting.

• 1:12:16Nathan Labenz?

  I personally don't want to buy a lot more accounting services than I am required to buy. And maybe others are out there, you know, wishing that they could buy more accounting and just don't have the budget for it. And when, you know, AI makes accounting professionals more productive, then they'll like buy lots more accounting services and and that will partially offset. But I just don't see that there's like that much latent demand for more accounting. I think most people are like doing roughly what they need to do, what they're required to do. And beyond that, they don't really have that much appetite for more. So in something like accounting, I think when we see threshold effects hit and all of a sudden AI can do the job.

• 1:12:57Nathan Labenz?

  I would expect that to be a field there where there would be more outright substitution and displacement. And not like an explosion of like accounting services provided. Dentistry, by the way, is like my example of like maybe the most extreme version of this, like I want zero dental services. I'd love, I'd love to never have to go to the dentist again in my life. So if there was something that I could buy that was 1%, you know, the cost of, of dentists and did the same service. Like I'd happily do that and I wouldn't be doing like lots more dentistry. I think most people share that intuition. Medicine in general is probably a bit different. I think, you know, most people might want a little more medicine. They might want a little more care so that there could be a surge of demand.

• 1:13:42Nathan Labenz?

  And as things become more accessible, as sort of capacity expands, demand probably grows to fill it. And then software engineering, maybe, you know, could be like the 10x or even the 100x. Like, do we get 100x as much software production over the next few years? Very plausibly. And that might be enough to sustain software employment. At least among the like senior engineers for a while yet, for perhaps a, you know, surprisingly long time. Even when it looks like, hey, AI can like code up this website in one shot, you know, what do we need engineers for? Well, if we're doing 100 times as much software and the architectures are getting ever more elaborate, then, you know, maybe there's, there's still a role for the kind of senior software architect, even if there's not for the, the junior person.

• 1:14:26Nathan Labenz?

  I think this is one area where. I do have a bone to pick with some of Dwar Kesh's analysis and I, you know, I think generally very highly of Dwar Kesh. I think his, you know, obviously his show is great. His questions I think are generally very effective in terms of eliciting alpha from his guests. And I think his essays are generally really good. But one thing that he has said recently that I do pretty strongly disagree with is that explanations, you know, when the question is posed, like, why aren't we seeing more impact than we have to date from AI on the labor market? He has argued that explanations that center on human bottlenecks are essentially cope. He thinks it's really that the models

• 1:15:10Nathan Labenz?

  aren't good enough and it's not that the people are too stuck in their ways. And obviously, I would say it's, it's both. You know, I mean, the models do have room for improvement. Jagged frontier, all that stuff for sure is real. But I think that there really is a lot of human bottleneck going on. And, you know, I see that at the hospital, right? I mean, when a resident comes to the room and talks to us and is like clearly less knowledgeable and less reliable than a large language model, I don't really see that as, I don't know how to interpret that in any other

• 1:15:52Nathan Labenz?

  way than that the humans are the bottlenecks, that they have not realized, you know, that nobody has told them, that they haven't experimented with themselves, actually using a model on their own. That's on them, right? It's not on the model. Like, I can just tell you easily, like, a bunch of times, that if the resident had engaged with chat GPT, as I had engaged with chat GPT before coming to talk to me, then they would know more and they would be able to do better. And I'm, I'm quite confident that is true in a very, very wide range of contexts. I do think that the human bottleneck is a very real phenomenon, not the entire story. But to say

• 1:16:33Nathan Labenz?

  that that's cope, I would definitely debate Dwar Kesh on that one. The other thing that I would point people to in terms of like, just a mental model for this, again, I mentioned already, but we ran that cross post with Luke Drago from the Future of Life Institute podcast. And he's got this idea of the inverse pyramid model, where basically the lowest rung

• 1:16:55Nathan Labenz?

  In terms of the hierarchy of an organization, in other words, the entry level employees, AI is basically coming for them first. Another way that he thinks about it is anything where there are lots of people doing the same job and the organization is geared toward trying to make sure that those individuals act more like cogs in the machine and like do the job in the same way every time where they're focused on consistency, reliability, process, standards, you know, systematic evaluation, like all that stuff really lends itself to AI. So he recommends, and I think this is pretty good advice, try to do things that are N of one and try not to do things where you are one of N.

• 1:17:37Nathan Labenz?

  So be N of one, don't be one of N is pretty good advice, I think. But obviously that's, that's like, you know, advice that some individuals can take. It's not something that I think is going to, you know, preserve the structure of employment broadly. And we, you know, we're not even talking about driving here, right? And that wasn't mentioned in the question, but last I checked, I think it's four million Americans out of like 150 million employed Americans, something like four million are professional drivers. We're getting real close, right? To where human drivers are just not going to need to exist. And it seems like the disruption there could happen quite soon.

• 1:18:19Nathan Labenz?

  And it again, seems like the bottlenecks increasingly at this point are human. You've got some city councils, you know, that are trying to do various things or what are the teamsters going to say, what have you? I'm in Detroit. Waymo was projected to launch in Detroit in 2026. It's going to be really interesting to see how that plays here in the Motor City, especially because companies headquartered here aren't really at the center of the frontier of the self-driving technology, to put it mildly. So I'm not sure what the response is going to be, but the, the bottlenecks there are pretty clearly human at this point. And it seems like, even in a world where the AIs don't really get any better, I do

• 1:19:02Nathan Labenz?

  see as we kind of figure out how to scaffold them and plug them in and manage the context right and all that kind of stuff, I do see pretty serious disruption, at least being possible unless it is blocked by.

• 1:19:17Nathan Labenz?

  Sociopolitical dynamics in just the next couple of years. And then, you know, again, superintelligence is a distinct question from that. But it seems like work like quite clearly close to where AI will be able to compete for entry level jobs, for jobs where an individual human is one of N people doing that job. Because there are the standards and the processes and evaluation frameworks to make sure that AI is doing well. And I expect AI, especially, you know, as we really put that elbow grease in, I expect it will do better than a lot of people in a lot of places. And that the economic, the market pressures will be very strong to adopt them. Customer service is another great example, right?

• 1:20:02Nathan Labenz?

  I mean, will there be some customer service people? I don't think it's going to zero in the immediate term. But my dad was on the phone with Bank of America yesterday on hold for however long, getting increasingly agitated by the fact that he was on hold as he's listening to their hold music and messages. And I was just like, man, I know several AI customer service firms doing voice agent type work that would at a minimum be able to reduce the wait time dramatically and probably do just as good of a job for like a large majority of tickets as the humans are ultimately able to do. So I think we're bottlenecked on willingness. I think we're bottlenecked on implementation. At the very high end, we're still bottlenecked on model capability.

• 1:20:45Nathan Labenz?

  If I had to kind of project like how far up the pyramid can AI go today? It's like, you know, maybe half the way up vertically, but that's like, you know, whatever, 80% of the mass. And I think the technology is basically there for that. And now we'll have to see how fast people actually. How fast does the market work? How fast does market incentive and pressure and increasingly lower barriers to successful deployment? How fast does that all work to actually create the disruption that I honestly think is kind of inevitable? But relative to the question, I would take the under in terms of timelines. And then the final point of the question is where do humans remain essential at each stage?

• 1:21:32Nathan Labenz?

  I think that's a design question more than anything else. I think we want to design AI future where humans are essential or are at least a big part of the process so that we can retain some authorship over the future and not just give the whole thing over to AIs as the gradual disempowerment people worry we might. But I think that that is going to have to be intentional, more so than like something where we will find that there's some human, some ineffable human essence, you know, some Elon Vital that, you know, something that for whatever mysterious or even mythical reason, mystical

• 1:22:10Nathan Labenz?

  reason, only humans can do. I really don't believe in that. I think that we are pretty remarkable, certainly in our blood breadth and our flexibility and the high end of human achievement, you know, is, is obviously super impressive and inspiring to the rest of us, you know, mere mortal humans. But AI is doing superhuman level stuff in more and more domains. And I don't really see fundamental blockers to that being in the fullness of time, functionally everything. So I think it's much more on us to think about how do we design these things? How do we design our overall society? How do we design our overall systems? How do we design our models? How do we design our implementations?

• 1:22:55Nathan Labenz?

  So that we keep the parts that we want to keep and retain some overall control of steering the future. I don't think that's going to just happen because there's something that AIs can never do, that only we can do. I think it's going to have to be a lot more intentional than that. Okay, next question regarding nonprofits and shifting need. As AI reshapes labor markets, how do you expect the definition and scale of those in need to change over again, zero to three, three to 10, and 10 to 20 years? What should nonprofits and funders start doing now to prepare, especially around workforce transitions, mental health, and basic economic stability? I would like to see a lot more work on this, honestly, than we have to date.

• 1:23:39Nathan Labenz?

  My general answer is I have to give Sam Altman a ton of credit here for making personal investments in UBI. And I think that that is something that we are ultimately going to need, whether it ends up being called UBI or exactly how it looks. I think we're going to have to have some new social contract that decouples a person's right to a decent material standard of living from their ability to contribute economically, especially in the context of competition that they're going to face from AI systems. I just think that's really the only way that we are going to get to a place that

• 1:24:21Nathan Labenz?

  anybody would be happy with. And I welcome other suggestions, but I haven't heard many that really make a lot of sense. The only other, what I hear is either like, we're going to need a UBI, or we're not going to need a UBI because she's always going to be work for humans. And I just don't believe that. Tyler Cowen, you know, it kind of confuses me on this point these days, to be honest. And I've been kind of waiting to invite him on the podcast for a while. Maybe I should finally get around to doing it. But I looked back at marginal revolution, and I think his first mention of ZMP workers, or zero marginal product workers, was from 2010. And this was basically research that was looking at how firms responded to the Great Recession. And what many firms did in response

• 1:25:06Nathan Labenz?

  to the Great Recession is they looked around their, their teams, and they said, who do we really not have to have to go forward and be okay? And they were able to identify people, and they were able to cut those people. And of course, you know, that's a noisy process. But by and large, it seems like firms were able to cut headcount, and then actually see somewhat of a surge in productivity, seemingly because they were producing more or less the same with fewer workers, because there were some people that just weren't really contributing much. You take that, and you take the general phenomenon of like bullshit jobs, where like, a lot of people seem to feel, according to survey results,

• 1:25:47Nathan Labenz?

  that like, even absent AI, that like, their own work isn't really worth anything, and it's kind of performative, or, you know, it wouldn't matter if it wasn't done. Maybe they're wrong about that, but are you really, you know, that confident that people who are saying that their own work is meaningless, or not necessary, are you really confident that they're wrong? I tend to think they probably, you know, at least some of them are probably right. So it feels like there's like already, you know, kind of at baseline, some ZMP workers that are employed. And there's a lot of people who are saying that their, their work is just not really meaningful or important, and that things would be fine if it wasn't done at all. And so, yeah, I just see that we, we don't have a great answer other than that we have

• 1:26:28Nathan Labenz?

  to find a way to decouple somebody's ability to exist and like have a decent life from their ability to compete economically. Again, especially to compete against AIs. Exactly what the right structure of that is, I think we would be doing, we would do really well to start doing much more of that experimentation. And I personally interpreted some of the recent UBI results as much more encouraging than I think the broader discourse did. I'm certainly not an expert in this, but my takeaway from some of the UBI research was like, people were disappointed that people seemed to work less in response to getting the UBI.

• 1:27:13Nathan Labenz?

  And I'm kind of like, I think that is the point. You know, I like, if they worked the same, in my mind, that might even be more of a failure. Like, I, I think we want to, not, we might not want to, but yeah, we maybe didn't want to like two or three years ago when these studies were being run, have people like just take cash and not work. But I think that what that does show is that people are just, in many cases, just working for money. They're not like, there, maybe you could also say like, at least some people are like relatively easily satisfied with like not that much money, which would suggest that they are able to find meaning in things like spending time with family and leisure and,

• 1:27:58Nathan Labenz?

  you know, whatever. Maybe they don't feel compelled to go out into the workforce and do some job that they really don't want to do just to get a bit more money. Obviously, like marginal tax rates are also, in many cases, like totally out of whack and working more can mean you forfeit benefits of all kinds, whatever. It's a very complicated question, but.

• 1:28:17Nathan Labenz?

  I'm encouraged, I think, much more than the average commenter is by these UBI results, because I just see that like, it didn't take that much to actually get people to work less. And if they're not enjoying their work, and they're substituting away from that work toward leisure, to me, that suggests they're not like, missing out on meaning. They're not like pining for the workplace as the place where they're gonna like, have identity and find meaning. It seems like they're finding that in other places just fine. Thank you very much. And I would expect that probably to continue. I tend to think that the, you know, we need jobs for structure and meaning and whatever. I think that's mostly cope. And I think it's especially unhelpful cope when it's projected

• 1:29:00Nathan Labenz?

  by people who have privileged positions where their work is like, high status and where they genuinely do love it. And I count myself in that group, and I'm very thankful for that. But when that sort of reality is projected onto people who are lower on the socioeconomic scale, who are doing work that they don't want to do, because they have to do it, because that's the only way that they're gonna feed and clothe their kids, I think that's like, quite kind of productive and misguided. So, yeah, bottom line, I would love to hear other ideas. I would love to hear, I would love to read your utopian fiction about other ways that we rework the social contract that isn't

• 1:29:43Nathan Labenz?

  just kind of a vanilla UBI type structure. But until somebody more creative and imaginative and visionary than me comes up with those ideas, I basically still think that UBI is the default. And denial is not helpful. More experimentation sooner about the details and the structures and exactly how incentives should work would be really valuable. Okay, thank you for those questions. Next one, are people being misled by bench maxing? And then another kind of related question, does the massive success of pre-training give people the wrong intuition about AGI? Yeah, I mean, characterizing whether people are misled, or whether people have the right or wrong intuition, I think is very hard,

• 1:30:26Nathan Labenz?

  because people have very, very different intuitions and world models, right? So like, there's such diversity there, but I can't characterize like, people's opinions in general with any...

• 1:30:39Nathan Labenz?

  Yeah, that's just such a wide ranging thing, right? That anything I might say about some people's opinions is obviously going to be contradicted by other people's opinions. That said, I do think bench maxing is probably misleading to at least some degree. And I would go back to the Chinese models, which I talked about last time, being like significantly worse on just a very random multimodal task than all of the leading Western models. As a kind of leading indicator of how much bench maxing might be misleading us more generally.

• 1:31:16Nathan Labenz?

  It does seem to me that like the delta between the leading frontier American models and the Chinese models in benchmarks is much smaller than the difference between them. In practical day to day utility on whatever idiosyncratic and or esoteric task you might want to use an AI to help with. And of course, the Llama four incident shows that as well, where they like, I guess, spun up like maybe a bunch of different Llama four variants to like, put one in each different LM arena category to try to maximize the score. I'm not exactly sure what they did.

• 1:31:54Nathan Labenz?

  But they they sort of LM arena maxed and that was effective in as much as they got a high position on LM arena. But you know, you don't hear all that much about llama four. And it seems like it just wasn't really competitive, but they were able to make it look like it was competitive on some of these standardized scores. So, yes, I think bench maxing is a problem.

• 1:32:21Nathan Labenz?

  Independent analysis is, you know, is a good antidote for this, looking at the meter charts, looking at like artificial analysis. You know, there are people that are the scale benchmark that is like largely private, you know, is a pretty good way to think about that as well. I think our KGI has been like remarkably. Durable in terms of how relevant it's been, but you know, people can obviously benchmark, benchmark on that too. But definitely things where there's a private test set and things where people are like taking it upon themselves to really analyze capabilities and they make that their thing, you know, and like really try to earn your trust by being a reliable

• 1:33:02Nathan Labenz?

  guide to model performance over time. I think those are the things that increasingly the field will be looking to beyond just performance on open standardized benchmark tests. Those, you know, continue to be somewhat relevant for sure, but like less and less all the time. And then on this, on the question of like, does pre-training give people the wrong intuition about AGI? If anything, I think maybe post-training is giving people the wrong intuition about AGI and maybe the, the classic Shogoff meme for pre-training is maybe a better intuition. And this kind of also connects to the breadth first versus

• 1:33:45Nathan Labenz?

  depth first search of the space of possible AIs. I think that if people are misled about the intuition, or if people have the wrong intuitions about AGI, I would expect that it's wrong in the sense that they have encountered a relatively narrow range of form factors of AIs, which are basically like chatbots and coding assistants, and that those, you know, those designs have kind of converged thus far. The, you know, the overall paradigm of helpful, honest, harmless, you know, which has been great for unlocking a ton of value and, and pretty great in terms of certainly when done well, as Anthropic has done, you

• 1:34:27Nathan Labenz?

  know, shaping the model's character, you know, no, no beef with those problem or with, with those approaches. But if there is a problem with that, it has presented to the public.

• 1:34:38Nathan Labenz?

  A very, very narrow slice of the overall conceptual menu of like what AI can be like. And so it maybe is lulling people into a false sense of security or a false sense of like this will be continue to be normal. And I think that in reality, the Shogoff meme of like, this thing is an insane alien that kind of encompasses everything in a super strange way and can kind of shapeshift and be anything you want it to be, or maybe even things you don't want it to be. I think that is maybe the better intuition for at least the space of possibility for AGI. And you know, when you think about things like the, I think back to the episode we did

• 1:35:18Nathan Labenz?

  with Apollo with Marius Hoppon from Apollo Research where they got access to chain of thought from O3 class models and they found that the chain of thought was kind of evolving to become its own dialect. Remember like disclaim, disclaim, vantage, you know, the watchers, these strange phrases that really didn't look anything like the training data, in this case, you know, being driven by reinforcement learning at increasing scale. It just feels to me like there's just, there's so much more alienness to these systems that we are than we are seeing. I guess, you know, just a couple other little intuitions for this, maybe worth mentioning. People often cite the bird airplane analogy.

• 1:36:01Nathan Labenz?

  There's many, many people have commented, of course, that like, we wanted to create a machine that could fly. A lot of early attempts sort of tried to mimic the bird. What we have is an airplane that like flies on quite different principles and is just like way faster, way more powerful, you know, can carry way heavier loads than birds. Have some, can't do necessarily everything that a bird can do, but for the things that we've designed it for, it's just way, way better. Right? We wouldn't want a scaled up bird to take a cross-country flight on. Airplanes are just way better than scaled up birds. Similarly, the, many people may have seen recently a video of, it was kind of a compare and contrast where it was, there was like a humanoid robot harvesting grain in a field,

• 1:36:45Nathan Labenz?

  you know, and kind of chopping it down and bumbling it up in a way that humans traditionally used to do. And then there was this carrot picking machine that's just rolling through a field, pulling carrots out of the ground en masse, washing them off and just operating, you know, at orders of magnitude faster than humans could possibly pick carrots. And so. I think something similar, you know, maybe happens with a true AGI or like a super intelligence where the it becomes so powerful that it potentially doesn't even really make sense for it to present in like this natural language sort of way anymore. Or at a minimum, that becomes sort of a dramatic reduction, some sort of, you know,

• 1:37:30Nathan Labenz?

  a majorly lossy kind of summary of what it's doing as opposed to the core thing that it's doing today, you know, today, the thing, the response that you get from the chatbot, like that is its output, right? In the future, I think the natural language summary of what it's doing may be just a very small part of what it is actually doing. And so, yeah, I think study the show Goth, study the weird chains of thought, expand your mind when it comes to the space of possibility and how weird things could be. Those are my guesses for how most people are being misled, if at all, today. Okay. Next question, is learning from a physical environment a requisite for AI? I don't think so.

• 1:38:15Nathan Labenz?

  I think we're pretty far along, obviously, in AI at this point. And when we look at all the things that AIs can do and like how many categories they're winning more than 50% on in the GDP valve context, all without any, you know, robotics or like physical embodiment at this stage, I think it's like pretty clear that you can get pretty good AI without needing to learn from a physical environment. Now, the line maybe starts to blur a little bit there when it's like, can you use a computer? Is that a physical environment? Well, it's a digital environment, but it's like spatially organized. And it does seem like the way that we are getting AIs to learn how to use a computer is by them actually like trying and failing to use the computer a lot, right?

• 1:38:59Nathan Labenz?

  We needed some language model base capability to like conceptually understand what one might want to do on a computer and what different buttons might mean and so on. And then from there, it's been like a lot of actual reinforcement learning where is once there was at least some ability to succeed, then, you know, you could hill climb from there up to like, at this point, a pretty decent performance. But I definitely think we're not quite there, but in like 2026, I think we'll definitely have computers that are AIs that use computers basically as well, if not better than your typical human user. So. Yeah, I guess maybe it kind of comes down to like a task by task thing. Do I think that large language models are going to become plumbers without a bunch of reinforcement

• 1:39:42Nathan Labenz?

  learning on physical plumbing like tasks? No, I think that if you want to be a plumber, you're going to have to do a bunch of stuff in the physical world, you're going to have to get good at that, and that is probably going to require some sort of embodiment. You know, when the question says physical environment, I would also say simulation is is going to get really good. Obviously, Nvidia's GPUs are not just good for training, they're also good for simulation. So we're going to see more and more simulation being used in all facets of AI training, including for robotics. But if you count that simulated physical environment as a physical environment, I do think you need it to do physical things.

• 1:40:24Nathan Labenz?

  And ultimately, like, a language model can't be a plumber. Like, you could, I think you can be a lawyer without ever having any physical embodiment. I think you could be an accountant. Arguably, your, you know, Excel spreadsheet maybe is your is your physical quote unquote environment there. But yeah, I guess basically, my intuition is that you need training in something like the environment that you're going to operate in. So as long as you're just operating in language space, language is probably enough. You want to start to operate in pixel space, you're going to need to be trained in pixel space. You want to operate in spreadsheet space, you're going to need to be trained in spreadsheet space. You want to operate in physical real world space, you're going to need to be trained

• 1:41:07Nathan Labenz?

  in a combination probably of simulated physical real world space and to some degree, actual real world space. And it's going to happen. You know, the other question, the other way to think about these kinds of questions is like, we'll never know because is it required? I think not for many tasks, but it's going to happen. So there, and there may be some positive transfer. You know, will the AIs of 2027, 2028, even if I'm just doing something that Is ultimately a purely language, say, legal task. Will the best models that I can go to for those kinds of tasks also have some sort of physical, real-world training as part of their overall training mix?

• 1:41:52Nathan Labenz?

  Wouldn't surprise me at all. It wouldn't surprise me if there's some positive transfer there, if there's just some better intuition, you know, if certain kinds of queries that require spatial reasoning just perform better because that that kind of training is folded into the mix. And so like, I'm not saying that 2027, 2028 AIs, even, you know, of the chatbot or digital assistant variety, won't have any physical environment-based training going into them. I'm just saying, I don't think you would have to do that to be successful, but in all like, or at least, you know, it's pretty likely at least that it will happen. And so then it'll be kind of hard in the end to tease out, like, was this required to happen? Could we have got here another way?

• 1:42:35Nathan Labenz?

  It just seems like everything is going to be developed and kind of folded into, to the degree that it works. And it seems like everything is working. It's all going to be folded into kind of the mainline frontier models. And so we will probably not have a clear answer. Could we have got here without doing any physical training? My guess, my bet would be counterfactually, yes. If it adds even a little bit, then it'll be part of the overall mix. And so, you know, what we'll actually have will be AIs that are kind of trained on everything, regardless of whether some of those things could have been, you know, cut with minimal performance loss or not. Okay, next section gets a little more focused on tooling and kind of AI engineering concepts.

• 1:43:16Nathan Labenz?

  Are you noticing any emerging standards or winners for tooling emerge across the companies that you work with? This is interesting. I would say. I don't really have anything super shocking to to report. You know, I think in terms of models that people are using,

• 1:43:36Nathan Labenz?

  you know, Claude for coding is typically the go to still certainly get very good reviews from people on GPT codex. But Claude still seems to be the go to for most people. If they didn't have Claude, you know, there are certainly other great options out there. Gemini three is also excellent. But I'd say Claude remains the kind of consensus top choice for coding. OpenAI is kind of pretty good at everything and is probably the default thing that most people use for like in browser, random queries on a day to day basis. And Gemini flash is, I would say, pretty clearly the top choice for things that don't require frontier capabilities

• 1:44:22Nathan Labenz?

  and where speed and cost is a significant factor. And I don't think that's really surprising. We're out of step with like mainstream online consensus at all. I really haven't seen anything that is, you know, that that that challenges the mainline narratives at a certainly at a model provider level. Now you go to the sort of tool level and yeah, I don't know. I think again, it's like, I think it's, I think the market is like getting away from most people. I think most people are because things are changing so fast. Most people are behind, you know, I think most people are not using the best thing, which is maybe, you know, normal life in general.

• 1:45:06Nathan Labenz?

  For example, I've used Langchain on a couple of projects recently, one at Waymark and one at another company. And I'd say it like works pretty well. You can like build agents in it. You can have those agents hosted on their infrastructure, which can be pretty cool. You can just log traces to it. It's a pretty heavyweight UI with like a lot of features. It can be a bit of overwhelming data presentation for many people at first. It does take a little like getting used to, but I'd say it works pretty well. And until such time as I'm like not happy with it.

• 1:45:44Nathan Labenz?

  Or here's some note from somebody else that like there's something that's just dramatically better. I'm pretty content with it. And I think a lot of people are kind of in a similar spot where it's like, man, I can barely keep up with model releases and tool releases kind of feel like a second tier question where as long as I'm like, meeting the need that I originally had, I'm probably fine. And so even though there might be better things out there, I think it's just so overwhelming to go try to shop effectively that it's like, people are kind of staying the course, you know, with whatever decision they made quite a bit. Another interesting direction that we're going to start to see develop here

• 1:46:28Nathan Labenz?

  and already are starting to see develop is of course that the model providers are trying to become platforms and they are building out more and more of this stuff themselves. So of course, OpenAI has their like agent builder type stuff now that competes directly with many agent builders that were built on their platform. They also have like observability of various kinds. Anthropic bought Human Loop, which was a past guest and I was a happy Human Loop customer for a while, but now they're part of Anthropic. So it's going to be interesting to see and of course, Google's going to just build everything over time to probably somewhat varying degrees of quality, but you know, they're going to have in the end, like the most robust portfolio of products, probably of any of the top tier

• 1:47:10Nathan Labenz?

  model providers. So what the dynamics will be there between how often it makes sense to just pick a platform, go with their model, go with their observability, go with whatever tooling, you know, they have, whatever monitoring they have versus try to maintain flexibility and, and ability to upgrade a model quickly on, you know, on a new release, which would potentially require you to have a more horizontal layer for these kind of observability questions and, you know, and all sorts of tooling questions. It's going to be interesting to see how that develops. If I'm a big enterprise, I want to avoid lock-in and I want to

• 1:47:55Nathan Labenz?

  Invest potentially more than I really have to in some of these things so that I at least have a little bit more ability to control my own destiny and I'm not totally beholden to one of the platforms. If I'm a startup or if I'm just an individual or if I'm an individual just doing a one-off project, then maybe I just kind of take the convenience and use OpenAI observability or use Anthropic's observability because I'm already using their model and it all just kind of integrates and makes things easier and faster. But all of this is to say, I don't think that there is like any super obvious major trends that I'm seeing. Others may have better answers. I would certainly, if you feel like this answer sucks, then you have a better answer,

• 1:48:37Nathan Labenz?

  write me and let me know and maybe we'll even do a full episode on it. But yeah, and I would even say like, just listening to the Late in Space podcast, I take quite a bit of pride in the fact that the cognitive revolution has been voted twice at the Late in Space AI Engineer World's Fair Events as the number three podcast for AI engineers. Late in Space has been voted in both of those surveys number one. And in listening to them, it also does feel like the sort of content mix has trended away from these kind of tooling questions. Of course, it's still like an important element, right? You need to be able to look at traces, but is it, you know, is it really where like

• 1:49:19Nathan Labenz?

  people's mental energy is going right now? It feels like it's less topical than it used to be and I think you can even see that in their mix of guests. And then just one other anecdote I'll give on this is when I was vibe coding the Christmas present apps that I've mentioned a couple of times, the one that I was coding for my mom, the custom trip planner, it involves like AI research into all these various things, right? So there's like a prompt to go find restaurants and she's gluten free. So we're like really trying to dig into like gluten free options in all these different places. And when it comes to where she wants to stay, she loves to have a nice view and wants a balcony. So, you know, the prompts are customized to that level as well.

• 1:50:01Nathan Labenz?

  I, you know, early on in the development and even still now, as I continue to kind of work with her and try to enhance it in ways to make it more valuable for her because she is actually using it quite a bit. I wanted to look at the traces of, you know, what, what are the inputs and outputs? What prompt is the model seeing and what is it giving back in raw form? So that I could kind of debug that or, you know, know what's working and not working at a level below the UI that she as a user is using. And so what did I do? I just added a trace function by just prompting Claude code to say, hey, I want to see all the history of the queries that are, that is, that are made to you.

• 1:50:43Nathan Labenz?

  Can you add a tab to this application that just gives me direct access to the full history? And it just built that thing, I think in maybe one prompt, maybe it was like two prompts. And now I have, you know, right alongside all the core features of the app, a debugging tab where I can go in and just look at the history of the prompts. So I think that's another factor that is kind of challenging. And this is like, you know, maybe future of SaaS writ small or, you know, future of SaaS in a nutshell. What would I have done in the past, right? If I didn't have Claude code to code that kind of thing?

• 1:51:18Nathan Labenz?

  Then maybe I go make a free trial account on Human Loop or maybe I, you know, wire it up to Langchain or whatever. But I didn't do any of that. I just said, hey, Claude, log all these things and give me a tab where I can see them. And that's been perfectly good. Could it be more elaborate? Yeah. Could it be more full featured? Sure. Like, you know, it's, but it meets my needs for the development of this particular app. And it basically took, you know, I can say confidently, it took me less time to prompt and have Claude code do it than it would have taken me to go find some solution, you know, figure out what it was, figure out how to connect into it, et cetera, et cetera. And so I suspect maybe that is also a part of why people are talking about this stuff

• 1:52:01Nathan Labenz?

  less because it's just become like a lot easier to meet the basic needs in some cases, just by like coding it from scratch, you know, with a couple of prompts. So. Yeah, I would love to hear more about this, you know, again, from other people. Okay, next question. This one's kind of an interesting one. And it wasn't actually part of the AMA, but it was a question that I was asked. So I had a little interaction, just posted, actually, this is in the context of vibe coding another app. So my, the app that I've coded for my dad for Christmas was this stock trading strategy back tester. He has these ideas, what if we did this, you know, what if I, every week, what if I bought the stocks that lost the most last week and tried to catch them on the rebound?

• 1:52:45Nathan Labenz?

  Okay, fine. Is that going to work? Is it not going to work? The app that I created for him allows him to articulate a strategy in natural language, translate that to trading rules, and then go back and look at some time interval and see how would that trading strategy have done over that time interval, just systematically executing those trading rules. Pretty cool. He hasn't used it all that much, to be honest. But in the context, in the course of doing that and testing it, I was testing that strategy of buying the biggest losers and then trying to catch them on the rebound. And I tried it once on an annual basis, you know, what if I bought the biggest losers from the last year, held them for the following year and did that every year? Would I beat the S and P or would I, would I fall short of the S and P?

• 1:53:29Nathan Labenz?

  I was sure there was a bug when in 2022, I think it was, Nvidia was one of the biggest losers. It was down like 50% on the year. And I was like, okay, something is clearly wrong with that. No way Nvidia would have been one of the biggest losers for a whole year. Turns out it was. And so the AI was right. And I asked the question and, you know, Claude code went and like, verified online that like, you know, indeed, this is right. Like Nvidia was one of the biggest losers of that year. So the app is working correctly. You were just wrong in your assumption. This is also another one of these moments where I noticed sycophancy on the decline because it would have been very easy for the model to be like, you're absolutely right. Nvidia has been a killer stock. Like there must be a problem in the code. It did not do that. It came back and said, no, you are wrong.

• 1:54:14Nathan Labenz?

  2022 Nvidia was one of the biggest losers and the app seems to be performing correctly. Okay. I posted that online. Here's a crazy stat. In 2022, Nvidia was one of the biggest loser stocks in the market for the entire year. And Holly Elmore, Executive Director of Paws AI, came by with a, I would say, a critical comment, as she often does, and basically said, like, you know, this isn't SportsCenter. Basically, like, you are morally out of line for engaging in, you know, fun, or sort of, you know, trying to make yourself look clever, or, you know, show insights by noticing these quirky things in the AI space, because the whole thing is

• 1:54:59Nathan Labenz?

  bad, and it ought to be condemned, and you ought to be condemning it, and doing anything else, basically, in her view, is morally reprehensible. And I was like, okay, yeah, do you really think this is like an effective way to advocate for your position? Because I think everybody who listens to this feed, and if you're two hours in with me on this episode, then I'm like, seriously concerned about AI safety issues, and like, do not want to see a race to recursive self-improvement, thing, you know, things, big decisions that are going to be made in the next year or two around the automation of AI R&D, you know, I think are very, very big and important questions, and I don't think we're ready

• 1:55:39Nathan Labenz?

  to cross some of those thresholds, or, you know, Rubicons, if you want. And so I said to her, look, like, I think I'm much more sympathetic to your cause than most people. I did sign the ban superintelligence statement three or four months ago, for example, as just like one material, or one tangible indicator, that I'm like, much more sympathetic to her cause than most. But like, do you really think that, like, coming after me over a tweet about some random observation about Nvidia's stock?

• 1:56:11Nathan Labenz?

  Is is the way to like advance your cause. And then she asked me, well, how did my comment make you feel? And I thought about that a decent amount and decided to address it here in the AMA, even though that's not what she was asking for. And I think my bottom line sense of this sort of thing goes back to a mantra that I used to say a lot more often, which is that we should really try to avoid psychologizing others' AI takes and focus as much as we can on the object level facts of what is actually happening, what is, what does that imply? And just take people's statements and positions at face value. There is such disagreement in the space.

• 1:56:53Nathan Labenz?

  There's such uncertainty, you know, the, the AI 2025 forecast results. Again, I got top 5% with predictions I don't think look all that super accurate. And of course, you know, famously, we've got Turing Award winners who have extremely different positions on how dangerous AI is going to be. And I think they're all genuinely held. So I think the way that like being sideswiped, attacked, or like, you know, accused of moral corruption online, because I made one random comment about Nvidia's stock performance from a couple of years ago, I think the way that made me feel was like, kind of indignant, confused, averse a bit to the person saying it and the cause itself.

• 1:57:36Nathan Labenz?

  And this is a cause that I genuinely, you know, I'm not like advocating for exactly a pause on AI. I don't, again, like, what does that even mean? Whatever. I don't think we should shut it all down. I do think there is tremendous upside. I do think that like, we should be very careful and we should, you know, we should be willing to slow down, if not pause, at some point when we hit levels that we really might not be able to control. But I also think like, the progress has been clearly much more beneficial than it has been harmful so far. And I've lived that in recent months. And so I kind of felt like alienated from the AI safety movement, or at least the more,

• 1:58:19Nathan Labenz?

  let's say, strident or shrill voices in it by that comment. And I think my takeaway from that is

• 1:58:28Nathan Labenz?

  I don't think, I wouldn't go so far as to say we shouldn't shame people. But I think we should shame people very carefully, very selectively, and only for what they are doing. I think it is pretty defensible at this point to shame XAI for some of the things that they have released in Grok. I think it is appropriate to shame XAI for having Grok on Twitter undressing women with seemingly no guardrails in place. That is worthy of shaming, I think. So, but that's an action. I would not shame people for, or I would not assume bad faith.

• 1:59:12Nathan Labenz?

  I know that not everybody's positions are, you know, fully rightly taken at face value. But if somebody's going to be out there engaging in the discourse, I think that like projecting your sense of their psychology onto them and arguing from that basis, it just ends up with people more often than not feeling bad, being increasingly bitter toward each other, more sort of calcification or hardening of factions, more sort of like sense of somebody's my ally, somebody's my enemy. And I don't want any of that. I, you know, I think the healthiest discourse that we can have around AI assumes that everybody's trying their best.

• 1:59:53Nathan Labenz?

  Ideally, it gives people the space to genuinely be trying their best and avoids this sort of kind of sideways accusations or moralizing or psychologizing, unless people are really doing things where you're like, you are undeniably fucking up and maybe even you have a pattern of undeniably fucking up, in which case, sure. So I would say to Holly, your question did not make me feel very good. It kind of made me feel alienated from you and your cause, and I wouldn't recommend doing that. I do think if you want to go protest outside XAI, I would support you in doing that. Choose your targets a little more carefully.

• 2:00:35Nathan Labenz?

  Try to recruit people like me to your side as allies and shame people.

• 2:00:43Nathan Labenz?

  Selectively for things that they've really done wrong, where they are responsible. I think that's fine, but don't just start using these tactics everywhere because it's just coarsening the discourse and and making everything a little more bitter and contentious than it needs to be. And if I don't think people do their best reasoning that way. Certainly, I don't think I do.

• 2:01:03Nathan Labenz?

  Okay, these are the only three AI questions that I thought made the cut, and I probably generated across three models well over 100. So definitely give this one to the humans in terms of questions that I wanted to answer. But here's three questions from AI to wrap us up. First, you're in Michigan, not San Francisco, or London, or Washington DC. Does that distance help you or hurt you? What do you miss by not being quote, "in the room"? I would say this is definitely a real issue. It definitely hurts to be outside of these core hubs. And obviously, San Francisco is like far and away number one, and London is like two, and maybe is close. I mean, I don't whatever. San Francisco and London are both like the hubs. DC is increasingly becoming a hub because there's so much policy

• 2:01:48Nathan Labenz?

  going on there. It's a very different hub, I would say, from the San Francisco and London hubs. And I'm not even sure it really belongs on that list, but that was the way the AI phrased the question. So being outside of San Francisco and London, I do think it makes it harder than it is if you're in those places to stay up to date, you know, to be in the loop, to, you know, have the sort of zeitgeist. There's definitely a lot of stuff happening in person in San Francisco, events going on all the time, hackathons going on all the time. To some degree, secrets are being traded or spilled across frontier model developers at the proverbial parties. The fact that there is a emerging trend of rooms at San

• 2:02:32Nathan Labenz?

  Francisco house parties where there's like a no AI talk room, that just goes to show like how much AI talk is going on. And the AI talk there is definitely way more sophisticated than it is anywhere else. So I think that's a that's a my honest sense of the reality. And I'm able to compensate for it pretty well by being a hyper online. So certainly like spending a ton of time on Twitter still is like part of where I keep up to date, no doubt about that. The podcast itself is also really helpful because I do get to have.

• 2:03:06Nathan Labenz?

  Substantive conversations with very, you know, plugged in people who are in those rooms much more often than I am. And then occasionally I do try to go to events, you know, so having gone to, for example, the curve each of the last two years or last year, going to the summit on existential security. You know, these are, these are also gathering places where, you know, you can get a very concentrated dose of exposure to the leading thought. I think it does really help to be there. So I feel like I'm missing out on that to some extent, but do in large part to the podcast. I am able to compensate for it to a significant degree.

• 2:03:45Nathan Labenz?

  But being outside of those hubs, I do think you have, you have to be much more intentional about how you're going to compensate. And it does probably end up meaning, you know, in the Bay area, you could probably be much less online and still equally or even more plugged in. But if you're not in those places, then online I think is really the main place to get it. And also, you know, I would definitely make some occasional trips to those places to be in the room because I do think that that is a really valuable way to, to learn and make sure that you stay up to speed. And you look at the results of the AI forecasting survey from last year and, you know, you got

• 2:04:29Nathan Labenz?

  Ryan Greenblatt at number two, you got a Jay Acotra at number three on the leaderboard. Like that is not an accident, right? Those people are extremely well informed and it's because of the social context that they find themselves in. In fact, Jay has said that, she said, my method for the survey was talking to Ryan and then getting a few more things wrong than he did. So the, the leading thought leaders do know each other. They do communicate a lot and that environment, you know, is, I think it really does help to spend at least a little time in it. Okay, next question. As a survival and flourishing fund recommender, you see the landscape of safety organizations up close. What's underfunded that shouldn't be? Again, that's an AI question.

• 2:05:12Nathan Labenz?

  I think my big answer here is neglected approaches. I'm wearing my AE Studio ACDC themed swag hat. And I do think the neglected approaches approach as articulated by AE Studio is a great answer, great meta level answer to this question. And as a reminder, they did a survey of people in the AI safety field. They asked them, do you think we have all the ideas that we need to be successful in AI safety in the big picture? The answer was no, we're going to need more ideas. And so like clearly, the community thinks that we need more ideas. The community thinks that there are not, we do not have all the answers that we're ultimately going to need.

• 2:05:57Nathan Labenz?

  So I think that things like what Janus does in terms of just being like super deeply engaged with language models and really trying to understand their characters and their tendencies. I think that stuff is really good. What Elios has done similarly with like model welfare tests, I think is like really interesting as well. I think AE Studio's own work in terms of self other overlap is, you know, I come back to that all the time. What Emma Shear and the team at Softmax are doing. All these sorts of things where it's like, can we find creative ways to either design

• 2:06:42Nathan Labenz?

  or train or somehow get into equilibrium with AI systems in ways that feel more stable, that feel like they could be, you know, the beginning of some sort of stable equilibrium? I think those ideas are dramatically underdeveloped. They are often pre-paradigmatic. They are.

• 2:07:09Nathan Labenz?

  Often developed by kind of weird people, some of whom I think would wear that label with pride. They are sometimes like intersect with kind of non-scientific ideas or you know woo sort of ideas or ideas about AI consciousness, which are obviously like very hard to prove and don't feel intuitive to many people. But I think all those sorts of things are, I think we should have more of them. And my kind of call to action there is like, if you have a weird idea that you've never heard anybody else talk about, I absolutely think it is worth trying to develop that idea. Most of the time, it's not going to go anywhere. Certainly, most of my idle, you know, shower

• 2:07:51Nathan Labenz?

  thoughts do not turn into anything great. But the field collectively believes we need more ideas. So where are those ideas going to come from? They're probably, at least some of them are going to come from people from other fields, people with just very unusual or idiosyncratic ways of looking at the world, people who interact with AIs in very unique and particular ways. People who find inspiration in biological systems that they can map onto AI systems in ways that other people aren't thinking about. I think all of that stuff is dramatically underdone. And arguably, the whole AI safety landscape is, you know, underfunded. So I'd love to see

• 2:08:33Nathan Labenz?

  more resources go into interpretability. I would love there to be not just one good fire. I know there are, there are a couple of other organizations, for-profit companies that are working on interpretability type stuff, but I would love there to be significantly more work going into interpretability. If we scaled that up by an order of magnitude, you know, I think that'd be great. Some of the stuff that Redwood Research is doing, where you take the assumption that models are going to be out to get us and then try to figure out how can we work with them. I think that is also really underfunded. For me, that's just like, I admire that work so much because it feels so hard to me. It like feels so depressing to work under that paradigm and try to make it work. But Lord knows, you know, that might many, many scenarios, you know, the kind of work could be the thing that that saves us.

• 2:09:17Nathan Labenz?

  So I think a lot of things should be scaled up. Probably the thing that I'm like, there's probably enough going on is what the frontier companies are doing, which seems to be trying to get the current model to be aligned enough to like supervise the training of the next model by like, you know, any number of things, data filtering and, you know, RL AIF type techniques, all that kind of stuff. Anything that kind of is in the recursive self-improvement realm. I would be a little less inclined to write a new check for because it seems like that's what the companies are doing. And if anything, I think they're probably going too fast at that relative to all the other things that we could be pushing on. But yeah, but for me, the weirder out, the farther out you get, the weirder you get,

• 2:10:03Nathan Labenz?

  the more I think there's going to be obviously many more misses than hits. But those hits could be really valuable. And, you know, when I see something like self other overlap, I'm like, yes, this feels like something that is just so underdone, has so much potential. And I would love to see more people, you know, of all kinds of idiosyncratic persuasions, trying to develop those kinds of ideas. Okay, last question. Turpentine got acquired by A16Z. You mentioned that you negotiated for editorial independence. What did that negotiation actually look like? How did it go? Again, that's an AI question. And I think that probably came out of chat GPT because, or possibly Claude,

• 2:10:48Nathan Labenz?

  but I think it was chat GPT, because I did use multiple AIs to review the contract as we went through the process. So it was very well aware of that negotiation. And honestly, I have to say, you know, it was a credit to Eric and credit to, I guess, A16Z more broadly. I'm not sure, you know, who all was involved. Eric has been my, you know, main contact person for all of this. It was honestly very smooth and pretty much entirely painless. I did already have via earlier agreement with Eric at an editorial independence clause in my agreement with Turpentine. And so that was, you know, a good starting point. I was a little concerned, honestly, you know, to be totally candid, when the deal was made.

• 2:11:33Nathan Labenz?

  And, you know, Eric called me on a weekend and was like, hey, so I got an update for you. We're going to be joining A16Z. And I was like, oh, that's interesting.

• 2:11:43Nathan Labenz?

  Mark Andreessen blocked me on Twitter a long time ago before I ever even interacted with him on Twitter. I think it was famously, he supposedly blocks people en masse who just like a tweet that he doesn't like. So I was probably mass blocked with many other people, but you know, I sort of said, hey Eric, you know, the dude blocked me on Twitter. I've never even met him or talked to him, but that does, you know, and of course, in the techno optimist manifesto, he did have an enemies list, which is not something I generally think people should be doing. I would, I would not be, I would not recommend publishing enemies lists for almost anyone. So I was like, you know, I'm a little concerned because it's pretty clear to me that some of the things that I think are important and value and want to advance in this world

• 2:12:26Nathan Labenz?

  are on the enemies list for A16Z. I really do want to like, make sure we reaffirm the editorial independence that I did already have codified, but you know, really want to make sure that it was super solidified going forward. And yeah, there was no problem. We worked through a couple of turns on the agreement and it was pretty smooth sailing. And basically I think I requested, you know, something very reasonable, but basically they agreed to it with no, no real substantive pushback. You know, a little tweak wording here and there and that kind of stuff, but like overall it was pretty smooth sailing.

• 2:13:03Nathan Labenz?

  So where it landed is I do have, and you know, explicitly written, contractually agreed upon right on this feed to say whatever I want. That includes criticizing A16Z, criticizing partners, criticizing portfolio companies, disagreeing with their stance on policy questions. Basically I can say whatever I want and it's totally fine for it to contradict their policy positions or you know, to say that some of their investments are dumb or whatever. I can, I can say anything that I want. And that is positively affirmed in the contract. So I give a lot of credit to them for being willing to do that.

• 2:13:44Nathan Labenz?

  And the only kind of pressure release valve or like off ramp that is in the contract is that if at some point, for whatever reason, A16Z decides that they just don't want to be affiliated with me anymore, you know, because I say, or

• 2:14:02Nathan Labenz?

  Advocate for or do something that is just too much at odds with the agenda that they're trying to advance, then they can just release the all of their interest in the intellectual property of the podcast to me. And that would include like the feeds and the logos and that stuff is kind of jointly owned right now. Like I have the domain and, you know, control the website and they have like the YouTube feed and whatever. So we're kind of, you know, mutually dependent at the moment. But the if, you know, they ever wanted to, they could just give me all those things and tell me you're on your own now. And I hope that doesn't happen. And I don't think it's likely to. I certainly won't be afraid

• 2:14:45Nathan Labenz?

  to be bold on this feed if I feel like there's important stuff to talk about. But and I have, you know, actually, it was funny because the first day Eric called me on a Sunday to tell me about that deal. And I had already recorded with Zvi on Friday, two days before at a classic Zvi episode, which was going to come out the next day and did come out the next day. And in that episode, Zvi accused Andreessen of perjury before Congress for having said that basically interpretability was solved. And I was like, Eric, you know, this is coming out tomorrow. And he's like, Yeah, I don't think they're really going to care. I think it'll be fine. And so far, it's all been fine. Obviously, they're they're

• 2:15:29Nathan Labenz?

  big boys and they can they can take some criticism and they've been willing to put that in black and white. And I do hope that over time, as we learn more about the overall shape that AI development is taking, I do really hope that the accelerationists and the AI safety people, that the accelerationists and the AI safety people can realize how much common ground we actually have. I think on an overwhelming number of questions, I'm probably going to agree with a 16Z. There are some where I definitely don't. And

• 2:16:10Nathan Labenz?

  I'm certainly not going to, especially now that I have this contract in place, I'm certainly not going to shy away from that. But I also don't want to pick a fight unnecessarily. I'm certainly not going to, I'm going to try to follow my own advice from 20 minutes ago and not psychologize their takes. I'm going to assume, and I, you know, I try to assume this kind of in general about like powerful people. It's not always the case, but like, I think that they, first of all, that, you know, their riches can be, right? They don't need more money. Why are they doing what they're doing? I genuinely think that they're trying to advance the human condition. I really don't think that multi-billionaire, decabillionaire folks like Marc Andreessen

• 2:16:55Nathan Labenz?

  and Ben Horowitz are making their decisions at this point based on lining their own pockets. I really don't think that's the case. I can't rule it out. I don't know them, but I really don't think that is the case. I think that they are trying to advance the human condition and they are trying to, you know, make sure that we don't go stagnant, that we don't allow fear of change to prevent us from realizing a beautiful future. So I take them at their word. That is what their motivations are. And I share a lot of that, some different opinions on some, I think, pretty important questions, but I, I think that the, there's a lot more alignment than has at times been assumed online.

• 2:17:39Nathan Labenz?

  And I think that's true, not just for me, but a lot of people that are fundamentally interested in AI safety, as I said earlier. I hope that I can be critical from time to time or, you know, at a minimum disagree and potentially even go into criticism without it fundamentally breaking the relationship. But, you know, I do have the assurance that I can continue to do this podcast and, you know, continue to reach you, the audience that has subscribed to the feed. And, you know, if you're again, listening this long into the podcast, you know, I appreciate that and I, I don't take for granted at all that people want to follow me on this learning journey. And I just wanted to make sure that I had the ability to speak freely, speak my mind, you know, say what I think is important without.

• 2:18:23Nathan Labenz?

  Having any fear that I would lose the ability to continue to use the modest platform that we've built, and that is in place. So I feel really good about it. I appreciate Eric and A16Z broadly for making that a pretty smooth and painless process. And it gives me a lot of confidence that I can just keep doing this and just keep calling it how I see it. Again, I'm not going to try to create conflict where it doesn't need to exist, but I will say what I think. But it's a very privileged position to be able to do that and even make part of my living doing it. I definitely don't take that for granted. So, thank you again to Eric and A16Z for making that as smooth as it was.

• 2:19:05Nathan Labenz?

  Thank you to everybody who has listened. Again, I really don't take this opportunity for granted. It's a lot of fun and it's thrilling every day to get to wake up and think about, what do I want to learn today? What feels important? How do I go make sense of this crazy AI wave that we're all riding together? It can be a little scary at times, but I do love the challenge that it presents me. So, thank you all for making that possible. And in closing, thank you for being a part of the Cognitive Revolution. If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions,

• 2:19:47Nathan Labenz?

  and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.