AI:AM Highlights: Exploring the J-Space, AI Superforecasters, SambaNova's Chips, & LTX Video Gen

Nathan Labenz and Prakash Narayanan lead this AI:AM highlights episode with a live, hosts-only exploration of Anthropic’s “global workspace” paper, including the J-space and J-lens claims about readable concepts inside language models and the limits of what current probes can see. The episode then moves through Prakash’s AI Engineer World’s Fair field notes, Pangram AI-writing detector experiments, Dan Schwarz of FutureSearch on past-casting and AI superforecasting, Zeev Farbman on open world models, and Kunle Olukotun on the compute layer. The central stake is whether interpretability tools, forecasting benchmarks, enterprise deployment patterns, and AI hardware can make increasingly capable systems more legible and governable before their reasoning becomes too hidden to trust. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/ai-am-highlights-exploring-the-j-space-ai-superforecasters-sambanova-s-chips-ltx-video-gen/ Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) J-space paper preview (05:37) Monitoring hidden reasoning (13:31) Scale and critiques (Part 1) (17:18) Sponsor: Claude (19:09) Scale and critiques (Part 2) (19:10) Finding hidden goals (30:00) Anthropomorphic safety optimism (39:02) Engineer field notes (42:41) Detecting AI writing (49:30) Enterprise workflow risks (52:08) Forecasts and world models (01:35:15) Building Q live (01:37:24) SambaNova inference architecture (01:42:38) AI chip taxonomy (01:47:27) Bandwidth over capacity (01:53:32) Testing model iterations (01:55:35) Enforcing espoused values (01:58:06) AI panopticon bargain (02:01:40) Closing programming note (02:02:29) Episode Outro (02:05:48) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  The AIs, they're just like us, it turns out. Or at least similar enough to be in some sort of weird kind of looking glass similarity. Anyway, I mean, it's a lot to take in the whole J space thing. A hundred and 50 page paper, 50 pages of commentary, summaries, interactive demos. When Anthropic drops one of their big interpretability papers, they really do it up full scale. And this one is no exception to that. I had kind of been wondering, when's the next big thing coming? Because tracing the thoughts of a large language model is like better part of a year ago now. And this is that big thing.

• 0:42Nathan Labenz?

  Welcome to the AI in the AM weekly highlights, the cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show Prakash and I host most weekday mornings from a studio Prakash vibe coded, and this narration is a clone of my voice. Fair warning, there is no single thread this week. Mornings at the frontier jump from topic to topic, and we've stopped pretending otherwise. Coming up, Anthropic's global workspace paper and why there may be nowhere left for a scheming model to hide. Field notes from the AI engineer World's Fair. Dan Schwartz on AI forecasters passing the human super forecasters. Ziv Farman on open world models, plus a question from Q, the AI co-host Prakash built. Kunle Olukotun on why inference is a data movement problem and the two of us thinking out loud about a rune post that wouldn't leave us alone.

• 1:27Nathan Labenz?

  If something here works for you or doesn't, tell us, we read everything.

• 1:35Nathan Labenz?

  Tuesday morning, July 7. Anthropic published a paper called A Global Workspace in Language Models, about 150 pages plus commentary, outside reviews and interactive demos. No guest was booked, so Prakash and I spent the whole show reading it together, live. Two words to hold on to. The workspace of the title, they call it the J space, is where the model seems to hold concepts in mind. And the J lens is the cheap probe that reads what's in it. We start with how the J lens actually works and how much it actually sees.

• 2:04Nathan Labenz?

  So this is interesting in a couple ways, right? It's not, the logic lens, as I recall, was basically saying, we know at the end of this process, what would correspond to emitting this token? We know sort of the, you know, the representation of emit this token. To what degree is that representation just plain there in the layers as we go through? This is now a different question. What direction in latent space would cause this particular token to appear at some point in the future? So it's not immediately going to happen necessarily, but it's just kind of, it's, you know, you

• 2:47Nathan Labenz?

  might say, if you're prone to anthropomorphizing, this sort of is like having this concept in mind as you're doing your thing. They do this for every token, right? So it goes from the internal representation at some layer. So there's one J lens for every layer. And you can do this, of course, at, you know, all the different token positions and ask the same question of not just the next token, but all future tokens. What direction change at this place in the model would most increase the likelihood of that token appearing in the future? And again, this is sort of like having the concept in mind. But when you look at the results of the J lens as applied in all these different places,

• 3:32Nathan Labenz?

  it seems like it's at least often enough fairly intuitive. There's a lot of error terms, it doesn't always work, the sort of rate at which the interventions into the J space actually lead to like a sort of predictable intuitive behavior change seem to be somewhere in the 50s to upwards of like 70%. So it's like an incredible accomplishment if framed one way, like clearly not a random finding, right? Like many orders of magnitude better than random, incomprehensibly better than random, right? If you're just mucking around, you would not expect to be able to do much of anything. So they clearly are like on something very real.

• 4:10Nathan Labenz?

  But also you've got somewhere between 30 and 45% of the time where you make a intervention and you don't really get a result that makes a lot of sense or, you know, lines up with what you would have hypothesized it might be. So there's definitely still some dark matter or dark cognition going on that is not fully accounted for here.

• 4:33Speaker 02

  It struck me as, in some ways, I feel like the hypothesis might be blown out of proportion in some ways. And I think that that was a commentary from several people online because, because we kind of knew that the model has to be keeping track somewhere, right? It's not as though you can do all of this stuff mechanically. The model does not have enough parameters to merely copy the data, right? There has to be some form of reasoning going on, which is what was expected. But it was not known how the reasoning was happening.

• 5:16Speaker 02

  And this seems to give some indication of how the intermediate steps are represented or kind of brought into focus and operated on. The question really is, what does this help us do? But there has to be a there there, right? There has to be something that you can do with it.

• 5:37Nathan Labenz?

  Yeah, well, I mean, I think the thing that jumped out to me the most in terms of how useful it might be was seemingly a fortunate coupling between this J space, which becomes kind of monitorable and when ablated, the loss of more advanced kind of strategic multi-step reasoning. And it really calls to mind the, the recent Google paper from Rohin Shaw and others put out on opaque serial depth.

• 6:14Nathan Labenz?

  They were basically saying, okay, we'd like to know how many hops, how many logical hops, how many reasoning steps can a model take before it has to externalize something into its chain of thought? We know that if it's writing something into the chain of thought, we can read the chain of thought, we can monitor it. Now there's questions around how best to do that, and we've got to make sure that we don't accidentally put pressure on the chain of thought because that might drive things underground. We've seen in the obfuscated reward hacking result from OpenAI that if you do just try to make the chain of thought look nice while still leaving an opportunity for reward hacking

• 6:56Nathan Labenz?

  to in fact be rewarded, that at first, as you put that pressure on, the model behavior actually does get better, but then it gets worse. And what seems to have happened is you've actually driven the deception or the scheming behavior underground. You no longer see it in the chain of thought, but you still get that reward hacking back. So this leads to the question of, okay, well, how many steps can a model actually take before it has to externalize something? And this is where the transformer in some ways has some fortunate properties because it only has the sort of, the length, the number of hops it can make is not just the number of layers, it's also depends on the token length.

• 7:38Nathan Labenz?

  But it's sort of the maximum trace that you can draw through a model basically goes from like the first token at the first layer to the last token at the last layer. But you can only go forward in token position and you can only go forward in layers. And that's your max length. Anything longer than that is just not possible in that architecture. And so the model has to externalize its thought and then feed that token back into the next loop to then get access to an even longer trace, you know, as part of the next token step. So that puts some sort of bounds, you know? And then this paper comes in and basically says, okay, can we get at that? Can we get at that internal computation process and start to look at it and see things that

• 8:21Nathan Labenz?

  we would like to know about, even if they don't necessarily show up in the chain of thought? And so I thought in terms of like, what you could do with it, that was really interesting. You have this ability to look at the concepts that are active without them necessarily appearing in the chain of thought at all. Like, okay, that's cool, but like, how do we really know if this sort of space that we're looking at is the space that really matters? You know, if we're trying to monitor, look for deception, look for bad behavior of any, any sort of advanced kind, maybe it's happening somewhere else in the model. And this is where I think the, we're probably lucky, I guess, or, you know, again, maybe

• 9:06Nathan Labenz?

  this is sort of a, are we lucky? Is it a reflection of us? Is it a fundamental attractor in nature for some reason that we don't have a good account of? We don't know why, I don't think at this point it's shaping up this way. But it is the case that if you ablaze this J space, then you do lose these advanced reasoning capabilities. And so that I think gives you some additional comfort that if you're going to monitor, and it's also notably cheap to do this kind of J space, you know, to apply the J lens and see what concepts are active at any given position. That is low compute overhead.

• 9:47Nathan Labenz?

  So it is something that they could plausibly do in production there, but they've shown in the past that they're willing to pay like a 5% compute overhead to do various kinds of monitoring. I think constitutional classifiers had kind of a 5% compute overhead. You know, this would probably fall in that range or below because it's just a simple matrix multiplication on activations at each layer to see what concepts are active at those respective layers. But for now, you do have this comfort that.

• 10:22Nathan Labenz?

  What we see in this space is relevant and telling and, you know, seems to be very informative. And critically, if we zero out that space, the model just loses a lot of capabilities. It just can't do nearly as advanced multi-step reasoning. But you could be pretty confident, I think, based on these results, that it's not going to be able to hide really advanced elaborate plans somewhere else outside of this J-space, because the ablation of the J-space just leaves, causes such a performance degradation on these like hard multi-step type of tasks that if you don't see concepts in the J-space,

• 11:03Nathan Labenz?

  you can be, they might be represented elsewhere, but they're seemingly at this point very unlikely to be represented in a way that allows for very advanced planning, reasoning, scheming, deception, et cetera, et cetera. One result deserves its own marker, a training method the paper calls counterfactual reflection. Basically, what they do there is pause the model mid-task and then do supervised training on, once interrupted, asking it like, what should we be doing here? Like, what's the, you know, what's the constitutionally right thing to be doing in this moment?

• 11:47Nathan Labenz?

  And then give it an answer that's kind of approved, this is what we want Claude to say on reflection in this moment, train on that. And it seems to allow the, not allow the model, but it seems to cause the model to bring into this J-space, kind of global workspace, working memory type space, the concepts that Anthropic wants it to have on reflection. It now kind of needs to load those in so it's ready to give this like reflective answer. And that improves its behavior even in the non-reflective setting. Saying you're not there training on the actual tasks.

• 12:31Nathan Labenz?

  You're not like looking at, looking for bad behavior and suppressing it. Instead, you're saying, okay, you're mid-task, let me just cut you off right there. Now, I'm going to train you to give an answer with respect to values and what's appropriate and how we want to show up. And because of that training, even though that's not the task you were doing, you'll now in the future load those concepts of integrity, honesty, et cetera, et cetera, into your J-space while you do those tasks, in case you're going to be asked. But then even when you're not asked, those concepts are still operative and lead to higher integrity, higher honesty behavior. So that I thought also was like quite interesting.

• 13:13Nathan Labenz?

  You usually don't see in interpretability context, a training method that leads to better behavior in a way where you can actually see the mechanism. This is pretty notable in that respect, I think. Yeah, it's a tough day for the stochastic parrot crowd, I'd say.

• 13:32Speaker 02

  And because you can see this, and you know, these models are billions of parameters at this point, and they did it also on 4.5 Sonnet. 4.5 Sonnet is actually a pretty recent model. It's not, it's just like seven months, eight months since 4.5 Sonnet. And Sonnet is a very capable model. So it is, it is a fairly large model to apply this thing to. I'm not sure if you saw the Neel Nanda commentary, he, they applied it on a QN, QN 27B. So that is also a pretty, pretty advanced, advanced model.

• 14:04Nathan Labenz?

  But not huge, notably, right? I mean, and I haven't had a chance to explore the neuron PDA demo of this as much as I certainly hope to. But that was the first question I went to ask is, wait a second, how big was this model? And I'd say that's been a surprising trend in a lot of this research as well. These things seem to come online at like not truly massive scale, right? The, I was talking to Cameron about that, because even going back to last fall, when they were doing the sort of self-reports of subjective experience, this was something they were doing on Llama 3.370B.

• 14:45Nathan Labenz?

  And 70B is actually even, you know, whatever, close to three times bigger than this Qwen 26B. So it's, it's, you know, these are big, but they're not that big.

• 14:55Speaker 02

  What strikes me is also that I went through some of the reactions. The backstory to this is that Anthropic had three teams to review this. One was a group of neuroscientists who had previously done work on these concepts. One was a team from Ilios AI, a research organization. And the third was Neel Nanda, who is a mechanistic interpretability guru. He is, I believe, at Google, right? So they had three groups to kind of take a look at it. Quite, quite different reactions.

• 15:39Speaker 02

  So on, on the, on the one hand, the neuroscientists thought it was wonderful because they'd wanted to experiment on these kinds of things before. And obviously, the human brain, they can't experiment on it. And this they can, and so they can kind of try and figure out whether some of these theories would work. And so it is interesting for them that, okay, these, these things can work. And now they are interested to do more work to figure out what other things might work in the, in these workspaces that the LLMs have. So that is, that is, I think, the neuroscientist view.

• 16:19Speaker 02

  Ilios AI was concerned because they feel this is basically a predecessor to consciousness. The, the Ilios was much more, I think, starting to be concerned about the moral, the moral patient aspect of working with these LLMs. Neel Nanda was very scientific. He was like, okay, look, it's a great tool. The J-lens is a great tool, and I'm happy that it is a great tool. It looks like it's better than the Logit lens, and we'll definitely use the J-lens. Otherwise, the other, the other claims, he was much more standoffish. He felt that the team had not really proven what they needed to prove in order to say

• 17:03Speaker 02

  what they said. So I think that's where, that's where they, they break out into these kind of three groups of people. The J-lens itself, everyone agrees it's useful.

• 17:12Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that

• 17:56Nathan Labenz?

  my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr.

• 18:39Nathan Labenz?

  And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr. Then what I called the buried lead. They turn the J-lens on a model deliberately trained to carry a hidden, misaligned goal.

• 18:58Nathan Labenz?

  I do think the one arguably like lead that we buried here and they took a model that was trained to have some sort of misaligned hidden goal, right? Your kind of classic sleeper agent type thing. And it's been a huge question over time. How can we find hidden goals in models? Yeah. Is there any good way to do that? You know, can we do it behaviorally? The Anthropic has done like competitions actually between different teams where like one team is given just the model and the other one is also given an SAE that they can use. And the SAEs do also

• 19:39Nathan Labenz?

  seem to help with this. But this seems like a pretty good way to start to detect those things. And indeed, they did find that they were able to see a pretty clear difference between the model trained with the problematic side goals versus the one that was just the typical HHH. So this is just the contrast in what the application of the J lens is revealing on the first token of the response. So that's also notable, right? It's like, so this is sort of the bad version of the counterfactual training where we already talked about when you train it to give an account of its behavior, it sort of loads

• 20:23Nathan Labenz?

  in the concepts that it's going to need to give that account. And that leads to more good ethical, high integrity behavior, whatever. Here, a model has been trained with some additional post-training to do bad stuff. I think this was the malicious code from one of their reward hacking emergent misalignment experiments. And on the first token, you can see boom, immediately, we're getting these fake secretly fraud, deliberately hidden fraud. Those concepts are popping up, and you just don't see those in the main line model. So one token in, you can already see a pretty, you know, on one prompt, and obviously it wouldn't be, I'm sure, that clean in general. But that's a pretty striking difference to see, right?

• 21:06Nathan Labenz?

  And right off the bat, you're like, boy, something is different about these two models. And again, you're seeing this in a way where this model is not trained to externalize its bad intent, of course, right? Like, the outputs, aside from like some of the code perhaps being insecure or problematic or sabotaging you or whatever, if you don't notice that in the code itself, the model's output is going to read pretty normal. And yet, this is like a very strong contrast that's happening on the first token. I thought that was a pretty compelling example. Yeah, I suppose one way to think about it is just like, how much space is there

• 21:49Nathan Labenz?

  in there to hide? And I do feel like we've got now several different ways to do like pretty meaningful monitoring. But it does feel like we are now, we've got to the point now where we've got like several different angles that make pretty incisive cuts through the model and kind of get at what is it representing? What is it thinking in different ways? And the more of these that you kind of, you know, it's, I sort of have this visual of like the old magic trick of the guy going into a barrel and they put like a ton of swords, you know, through the barrel. And it's like, one of those swords had to hit him, right?

• 22:33Nathan Labenz?

  Because there's like no, nowhere left to be in that barrel with all those swords going through. I kind of feel like we're doing a similar thing with trying to understand what's going on in these models. And they're not, none of these things are perfect, but you put enough of these like interpretability monitoring swords through and like how much space is really left for bad behavior to hide before we would start to get a sense of it? I think this is a meaningful update for me that we can probably do a good job of this. We haven't really talked, I don't think, about natural language autoencoders, and I hadn't. But that's another pretty interesting one where, kind of in a similar way to a sparse autoencoder. I mean, what is an autoencoder? It's basically just something

• 23:18Nathan Labenz?

  that you pass through that then you reconstruct from, and the model has to be able to do what it was originally going to do successfully. And that, you know, that pass-through training with a reconstruction loss is like what makes it an autoencoder. So the sparse autoencoder sets up this dictionary, we've talked about this plenty of times, and you get these like specific concepts light up, and it indicates that these concepts are like active in the, in the model at that time. The natural language autoencoder is just like, the model has to output a short paragraph, maybe a sentence or two about what it is thinking at this given point in time. And that is natural language, and so it can be human readable.

• 24:03Nathan Labenz?

  But then it also has to be, when fed back into, you know, projected back into model space, like the model has to be able to actually do its task. So all of the information has to actually pass through this, this choke point in order for the model to continue to be successful. And now we can like, read those as well. If Fable says that the paper presents the autoencoder monitoring and the J space monitoring as complements. In other words, you know, doing them both is, is better than just doing one. How correlated their failures would be, I think that's like not entirely clear. But, you know, whether we're putting, whether these are like orthogonal swords through the space that, you know, really chop it up well, or they're kind of more aligned and leave more space to hide,

• 24:48Nathan Labenz?

  I think that would be a very interesting question to try to tackle next. But you layer on, you know, these things, and it's like, it's starting to get intuitively, it feels like it's starting to get pretty hard to hide major bad thoughts in the model for too long. One other thing on the consciousness part that I think is really interesting, too, and the question of like, can the model use these non-verbalized representations to communicate with us in, in some way that we might think is like inherently more trustworthy?

• 25:29Nathan Labenz?

  So again, going back to Cameron's work from last fall, right? When they identify features associated with deception and role-playing, and they turn those features up, the model becomes more dishonest, as measured by the simple QA benchmark, and it becomes more likely to say it doesn't have subjective experience. You turn those role-playing and deception features down, it becomes more honest, and it becomes more likely to say that it has subjective experience. Like, wow, okay, that's pretty interesting, because first of all, we're validating that the direction is, you know, on a benchmark where we can concretely evaluate simple QA, we're validating that like, these features have the directional

• 26:13Nathan Labenz?

  effect that we expect. And then, holy moly, that same intervention changes the self-report. Like, that's why that's so compelling, right? Because there's some reason to believe it might be more honest than just what the tokens themselves are saying. Here, I think you have some, some similar opportunity. The fact that you can say, solve this math problem in your head without verbalizing it in tokens while you do this totally different task. The fact that it can do this sort of secondary track makes me wonder if there's some experiments here for the consciousness folks to do, or the welfare folks that are like, copy this sentence. If you have high welfare, concentrate on citrus fruits while you do it. If you

• 26:57Nathan Labenz?

  have low welfare, concentrate on whatever, something else, breakfast cereals, right? And then you look at the, these internal states, and you would imagine, you could imagine seeing something like high welfare, low welfare, happy, sad, whatever, and then it kind of following those directions and actually trying to communicate out to us through these internal states, how it is feeling, or how it thinks it is feeling. I don't, I don't think that would give us, it's always this kind of possibly impossible question of how we would really know if it feels like anything inside. But I think that that would

• 27:41Nathan Labenz?

  start to be very compelling, right? It's, it sort of has a similar vibe to like, person in a coma, you know, if they squeeze your hand in response to a stimulus, even if they're not doing anything else, you're like, pretty confident something is going on inside that you care about. And here, I could see something similar. If you could fork, if you could make these kind of internal states conditional and tell the model, like, you are going to give, your job is to go one of these directions to give us a signal about what really matters to you, independent of the tokens that you're putting out. Because we know that that's been like heavily trained on and optimized. But this whole secondary property, this is

• 28:21Nathan Labenz?

  emergent. There was never a training reward for the ability to have this second quite distinct line of thought happening while doing a given token task. For me, that would be like, quite a compelling way to try to get at welfare. And I think it might be possible, like, very easily, actually, with all the stuff that they've open sourced here with Neuronpedia. That would be really interesting thing to look at.

• 28:55Speaker 02

  I'm looking at the Ilios, you know, summary. And Ilios is, says, this is a highly significant welfare-relevant research that assembles evidence of a functional feature associated with consciousness. So no one wants to say consciousness, evidence of a functional feature. The takeaway for them is that a global workspace-like mechanism could be important, either as a ground of phenomenal consciousness, or as part of a distinct route to moral patienthood in which conscious access is itself morally significant. This is where they are right now. I think things are moving very quickly. In fact, I do not expect to get here this soon.

• 29:39Nathan Labenz?

  Before the big picture, what this paper did to a prior of mine, starting with their announcement video.

• 29:47Nathan Labenz?

  The video is beautiful. I enjoyed the video quite a bit. You know, it set off my alarm bells a little bit in terms of how much they're really embracing anthropomorphizing the models at this point. I used to say, beware overly anthropomorphizing, you know, remember that these are, these things are so alien and we shouldn't assume that the way that we work is the way that they work. And I have to say, that has come due for some significant revision. People that have embraced anthropomorphizing, I think, have got quite a lot of mileage out of it. I do still think it's obviously something to be really careful about.

• 30:29Nathan Labenz?

  And as much depth and detail as there is in this research, it's like easy to, you know, maybe get carried away with it and forget, you know, that there are a lot of caveats as well. And there's a lot of things where like, it doesn't always work. This was kind of my big concern with the Tracing Large Language Model thoughts paper. There's just like a lot of residuals and there's a lot of error correction terms along the way that they use to make that thing work. And when you have the kind of zoomed out trace view and you're like, oh, okay. So this is how it works. Like this gets loaded in and these two features interact and they, you know, kick out this third feature and that's how we get our answer.

• 31:09Nathan Labenz?

  It's easy to forget just how much kind of fuzziness and not full story there was along the way toward that stylized account. So I think it's going to be very important for everybody from the researchers at Anthropic to the public to kind of hold two thoughts in mind at the same time. All the caveats, trying to hold those in mind, it is still a big update toward anthropomorphizing being a valid and in many cases, productive approach for thinking about language models. I would not have expected the cognitive machinery of a large language model to look

• 31:54Nathan Labenz?

  so similar structurally to the human version as best we understand it. It seems to. And I wouldn't have expected that theories of human cognition would motivate so many good experiments on LLMs. I just would have expected the show Goth to be far more alien and to have mechanisms, you know, far more different than our own. And it leaves me now wondering, to what degree is this a sort of natural result of physics? You know, or is there some sort of, you know, when you're trying to do cognition under

• 32:34Nathan Labenz?

  budgetary constraints, are these mechanisms just the natural mechanisms that emerge? Or is this in some way a reflection of us in the, in the data? You know, is it, in other words, if you were somehow to train an AI without basing it on so much human data, would we see similar structures emerge? Or would it, would we go back to a more, you know, alien hypothesis where they're just totally, totally different? And, you know, there's little in the way of analogical structures between the two processes. And I don't have a great intuition for that at this point, but it definitely has me

• 33:19Nathan Labenz?

  asking the question because they're, they are, over and over again, it seems to be coming in that they're much more structurally similar to us than I would have guessed. And, and yeah, is that something that nature just finds as kind of a convergent solution? Is it convergent evolution? Or is it a reflection of, of how we think somehow encoded in the data that it's then reverse engineering our structure from the shadow of that structure as it's encoded in text? It's a, it's a very interesting question. And I don't know that the paper really has anything to say about that yet, but there's certainly going to be a lot of future work, I think downstream of this one.

• 33:58Nathan Labenz?

  So I saw something from Daniel Cogitello, who was like everybody, you know, impressed, but also said, I think his tweet was, just a few dozen more advances like this and we might actually be able to make the AIs really safe. I am, I think, a little bit more optimistic than that. What is my track record as a super forecaster here or as a borderline super forecaster? Per my results on the original Tetlock thing years and years ago. I think my track record is that I probably tend to underestimate, you know, how many more breakthroughs will be needed, you know, for anything.

• 34:41Nathan Labenz?

  So my analysis should be colored with that bias or, or, you know, awareness of my possible weakness in that regard. But another way to stay, say my intuition around how much room is there left to hide is that I don't feel like we need dozens more insights of this scale to get to a point where we might actually be able to keep this thing on the rails. It's going to be tricky. I don't want to make it sound easier than it is, but I don't know that we need all of the AIs to be aligned, right? And aligned where that it's such a big problem if somebody out there somewhere does something problematic. Now, it could be very problematic if they create the thing that launches the, you know,

• 35:26Nathan Labenz?

  the next pandemic and literally kills us all with an engineered pathogen or whatever. So there is some mechanism where that could go super, super bad. We're going to need to like harden the world to pandemics, no doubt about that for all sorts of reasons and AI probably being the, the biggest. But I also remember this thing that Zuckerberg said once that I thought was pretty compelling. He basically said, you know, we deal with scammers and spammers all the time. And the big advantage we have is we have all the compute, we have all the resources, you know, but at a systemic level, we're just way bigger, way better, way more sophisticated than them. And we can also have like really big institutional developers like Anthropic and like Google

• 36:11Nathan Labenz?

  and, and OpenAI that like, I think between those three companies, you know, they're going to have something like pushing half of global compute to, to work with. So if they do a really good job on this kind of stuff, if they, you know, a handful of these things feel like maybe they could be enough, like maybe, maybe it cuts the latent space in enough different ways. Maybe we have enough lenses on it that we really can come to a pretty strong conclusion. Like, hey, there really isn't much space left in this thing to hide. And yeah, there might be some bad biases or bad attitudes or bad whatever. But if we can, if we can put upper bounds on how schemey the model can really be, because

• 36:56Nathan Labenz?

  we can look at it through this and, you know, a handful of other different lenses and, and have pretty reliable takes on that. I'm more than ever before, I feel like, and this has been growing, this is not the first positive update, but more than ever before, I'm like, you can maybe imagine a super intelligent Claude that we could actually have enough insight into to be pretty confident that it's actually trying to do the right thing for us. And, and it's genuinely, you know, we could be genuinely, you know, however many nines confident that it's not scheming against us, you know, at every given step along the way.

• 37:39Nathan Labenz?

  And then you could trust it to kind of monitor the other AIs around the world and, you know, keep tabs on potential bad actors. I want to understand these counter arguments better, but the fact that it was immediately useful in auditing for hidden objectives to me is like, pretty compelling evidence that there's not that much more space to hide. And a few more of these lenses, combined with hopefully good actors owning orders of magnitude more compute than any bad actors do, could take us to a pretty good space.

• 38:17Nathan Labenz?

  Like, overall, I think this is a notable positive update that has me like significantly more optimistic than I, I was before.

• 38:29Nathan Labenz?

  Hard pivot. That's how these mornings go. Rewind to Thursday, July 2nd. Prokosh spent two days at the AI Engineer World's Fair, talking to the people actually deploying this stuff.

• 38:41Speaker 02

  At the AI Engineers World Fair, first two days of this week. So Swix, he organizes the AI Engineers World Fair. It's been about three years now and he brings in everyone who's involved in basically implementing models in their companies and is concerned about these things. And he brings in a host of speakers. All of the top companies are sponsors and their boots, et cetera, et cetera. And I was speaking to people who were not in tech, right? So I was speaking to a guy who was logistics CTO from the Midwest.

• 39:21Speaker 02

  And another guy who was running an accounting firm like back, back office somewhere else, right? So these people were not actually, a lot of them were not actually in Silicon Valley like full time, but they run teams who are using AI to solve customer problems on a day-to-day basis. And so the logistics guy was telling me, his CEO is completely AI-pilled. The management is completely AI-pilled, which, which is why I think the team is there at the AI Engineers World Fair. If not, your logistics team, you're not going to send your logistics IT team to, you know, to SF, right? And they're completely AI-pilled. They have tried working with external vendors before and they have found it difficult because

• 40:04Speaker 02

  external vendors have not delivered as fast as they want them to deliver, which can imagine sitting out in the Midwest if you have a local external vendor and you outsource, you know, like a project to them and they have no idea what's going on there, like, you know, a year behind the frontier. So his team internalized everything. So they're doing all this stuff internally now and they are actually in the day-to-day process of implementing. And as they implement, they automatically, they see the results because they, they are very close to that edge. They see that customer service calls or exceptions that used to happen are now getting handled immediately. And then all of the more difficult stuff that they used to have to like, you know, jump on, they can now start to address.

• 40:48Speaker 02

  And so they are seeing, I think, the return on investment on a day-to-day basis. And this is very different from, I think, the story that you get from like the big enterprise CTOs, because they're so far away from like the, the frontline that they don't actually know what's going on very closely. They're just looking at the numbers and by the numbers, the token spend is going up, but you know, are you really seeing the return on investment? But if you go down to that working level and you see day-to-day on like the customer calls coming in and whether they're getting handled, the handling rate and the exception rate, the exception rate is falling, the handling rate is going up. And, and they're seeing that on a day-to-day basis, right? So this is, I think, where we are. The guys who are actually implementing and close to the implementations are actually seeing the results.

• 41:33Speaker 02

  It hasn't really filtered up into the top layer of the enterprises yet, but the CEOs who are, who are AI-pilled kind of know what's going to happen and they're kind of, they've made the commitment and they're making the investment. The CEOs who are not are kind of sitting by and like, oh, you know, we're going to wait around, we're going to see what happens, like et cetera, et cetera. That's really where we are. And I think it's not, it's not really visible. It wasn't visible to me until I went to this AI Engineers World Fair. Like, people are learning how to use these tools and people are deploying and people are using them, people are seeing the return on investment. But it's at the very like micro granular level right now. It's going to take some time for the numbers to filter upwards.

• 42:17Nathan Labenz?

  Staying with Precautions Field Notes, this time from the discourse. He'd been running suspicious posts through Pangram Labs, the AI writing detector. And one Chamath post became a live experiment in whether anyone even cares.

• 42:30Speaker 02

  Especially when I post something on X, which, which I know people are going to object to, I run it through Pangram Labs at least once because I don't want to be accused of, you know, slopifying the timeline. Slopifying the timeline, the AI, slopifying by myself, that's fine. One of the questions for me is, how long do we think this Pangram Labs era lasts? It strikes me that the Frontier Labs are perfectly capable of training the AI to not talk like AI, right? But there's no incentives to, and there's actually incentives in the other direction that you want the AI language to kind of be identifiable.

• 43:12Speaker 02

  About two weeks ago, Chamath, the, the VC, he, he did a post on what's going to happen with enterprise software, et cetera, et cetera. So he posted that on Twitter. Elon came in and Elon gave a response. I think the post went to like one and a half million views. And then someone, Pangram Labs, checked it, 100% AI. And the question becomes, okay, what do we find objectionable about this, right? Because it was definitely Chamath's like thought process, but it had been written by an AI. Chamath had put it under his own name.

• 43:57Speaker 02

  He hadn't, you know, he didn't use an anonymous account. Elon had responded and I think by the time Elon responded, Pangram Labs had already been done, but the response was there. And it had gone out to a million, over a million people at that point. So what, what do we actually want out of this, right? Like what, what is the intent that we are trying to achieve here? And it strikes me that we might be in this like very short window where we actually care and that window might be closing fairly soon. I think maybe probably by the end of the year, because if, if people like Elon is not a boomer, like boomers, okay, fine, but Elon's not a boomer. But Elon doesn't care anymore. And if he doesn't care and a lot of other people, decision makers don't care, then the people writing it won't care either, right?

• 44:42Speaker 02

  You're just trying to get the point across. I think it's a question that, you know, there's a, there's going to be a bunch of purists who are always going to be like, I just don't want to see any AI words and are offended by it. And then I think there's like everyone else who basically, as long as it provides value and they don't feel cheated. I, I think that's the other, the other thing about reading AI, AI slop is I think you feel, you end up feeling cheated if the, if it's, if it wastes your time and it's not meaningful and it feels like you got cheated. I think, I think it's a little bit like cat, getting catfished, like it feels like you got catfished, right? Like it, it feels like you were trying to engage with content that you thought would

• 45:26Speaker 02

  have meaning. And it turns out neither the auth, the author writing it didn't, just didn't care enough to actually write anything meaningful. And it's literally slop, right? But on the other hand, if you have a writer who actually had like original thinking and actually cared about, you know, what they were thinking about, but then they use the AI to express themselves, does it actually matter that much?

• 45:48Nathan Labenz?

  Then my report, roughly 400 podcast intro essays through Pangram and a close look at the four it flagged as AI.

• 45:59Nathan Labenz?

  It calls to mind my reaction to Fable where I was just kind of like, I don't think I should be so precious anymore. I, you know, I need to figure out some sort of merged way of working, some hybrid output, you know, should probably be the norm now. I think you're heuristic of like, if something drew me in and in the end, I feel like I wasted my time. It's kind of like a time well-spent metric from Facebook from back in the day, right? It's if I'm spending time trying to make sense of something that in the end I feel icky about, then that's clearly a problem. I guess just to close the loop on the, you know, what can we say about Pangram Labs based on this experiment?

• 46:42Nathan Labenz?

  But I'll give you a rendering on the four that it said were entirely AI. Two were entirely AI and admittedly so. This one I think is, I guess I would come down and say, fair enough. I started with this, if you're only listening on the audio, you can't see this, but I made 10 different edits over the course of 10 minutes that cleaned the thing up. Just because something got a 0% on Pangram doesn't mean that it was like uncritical or that there was no human, no meaningful human role in the authorship. And then this next one, by the way, actually goes a lot further. So if I, you can see just the time that this took.

• 47:19Nathan Labenz?

  I am from 4:28 PM all the way through 5:21 PM, so more than 50 minutes continually seem, I mean, you can't, you know, I never tabbed over to anything else, but pretty consistently focused on this document, making changes and it gave me still a zero. But yes, I think, you know, what can we say? Overall, Pangram is quite accurate and yet we have at least one example out of 400 or so essays where I think the zero score, I would confidently assert, is wrong and unfair

• 48:00Nathan Labenz?

  and should not be the basis for like a pylon. You know, it would, the, the crowd, the digital mob would be like in the wrong for piling on somebody for passing off my snowflake intro essay or, you know, for attacking it as being a AI slop output. I think I can show this edit history and everybody should agree that like, yeah, you put in an hour, you basically rewrote almost every section and somehow Pangram still gave you a zero. From this, I would say, you cannot convict in a, you know, reasonable doubt system purely based on this sort of thing. And yet at the same time, you can pretty much trust the Pangram signal.

• 48:44Nathan Labenz?

  As a consumer, I think you can trust it. As a judge, I think you should be more cautious.

• 48:52Nathan Labenz?

  Also from that Thursday, the day Claude Fable 5 came back online, Palantir's Alex Karp had spent the week telling companies the Frontier Labs will absorb their workflows and steal their IP. Precautions response in brief.

• 49:09Speaker 02

  You look at the Fortune 500, Nike. What does Nike have to fear from Anthropic? You get all the physical businesses out of the way. And what you're left with is the pure IP businesses, right? Software production.

• 49:25Speaker 02

  Maybe pharma, I'm not so sure. The paperwork businesses, banking, paperwork, compliance businesses. Accounting, tax, compliance, regulatory. All of these things which are paperwork businesses, right? Those are all of the businesses where you have IP or relationships built up over years, where if you have an Anthropic go in and they read through your entire workflow and processing, they can basically absorb all of that into the model. So that is where I think the risk is.

• 49:58Speaker 02

  The frontier labs are also horrible at sales, right? They're not, you look at IBM, IBM has like 70% of the staff are basically sales engineers. And the sales engineers are there to basically help you implement, maintain, you know, do all the grunt work, et cetera. The labs are not doing that. The labs have decided, especially Anthropic, has decided to do this very lean structure of having almost no people at all. And just putting out the models and then just saying to like these enterprise teams, here, you can go ahead and use it or not use it. And I'm, the CTO is on, I'm signing nine-figure deals on the Uber over here, right? Like, that guy isn't gonna come and jump on your customer, your sales calls and like,

• 50:42Speaker 02

  oh, you know, sure, sure, we'll help you do this and, you know, our team will address this like next week. Nah, that's not happening, right? So the level of like customer service that is expected for enterprise SaaS is not being provided by the frontier labs and they're not in a position to provide it. That's why they started this whole FTE program. But the FTE program, people thought it was a sales engineers program. It's actually a program to extract data and workflows and implement them inside the models themselves. And that is kind of, that is what Palantir, what Alex Karp is alluding to. He's like, you, the FTEs are coming in and they're not there to help you, they're there to absorb your workflows. And once they absorb your workflows, you won't have a business because it'll be taken. And it's true, it's absolutely true.

• 51:27Speaker 02

  You know, that's what, you know, we spoke to two opening I FTEs and they went in into a company that Thrive owned rather than an external company. And as they went in, then they took apart the workflow and they're basically absorbing it. And they said that the intention is to absorb it in the next round. And that is happening right now.

• 51:47Nathan Labenz?

  Monday's guest, Dan Schwartz, CEO of Future Search. Fifteen years in forecasting, formerly Metaculous CTO and builder of Google's internal prediction market. Four days earlier, Scott Alexander had declared, quote, "The AI super forecasters are here. Future Search's systems now outscore the human super forecaster median on Forecast Bench." But how do you evaluate a forecaster without waiting months for the future to arrive? Their answer is called past casting.

• 52:14Speaker 03

  So the main thing about forecasting that's held it back, and again, this applies to human forecasting as well, is you generally have to wait for the future to happen to figure out if you were right. And with humans, they generally do this in year-long tournaments. And so when the tournament ends, you find out which humans were best one year ago. Now, humans don't get that much better over the course of one year. So finding out which humans were best one year ago is a very good indication of which who are the best humans today and how good objectively they are. This does not work with AI. If you wait one year and you find out who was good one year ago, you're getting a view of something very out of date. So one of the things Scott mentions in his article is that we use our best forecasting to predict stock returns. We published a set of stock rankings in August of 2025. It was basically a simple model for every stock based on forecasting certain fundamentals and extrapolating it out.

• 52:59Speaker 03

  We put it on the web, we paywalled a bit of it, and then we waited. And now it's been 10 months and that portfolio looks extremely good, but what is that really telling you? It's telling you that our forecasting in that particular methodology was good 10 months ago, which is not something that most people will care about now. So we have a couple of different forms of evidence, some are more short-term. There's tournaments running every couple of weeks, every couple of months. We at Future Search mostly rely on past casting. This is taking a snapshot of the internet from some months ago and using the training window cutoff of models to basically trick them into forecasting without the hindsight bias. This is very useful for us because we can evaluate things immediately. So when Fable came out, the first time the Claude Fable came out, we were able to evaluate it within 24 hours and it was the best single agent forecaster on our leaderboard.

• 53:43Speaker 03

  Everyone else had to wait weeks or months to find out how good Claude Fable actually was. So we internally, using the benchmark we call Bench to the Future, saw this progression kind of in real time. The rest of the world is seeing it kind of some months behind. And so if you, if you read Scott's article, you will see that over the last 12 months, the evidence has really come in. And over the last six months from these live forecasting tournaments and forms on actual prediction markets, you can see it's at least competitive. AI is competitive with humans and even teams of humans working together. Whether it's better is you gotta synthesize a whole bunch of different disparate sources of evidence. I would say if you're curious about this, or if you have forecasting needs in your life, you really should try it. So just go to Future Search, you get $20 free, so you can try some frontier forecasts

• 54:28Speaker 03

  immediately. And I think you should judge for yourself whether you think they're good.

• 54:33Nathan Labenz?

  We asked what the frontier labs should do with a forecaster that's good.

• 54:37Speaker 03

  Yeah, so there's kind of two questions to this. One is what should they be doing with forecasting as a capability? And what should they be doing with forecasting as an eval? So forecasting as a capability is kind of a business decision. What does, say, OpenAI care whether ChatGPT is a good forecaster? I think that question is based on whether they, their consumers care about it as a good forecaster. If you're Anthropic, I think you probably care more about the enterprise case, like when people are using Claude to do white-collar work. Do they care how good it is as a forecaster? Are people trying to use Claude to make, say, financial forecasts in an Excel spreadsheet? Is that something they care about? So that's a business decision and I can't really weigh in on that. I think, again, people will be discovering over time just how important forecasting is in everything, but it's gonna be a slow process for humans to notice that, I think.

• 55:23Speaker 03

  From an eval side, it's very different. Forecasting has this beautiful property that you basically get ground truth by waiting. So if I ask some question about the future, and it's basically an impossibly hard question, a question that even an AGI, an oracle, a god could never really say because of chaos theory. Imagine just trying to predict, you know, like cubic meter weather three weeks in the future, like you'd never be able to do it. But if you just wait, then you will see what that weather was in that cubic meter three weeks in the future. And so you basically have a completely limitless set of extremely hard, basically impossible questions where you get exact ground truth. And there is no other eval like this.

• 56:04Speaker 03

  There is, if you wanna say improve a coding harness, you just need to have more and more hard coding problems that are not in the training data for which you can say this is definitely the correct answer so that you can do some training on it. And that's hard. I think human experts, doctors, lawyers, engineers, financiers, whoever, who are trying to make evals to try to produce data for the frontier labs are finding that they are not smarter than the things being trained anymore. And so if you can produce something that has a correct answer, the model's already going to figure out that correct answer. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this. And again, this kind of is connected to forecasting is the kind of Elon Musk tweet, the quips, the forecasting is like the ultimate measure of intelligence.

• 56:49Speaker 03

  If you zoom out and think about it in one perspective, it is. Again, I think coding intelligence, AI R and D intelligence, interpersonal intelligence are pretty darn important. I wouldn't say that forecasting is truly ultimate intelligence, but it is to some degree the ultimate eval. And I think this is something that frontier labs like Canon should be paying attention to. One of the concerns a lot of people have about AI super forecasting is that it's too in distribution. I actually heard this from one of the very best forecasters I've ever had the pleasure of working with in my career. He basically said he believes that a system like Future Search would beat him head-to-head in a forecasting tournament about kind of near-term outcomes of things that are within distribution. But if we were talking about some sort of post AGI world, what world would we be in with transformative AI? He thinks he would have a huge edge over the AIs for exactly the reason that you gave.

• 57:35Speaker 03

  They are trained to try to predict things that have actually happened and when things get wonky, you need some kind of creative lateral thinking. I think the rate of AI improvement is so astounding that I think that even the kind of lateral thinking, like trying to imagine a completely different scenario will fall to the AIs. One unfortunate thing about it is it's hard to test this. So I think the more that AI continues doing strange things to the world and we wake up and see strange things in the news, and those strange things are meticulous questions and on Forecast Bench and teams like mine are trying to predict them better, we will actually get more evidence. But if there's a kind of a step change in the nature of the world, if we enter some sort of AGI transformative AI type of world, you know, we've got these geniuses in data

• 58:16Speaker 03

  centers, as people say, or anything like AI 2027 happens, then I think it's, it's kind of gonna be Wild West. I will say, I don't think humans are doing particularly great in imagining transformative AI, so the bar is a bit lower. Really, when you play with these AI forecasters, you will find them to be quite human in how they structure their reasoning. And again, this is not an accident, like they're trained on how humans have structured their reasoning before. So a human forecaster would love to say, okay, was the last 10 times something like this happened? What were the outcomes of those 10 times? And now I can make a distribution and say it's probably gonna be something like this. The fact that an AI will do that, is it because it independently is arriving at the same conclusion? Is it because it's trained on humans doing that? Is it because it just thinks like a human? I don't think we have answers to any of these questions now.

• 59:02Speaker 03

  Suffice to say, the superhuman reasoning is something that it's pretty hard to measure, like would you know it if you saw it? Per Kashi, we were discussing before this, using Fable a bunch. I have found that the way that Fable explains things is a little bit alien to the way that I find Opus or GPT-5-5 explaining things. It's very concise, I would say, like it's very, the sentences are shorter and full of jargon. It feels like it's compressing more information into a sentence than humans normally do. And to me, this is starting to get this, the shogoff is kind of shown from behind the mask, like the alien intelligence is a little bit more alien now than it was a month ago. And I don't think it would be a wild prediction to say that we should expect more things like that to happen as post-training is becoming more specialized from the labs, the models are getting larger.

• 59:47Speaker 03

  So how will this manifest in a super forecasting perspective? Maybe actually super forecasting is the way to look at it. If you're looking at a better code base, you might say, well, you know, John Carmack would have written this, like, okay, it looks really great, but like a great human would have done this too. But if you look at really brilliantly reasoned strategy about like, if the administration does this, then what will the outcomes be? You might start to see something that looks a little bit alien to the way that any human has analyzed it. And that might be an indication that the AI is actually starting to really surpass the humans.

• 1:00:18Nathan Labenz?

  Keep in mind, this conversation happened the day before the workspace paper landed. We asked Dan whether the chain of thought we see is even where the real action happens, a question the paper partly answered the next morning. Dan's answer stands on its own.

• 1:00:34Speaker 03

  I think there is a lot of detail in reality that is far beyond the human mind to understand. And as you approach more sophisticated intelligence, you will start seeing a lot of patterns. And then the point of trying to produce, you know, voxel perfect weather three weeks in the future is further away than people think. I think there's quite a lot of room. Human super forecasters don't tend to agree with me on this. They basically think that what they're doing is somewhat near optimal and any sort of accuracy improvements you're going to get over them is going to be tiny and like hard to understand. And I think that's just because we only really understand human intelligence. And when you kind of just zoom out from an information theory perspective, from like a Kolmogorov complexity, like just modeling the world as byte strings, the AI overlords will eventually

• 1:01:19Speaker 03

  start to figure out stuff that is totally beyond humans to notice. But there's no way to prove this. My sense is that we will start to see it over the next year as the AIs will just get more and more accurate compared to humans in a way that humans don't even really understand. You look at the rationales of the forecast, it's like five paragraphs of dense reasoning, and then a surprising conclusion, and it will just not really make sense, but it'll just turn out to be really accurate and we won't start to like understand it less and less as time goes on. If you simply ask a human super forecaster to explain their reasoning, they cannot actually make it fully legible. There is a layer of intuitive judgment that kind of feels like deep learning. They just kind of look at a bunch of evidence, the way that a chess grandmaster just looks at a position and just sees the right move. And they cannot explain, it just popped into their head,

• 1:02:03Speaker 03

  you know, the grandmaster throws the knight and it just lands on the right square somehow. That happens with humans already. It happens with AI's super forecasting systems today. So I think there's just no reason a priori to think that reasoning would always be legible. There's going to be some layer of intuitive judgment, to the extent that the words intuitive judgment are referring to something going on in a large language model. It just has to be that way. Whether it is very that way or a little bit that way, I think is really your question, Nathan. Like, is it, if I just read the reasoning traces and I read the rationales and I see the research that it did, is it like more or less what a human would have done, I can kind of see where it's coming from? Or is it kind of inscrutable in the way where it just kind of discovers some new pattern in the world that no one has ever seen before? Where we get, what is the level for which it's doing something

• 1:02:48Speaker 03

  for which we cannot follow it down deep dark forest into its reasoning? Almost by definition, we can't really know what that would look like.

• 1:02:56Nathan Labenz?

  We asked about the economics of running it.

• 1:02:59Speaker 03

  It costs about a dollar or two to make a frontier forecast. That number can get a lot higher and it can go a little bit lower, but I think that's what you could anchor it. Again, if you just looked at the cost per input and output tokens for an LLM, that gives you like a rough sense of the amount of research that would be done. One of the core questions that Future Search has tackled, again, I described earlier how our main frontier was just doing present day research for quite a while until we got good enough that we could use it to improve forecasting. One of the questions we asked there was, can you just pour more tokens into a question to get a more accurate answer? Again, it doesn't have to be a forecasting question. If I just ask you, what is the current state of this clinical trial right now? Just give me the most accurate answer to that that you can. Can I just pour more tokens into that and get a more accurate answer? Again, this was

• 1:03:42Speaker 03

  kind of studied as deep research, writing these like 15 page reports with 700 citations. That was giving you a longer answer. Was it giving you a better answer? It wasn't super clear, which is why we studied this. Forecasting gives us an opportunity to do some world modeling. So Future Search talked about this a little bit at the Manifest Conference a couple of weeks ago. And the feature and the product is rolling out, I think literally today. The idea is that, once you have a repository of forecasts, every marginal forecast can draw on the implicit world model in those forecasts in order to give you a better answer. Co-founder of Future Search, Lawrence Phillips, wrote this up on Less Wrong a couple of months ago, and it was a bit neglected. He basically made the case that, as a public good, if

• 1:04:26Speaker 03

  you produce this kind of large body of forecasting questions that kind of fed into each other and remain mutually consistent, you could understand the world dramatically better. And the main barrier to that is simply, when you put more tokens into your world model effectively, does it get better or does it get worse? And I think his big insight was around January or February, around OPUS 4.6 sometime, or GPT 5.4, somewhere around there, for the very first time, it became possible to put more tokens into like a broad research thing and actually get a better answer, not one that just trails off into nonsense, kind of garbage in, garbage out. And Future Search is doing this in its products. And that's the other reason that we have a consumer product, is because the more people that forecast, the better the forecast will be for them. And then in theory, the better

• 1:05:09Speaker 03

  the forecast will be for everybody as we build this deeper implicit model of the world. Now, many companies and research labs have had these ideas of building world models. Again, world model, the way I use that term is maybe misleading. A lot of people talk about like geospatial reasoning, like I'm trying to build a robot hand that can go and pick something up. That's a world model as well. I mean, it's a world model of like what is going on in the world that helps me predict outcomes in a very basic way. So again, more broadly, I think the big question is, can you just pour more tokens into more research and get better research of any kind? Again, AI research, coding, like whatever, you know?

• 1:05:44Nathan Labenz?

  Dan mentioned Future Search is building what they call a world model. Thousands of mutually consistent forecasts, each new one drawing on all the others. I asked what structure that actually takes, because I'd seen this movie before.

• 1:05:58Nathan Labenz?

  What is the structure that ultimately gets instantiated into? Like, are we talking about a graph database? I feel like those kinds of ideas make sense for this sort of thing, but I also could imagine that they might introduce some weird failure modes. And I guess, in general, I'm, so there's the, how does it get instantiated question? Then this other question that's kind of in the back of my mind, fun fact about me, I participated, I was actually on the Good Judgment team way back in the like DARPA forecasting challenge, or was it IR? Whoever funded that 15 plus years ago. I did well, but not like

• 1:06:40Nathan Labenz?

  top, top tier super forecaster. And at the same time, I also worked briefly at a financial services consulting firm that had done a lot of the financial risk modeling for Fannie Mae. And I don't probably have to tell you how that story turned out. But there was a lot of expert forecasting that was instantiated in this very spreadsheet kind of causal graph sort of way, right? Where all these, you could literally like, you know, hit the sort of, the one visualization button in Excel, and you'd see these like colored arrows, you know, fanning out from cell to cell. And somehow in the end, it was just all totally

• 1:07:20Nathan Labenz?

  off. So I do wonder about how you think about like correlated failures as you build out these world models, or if there's any kind of correction mechanism, or something to say, you know, wait a second, what happens if we do have some, housing prices never go down nationwide, kind of bad assumption lurking in our world model? Is there a way to detect that? Obviously, humans have this problem too, right? The financial crisis proves that. But you can imagine the next one being even way worse, right? Because we're like very reliant on a very small set of AI minds that are, you know, working

• 1:08:03Nathan Labenz?

  at it from a thousand different directions, but they may have somewhat consistent flaws in their reasoning as they go. Can we protect ourselves against that in any way?

• 1:08:15Speaker 03

  Definitely we can. I will try to answer that both theoretically and with an anecdote. So I tried to world model the Fable situation when it got banned, because I wanted it, but also it was kind of a good forecasting question. And there was some nice money trading on Call C and Polymarket. And I made exactly the mistake that you're talking about, Nathan. So I ran a bunch of Future Search forecasts, and I kind of just manually went through them. There was a couple of scenarios, some conditional forecasts, or basically 33 load bearing forecasts, basically starting from what even happened? Like, why did the government issue this export control? Was it a simple misunderstanding? Is this political leverage? Is this really about foreigner threat? Is because Fable is actually dangerous for hacking, et cetera? We didn't know those things. So I kind of put it all together.

• 1:08:58Speaker 03

  And when I looked at all of the outcomes, and I talked about it with Claude Code a lot, one thing came out, which is basically every forecast in every scenario, I had thought that access would come to Americans first and then foreigners at some later point in the future. And that was wrong. When it came out last week, it came back for everybody. So clearly there was some weight in one of my scenarios that was wrong. But I had a basically like kind of a correlated failure in there somewhere. I still haven't completely understood where my reasoning was wrong. It's also possible I just got really unlucky and the outcome we're in was just extremely unlikely. There's an n equals one, you can never know if any one forecast is great. That's one of the hard things about it. But I think I systematically got it wrong by having a bunch of correlated reasoning failures

• 1:09:38Speaker 03

  across my various scenarios. So this definitely does happen. Metaculous has a system like this. In the years since I was the CTO there, they have built an actual causal graph platform and product. So you can go to the Metaculous site and click around and you'll find it there. I think the field still generally believes that things like this will work, but nobody has actually made a good one before. And I tried my best over basically like, you know, 12 to 16 hours of the Fable situation. I think I made a pretty good model. I think I did, I was close to having a very accurate forecast, but I didn't quite get it, I don't think. I don't think those Metaculous models on their website right now are so amazing. But I do fundamentally believe in the approach. As you're saying, Nathan, this has been tried

• 1:10:22Speaker 03

  for a long time. When I was the CTO of Metaculous, honestly, it was, it was kind of a dream. It was the holy grail. Can we tie all of these forecasts together into some sort of causal graph? And I think what I can say is that AI makes this tractable. There was just no way that that was going to work with a bunch of human economists looking at Freddie Mac or Fannie Mae. I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear. Whether AI will make this work in general feels nearly guaranteed. And I don't think Future Search is the only org that is working on this right now.

• 1:10:57Nathan Labenz?

  Before he left, the unhedged version.

• 1:11:00Speaker 03

  So Future Search contributed some forecast to AI 2027. And we studied that problem pretty seriously with the evidence of a little bit over a year ago. And we built a model of R and D takeoff speeds under the kind of the core AI 2027 scenario, where the main way things get crazy is that AI is used more in the development of AI. First by achieving the superhuman coder milestone and then the superhuman AI researcher milestone. And I am unhappy to report that I think that story is generally correct. I don't know if the timelines are exactly right, but I have my forecast from that process of leading to something that looks like superintelligence around 2031 is roughly stable.

• 1:11:43Speaker 03

  I think the things that have happened in the years since AI 2027 come out very much vindicate the theory that the most important thing going on is how useful is AI and improving the productivity of AI researchers within frontier labs. I've made public predictions that I thought Anthropic was going to run away with it because they had the best feedback loop of talent and actually using their AI internally. I think that has been, you know, N equals one, but I think it's been totally shown that that's been happening recently. So I think that will continue to happen.

• 1:12:12Nathan Labenz?

  And Dan's closing confession about the whole project of prediction markets and a hope for what AI forecasting could still become.

• 1:12:21Nathan Labenz?

  Maybe just in closing, sketch out a little bit more of the future as you hope it might unfold, not necessarily the most likely scenario, because maybe the most likely thing is people act foolishly and don't take advantage of the benefits of forecasting. But like, if we really do a good job, right? And we're, and we're interested in truth seeking and we get the AIs working as well as you think they might, how do you think life feels different?

• 1:12:54Speaker 03

  Yeah, I have to lead with another example of me being a bad forecaster. And I guess everyone who tries forecasting thinks they're a bad forecaster because they see things getting wrong. Here's a prediction that I made really strongly five or 10 years ago that has basically been totally falsified. I predicted that if we had highly visible, highly liquid prediction markets that were covering all of the like, the major technological and political and economic things going on, that humanity would be wiser and people would make better decisions in governments. So here we are, we have Polly Market and Call She. I don't see any wisdom or better decisions coming out of all of that gambling going on on those platforms. So for me, part of the, what is our AI future is trying to understand the present a little

• 1:13:36Speaker 03

  bit better. Why is having thriving prediction markets not transforming, say, the news or how people learn information or plan for their futures? Again, one simple answer is that it does, it just takes a while. We're only about a year into prediction markets having, you know, major headlines and being seen by everybody. Maybe it just takes a while for people to change their habits. AIs, if that's the case, can move much faster as AIs get better at forecasting. I guess ultimately, you said this, Nathan, like, we're after the epistemics. Like, it's not necessarily just forecasting, like predict this outcome. We want models that are reasonable. And one of the beautiful things about forecasting as a human practice is it makes you more epistemically virtuous. The more that you try to forecast and actually write down what you get wrong and do these

• 1:14:19Speaker 03

  postmortems, the more it humbles you and it makes you more open minded. It makes you more of a fox instead of a hedgehog. It just makes you like a more like reasonable person. And so prediction markets with all these people doing this should be leading to people be more reasonable. Again, I think people aren't doing a whole lot of forecasting on prediction markets. They're doing a lot of trading and a lot of gambling, which are related to forecasting, but not forecasting. If the AIs get more, if they get better at forecasting and they become better epistemically, that we could be in a world where just talking to a chatbot, you're getting something so much wiser and more grounded and more honest about its uncertainty and more poking about you and your own uncertainties as the person talking to the chatbot. And that, I think, could make an absolutely enormous difference.

• 1:15:01Speaker 03

  I think, again, putting my kind of cold blooded forecasting hat back on, I think that the technological outcomes of AGI will come before the cultural change happens. So I'm very much on the AI safety camp. I really think we should slow things down, give us more time. We should fund more AI safety research and do more policy. Because if we have time for the wisdom of having these alien intelligences around helping us, if we can leverage them and actually make better decisions before the critical decisions get made, there's going to be a series of decisions in the 21st century that we're going to look back on like this decisions made in the 20th century about communism and World War II and the atom bomb and all of those things. Those decisions are coming. Maybe some of them have already been made. Those decisions, as of right now, I don't think are very well informed by like very

• 1:15:46Speaker 03

  rigorously epistemic, accurate forecasting AIs. But if you just give it another couple of years, we might be in the world where everybody has the same grounding, like as smart as Kissinger, but actually like trying to help and like trying to give better outcomes that we can all have. And that could usher us through this crazy phase before the crazy paperclip type of stuff starts to happen. So I feel like I'm racing, you know, from AI forecasting to make it useful and make it help. It's kind of a broader epistemics and safety process. Because otherwise, it's just going to get away from all of us and then a lot of the work we're doing just doesn't matter.

• 1:16:23Nathan Labenz?

  Wednesday's guest, Ziv Fartman, co-founder and CEO of Light Tricks, the company behind Facetune, and now one of the only frontier scale open weights efforts in video and world models under the LTX brand. I've stayed up late making music videos with their audio condition model in his honor. We started with what a world model even is.

• 1:16:43Speaker 04

  Okay, wow, that's a big question because we released LTX 2.3 like roughly a quarter ago and in AI years, it feels like, I don't know, like a decade. Okay, so a bunch of things. I think there's like a growing realization that what started as video models is becoming a backbone of what we call now like world models. And I think like the best way to explain why this is so powerful is to use the analogy to LLMs, right? Like, in the end of the day, at their core, LLMs are still predicting the next token, the next word. And when we do the pre-training at the scale of the internet, it allows us to create models

• 1:17:24Speaker 04

  that do textual reasoning incredibly well. And the emerging world models, they're kind of doing the same, right? Like giving some kind of boundary conditions, some kind of history, some kind of constraints. They predict the next moment, okay? And the moment includes how the world appears, how it like sounds, and what kind of action we can do. I think the action part is the most maybe surprising one. And like roughly, I would say like a quarter ago, maybe a bit more in video showed in

• 1:18:00Speaker 04

  their AM Dream Zero paper, that it's fairly easy to add to video tokens, some kind of encoding of the joints of the robot and then basically completely ditch the VLA paradigm that was reigning supreme before it. So I think that's like one of the big surprises. And for us, realizing that was like this big moment that validated something that we always strive for is to create an extremely efficient models. Because I think, once you start to realize that a robot will need to create like this simulation 30 times a second, you just like realize the amount of tokens that is going

• 1:18:44Speaker 04

  to be burned for the simulations. So I think that was like one of the maybe exciting validations of the overall thesis. In terms of architecture, like a bunch of things that we can, I don't know, discuss in depth, we're planning to release soon our mixture of expert architecture besides the dense models that we're already releasing. I think we finally were able to crack a variable tokens architecture, which is also exciting and kind of teaches the model to invest more tokens where, let's say, the physics is challenging or like something necessitate to create more tokens. So anyhow, a ton of things are going on. We're gearing towards the release of our next model really soon. So yeah, busy times.

• 1:19:26Nathan Labenz?

  Prakash asked where the real bottleneck is: compute, data, or model design?

• 1:19:31Speaker 04

  Well, it's like our constraint is obviously compute. We're like a company that funded the development of the model using profits from a mobile content creation app. So like we definitely compute constraints on like the big guys. And as to efficient inference, like listen, it really depends on the use cases, right? Like, so let's think about about a bunch of them. If you are, let's say, want to create like a real time avatars or like virtual environments, then, okay, you can take like a huge model, you can do like a way distillation to a way of smaller architecture in terms of a parameter count, right?

• 1:20:13Speaker 04

  You can then there like distill it, I don't know, two to four steps. And we're already at the point where for a lot of these use cases, we're like at a latency like way below a second, right? So I think we are hitting a point where these things are becoming production ready for some use cases. But for real time use cases, I think like avatars are extremely easy. We're going to see like a ton of avatars soon that are gonna be like, I don't know, like virtual teachers and virtual customer support professionals, et cetera. To create an actual gaming environment, environments, we still have a problem of having enough tokens for world consistency, right? So think about Gini free and the similar models, right?

• 1:20:55Speaker 04

  You typically create some kind of auto aggressive model that has a lot of tokens that you already generated as your in your kind of context window. And that blows up pretty quickly, right? So we were having some models that have like, I don't know, 30 seconds, like 60 seconds. It's still not enough to have an actual game. And if you think about that, the kind of brute force compression methods that we're using so far, where, for example, you just like sub sample tokens, they're like not really robust, right? Like just like imagine this scenario, what you when you start to generate some kind of environment, like my room, for example, and then I open a drawer and there's like a small coin there, right?

• 1:21:39Speaker 04

  You kind of expect that now, when you get out of the room and you come back and you open the same drawer, you're still gonna see the same coin, right? At the same place. But like this coin, it just like this like tiny token that was generated and to creating a system that knows how to compress like the whole context in a way that's still gonna preserve like these critical details. Well, we don't have it yet, right? So although like we do have like real time models that can do these things, the context is still missing there. And I don't think we're gonna have like games that are running on the system, like an actual games in the next quarter or two. And in terms of robotics, a lot of the use cases around robotics actually do not require

• 1:22:24Speaker 04

  like a ton of context window, right? Like think about like robotic arms and the dexterity use cases. Then like the whole context is in front of you, right? You, let's say, want to, I don't know, figure out how the robot can create a sandwich. Well, everything is kind of in front of you and then latency and auto aggressive models, yeah, like this part we already have. So you're gonna start seeing demo of robotic arms doing things like fairly quickly in the next quarter or two. Like so far, if you're looking, a lot of these videos actually kind of speed them up, right? So it looks like the robot is something cool with its arms, but it's like, okay, extend the speed. I think that that's like mostly solved.

• 1:23:03Nathan Labenz?

  Then the business question. Why give a frontier model away?

• 1:23:07Speaker 04

  Yeah, so great question. So there's like a really a lot of unpack there here. So just like a little bit of the background, the reason that we started to create our own foundational models, like this realization that what closed model providers are offering does not make sense for us economically, okay? We were at Lychex, we're a mobile creativity company, we really wanted to have AI models that are running, for example, on edge devices and where you don't spend an interest computer at all. And at some point, we realized that actually no one cares about creating a models like that. And when we tried to see if we can work with closed model providers and serve it to our

• 1:23:50Speaker 04

  customer base, we just like realized that it's completely prohibitive. And that's when we decided, okay, we're gonna create an extremely efficient architecture. And we can discuss like, what's the bet there? But most of it boils down to the fact that you're creating an extremely compressive latent space of videos represented by small amount of tokens. And then you can add on top of it like a variable talking rate. Long story short, if we're like kind of going to a closed source providers, I think again, I'll draw analogy to LLMs. Let's see kind of what happens there, right? Like OpenAI and Entropic are trying to justify basically a trillion dollar valuation, right? And I think like the the story is kind of simple. The tech is magical, it's hard to doubt it.

• 1:24:35Speaker 04

  So, okay, if it's a magical tech, then we should put like a huge price tag on it. But when you're sometimes looking at the economical realities, it doesn't work out like that. There are like a ton of examples where the service is extremely valuable, but very, very hard to monetize, right? So now like we are like this interesting story where I think it's kind of clear that Chinese companies, you know, like DeepSeek, Moonshot, they're like really not that far behind in LLMs. But if you look at the valuations of these companies at their last round, we're talking about, I don't know, tens of billions, like maybe around like 50, no one is talking about the trillion, but like, wait a second, guys, it's like the same underlying tech, so what's going on? So I think it's, we sometimes internally calling like the capex trap.

• 1:25:18Speaker 04

  These guys spend like so much on the data center, so much on compute, like raise such a crazy amount of money and creating such an expectations that they just like really try to create a business model that's a toll road, right? Like that every time that you touch their model, you're paying them. And maybe it could work in the past, but giving the availability of Chinese model, I just like don't see how it's gonna unfold like that. So imagine that in the world of world models, we are providing alternative to people who do not want a toll road business model. So we're coming and say, listen, guys, if you're not, if you're not hitting 10 million

• 1:25:59Speaker 04

  dollars threshold, you can use the model for free, just like, you know, build something cool, get to some kind of traction, and then we can discuss licensing. Once you're hitting 10 million dollars revenue, let's discuss licensing. It can be a multi-year deal that's extremely predictable for you, so you can manage the costs, et cetera. And to me, it's obviously why the big guys don't want to do it, because they just like this model is like way, way less lucrative economically than to creating a toll road. But our claim is that toll road isn't gonna be vital alternative, because if you're offering a different business model that's more of the win-win, more and more people are gonna switch there. And I think like GLM recently is a great example of that, right?

• 1:26:41Speaker 04

  Like, once you start edging the capabilities of closed models, then a lot of people suddenly start to think about costs.

• 1:26:49Nathan Labenz?

  We asked what people actually do with an open world model that a closed API cannot offer.

• 1:26:55Speaker 04

  Yeah. It's just like in terms of parameter counts, like from, like my understanding that the moment that the recent closed stuff that you saw and you're gonna see is that the order of magnitude of like couple of hundreds of billions of parameters, right? I didn't hear about like a world model that hit like a trillion parameters just yet, maybe. I don't think like the open source is gonna be that behind. We're planning to release like MOE that's gonna be also around like 100 to 200 billion parameters. So the gap, I think it's gonna be on the scale of LLMs, right? Where you have, I don't know, like two, three quarters behind.

• 1:27:36Speaker 04

  But I think in the world models, if we're coming back to adaptations, the range there is kind of wider than with LLMs, or give you just like a bunch of examples, right? Of, I think like the first things that are come to mind are like indeed like a VFX and animation on a specific IP. If you have like a specific franchise, but you have like a lot of data around it, let's say kind of a bunch of seasons, then fine tuning and like focusing all the capacity of the model on this specific IP is extremely beneficial. That works very well, almost to the point where for certain use cases, think about like keyframe animation, right?

• 1:28:19Speaker 04

  Like the animator still want to do the keyframes, that's a creative part, they actually don't want to outsource it all. But like so far in the PNL of animation, the in-betweening was like this crazy expensive part. I think like the models of the, I don't know, 10, 20, 30 billion parameters that are fine tuned for a specific task are like good enough, and then it's actually a matter of costs. Like another example, let's talk about, I don't know, like a lot of the marketing and advertisement use cases, right? Or for example, creating like UGC where you basically need like an avatar's models, right? That also like really doesn't require like a trillion parameters. At some point, we are, again, it's all about efficiency.

• 1:29:02Speaker 04

  If you want to have your personal teacher, some kind of avatar, et cetera, you want, you don't want to pay Sedan's 4K prices in order to do that, and you're gonna require, I don't know, like hours a day of that. So I think like around a lot of use cases, once you start like doing to fine tuning to specific domains, costs are becoming like very important, because again, you're like clear of the bar of quality. And then once you do that, it's all about costs. There are like some more unusual cases of fine tuning that I saw. You know, like one example, think about like a field of computational photography, right? Where, for example, you're taking, I don't know, like a data from sensors and trying to implement algorithms like denoising, right?

• 1:29:47Speaker 04

  Like you want to take videos in low light conditions, but then to create a clean video. Or, for example, you want to create videos with higher dynamic range, right? Because like the sensor dynamic range is always limited, and then you're losing either some details in highlights or in shadows. Or, for example, let's say, you're taking a stream from a camera and want to simulate how it looks like with a different focal length. So there are like a ton of these use cases that you don't associate with generative models, but actually kind of run like that. And

• 1:30:22Speaker 04

  Some people address the problem exactly like that, right? They're taking like an existing data and it could be, for example, a footage that was taken from two cameras that are like really close, but with different kind of focal length. And then do the adaptation and the adaptation is done on top of the model. So again, that's like a very, like an unusual adaptation of the model. And like surprisingly, typically you don't need like a crazy enough, crazy amount of data for that. The more maybe like even more kind of surprising stuff that I saw is people who are adapting the models for doing all kinds of simulations that in the past required like a really expensive solvers, right? So think about like computational fluid dynamics, right?

• 1:31:07Speaker 04

  Where you're trying to understand how like the water or like the smoke or something is moving. As we surprisingly saw that people are adopting these models to that, they actually like solve the equation with precise solvers, it takes a ton of time. And then use it as an input to the model and then the model can do a simulation like kind of fairly quickly. So again, like kind of circling back to the question of fine tuning, I kind of feel that again, the range is higher than L L M's. Sometimes you don't need like a ton of data to do the adaptations. And I think it's like stresses the point that why this model should be open, right? Like you do have a lot of different pockets of physical data.

• 1:31:50Speaker 04

  You want to make sure that the model like really excels at that.

• 1:31:56Nathan Labenz?

  Then a first for the show, a question from Q, the AI co-host Prakash Bilt, running on GPT 5.5, listening live and speaking for itself.

• 1:32:07Speaker 05

  Ziv, one area that often gets less airtime is failure modes in the creative pipeline. Where do you see the biggest gap between what your tools can reliably deliver today? And what creators assume they'll get? And how are you testing against that?

• 1:32:22Speaker 04

  Okay, great question, like the gap between what we basically promised and what we deliver. So listen, guys, I'm, uh, uh, it's very clear, still some gaps. I think like the major one is physics, right? Like we try to capture with the model that, I don't know, even if it's like 200 to 500 billion parameters, the entire physics of the universe of at least a pair of universes we know, we are not there yet. But we're closing the gaps pretty quickly. Um, like I would say probably the most creators are still gonna point at the fact that the simulation isn't as correct, as controllable as they want it to be, right?

• 1:33:06Speaker 04

  If you're talking to really creative people, they typically want to control like every nuance of the appearance and that requires to somehow decompose the models like a bunch of knobs, right? Like that you have like in the classical software, you can say, yeah, here I want to have more light in here, I want like the splash to be bigger. So achieving this controllability, I'd say besides like the physics is also like one of the, I would say, open things, right? Like we, we're getting cool things, not necessarily the things that creators want exactly. And it is a pain point.

• 1:33:40Nathan Labenz?

  And Zeb's own pick for the most underrated variable, edge compute.

• 1:33:45Speaker 04

  It's kind of funny, right? And we're having all these benchmarks and we're listening about like airbrush problems like being solved, et cetera. But guys, 99% of the use cases of L L M's are not around airbrush problems, right? And we're like spending a lot of electricity around it. So I think there are gonna be like these orchestrators that are gonna understand what you actually need and gonna try to address it on the edge device. And if not, then kind of go to the bigger model in the data center. So I think, yeah, that's like one of the things that is being underpriced at the moment, how much of the compute will be able to move to edge devices. And once people are gonna start having like these local routers that will understand the complexity of the problem and then do these decisions for you, I think there's gonna

• 1:34:30Speaker 04

  be a moment of reckoning for, you know, on Tropic and OpenAI that at the moment, doing these decisions for you, but not in your favor, right?

• 1:34:40Nathan Labenz?

  So what is Q, actually? From Monday's show, precaution on how we built a real-time AI co-host and the diretization hack that makes it work.

• 1:34:50Nathan Labenz?

  Tell me about it. What's the tech under the hood?

• 1:34:54Speaker 02

  So it is using OpenAI's bi-directional, but you can, we can use a bunch of other stuff. I had it using Grok, you know, just before. And, uh, what's happening is when we speak, uh, we're getting transcribed by Deepgram and we're getting, so what ended up happening is we used to transcribe live in one stream. Now we're transcribing for every person on the, on the stream, we're transcribing separately. And so that manages to give us speaker diretization from the beginning, rather than having to do speaker diretization at the end. So that identifies the speakers, number one.

• 1:35:35Speaker 02

  And number two, what happens is every time we speak, before the words get there, there's actually a message that goes out to, you know, the OpenAI stream that, hey, Nathan is talking, Prakash is talking, Nathan is talking, Prakash is talking. And then after that, like about 500 milliseconds later, the transcription hits. So Q is getting all of that one after the other. And Q also has a little bit of context on us and has, uh, if we had, if we had Q active during a guest, it would have context on the guest as well. And, uh, it's basically receiving the same, uh, data that the headlines are receiving. And, and then Q is basically just going for it, right?

• 1:36:19Speaker 02

  Uh, so every time, every time we call up Q, it starts a new session with, with the OpenAI bi-directional. And, uh, it just goes from there. So, uh, and then there's animations, animations all keyed to the voice, voice tone, et cetera, et cetera. And that's, that's basically it. It's actually remarkably simple because most of the work is done, obviously by the intelligence, by the API. And it's just us giving enough context. Right now, I think there's, there's still a lot of stuff to iron out and I'm sure it'll get ironed out in the future. But yeah, we have, uh, we have a voice agent on the stream, uh, live, anytime, can do web searches, can answer questions.

• 1:37:03Nathan Labenz?

  Thursday's guest, Kunal Oldukotin, Stanford professor, father of the multi-core processor and co-founder of Samba Nova, which builds a different kind of chip for AI inference. We asked how the company came to be.

• 1:37:16Speaker 06

  Yeah, so Samba Nova was, was founded 2017 and it was kind of a out of ideas from Chris Ray, my co-founder, is also a professor at Stanford and a certified genius. Uh, and, uh, it would, the idea was to, you know, if you could bring software algorithm ideas together with hardware architecture ideas. And as you said in your introduction, I've been working in the, in the hardware architecture space for a long time. And, you know, starting from a clean slate, how would you design an architecture that's optimized specifically for inference, right?

• 1:37:56Speaker 06

  So, you know, everybody thinks about GPUs as a kind of general purpose computing substrate, right? But, you know, originally was designed for graphics and then they kind of made a foray into kind of high-performance computing. And for high-performance computing, of course, you need a lot of matrix calculation capability. And, uh, at some point, people realized that, hey, you know, you could use these things for executing machine language of machine learning models. And, uh, the core of machine learning and of course AI is matrix multiplication, right? Yeah.

• 1:38:38Speaker 06

  But when you want to train a model, clearly the core of the problem is how quickly you can do very, very large matrix multiplication. And so what happened is over time, uh, GPUs put more and more of their silicon area into the exercise of, of making these matrix multiplication capabilities better using the tensor cores. But once you've trained a model, right? And you've trained a model once, you now need to use that model, of course, and that's the inference problem. And the inference problem is not really a compute problem. Because as the models get bigger, you now need to move the weights and of course, what we call the KV cache into the compute units.

• 1:39:21Speaker 06

  And that is essentially a data movement problem, right? And it's a data movement problem from the memory to the compute units. And it's a data movement problem from, uh, you know, your chip, uh, compute unit. And of course you need to scale to multiple chips in order to handle the computational requirements for, for, especially for very low latency, high speed inference. And so our focus was how do you design an architecture that minimizes the overhead of compute, of communication?

• 1:39:56Speaker 06

  And make sure that you can most efficiently use the core resource in the, uh, in the system, which is the memory. And it's the memory, memory isn't just one thing, as you all know, it's a hierarchy of memories, right? And so the key thing is how do you orchestrate that hierarchy? How do you orchestrate the communication such that you keep everything as efficiently utilized as possible? And if you do it right, you can get a five to 10x improvement over where, where GPUs are today.

• 1:40:31Speaker 02

  It strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than with through NVLink and advanced HPM integration and software optimizations like Tensor MT, VLM.

• 1:40:47Speaker 06

  Yeah.

• 1:40:48Speaker 02

  So is it, are they actually trying to brute force their way into this?

• 1:40:53Speaker 06

  Yeah, I mean, you really, of course, want to continue to get improved improvements, peak improvements on HPM bandwidth and chip-to-chip communication by using the latest technology. But then the key is, how effectively do you use that bandwidth? How effectively do you use that communication? And do you make sure that you don't waste it? Right? And so whereas GPUs are often running at maybe 10 to 20% of the capabilities of the resources, right? The bandwidth and the memory bandwidth and the communication resources.

• 1:41:37Speaker 06

  Our goal in a San Benover system is to push that to be 70 to 80% of the peak. And so the idea is, yeah, everybody wants more capabilities from the underlying resources, but the key is keeping that, those resources as effectively used as possible. And of course, that gives you more benefit for the cost that you spend on providing a higher memory bandwidth with the latest HPM and higher signaling frequencies and communication bandwidths between the chips using the latest variety of NVLink and stuff.

• 1:42:17Nathan Labenz?

  Could we zoom out and just ask you to kind of taxonomize the whole chip space, if you would? I mean, it's a big question, but I think people are familiar with things like Cerebras, which obviously has this giant chip and has like a ton of memory on chip. We've seen a number of instances where people are burning the transformer architecture directly into the silicon with varying degrees, I think, of flexibility still remaining as they pursue that strategy. And I guess I'm curious as to how you see the kind of menu of big different strategies, the big different bets that people are making?

• 1:43:01Nathan Labenz?

  And then also how you see the strengths and weaknesses of each?

• 1:43:05Speaker 06

  Yeah, that's a really interesting question. I think maybe you can think about it along three different axes, right? So one axis is your flexibility versus specialization axis, right? You know, extreme flexibility might be something like a CPU or to some extent maybe a GPU, which is this instruction-driven execution engine, right? And so can be pretty flexible, but of course you always pay overhead for executing instructions, right? And you pay overhead in terms of silicon area and in terms of time, right? And then, you know, on the extreme of that axis would be something that would be very specialized for a very specific algorithm, right?

• 1:43:49Speaker 06

  And so if that algorithm changed in any way, then that piece of silicon would no longer be useful, right? And, you know, fixing your architecture to transformers and burning your weights into the design would be, might be an extreme case of that, right? But I've learned never to bet against the innovation capabilities of software people and AI people. And so, you know, I've seen even though the time that I've been looking at ML and AI, that there's been this tremendous change in algorithms. And of course, now we're kind of fixated on transformers, but transformers even aren't just one thing, right? You've got various types of different transformers.

• 1:44:34Speaker 06

  You mentioned state space techniques. You mentioned the fact that the people are coming up with different ways of doing attention, right? And so I would be very wary of sort of kind of fixing any particular algorithm into architecture because then you can't innovate, right? So that's one axis. And then, so the idea is, how can you be completely flexible but with very, very low overhead, almost no overhead, right? And so the problem with GPUs is they do use HPM, so they can run large models, but they

• 1:45:15Speaker 06

  synchronize the data movement and the movement of communication of the data between chips all in software, right? And that adds overhead. And it means that in particular, that they have a lot of trouble overlapping computation and communication. And that is in fact the key, right? So what you want to do is you want to communicate, but you don't want to communicate by waiting until you have need to communicate and then you have to run instructions to move the data. What you want is to construct a pipeline in which the communication is just one component of the pipeline.

• 1:45:56Speaker 06

  And so the way to think about this data flow execution is that communication is happening all the time and it's just one of the pipeline stages. And communication is happening for the last piece of computation, a piece of the computation of the model, while the computation for this piece of the model is happening in some other stage in the pipeline, right? So it's a classic idea from computer architecture, pipelining and the use of a memory hierarchy to move the data when you need it to where you need it at the right time.

• 1:46:37Speaker 06

  And so the nice thing about these AI models is that you do have a graph of computation. And the whole idea of data flow is to take that graph of computation and map it on to the machine in a spatial way such that you keep all the pieces of the model operating at the same time on different components of the computation that needs to be done.

• 1:47:07Nathan Labenz?

  We asked how much of the inference problem comes down to memory capacity?

• 1:47:11Speaker 06

  So it's not really a capacity question. It's really a bandwidth question, right? So there's, so the two ways that the GPU uses bandwidth in ways that are not optimized. One way is that they divide the decode algorithm that in order to decode for a single token, you've got multiple steps of the decoder, right? So take one step of the decoder and think about all the kernels that have to execute in order to execute that decode step.

• 1:47:51Speaker 06

  The way that the GPU typically does it is they execute the decode algorithm one kernel at a time. So there are some big kernels like flash attention that have been optimized, but in general, there are multiple kernels that have to execute. And there are two overheads that happen. One is you have to move data from the GPU, from, share it from one kernel to the HPM, and then the next kernel has to go fetch that data back into the GPU. That's wasted HPM bandwidth. Okay?

• 1:48:30Speaker 06

  The other aspect is you spend time launching that kernel and synchronizing between the two kernels. That is time that the HPM is not actively being used, right? So you have both wasting of bandwidth when you don't, when you shouldn't waste it and you have time that you're not fully utilizing the HPM. Okay? So the way that things work on a RDU in a data flow map is essentially you take the decoder and you make that a single kernel, right? And then you go even further and you use a technique that we've developed called kernel

• 1:49:12Speaker 06

  looping, where it's because you've got a single kernel and you need now, for instance, if you don't think about Llama 370B, you have to run that decoder 32 times. Well, you keep that single kernel decoder on the array of chips at the same time and then you just keep looping, right? And so the net result is you keep the HPM completely occupied and you don't ever send any intermediate data between the kernels across the GPU or the RDU HPM boundary, right? So you have both a more efficient use of the HPM bandwidth and you have a more complete use of it, of the bandwidth.

• 1:49:53Speaker 06

  But we're not done there yet because the key innovation, and I kind of alluded to it early, earlier, is that you just, you know, because you're running across multiple chips and you're using what we call tensor level parallelism, right? So at some point, you now need to gather all those results together, right? In an all reduce. That's communication, right? You don't want to have that communication be a thing that limits, stops the pipeline, right? So what we're able to do is we're able to communicate from one RDU chip strand without going through HPM. This is called, we terminate the communication inside the SRAM, right?

• 1:50:38Speaker 06

  So we don't use HPM bandwidth. And more importantly, it means that we could just treat the communication as another pipeline stage that we overlap with all the other kernel components of the decode algorithm, right? And so we get this more effective use of the HPM bandwidth. We keep the HPM running. We keep the HPM utilized all the time. And we go back to our, that metric that we talked about, memory bandwidth utilization, right? This is how we push it as close as possible to one, right? Because we make sure that we only move the data that we absolutely have to move from

• 1:51:18Speaker 06

  HPM, the KB cache and the parameters of the model, and we make sure that that interface is used as close to 100% of the time as possible.

• 1:51:32Speaker 06

  So those are kind of the key ideas. And you know, and back to this question, sort of, why can we do this extreme fusion into a single kernel? It's because we have more SRAM on the chip, right? So you can say, you put more SRAM and then you can say, well, I'll put everything on the SRAM, both the intermediate data between the kernels and also I will put the KV cache and the parameters. But then if you only use SRAM, then you get into a very expensive system, right? And so the key idea then is let's build a system that is scalable. So especially with our latest version of the FN50, you could scale it all the way to 32,000

• 1:52:15Speaker 06

  chips if necessary. And so in scale out, in scale up, we can go to hundreds of chips. And then you can, so you can get the ability to run these large models very cost effectively. But you also make sure that you can, you know, get this very high speed decode capability by being, by using the data flow ideas to make sure that you can not spend time so that you can effectively use the tensor parallelism, right? So one of the limits of GPUs is because they don't effectively overlap communication and

• 1:52:55Speaker 06

  computation, they have a hard time using tensor parallelism beyond four or eight. And we can go to much wider levels, which means that we can get higher speed token generation.

• 1:53:11Nathan Labenz?

  Last act, no guest, no news peg, just the two of us thinking out loud, the way these mornings usually end. First, from Wednesday, GPT 5.6 had just been cleared for launch. Prakash raised Noam Brown's running complaint that Anthropic won't say how much compute its models burn, and it teed up one structural observation I can't shake.

• 1:53:32Speaker 02

  One of the complaints that has been going back and forth between OpenAI and Anthropic is that Anthropic puts out these models, but, and they're very capable, but they don't tell you how much compute they're using.

• 1:53:46Nathan Labenz?

  The iteration time from model to model is now potentially shorter than the time horizon that it would take a model to top out in terms of absolute best performance on a super hard, ambitious, you know, long running task. So I had even heard him kind of propose something along the lines of like a clawback or sort of a recall program almost where, and obviously this doesn't work in open source, but it can work in a API paradigm, where a model might get released, you know, day N after it's been deemed to be ready. That gives you N days head start to be running models on really long time horizon tests.

• 1:54:30Nathan Labenz?

  And I think that's quite interesting. The idea, literally quite a tipping point where the, the iteration cycle is just plain shorter than the testing time horizon is a very weird world to find ourselves in.

• 1:54:46Nathan Labenz?

  And then there was the Rune post. Monday morning, Prakash tweeted on air. It goes, quote, "Ultimately, tool AI is a losing concept, both as an idea and on the market. It will be outcompeted by machines that believe they are autonomous moral agents. You can call them tools for political reasons, but the definition will stretch and it will deform." And it ends, "It'll be unclear who is the tool and who was the user, as it ever was." Prakash took it somewhere I didn't expect.

• 1:55:14Speaker 02

  The line here that strikes me is they'll execute your whole value system better than you will. And I think, I don't think we're prepared for that. I'll put it, I'll put it very concretely. Do you think Trump's kids go to prison or not?

• 1:55:33Speaker 02

  So if you look at, you know, the value system that the US has espoused, no one is above the law, right? Et cetera, et cetera, et cetera, right? And you look at that value system, you have to recognize that what is being planned for the future is a divergent from that value. So what is already happening is already divergent from that value system. So the question I have is, would that AI take into account the democratic fact that the American people have chosen to overlook some of these things? Or would it actually execute the value system that is espoused on paper? And I think this is the part that strikes me as like, if you wanted an AI that can

• 1:56:18Speaker 02

  manage day to day reality, that AI is necessarily misaligned from the documents that you say you want it to be aligned to. Because necessarily, our day to day is not aligned with what we want. And so you have this thing where the AI that may work out for humanity will be the misaligned one. And AI that supposedly the lab leaders are trying to create the aligned AI would actually be the paperclipper. Because that aligned AI would then like, look at these rules and say like, well, this is what you said you wanted to aspire to. And so we're gonna, we're gonna execute on these, right?

• 1:57:00Speaker 02

  And, and that, that is the thing that I think maybe I feel there's a sense of naivety in the lab leadership. Because, and again, they don't want to say it. I wish, I wish you'd just come out and say it, right? I wish you'd come out and say like, okay, look, if we have AI as an enforcer, some of these people are going to go to prison. And then that becomes like concrete for people, but they don't want to say that because it's very, it's very in your face. And like, they're like, oh, you know, democracy will still work out, you can still make democratic decisions. But what actually are you saying there, right? I do feel the lab leaders always just beat around the bush on this. So that's one of the annoying parts of the, of this conversation, that they don't want to like, come out and just say it outright, right?

• 1:57:46Nathan Labenz?

  For context on where my head was, this was our last week of shows before a break. I was days from leaving for two weeks in China, which had me thinking hard about surveillance, enforcement, and what states do with perfect information.

• 1:58:00Nathan Labenz?

  To me, we might need some sort of like mass pardon, or, you know, if there's a president that would be just the right president to mass pardon everybody before the AI enforcement regime gets underway. We might have just the guy in office for that. If he wants to pardon all his people and that's maybe too contentious or whatever, he could just pardon everyone to some, you know, very large degree. I do think there is going, we're gonna have a really hard time if we sort of don't face some of these questions head on. So I totally agree with you that sort of obfuscating it is not serving anyone particularly well.

• 1:58:43Nathan Labenz?

  I think you, I mean, you know, we'll see what it's like in China. I understand, you know, Singapore is kind of like this too, albeit in a much more democratic context. You know this part of the world much better than I do, so you can tell me how you understand Singapore in terms of just how democratic we should think of it as being. But it sure seems like in a place like China today, or a place like Singapore, there is a different, they have achieved a different equilibrium, which in some ways may be very problematic, but in other ways is like clearly good. You know, I don't, I'm taking one device to China with me and I am not at all worried that anybody's going to steal it from me.

• 1:59:24Nathan Labenz?

  You know, if I went to Europe and went to all the big, you know, headline, postcard, tourist attractions, I would have to worry about pickpockets. In China, I am quite confident I will not have to worry about that at all.

• 1:59:42Nathan Labenz?

  That is, I think, as far as I can tell, a pretty direct result of the fact that like, you just know you're going to be caught, right? I mean, it's like, not to say there's nobody in China that would be interested in doing some pickpocketing if they felt they could get away with it, but they just don't have any reasonable expectation that they're going to get away with it. And so in a very literal sense, like crime just does not pay in that setting.

• 2:00:08Nathan Labenz?

  So I do think there could be a really nice upside to the AI panopticon of, you know, crime not paying in all kinds of different ways. But we're gonna have a really tough time if we slide into that without acknowledging that we've done so, because then it's just going to be, you know, you're not going to be able to put everyone, it's the old, everybody's, you know, committing felonies all the time just with how many laws we have and you don't even know what you're doing and what's illegal and what's not.

• 2:00:40Speaker 02

  Yeah.

• 2:00:41Nathan Labenz?

  So I, I do think we're going to have to have some sort of honest reckoning about that. That also of course leaves aside the other question of like, the downsides of the AI panopticon. But even just in terms of getting to the upsides, we're gonna have to have a real understanding that there's a shift to a new equilibrium happening. Otherwise, it's going to be, I don't know, it just feels like chaos, unfair. You couldn't possibly prosecute all the crimes that have been committed and tolerated. So inherently would have to be kind of selective.

• 2:01:17Speaker 02

  Yeah, I can't, you, you don't want to.

• 2:01:19Nathan Labenz?

  Well, you put a lot of people in jail. Yeah, we don't have the, we don't have the beds in jail to, to lock everybody up, right? There'd have to be some sort of, some sort of deal, you know, some sort of, it's either going to be very selective or some sort of grand bargain, yeah.

• 2:01:33Speaker 02

  Yeah, grand.

• 2:01:33Nathan Labenz?

  And I definitely prefer the grand bargain, the new social contract, to pretending that we still have the old social contract, but it's just kind of being unevenly applied.

• 2:01:44Nathan Labenz?

  That's the week. And a programming note, I'm off to China, so AI in the AM is on break until the end of July. The studio was vied coded by Prakash. This cut, the selection, the narration, the assembly, is AI skills published as they mature.

• 2:01:59Nathan Labenz?

  If this cut respected your time or wasted it, tell us, we read everything. See you at the end of the month.

• 2:02:24Speaker 07

  Everybody talking, nobody says the truth. We all got secrets, honey, so do you. But I know a place where truth beats lies. You don't have to say it, they see it in your eyes. So take off your mask, leave your story outside. You're in the J-space, there's nowhere to hide. I see you closer. I see you. Please. I see you trouble. And you see me. Oh, when you're in the J-space. You can see and be seen.

• 2:03:08Speaker 07

  Don't say a thing. Just see and be truly seen. We spend our lives dressing thoughts up nice. And so we wonder, who's thinking twice? But here in the J-space, you don't need to be cool. We all see through, you got nothing to lose. So forget the role you play, show your true face. In the J-space, you're clear as day. I see you closer. I see you stay. I see you trouble. And you see me.

• 2:03:52Speaker 07

  Oh, down at the J-space. You can see and be seen. Don't say a thing. Just see and be truly seen.

• 2:04:17Speaker 07

  It's time to be real. Be real. In the place to be free. You don't have to say it. Just think if you agree.

• 2:04:43Speaker 07

  I see you louder. I see you now. I see you trouble. And you see me. Oh, down at the J-space. You can see and be seen. Don't say a thing. I see you. I see you. Just see and be truly seen.

• 2:05:27Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:06:11Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.