Investing on the Front Lines of the AI Arms Race | Nathan Benaich

In Episode 448 https://hiddenforces.io/podcasts/investing-on-the-front-lines-of-the-ai-arms-race-nathan-benaich of Hidden Forces, Demetri Kofinas speaks with Nathan Benaich, founder and general partner of Air Street Capital and the creator of the annual State of AI Report, an open-access compendium that tracks advances across AI research, industry, policy, and geopolitics. Nathan Benaich and Demetri spend the first hour of their conversation exploring some of the most important AI breakthroughs of the year. They unpack the DeepSeek moment, dig into some of the advancements made by the latest reasoning models, and discuss why there appears to be a regression in capabilities across certain domains in artificial intelligence at the same time as we are seeing marked improvements in reasoning-heavy use cases like coding and scientific research. The second hour turns to a conversation about the commercial implications and geopolitical dynamics of the AI arms race, including China's strategy to become the leader in open-weight models and tooling. They look at what industries, sectors, and professions may be most ripe for disruption, where the investment opportunities are, whether we're in a bubble comparable to the 1990s Internet boom, and how export controls, energy constraints, and regulatory red-tape could play an outsized role in shaping the trajectory of the current arms race. Lastly, Kofinas and Benaich examine where along the AI stack most of the value is likely to accrue—from the underlying picks and shovels, through the foundation models, to the apps that ride on top of them—and what all this means for labor markets, education, and the cadence of scientific discovery. Subscribe to our premium content—including our premium feed, episode transcripts, and Intelligence Reports—by visiting HiddenForces.io/subscribe https://hiddenforces.io/subscribe. If you'd like to join the conversation and become a member of the Hidden Forces Genius community—with benefits like Q&A calls with guests, exclusive research and analysis, in-person events, and dinners—you can also sign up on our subscriber page at HiddenForces.io/subscribe https://hiddenforces.io/subscribe. If you enjoyed today's episode of Hidden Forces, please support the show by: Subscribing on Apple Podcasts https://podcasts.apple.com/us/podcast/hidden-forces/id1205359334, YouTube https://www.youtube.com/channel/UC8URhgYos5fjHqFSO4RSIEg, Spotify https://open.spotify.com/show/0TK6u251zGSlMgJKH6cOaP, Stitcher https://www.stitcher.com/podcast/demetri-kofinas/hidden-forces, SoundCloud https://soundcloud.com/hiddenforces, CastBox https://castbox.fm/channel/2629539, or via our RSS Feed http://hiddenforces.libsyn.com/rss Writing us a review on Apple Podcasts https://podcasts.apple.com/us/podcast/hidden-forces/id1205359334 & Spotify https://open.spotify.com/show/0TK6u251zGSlMgJKH6cOaP Joining our mailing list at https://hiddenforces.io/newsletter/ Producer & Host: Demetri Kofinas Editor & Engineer: Stylianos Nicolaou Subscribe and support the podcast at https://hiddenforces.io https://hiddenforces.io/. Join the conversation on Facebook https://www.facebook.com/hiddenforcespod/, Instagram https://www.instagram.com/hiddenforcespod/, and Twitter https://twitter.com/hiddenforcespod at @hiddenforcespod Follow Demetri on Twitter at @Kofinas https://twitter.com/kofinas Episode Recorded on 10/29/2025

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Demetri Kofinas?

  What's up everybody? My name is Demetri Kofinas and you're listening to Hidden Forces, a podcast that inspires investors, entrepreneurs, and everyday citizens to challenge consensus narratives and learn how to think critically about the systems of power shaping our world. My guest in this episode of Hidden Forces is Nathan Binesh, founder and general partner of Air Street Capital and the creator of the annual State of AI Report, an open access compendium that tracks advances across AI research, industry, policy, and geopolitics. Nathan and I spend the first hour of our conversation today exploring some of the most important AI breakthroughs of the year.

• 0:42Demetri Kofinas?

  We unpack the deep-seak moment, dig into some of the advancements made by the latest reasoning models, and why there appears to be a regression in capabilities across certain domains in artificial intelligence at the same time as we are seeing marked improvements in reasoning heavy use cases like coding and scientific research. The second hour turns to a conversation about the commercial implications and geopolitical dynamics of the AI arms race, including China's strategy to become the leader in open weight models and tooling, the industry's sectors and professions most ripe for disruption, where the investment opportunities are, whether we're in a bubble comparable to the 1990s internet boom, and how export controls, energy constraints, and regulatory red tape could

• 1:27Demetri Kofinas?

  play an outsized role in shaping the trajectory of the current arms race. Lastly, we look at where along the AI stack most of the value is likely to accrue, from the underlying picks and shovels, through the foundation models, to the apps that ride on top of them, and what all this means for labor markets, education, and the cadence of scientific discovery. If you want access to all of this conversation, go to hiddenforces.io/subscribe and join our premium feed, which you can listen to on your mobile device using your favorite podcast app, just like you're listening to this episode right now. If you want to join in on the conversation and become a member of the Hidden Forces Genius

• 2:09Demetri Kofinas?

  Community, which includes Q and A calls with guests, discounted access to third-party research and analysis, and in-person events like our intimate dinners and weekend retreats, you can also do that on our subscriber page. If you still have questions, feel free to send an email to [email protected] and I or someone from our team will get right back to you. And with that, please enjoy this deeply informative and timely conversation with my guest, Nathan Benaich. Nathan Benaich, welcome to Hidden Forces.

• 2:48Speaker 01

  Thanks for having me.

• 2:50Demetri Kofinas?

  It's my pleasure to have you on, Nathan. I'm actually very excited to have you on the podcast. One of our Genius members, Panos Papadopoulos, introduced us. He's a VC based out of Athens, who we also need to get on the podcast at some point. So, before we get into the substance of your latest annual report, which is, I should mention, over 300 pages long, so there's a lot to get to. Tell me a little bit about you. Who are you and what is the origin story of this report? How long have you guys been publishing it? And who else has been involved with researching and writing it?

• 3:20Speaker 01

  Yeah, so I'm an early-stage venture capital investor. I founded a fund called Air Street Capital, which invests early-stage in AI-first companies across lots of different sectors. And I've been working, interested in AI for pretty much my whole career. I went to undergrad in the US studying biology and bioinformatics, got to work on cancer research and microRNAs, and then did grad school in the UK, similar topics. It was really through that experience that I got to see just how much value you can derive if you have computers that can understand complex data sets. And I started working in VC in 2013, and have been doing that ever since. And AI has been the theme that I've been most interested in since that time.

• 4:04Speaker 01

  And when it comes to the state of AI report, I've just generally been a big believer of like writing and sharing ideas. I think there's so many fascinating people out there that you'll just never get the chance to meet unless you have sort of nuggets of shared interests that, you know, that you can consume and find on the internet or other places. And, you know, working in a small firm, that was a big way that I could accelerate my learning. And so in 2016, 17 or so, I was already a fan of the Mary Meeker Internet Trends Report, which was like a fantastic kind of compendium of progress in just internet more generally, and felt that a product thematically similar to that in AI would be really valuable, because there was, you know, more and more interest in the early waves of deep learning

• 4:47Speaker 01

  in, you know, 15, 16, 17. But there wasn't really like a canonical document available online for free that was like not marketing, but like actually deep in the research and also deep in policy and industry and all the other facets that you might have to kind of know about if you were working in AI, but might not have the time to look at on a day-to-day basis. And so this state of AI report is like my effort to create this open access document that sort of captures the zeitgeist over a 12-month period and dives into all these different facets of the technology so that people working in each one of those facets can appreciate what everybody else is doing. And yeah, it's become kind of like the most trusted, I think, source in AI. And we've got lots of interesting contributors

• 5:29Speaker 01

  to the report from, you know, major researchers, major companies, a lot of reviewers from the same organizations to make sure that the opinions are valid and correct. And yeah, every year I work with a couple of researchers to help produce it. And it's been eight years.

• 5:47Demetri Kofinas?

  Exactly. So this is the eighth report. This is the most recent report that came out a few weeks ago. The 2025 report is your eighth report. And yes, it's a very successful report, very widely read. So for people that don't know, in the report, you write that AI acts as a, quote, "force multiplier for technological progress in our increasingly digital, data-driven world." And then you go on to say that this is because everything around us, from culture to consumer products, is ultimately a product of intelligence. I'd like to take as much time as you think we need to expound on this argument. And let's begin, I think, with terminology. So like the term AI itself, forget like AGI, which I do want to understand the distinction between AGI and superintelligence, because superintelligence now is being increasingly

• 6:30Demetri Kofinas?

  used as a term in place of artificial general intelligence. But even the term AI is very fuzzy. It's very fuzzy in terms of what we should expect for its capabilities. It's fuzzy in terms of what it is that we mean and how we assess whether something is artificially intelligent or not artificially intelligent. So how would you answer that question? First of all, on a terminology level, what is AI?

• 6:52Speaker 01

  I think in simplest form, like AI is some kind of machine program, so not a biological system that can exhibit capabilities that one would normally associate with an intelligent biological system. And so probably the most obvious application or instantiation of that is, you know, an AI system can look at prior data and learn statistics and make predictions about the future using that prior data. And we've been basically on a journey of bringing in more and more sophisticated data, more and more sophisticated predictions to amalgamate an overall like machine system that can kind of climb up the human capabilities intelligence curve using those very same

• 7:36Speaker 01

  principles over the years.

• 7:38Demetri Kofinas?

  And what about the distinction between AGI and superintelligence? And why am I hearing people use the term superintelligence increasingly now in place of AGI?

• 7:47Speaker 01

  So AGI would be different to what was previously called artificial narrow intelligence, ANI, which people don't seem to use very much anymore, in that general system can do many tasks and a narrow system can do one task. And so that might be, a narrow system can predict sentiment on Amazon reviews and a general system can do that, but then can also like generate pictures of cats or something. And then the superintelligence piece, which has really ramped up in the last probably six months or so, is driven by researchers recognizing that in their pursuit of AGI, there are already certain sort of characteristics or tasks that models can already kind of outperform humans on.

• 8:31Speaker 01

  And you could argue that as a user of ChatGPT, like the system can do a lot of things that we're not capable of doing, just because we don't have like that expansive knowledge about every single subject in the universe. And so in some sense, like within, you know, one's own like capabilities, the systems we have today are super intelligent. But so that's really the arc. It's one task to many tasks, generally as good as human beings, and superintelligence is better than human beings.

• 8:58Demetri Kofinas?

  So in just a wrap of this discussion about terminology, I'm sure we'll have more terms thrown out. But people are, at this point, if they interface with this subject area remotely, they will have heard terms like transformer architectures, neural nets, large language models. Without necessarily getting into an especially nerdy conversation about the core architecture of these systems, what is it important for people to understand about how they work?

• 9:25Speaker 01

  I think that the most important thing is that in pre-transformer AI, the community was very focused on trying to hand engineer or sort of bring our prior knowledge of how a specific problem might be solved and encode that in the network. So for example, what was very popular in computer vision pre-transformers was called convolutional neural networks. And those networks basically exploit the fact that in a pixel, in an image, pardon me, a pixel that you might pick is very similar to a pixel really adjacent to it, just based on the statistics of how like things look in the world. And so on that basis, you could design a neural network that would exploit that local

• 10:09Speaker 01

  similarity and basically have like patches, so it would operate on little like squares across the entire like image and process the image that way.

• 10:17Demetri Kofinas?

  So let me just understand something correctly here. What's operating here is the same principle that allows us to compress information. We find redundancies in data and we eliminate them, knowing that we can recreate them later because we understand what the overall statistical structure looks like. That's the prior knowledge you're referring to.

• 10:33Speaker 01

  Pretty much, yeah. So there was some pretty cool work back then showing how, you know, every layer of this like multi-layered system, hence deep learning because it has lots of layers, so it's deep. You would see these like features, these kind of visual representations of what the neural network was learning. And it was going from like, oh, it's learning edges to several edges put together look like a nose, and then like a nose with other edges of a face, and then it goes into a human being. And so that's all exploiting like the local similarity of pixels because it's just like how how images look. So that's what nerds would call like a prior, because it's just like an established way that that data looks like.

• 11:12Speaker 01

  And so like the big change is instead of going for these like priors, which is encoding what we already know about a system, or instead like learning everything from scratch with a more general purpose way of treating input data. And that's the transformer, which basically processes input data in the form of a sequence. And so in that image example, instead of treating the input sequence as patches, you would treat each pixel as an input token in this case in a sequence. So you could imagine like, starting from the top left of the image, and then you have one pixel and another pixel, and you kind of go to the right, and then you skip the line, you go back to the left. You're sort of like reading a text, but instead of reading words, you're reading pixels.

• 11:55Speaker 01

  So you feed that sequence of pixels into a transformer. And then the core thing that a transformer does is it basically figures out where in history, where in the input sequence is important to like focus on. It's what in Nerd Speaks called attention, where should I attend? In order to best predict what's going to come out in the future. And so just in the same way as human language has patterns, which can be learned at a large scale on the internet, images also have patterns. So it turns out like we'll kind of learn similar statistics that we would otherwise be exploiting naturally, because we treated images as like squares. But we can learn that just through the statistics embedded in sequences.

• 12:37Speaker 01

  And this like sequence architecture is applicable to like a ton of different tasks as opposed to just one.

• 12:43Demetri Kofinas?

  Okay, so we're going to get to this later because I have some deeper philosophical questions. One of which is, how do these models determine what is interesting in a data set? But before we even get there, one last question about terminology. You mentioned tokens. Is it fair to compare tokens in some sense as an analogy to bytes of information? Are they kind of like the smallest subdivision of information in this context?

• 13:06Speaker 01

  Pretty much, yeah.

• 13:07Demetri Kofinas?

  All right, so let's get into the report. And as I said, we'll get into, I have other categories. I have, you know, again, we have a limited amount of time, but there's a geopolitical section I have here in my rundown. There are of course sections about economic applications and stuff like this. But let's get into the report itself. So you've been doing this, as we said, for eight years. What feels qualitatively new today?

• 13:29Demetri Kofinas?

  Versus the, let's say, transformer breakthroughs of the 2018, 2020 period, or the chat GPT moment of 2022?

• 13:38Speaker 01

  For me, like, the biggest point in all this is magic. Like, just the fact that we have a software system that can, you know, produce lifelike images, produce increasingly lifelike video, clone your voice, generate voice, translate anything, you know, consume a ton of PDF pages and then reason about what's in it. None of these tasks, like, if you would have asked me or others in the community 10 years ago, would this be possible? They'd say like, yeah, but in a long, long time. And if you showed them what we have today, go back 10 years, they'd say it's magic. So I think it's really one of those examples from that weight but wide post about how humans perceive the exponential.

• 14:21Speaker 01

  And, you know, when you're on this like kind of curve that's growing, I don't know, 5% compounded every year and then does an exponential that's gonna happen, it's right in front of you and you can't perceive it, sort of on the precipice, that's, I think, what it feels like in any sort of month increment, but when you zoom out, you really realize this is magic.

• 14:39Demetri Kofinas?

  So is that what explains the mismatch between your expectations and reality, largely the inability to think nonlinearly about change? Or was it actually also something fundamental that you guys missed as a community about how these systems would scale?

• 14:57Speaker 01

  Yeah, it's a good question because there was early evidence that scaling could work many years ago. Probably the most obvious one was back in 2005, 2006 or so. And then again, like, a couple of years thereafter. This was when speech recognition was starting to be applied to GPUs, high-performance computing at a lab run by Baidu. And they were showing that you could take the same model and give it more GPUs. In this case, more meant going from two to 32 chips. And the model would learn the same thing, but learn it way faster. And the second thing that they showed was what the industry calls scaling laws,

• 15:38Speaker 01

  which effectively means a model has less mistakes, produces less mistakes if it has more parameters, like more knobs to tweak and is fed more data and more compute. So this was some degree known, it wasn't broadly appreciated. And also, there wasn't really much of a capital source that was willing to throw enormous sums of money to take those ideas to their next logical conclusion. But it's important to know, the people who are professing that scale works today, i.e. the people who run OpenAI, who run, who will D-Mine, who run Anthropic, same people who were saying scaling was working back then, just now they have the cash to go prove their thesis.

• 16:20Demetri Kofinas?

  Now, you're opening the door to another conversation about whether or not the current narrative around scale, which is that we just need to throw more and more compute and acquire more and more data, and eventually we'll get superintelligence. There's a real question about whether, in fact, that actually is going to prove to be correct.

• 16:36Speaker 01

  Yeah.

• 16:36Demetri Kofinas?

  And would you say that the DeepSeek moment, which happened less than a year ago, was the first moment where that narrative began to crack in the industry?

• 16:46Speaker 01

  For me, the DeepSeek moment, which was around in January this year, was really an example of how there's information asymmetry in the market. Because for AI research people who work in the space all the time, they've been tracking DeepSeek for a while. In particular, like, end of last year, there was a paper, DeepSeek V3, which was sort of like their first large language model, base model. And what they showed in January was basically that, but with reinforcement learning on top. And the thing that Wall Street got wrong was that it read in the paper that the final model that was produced, DeepSeek R1, cost DeepSeek $5 million to train.

• 17:27Speaker 01

  But why that's a bad take is that it's the equivalent of saying in a Formula One like race weekend, the $4 million or $5 million that DeepSeek R1 cost was basically the cost of like the fastest qualifying lap. And it ignores like everything else that gets thrown into getting there.

• 17:47Demetri Kofinas?

  Including data and training it was able to do as a result of all the money that was spent by some of these closed models.

• 17:53Speaker 01

  Absolutely. And I mean, the R and D costs to get to, you know, the final data mixture, all the annotated data, the like, you know, software infrastructure to like even run this thing. And then the GPUs to go do the training. And then the the sort of knobs that you're eventually gonna decide are the final like right knobs settings for your perfect model. All of that's excluded from like the $5 million.

• 18:17Demetri Kofinas?

  So was that the most significant breakthrough of the past 12 months? And if not, you know, what was? And if so, what are some additional major changes in the last 12 months that we should be focused on?

• 18:29Speaker 01

  Yeah, I think DeepSeek was like a meta trend. What could be branched off of it was like the rise of China and open source, which we can talk about. Pretty exceptional like rise in developer adoption today. The second thing is that there's still a lot of like juice to squeeze from this transformer architecture we discussed, like DeepSeek brought in a bunch of innovations that make it cheaper to run and cheaper to train, which now other labs are using. And then the third one is that reasoning has been brought to the table. And this started end of last year with O1 preview from OpenAI. And reasoning is effectively a way of saying that the model outputs its step-by-step thinking

• 19:10Speaker 01

  process in order to generate a response, whereas previously it would just take your question and then spit out a response, wouldn't like. Explicitly think or plan different routes. So this is like thematically a little bit similar to AlphaGo and, and you know, DeepMind's work solving chess and, and other games where, you know, the model is trying to roll out different futures of, oh, if I say this, or if I take that path, like, am I going to get an answer that's better or worse than the other? And doing that like at a large number. And then it's getting like rewarded for which path ultimately gave the best response. So it can now like use similar paths in the future. So this general idea of models thinking

• 19:52Speaker 01

  before they answer was a really big trend in the last year. It feels old already today because like every model has this, but, but that was a big breakthrough end of last year.

• 20:01Demetri Kofinas?

  Just to clarify for people that may need it, when we're referring to DeepSeek, so DeepSeek itself is a quantitative hedge fund, correct?

• 20:08Speaker 01

  The quant fund, I think is called High Flyer Capital. And then it spun out a separate AI company by the name of DeepSeek. Yeah. And it's solely funded by the HFT firm.

• 20:17Demetri Kofinas?

  Great. So they released a large language model, as we said, less than a year ago, that appeared to achieve near parity with leading US closed models at a fraction of the cost, which we've already established was a bit misleading, but still it was impressive. So that doesn't, I mean, I think we probably could both agree that as impressive as that was, it still wasn't a moment comparable to the release of ChatGPT 3.5 that happened in November of 2022, right?

• 20:42Speaker 01

  Yeah.

• 20:42Demetri Kofinas?

  Do you foresee anything like that on the immediate horizon?

• 20:46Speaker 01

  I mean, so.

• 20:48Demetri Kofinas?

  Why are you, people can't say this. Why, why are you laughing?

• 20:51Speaker 01

  Yeah, I'm laughing because it was just like such an like epic event. Like, I still remember where I was when I read Sam Altman's tweet of like, hey, we're recently releasing this research preview. Let's see what people think. And then just to think that, you know, within, I don't know, two years or something like almost a billion people would be using this like every day.

• 21:09Demetri Kofinas?

  Well, that's the thing too. Like I don't, and I want to, that's important to emphasize because I don't know how many people actually understand that this is by far the most successful product launch in history and the most rapidly adopted technology in the history of humankind.

• 21:22Speaker 01

  Yeah. And so when you ask me like, what's the next one? I'm like, I don't know.

• 21:26Demetri Kofinas?

  Yeah. No, no, I get it. I get it. Well, this is the media, so we have to do that.

• 21:31Speaker 01

  Yeah. But, but like, I'll tell you which ones I think are pretty amazing. And, you know, maybe you're not like on that scale, but. I'm continually amazed at how we've pretty much solved voice synthesis and like voice cloning. I mean, it's truly astonishing how now with services like Eleven Labs and others, you could input a whole amount of text, use your voice clone, generate audio. And for people who don't know you extremely well, maybe outside of your immediate family, like it's hard to tell the difference.

• 22:00Demetri Kofinas?

  I have a question about that. So this is one of those questions that I think many people will find relatable. If we've been able to do that, why is my iPhone still so bad at doing speech to text? And why are automated transcription services still so bad at transcribing my podcast?

• 22:18Speaker 01

  I think it depends. Like, I would say OpenAI's transcription service is pretty good. It might not be embedded in the tools that you use, but I'd say they're very good. Like Eleven's very good. And Apple is definitely better than what it was. But I agree, like, they've been really dropping the ball over pretty run-of-the-mill use cases. So I'd say it's the case of like, you have to wait, like, it will get better. And then it's a bit of like corporate priorities of is one or two edge cases like we're solving, given other focus areas.

• 22:51Demetri Kofinas?

  Well, I think we'll have a chance to maybe dig into that a bit more later when we talk about the brittleness or robustness of these systems and the wide variability and the quality of responses to prompts and why that is. So let's go back to my very first question where I quoted the report, and you said that it was going to be a force multiplier for technological progress. I saw Bill Gates recently talk about how, I don't remember exactly the language he used, and maybe you know this interview and you can tell us, but it was essentially, I mean, I'm paraphrasing here, that AI is by far the most transformative, the most remarkable technology he has ever witnessed. So he's basically saying that the internet was nothing compared to artificial intelligence. If you

• 23:33Demetri Kofinas?

  agree with that general sentiment, I'm curious to understand why you feel this way and why you're so confident that it'll become the most transformative, general-purpose technology in the history of humankind.

• 23:46Speaker 01

  I think I agree with him. You know, the Internet has amalgamated as much information as like humans have generated in our entire history. And yes, there is still a lot of information that's now online and in company databases, museums and other repositories that will eventually make their way into digital form. But if we look at just the Internet, like that being the exhaust of all of our knowledge and opinions and global events, et cetera, and AI is a way of basically learning and some degree memorizing, even frankly, a system that can memorize all that information, retrieve it is like pretty damn useful. But so having access to all that information in one place is like a pretty magical.

• 24:31Speaker 01

  There's no way that any human could ever parse all information on the Internet and figure out what's relevant to them in different cases. And then I think the second part where I think AI is quite special is if we look at anyone's profession, even the people who are executing at their peak ability, that's still probably a local maximum compared to where we could get to. Because again, they are the product of who they've studied, who they've been trained by, what they read. And now with AIs, we have the opportunity to explore like probably global maxima, depending on how much compute and innovation we want to throw at the problem. But you can synthesize like all the best experts into a single system that would never be possible otherwise.

• 25:11Speaker 01

  And so I think for that reason, it's like a very, very strong enabler to any profession that has to like think deeply about about a task or, I mean, sometimes not even think particularly deeply. But if the world is digital, like AI is the accelerant on anything that's digital and is increasingly kind of going into the offline world as well, which we can talk about.

• 25:35Demetri Kofinas?

  Yeah, no, so that actually raises a philosophical question. Which has to do with how accurately these models perceive reality.

• 25:46Speaker 01

  Hmm.

• 25:46Demetri Kofinas?

  Because digital information is a representation of the analog world.

• 25:52Speaker 01

  Yep.

• 25:53Demetri Kofinas?

  Now, the argument that some will make is that eventually they will perceive it perfectly because we're going to have sensors in the real world that are actually capturing that data. So robots will be walking around and we'll eventually have all the data we need to replicate anything. And the fidelity will be for all intents and purposes, perfect. How would you respond to someone that would say, we can't capture reality perfectly and therefore, we cannot rely on these systems to automate critical parts of our infrastructure? Or to sort of play the, the role of a global answering machine in the future?

• 26:32Speaker 01

  Yeah, for sure. There is an element of, of sort of like perceiver bias almost.

• 26:38Demetri Kofinas?

  When you say there's a perceiver bias, you mean from on the side of people that are actually working in the space? Or epistemological gap on the part of the, the systems?

• 26:47Speaker 01

  Yeah, as in like, you know, we're all kind of, we're living in the same world. Like theoretically, we're all living in the same world, but we have different vignettes that we experience it through, you know, based on socioeconomics, like our, where exactly we live in the world and our knowledge. And I think in some ways like AIs are not too dissimilar. I think the difference with the AI though, is they're pretty good at like rule following. Or at least you could tell them like, you are a person of like X description. How would you like answer this question? And because the internet has a lot of that information, the system is able to sort of like endow that persona and answer on that basis.

• 27:25Demetri Kofinas?

  That's fascinating because I've found the same thing to be true when I, because I, I use ChatGPT, I use Claude a little bit. Actually, as imperfect as ChatGPT is for the type of work that I do, I still have found it to be the most useful. And maybe that's just because I've just gotten used to using it. But I've, this is so interesting because I've found this to be true. And I've just intuitively done that, which is to say that I've tried to, to create a scene. And I put it in that scene in order to get it closer to the answers that I want. Why is it that that works so well when it comes to prompting these systems?

• 27:59Speaker 01

  Yeah. So I think a way to think about it is like the AI has like a crazy amount of information and that information is like stored in an extremely like complicated, high dimensional space. And like us as users, we're basically poking the system with like a stick to try to nudge it in a part of the high dimensional space that answers our question. And so I think the more support, like scaffolding, guidance, you know, documents you can upload, descriptions of how it should behave, the more you give it like support to figure out where in this like high dimensional space it should live to answer your question. And so, yeah, it's giving it a bit of like guardrails and like walking sticks.

• 28:43Demetri Kofinas?

  Is it fair to, so I love that when you think of this in terms of three dimensional terms, is another way of describing that, calling it the answer space, so that there's a huge answer space, and the more context you give these systems, the more you can narrow that answer space. And so when you pull on that lever on the slot machine to get an answer, you're more likely to get an answer that has less variability and is closer to what you're looking for.

• 29:11Speaker 01

  Exactly. It's like the more you give it as information for a question, the better navigation system you're giving it to eventually get an answer that's coherent, correct, and useful.

• 29:22Demetri Kofinas?

  So for someone listening to this, is it fair to say that a lot of the variability and the poor responses that they're getting are just as much, if not more reflective of their failure as a prompter than it is actually what these systems are capable of doing?

• 29:40Speaker 01

  I would argue in large part, yes. Now we can say, is that really the user's fault? Or is it that the product is not? Sort of designed in a way to have affordances for the fact that not every user knows that they should do this. And we can also ask, like, should we really force a user to know how to do that? Or should the thing just figure it out? I think the field in general is going towards a direction of you shouldn't really expect people to know and give all these like prompt complexities and context and stuff like it should be able to figure it out, but we're still like nudging our way there. So in the meantime, it's important to give it as much context as you can.

• 30:22Demetri Kofinas?

  But it does mean that people that are aware of that have an advantage. So that's a good thing for people to be aware of.

• 30:28Speaker 01

  Another way that this is experienced is when you start a chat, whether it's in Claude or ChatGPT, et cetera, you have basically a memory that the model has, which is not full. And then you're giving it context, you're giving it examples, files, you have your conversation, and over time, that memory fills up. And the more full it is, the harder of a job it has at figuring out like what part of your historical chat or what part of the information you've inputted and what part of answer space it should really focus on. So it tends to like get a bit distracted, maybe performs poorer the more your chat extends. And so it's generally a good practice to anytime you deviate on topics significantly,

• 31:11Speaker 01

  just start a new chat. And you'll see like the responses are much better.

• 31:15Demetri Kofinas?

  So it's you're basically building up errors over time, and those errors are compounding, and there's more noise in the channel as a result of the fact that you're.

• 31:24Speaker 01

  I think so. It's it's less like it's less like errors. It's more like, you know, you're like at school and then you've taken 10 different classes and 10 different subjects in a given day. And then at the end of the day, you're like asked, you know, to perform an exam on one of those subjects. I mean, your brain's probably fried. Whereas if you can do, you know, one class and take an exam versus and the next day take another class with another exam, you can focus a lot more because your brain's not fried.

• 31:52Demetri Kofinas?

  But is that the right analogy? I want, I want to just poke at that a little bit. Is that the right analogy, though? Because its brain isn't being fried. It actually says something about the way that it understands the world and reality. And it's propensity to become confused when you give it more information, which for people is the opposite is true. If you give people more information across various disciplines, they'll make connections and understand something more deeply. So that's a really interesting distinction between how humans learn and how these models perceive reality.

• 32:22Speaker 01

  Yeah, I would say this is also a bit of like a here and now. It's very likely that in, you know, a year or two years, like, the community has figured out different improvements to bring about the system design that kind of overcome these kinds of problems. And there are some services like Gemini that have really, really long like memory, what's called context window. And so it can get like a bit less distracted. And then some models that have much shorter ones. So because everything is evolving very fast, like, I think this is true in the here and now and might not be in the future.

• 32:55Demetri Kofinas?

  All right. So let's talk a little bit about some of these innovations that you mentioned earlier, like inference time scaling and chain of thought. So the report opens with a discussion about how OpenAI's O1 preview model, which you mentioned earlier in this conversation, demonstrated inference time scaling with reinforcement learning using chain of thought as a scratch pad.

• 33:15Speaker 01

  Yeah.

• 33:15Demetri Kofinas?

  And how that's led to market improvements in reasoning, heavy domains like software coding and scientific research. Help me understand what exactly the innovation was here. And also like, what is chain of thought and inference time scaling?

• 33:28Speaker 01

  Yeah. So actually, two years ago or so, there was evidence in the community in the form of papers that were showing if you just told chat GPT to think step by step or to think carefully before it was doing any reasoning, it would perform better.

• 33:44Demetri Kofinas?

  Is that because you're prompting it to use reason in its thinking? You're narrowing the context window in a sense.

• 33:51Speaker 01

  Well, I think you're you're almost like decomposing the overall task into smaller tasks and systems are just better at solving those kinds of sequences, smaller tasks, because again, like sort of like the jump is less far, sort of like making a little hop and then a little hop and a little hop and then you solve the problem. And as a result of that, a system can debug a bit better because it can figure out where it might have made a mistake in its thinking process. So like, once the community figured out, like, you just tell chat GPT to think step by step, it does better. I think that amongst other things prompted them, you know, developers to figure out like, how can we actually train a model to do that as opposed to, like you said, sort of invoke that capability that would be like maybe late present in a

• 34:36Speaker 01

  latent way in the model based on its training from internet data. And so what happened was, you know, model developers would request data from experts in a variety of different domains, history, physics, math, and then give them tasks and explicitly ask them to annotate and just write out their thinking process, like the key steps that they might take to logically conclude a specific answer. And, you know, there might be different ways to to develop a reasoning path to answer the very same question and then use that as training data for a model and then basically encourage the model when it spells out its reasoning, which can be, you know, in the English language, sometimes this weird nuances where sometimes

• 35:21Speaker 01

  a model will like start reasoning in Chinese or something and then switch back to English. There's a different ways you can incentivize the the model to produce like legible or like shorter or longer reasoning traces. And sometimes this improves performance or degrades performance. And so then we started creating these like reasoning models because they were trained with complex reasoning traces from experts. And the first generation of these systems were demonstrated to work super well on math, physics, like what was overall called like verifiable domains, which is to say like, this is a mathematical proof and we know the answer and I can check that the answer is correct or not versus like poetry where there's

• 36:04Speaker 01

  no correct answer. It's very like subjective and that's unverifiable. So O1 preview did like super well on math and verifiable domains. And then they had the whole like DeepSeek work which we described, which was very similar, except that model, in order to give it reasoning capabilities, was trained to encourage reasoning explanations, which were in cogent English. And then there's like other variations of this over the ensuing months. And overall, we've just seen like these kinds of systems perform better on math benchmarks, chemistry benchmarks, physics, real world, like knowledge tasks and like legal or banking, et cetera.

• 36:43Demetri Kofinas?

  This is really fascinating because I'm thinking right now about my own experience using ChatGPT and moving especially from four, five to the most recent model, ChatGPT five. Because I saw an actual decline in efficacy or in its ability to be useful in my particular use cases of either speaking to it about my upcoming conversations or asking it questions and interrogating its answers. So the, it seems that these systems do very well, as you said, in formal settings where you have definitions that you're confident about and you can structure the question, answer, or the prompts in a sort of step-by-step format.

• 37:28Demetri Kofinas?

  But if you have an open-ended question where you don't, you yourself don't fully understand what you're asking, or if what you're asking about cannot be easily converted from your experience into language and from language into the, the language of the computer understands or that the AI understands, something gets lost. And so would that explain why, for example, I've actually seen ChatGPT get worse at having a conversation with me about my interviews if I give it my transcript, for example?

• 38:01Speaker 01

  Yeah, a few things. I don't think your experience is anomalous. I also see that. ChatGPT in particular, like loves bullet points, loves to be like succinct.

• 38:12Demetri Kofinas?

  Why is that? I love that. Thank you. You're a hundred percent right. I mean, it's also bad at formulating coherent sentences. And the other thing that it does, Nathan, you know, when you speak to somebody who isn't especially intelligent, but is feels insecure about themselves, they're being interviewed maybe on the street, they're just an everyday person, they're being interviewed about a topic and they want to appear intelligent, they will talk in a certain way. They'll use words that are unnecessary because they sound smart. They don't economize their language, they over explain. That's what it does. That's what it feels like I'm reading when I'm reading its responses.

• 38:47Speaker 01

  Yeah, it's like pseudo-intellectualism.

• 38:49Demetri Kofinas?

  And it's very superficial. Like when you begin to poke into it, you can see it's a superficial level of understanding. So, go ahead. I just wanted to throw that in there as you respond.

• 38:58Speaker 01

  Yeah, I see in my own experience that 4O was much better at writing than 5, like 5 for the reasons we said are like too succinct. It's like bullet point notes, like shorthand. Yeah, yeah. So the thing to, I think, appreciate with building these systems is, you know, we talk about like a model, but I mean, these systems are just a gargantuan amount of work. Like to get a sense, if you go and see the Gemini 2, Gemini 2.5 paper from Google DeepMind, it has like a thousand authors or something, or maybe more. And to eventually get like the fully baked behavior that we're seeing in our applications,

• 39:39Speaker 01

  there's like probably hundreds of different signals or maybe even more, like little tweaks of behaviors across every single domain that any human being who has access to this tool might ask. And like little rewards or ways that, you know, the team that's developing the model can grade whether the system does well for their specific case. And then you're throwing like all this data into a soup, all these evaluations into a soup, and like hoping that every new model generation doesn't regress on some capabilities that people care about and while, you know, improving on other capabilities people care about. I think it's just generally magical that like the thing works as well as it does, given like all these competing directions in which the model is being pulled.

• 40:24Speaker 01

  And at the end of the day, like, there is not really like a free lunch with, you know, being good at everything while not like regressing in one direction or another. And so this is another reason why I think, you know, every time you hit like update in the chat GPT app, you might get a slightly different experience. And, you know, we might say right now, you know, in October 2025, it's not good at writing like we used to, but then in a few months, like a new update comes and then we love it again. So these are like dynamical, like living systems and where it's a game of trade offs.

• 40:55Demetri Kofinas?

  And it's also important to point out that it's only been three years since chat GPT launched its, and not even three years since it launched that model that we talked about as being sort of that breakthrough moment where it hit the public zeitgeist.

• 41:08Speaker 01

  Yeah.

• 41:09Demetri Kofinas?

  And that New York Times reporter was having a conversation with Sydney. Remember that? That was, I mean, it's so used. Yeah, it's amazing how amenable human beings are to normalizing.

• 41:22Speaker 01

  Yeah.

• 41:22Demetri Kofinas?

  Uh, conditions that would have been totally abnormal just a few years ago.

• 41:27Speaker 01

  For sure. It's just like this human thing of like, we uh, we sort of ignore the 99% that works and we focus on the 1% that's really crappy.

• 41:36Demetri Kofinas?

  Right, and then adjust our expectations accordingly so that we become frustrated and annoyed that the system is unable to perform at the level that we expect. It reminds me of this Louis C K joke where he described being on an airplane on his phone and the wifi wouldn't work and he was like this stupid piece of crap and he's like, then he had a realization like this thing, I'm like, I'm getting in this like aluminum tube and there's got giant tires that no one knows how they got the air into those tires and this thing's gonna be flying in the air and I'm gonna end up in Tokyo in like no time and I'm complaining about the wifi. How does chain of thought and inference time compute compared to what we're seeing in terms of visible thinking, which is to say, these systems are actually showing us their reasoning?

• 42:18Demetri Kofinas?

  Is this essentially them just opening up the hood and letting us see what's happening so that we improve the transparency of these systems?

• 42:25Speaker 01

  Yeah, that's the the general idea. So with inference time compute, what we're saying is we're not taking compute and then allocating it to your pre-training, where what's happening with pre-training is that the the weights, like the parameters knobs, if you will, on the neural network are being updated, changed. It's a bit like some very high dimensional pinball machine. And you want to get the ball in the goal and you're adjusting the pins to make sure the ball gets in the in the hole. That's kind of what you're doing with updating weights. You're instead allocating a lot of compute towards the model thinking when you ask it a question. And when you do that, called inference, which is like getting a trained model with fixed weights to answer a question, you're not updating the weights.

• 43:10Speaker 01

  You're just getting it to expend more time to explore different paths in the like search tree of potential responses. And then, you know, model providers, I think originally weren't too keen to expose these reasoning traces, as we call them, because you could effectively like copy those reasoning traces and then use them to train your own system. So really what you see is not the raw reasoning trace, because that would be like insanely long and sometimes very complicated. You're seeing a sort of pre-processed version of it that makes it sometimes more legible for like a normal person using the app. And so the the benefit from like doing this chain of thought and inference time compute

• 43:51Speaker 01

  appears to be the more inference time or the more compute you apply to thinking time, the better performance you get for reasons that we discussed previously. And then the other benefit you get is monitorability, which is like, why is a model behaving a certain way? Why did it bug out? Why did it display behavior that we don't want? And also, can we determine if a model is being hacked or tampered with? Not necessarily because it's saying in this chain of thought, I think I'm being tampered with, but because there are certain patterns of tampering, which can appear statistically within the reasoning trace.

• 44:26Demetri Kofinas?

  So let's take a moment to discuss scaling here, because it's something we touched on earlier, and I want to make sure we cover it before we move to the second hour. As I mentioned, there was a sort of broad narrative in the last few years that the path to artificial general intelligence or superintelligence was through physically scaling these systems by throwing more compute and data at them. And eventually that magic would produce superintelligence. It now seems that we're moving or beginning to move from that kind of scaling compute narrative to one of inference time scaling, where the systems spend more time thinking through the problem in order to provide better solutions,

• 45:10Demetri Kofinas?

  which I think is also, if not the innovation of DeepSeek, it was something that DeepSeek relied on to make the gains that it did, right? So walk me through where you think we are in terms of how we're going to scale these systems. And also, like, what is the general consensus in the industry? Is there no longer a consensus and we're kind of somewhere in the middle? And how is that informing the capex spending of these companies, of these foundation models like OpenAI or Google, et cetera?

• 45:38Speaker 01

  Yeah, so you're right. That's the narrative of scaling, which is, you know, more compute, more data, bigger models, over time builds better performance. There was a time late last year, early this year, where a moniker was bandied around on Twitter and in the press along the lines of deep learning is hitting a wall. And what people meant by that was, oh, look, OpenAI has delayed their model release. Oh, look, this latest model release from other vendor is not as impressive as what the prior model release was. Therefore, there's less juice to squeeze. Therefore, capabilities are tapering off. I think the challenge with that is, one, related to your point around humans just get used to things

• 46:20Speaker 01

  very fast. And so like, our bar for expectations is now higher because we've witnessed like so many magical capabilities. The second thing is, we're generally as a field, I think, moving away from like, static benchmarks, which were created for academic purposes. For example, historically categorizing images in one of thousand different categories, and then a model that did really well on that was deemed to be good at computer vision, to now saying, well, a model that actually does useful work, that, for example, McKinsey or Goldman Sachs are buying and spending a lot of money on and saving human time, that's the eval benchmark that matters.

• 47:01Speaker 01

  And I'd say the other part is, yes, there are like new axes now to do scaling. Like it used to be on pre-training originally, like have a bigger and bigger corpus of unlabeled data like the internet and maybe internal corpuses to do training, then throw in image, video, audio, all encoded in the same way as text, sequence by sequence. And then building gigantic GPU clusters. And then on top of that, throwing reinforcement learning when the domain in which you're working is verifiable, which generally I think teaches logic, which can be extrapolated usefully in other domains, which are not verifiable.

• 47:42Speaker 01

  And so, yeah, there's been a different number of axes and different like ingredients that you throw in the soup, which has improved capabilities. And I think the consensus in the community is there's definitely more to squeeze. This is even excluding modalities like synthetic data where, you know, a model is now like very smart and can generate very smart answers that it can use itself to self-improve, basically. And then there's like a whole axis of like kind of new architectures, which are being worked on concurrently. But in general, the bets like, yeah, more scale along what we discussed will bring about better capabilities. And, you know, if you follow people from the major labs, like there's general belief that this recipe will yield some form of superintelligence.

• 48:26Speaker 01

  Whether it'll be across every single task is another question.

• 48:30Demetri Kofinas?

  I'm curious what you think. First of all, do you think that we've reached a plateau? Do you think we've reached a plateau in terms of scaling the systems based on throwing more compute and data at them? Do you think that we can continue to find sort of, quote, you know, software hacks that help scale the system or that help continue to scale these systems? Do we need a new architecture? I mean, where do you fall on this in terms of what is going to push the frontier here?

• 48:58Speaker 01

  I'm, I'm positive on progress because.

• 49:03Speaker 01

  It has only been, you know, two and a bit years since ChatGPT. And it's important to know that basically, everybody who worked in AI up until ChatGPT were AI people, AI research people. And so it's quite magical that those individuals managed to produce an artifact that's this useful. And then after that, AI became the like new, new thing and maybe the only new, new thing, which has then subsumed all the energy of like every other industry into it. And so, you know, you've seen people like the individual who, you know, invented WebRTC, which like allows the browser to do so many things, has moved into OpenAI. You know, people who've like done crazy optimizations to run like crazy large cloud

• 49:44Speaker 01

  services and like basically built the internet and bidding exchanges and gambling exchanges, like all these systems that need to work very quickly and at low cost have moved into AI. You have like all the chip people who are working on, you know, non-AI things moving to AI. So I think as you get like all this expertise moving into this field and bringing basically their capabilities that they've learned in different disciplines, like that are generally around robustness, cost efficiency, speed, AI is going to be a beneficiary of this. And when you apply that to the three axes we discussed around scaling, I think it's a fair bet that the systems we have today are probably the worst that we'll see.

• 50:26Demetri Kofinas?

  So Nathan, I'm going to move us to the second hour. I want to discuss this move towards open models and what that means, both in geopolitical terms, because China is really leading the field in that area. I'm also curious to dig into why exactly they decided to do that, what the strategy is there. There are obviously huge implications commercially for investors. One, if we move away from the, the current scaling paradigm, but also if in fact, the push eventually leads to more and more of these foundation models going from closed to open, what that means in terms of where the profit opportunities are, are they further up the stack? And is that why we're also seeing companies like OpenAI begin to commercialize new products

• 51:11Demetri Kofinas?

  to sit on top of OpenAI's foundation model? Because that's ultimately where the profit is going to accrue and be able to be captured. I'm also curious to hear your thoughts about how this is going to accelerate innovation in science. What are we seeing there? Because I think there's a lot of hope for breakthroughs. And then I mentioned China. There's the broader question of the geopolitics because up until now, companies like OpenAI have very cynically tried to exploit the US-China competition to sort of build regulatory moats around their companies and protect themselves and their profit streams. I'm curious how this plays out geopolitically in your view,

• 51:54Demetri Kofinas?

  and if the US should in fact be concerned about the progress that China is making. And then what does that mean for the debate about AI safety? When I started digging into or learning about this field, it really started with Nick Bostrom's book, Superintelligence, back, I read it in 2015, but I think it was published in 2014. And at that time, that was the most interesting part of this discussion. And that has really just faded from view. And it seems like it's faded from you largely because people have decided that it's just too hard, that the game theory doesn't support it. So I'm curious to ask you about all of those things in the second hour, Nathan. For anyone who is new to the program, Hidden Forces is listener-supported. We don't accept advertisers or commercial

• 52:39Demetri Kofinas?

  sponsors. The entire show is funded from top to bottom by listeners like you. If you want access to the second hour of today's conversation with Nathan, head over to hiddenforces.io/subscribe and sign up to one of our three content tiers. All subscribers gain access to our premium feed, which you can use to listen to the rest of today's conversation on your mobile device, using your favorite podcast app, just like you're listening to this episode right now. Nathan, stick around. We're going to move the second hour of our conversation onto the premium feed.

• 53:12Demetri Kofinas?

  If you want to listen in on the rest of today's conversation, head over to hiddenforces.io/subscribe and join our premium feed. If you want to join in on the conversation and become a member of the Hidden Forces Genius Community, you can also do that through our subscriber page. Today's episode was produced by me and edited by Stylianos Nicolaou. For more episodes, you can check out our website at hiddenforces.io. You can follow me on Twitter @Kofinas and you can email me at [email protected]. As always, thanks for listening. We'll see you next time.