20VC: "Anti-Data Centres is a Chinese Psyop" | How Many Planned Data Centers Will Actually Get Built? | Is Energy AI's Biggest Bottleneck? With Thomas Sohmers, Co-Founder @ Positron

Thomas Sohmers is the co-founder and chairman of Positron AI, building chips to make running AI dramatically cheaper and more energy-efficient. https://www.positron.ai/aboutThe company recently announced an $875 million Series C at a $5 billion valuation, backed by investors including Gavin Baker's Atreides Management, NEA, Valor Equity Partners and Netscape co-founder Jim Clark. AGENDA: 04:30 Why Does AI Inference Need Different Hardware from Training? 10:25 What Is Nobody Telling You About AI's Token Economics? 13:10 Should We Really Slow Down the AI Frontier? 16:55 What Happens If America Slows Down—and China Doesn't? 19:30 Should We Restrict China's Access to AI Chips? 20:15 Why Aren't Zuckerberg and Jensen Backing an AI Slowdown? 21:25 Are We Being Misled About Data Centers? 23:25 Can the West Beat China While Drowning in Regulation? 26:20 How Many Planned Data Centers Will Actually Get Built? 27:15 Data Centers in Space: Real Opportunity or Elon Hype? 28:10 Is Energy AI's Biggest Bottleneck? 29:55 Could the Debt Boom Derail AI? 31:25 What Is KV Caching—and Why Does It Change AI Economics? 35:25 Does Compressing AI's Memory Make It Less Intelligent? 41:05 How Do We Solve AI's Exploding Memory Demands? 43:45 Will Every Company Own Its Own AI Model? 47:10 Why Bet on Ever-Bigger Models? 48:20 If We Already Have AGI, What Comes Next? 53:25 What Happens When Every AI Lab Builds Its Own Chips? 57:30 If DeepSeek Can Slash Costs with Software, Why Build New Hardware? 59:50 How Cheap Will AI Tokens Be by 2028? 1:03:15 What Would Be the First Warning Sign of an AI Bust? 1:03:55 Could Open Models Break OpenAI and Anthropic's Growth? 1:05:55 Could Mercor and Surge Become $200 Billion Companies?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I would say I'm overall opposed to the pace of the frontier direction going in. The scariest thing to me on the political spectrum and the way all of this is being treated is that it's now become a almost unifying issue on left and right about being anti-data centers. I think that is almost entirely a Chinese psyop. A single In-N-Out uses more water than the largest data centers in the United States.

• 0:22Harry Stebbings?

  This is 20VC with me, Harry Stebbings. Now, we have probably one of the most important times in history for technology. We have the biggest model providers saying we need to pace the frontier. But what does that actually mean in reality? How possible is it? What does it mean for the threat from China? What does it mean for the infrastructure layer moving forwards? We have a true expert of the space on the show today in the former Thomas Somas. He's the co-founder and chairman at Positron AI. They just raised an $875 million Series C at a $5 billion valuation. They've got some of the best investors in the business, including the one and only Gavin Baker at Atreides, and many more great names. Thomas did not hold back in this episode, and it's this beautiful combination

• 1:04Harry Stebbings?

  of incredible education on the infrastructure that powers this economy for AI, and then also, I don't know how to say it, but analytical gossip would be a more intellectual way of saying incredible discussion about what we can expect in the next few months from the biggest players in this space. But before we dive into the show today, the best model for your application might not exist yet. The most ambitious AI teams are training open models to beat the frontier in their domain. Now, on November 3 in San Francisco, Forge by Fireworks brings those teams together. People like Jensen Huang, CEO of Nvidia, Michelle Kauta, President and Head of AI at Replit, Lin Kao, CEO of Fireworks, and more to hear how AI leaders are taking control

• 1:46Harry Stebbings?

  of their differentiation, margins, and roadmap by owning their intelligence. Learn how they're building it with inference, training, and intelligent routing. Forge is free to attend, but space is limited. Apply today at fireworks.ai forward slash forge. While Fireworks AI powers product intelligence, Asana keeps the work moving. Most companies have tried AI. Most aren't seeing results. Not because AI doesn't work, it's because AI hasn't reached the workflows yet. That's the gap Asana is built to close. Asana is the operating system for human agent teams, your easy button for AI productivity across every team. Ready-to-go AI teammates pre-built for marketing, ops, and IT. No-promant engineering, no setup.

• 2:31Harry Stebbings?

  They show up where the work is happening, already on-boarded, in your workflows, ready to deliver. With Asana, your whole company can work on the same plan towards the same goal, whether you're a team of 10 or a team of 10,000. Asana, where humans and agents workflow together. Try it at asana.com. That's A-S-A-N-A dot com. While Asana organizes the work, Superhuman speeds up the day. Can I be honest with you for a second? Here's what my week actually looks like. Back-to-back meetings all day, and the real pressure builds up in the gaps between them. I'll come out of six founder meetings with a whole stack of follow-ups waiting, and my inbox is hundreds of messages. And somewhere in there is the one thing that really matters, which I always tell my mother is her.

• 3:15Harry Stebbings?

  I've tried AI tools for this, but they always lived in another tab. I'd have to stop, go find it, paste in the context, write the perfect prompt. It was just enough friction that it never really stuck. Superhuman Go, from the makers of Grammarly, totally different. It's right there where you need it, when you need it, in the doc when I'm reading, the email I'm writing. When a thread gets dense, I highlighted and asked Go to pull out what matters without losing my place. And after a run of back-to-backs, I asked what was actually decided, and Go turns it into clear action items or drafts, the reply I need to send. It's already up to speed, and I'm always the one in control. Try Superhuman Go from the makers of Grammarly and find out more at superhuman.com. You have now arrived at your destination.

• 4:01Harry Stebbings?

  Thomas, I am so excited for this, dude. I said to you just before this that I think there are some big questions that the world doesn't know, or things that they know, that I think we're going to correct today. So thank you so much for joining me.

• 4:13Speaker 00

  Yeah, great to be here. Thanks, Harry.

• 4:14Harry Stebbings?

  Can we just have a brief description of what Positron is and where does it sit in the stack?

• 4:20Speaker 00

  Yeah, so Positron's a fabulous semiconductor startup that's building hardware. So really, everything from the chips, the software directly running on top of that, all the way up to the full systems and rack-scale deployments to power generative AI inference. So effectively, everything in the hardware and sort of the low-level direct talking-to-hardware software stack that powers all of the applications everyone in the world is excited about right now. So everything from the likes of ChatGPT, Claude, et cetera.

• 4:50Harry Stebbings?

  How does the infrastructure stack required for inference, what you're working on, change compared to training?

• 4:58Speaker 00

  So training, I would say, from the underlying compute level fundamentally is a compute-bound problem. So it's a workload that the more flops that you have, and if you look at, be it from the regulatory and some of the export control frameworks are heavily focused on the just flops required. So how many floating-point operations per second can be done? And more or less, the amazing thing that the scaling laws of the past decade have shown is that the more parameters you add to a network and the more flops you dedicate to that during training, the better that that model is going to become. The big difference with inference and so the deployment of those models is the fact that for the actual math and the steps that you're doing

• 5:40Speaker 00

  is about half of what you're doing during training in terms of the steps not shouldn't be thought of as like the actual compute involved. But what it turns out to be is that that forward pass, that inference portion of it is heavily, heavily memory-bound due to the fact that basically for every single token that's generated, every little bit of output, that requires going through the weights, the parameters. You could kind of, from a biological sense, think of the neurons. You have to read the values of that for every single individual token. And so the way that that's, I think, very interesting about this. And I don't know if it says anything about the value or kind of actually saying that the inference is somehow. I don't want to say more important because, of course, you have to train.

• 6:25Speaker 00

  But fundamentally, when you're training, you already have the corpus. You already have all of the training data. And so with all that data, you can massively paralyze the token inputs, all of the sequences of words and sentences, paragraphs, et cetera, that are going into it. So that's something you can just crush through with a bunch of compute. But when you're inferring, because that's actually generative, you don't know what the token is five words down the line. And so you have to generate each and every one auto-aggressively or in order without foresight. And so that becomes a hugely memory-bound problem that can't just be massively paralyzed like training.

• 7:03Harry Stebbings?

  So I totally get that in terms of the shift from compute-bound to memory-bound. Is that what people mean when they say about the memory wall with regards to what you're doing?

• 7:12Speaker 00

  Partially, I mean, the memory wall as a phrase has been around for a long time before the hype around AI. And really, what it's come down to is if you look at the past 50, 60 years of computing, we've been able to have Moore's law, giving us more transistors per square millimeter silicon consistently. And while that has been able to result in greater raw compute flops, et cetera, the improvement of the memory technology has not kept up at the same rate. So roughly speaking, between 2014, just very early innings of the new AI era, till 2024, you had about 120x improvement in the flops of GPUs.

• 7:54Speaker 00

  So a single Nvidia GPU had about 120-fold improvement in flops. And that's what enabled a whole lot of the improvements over that decade. The improvement in memory bandwidth is only 17x. So I would say like the real embodiment of this is we had massive improvement on a per-device basis of the flops and then a whole bunch of elements on the periphery of improving the connectivity, et cetera, et cetera. But just like the ratio of the compute to memory bandwidth had this divergence. And so you had cases where if a problem was memory-bound and you couldn't just scale the compute linearly with that, you were getting more and more memory-bound as the decade progressed.

• 8:34Harry Stebbings?

  Why was there such a misalignment in the progression between the two? One's 100x, one's 17x. Why is that the case?

• 8:42Speaker 00

  Yeah, well, it comes down to a lot of technical implementation details, like the fact that if you if you look at the lowest level, the type of memory that is used on the silicon itself is called SRAM, static RAM. And SRAM is made out of, you know, six transistors with a bit line and word line, some other control logic around it. But that SRAM cell has not scaled in terms of the sizing of that with Moore's law over the past about fifteen years. So they have grown or shrunk, I should say, but much slower than just a group of transistors that you'll use for for other purposes. And I would say that there's just been a lot more architectural advancements that could happen on the compute side.

• 9:23Speaker 00

  While an S six T SRAM more or less has not changed in thirty or forty years from like an architectural, you know, primitive perspective. And so, you know, that's on the input side of raw technical capabilities on fabrication, et cetera, have not been able to improve. But I would also say that there wasn't the right motivations for most of that that decade. So with convolutional neural networks, so things that powered like Alex nets, which, you know, really launched the deep learning revolution in twenty twelve. And then ResNet and, you know, all of the, I would say, the advancements during the twenty teens was in the realm of machine learning models that were fundamentally compute bound. You could just throw more and more flops at CNNs and get better results.

• 10:05Speaker 00

  And you didn't really need all that much, be it memory capacity or memory bandwidth. But it was really with the transformer. And even though the attention is all you need paper came out in twenty seventeen. I would say it did not really get the attention, you know, pun intended, uh, it deserved until, uh, twenty twenty. Well, uh, GPT one and GPT two came out prior to that twenty eighteen, twenty nineteen. But it was really GPT three showing that, okay, you go from a billion ish parameter up to a hundred and seventy five billion parameters. And you actually get this massive improvement in capability. And that's really where I would say the transformer revolution started. And most people didn't really catch on to that until the end of twenty twenty two, when, uh, chat GPT came out.

• 10:48Harry Stebbings?

  When you look at token economics and token efficiency today, what does like no one know or talk about that you think should be much more front and center?

• 10:56Speaker 00

  I do find it compared to a year or two ago, there are now different prices listed for cash versus uncashed tokens. But I don't think people realize how any providers that charge the same amount, even with a lot of people's cash prices, how high margin that is. It's like insane. You you make all of your money on selling cashed input and output tokens.

• 11:20Harry Stebbings?

  Why is that? Sorry, just so I understand that.

• 11:22Speaker 00

  Oh, because for like when we discussed earlier, that both processing a cash token is essentially free. It's it's one one thousandth of the cost, you know, I'm, you know, order of magnitude of actually having to recompute and and generate that token. So there there is so much you you can juice out of selling those those, uh, cash tokens. And basically, all the providers, they charge you to cash a token. They charge a higher rate than just, you know, the normal processing fee for like an input token. And then they charge you a lower rate when you read from that. And it's great when you're paying that lower rate, but they're making obscene margin on that cash read.

• 12:02Speaker 00

  And there's a reason why, you know, Anthropic is being is reported to have, you know, eighty points of gross margin right now on, you know, API business.

• 12:09Harry Stebbings?

  Were you surprised by that eighty points?

• 12:11Speaker 00

  Not really. I'm impressed by eighty points of margin in basically any industry. It's it's difficult to get that margin. And great thing about capitalism is those margins will compress with competition. So I'm I'm confident and happy for that, even though those people are, you know, theoretically my customers and my margin is sort of based on their margin. But I care more about, you know, a healthy ecosystem long term. It it's more surprising to me how many people still today think that these are horribly unprofitable businesses and that the whole market's going to zero. It it's absurd to me that the meme of OpenAI, Anthropic, et cetera, are just burning cash. And eventually they'll run out of cash that they can burn. Like if they stop training, they'd be massively profitable overnight.

• 12:53Speaker 00

  And there's a ton of other levers that they have without pacing the frontier as as Dario just had in his essay. I I mean, I jokingly think that, you know, a little bit of the pacing the frontier discussion is, oh, this is a great way to to reduce costs, uh, ahead of IPO. But I I don't think Anthropic or anyone needs to do that. I think they're they're amazingly profitable businesses with their scaling rates and.

• 13:16Harry Stebbings?

  And it would reduce costs, just so I understand, because they would spend less on training because they would be slowing down the speed.

• 13:22Speaker 00

  Yeah.

• 13:22Harry Stebbings?

  Right?

• 13:23Speaker 00

  Yeah, I don't think that's actually the intention or anything, but yes, that that's the bulk of their costs.

• 13:27Harry Stebbings?

  How did you analyze the pacing the frontier? You brought it up. How did you analyze it?

• 13:32Speaker 00

  I have mixed feelings on on the safety topic. I I am a human, you know, would like to, uh, live old age. And and more more so than that, um, have have humanity, uh, continue to the stars and beyond. But I believe much more in the ability for this technology to revolutionize every part of humanity in in a positive way. I do worry that pacing in a lot of the ways that's being talked about, not necessarily how it will be implemented, has two big risks. One, a major pause and sort of that playing into, for lack of a better word, the blood eight sentiment that exists.

• 14:11Speaker 00

  And that by pushing for pause, it's actually giving ammunition, giving better basis for those that actually just want to stop the technology over completely. And I I see that as a major risk for humanity. The second piece is my actual P do, my, the thing that I'm most worried about of any, you know, AI outcomes. Is that technology and capability being concentrated to relatively few people? And having a lot of what's being discussed from a regulatory framework and limitations on on technology, et cetera. I think is like the modern, you know, road to serfdom. It's like the concentration of technological capability, making legal to do matrix multiplications.

• 14:53Speaker 00

  Is like the thing that will set us back to, you know, pre, uh, not just industrial revolution. It's like pre enlightenment capabilities, like that. That is like the biggest attack on classical, like liberal freedom concepts that I I can I can think of. Because while I do think that the vast majority of tokens are going to be produced by the big players. If the technology itself is restricted to just those, then they're going to be the new lords and and kings and everyone else's is back to serfs.

• 15:23Harry Stebbings?

  The greatest of respects is it not just lip service. Great, we'll stick meter in the corner. They can do their compliance. And then we can IPO. Sam can have a reason not to IPO because his numbers aren't as good as Anthropic's. It plays into both our desires.

• 15:38Speaker 00

  And and Elon wants time to catch up as well. And.

• 15:41Harry Stebbings?

  So it plays into everyone.

• 15:43Speaker 00

  Completely agree. And I I think that is the biggest internal reason for everyone other than Dario. Dario and and I would say the vast majority of people in Thropic are true believers, both in all of the promise and capabilities of the technology and the risks. And if I were in their shoes, I would also be taking the the massive amount of responsibility for that. But I will say, like, yeah, there there's a lot of strategic reasons of saying, okay, by having these auditors, et cetera, that removes some potential responsibility, culpability from like legal perspectives, et cetera. The risk that I just don't think that any of, I think this is a problem with a lot of very smart people, especially when they've amassed large wealth and power, et cetera. Is they think that they're going to be able to keep that.

• 16:25Speaker 00

  And that the scariest thing, kind of my my point, like the centralization of technology, like if it gets concentrated with companies, governments, et cetera. Is you've got people that think that they're the smartest people in the room, not realize that they are not going to be the ones to actually control it when they put these measures in place. Like Dario, basically, on one hand, verbally begging for government governments to take over Anthropic. He's in part saying that because he doesn't think it will actually happen. I would love to see his reaction if and when that actually happens and he realizes, oh, shit. I thought that if I was begging for regulations, they would then make me the regulator. And when that doesn't happen and it just becomes a bureaucracy that halts all progress.

• 17:08Speaker 00

  And the capabilities that currently exist basically get squandered to select bureaucrats. That's the worst outcome I can imagine.

• 17:16Harry Stebbings?

  When you consider the advancements that China are making, especially with their open ecosystem, but they are an incredibly talented ecosystem right now, moving forwards. If we pace and they don't, what happens then?

• 17:30Speaker 00

  I guess when I said that the worst possible outcome, I wasn't counting the Terminator outcome and I wasn't counting that. So I think, of course, everyone can agree Terminator or similar or similar is is very bad, but I think is extremely low probability. And and just, I'm not a believer in that doom scenario. For the vast majority of people would result in the same level of serfdom that I worry about with the scenario that I I described. Would be significantly worse for some number of people in a Chinese CCP controlled, you know, super intelligent AI scenario. You know, on one hand, their strategic angle right now is have technology proliferate through open source, et cetera.

• 18:12Speaker 00

  I think as soon as they get into pole position, the ladder gets pulled up with them in some way. I don't think they actually want the technology to be easily accessible to everyone. Now, I don't know if they will decide that it's okay if the rest of the world has some access to the technology. But they definitely will not let the billion people that are not CCP party members, uh, benefit equally from from the technology.

• 18:35Harry Stebbings?

  So just so I understand, do you agree with it? Because to me, I just didn't get it. You can't pace the frontier unless the global AI community paces the frontier, and I don't see Putin signing up.

• 18:46Speaker 00

  Agreed. And I, I think this is a little bit the same naivety that, that I described by these company leaders and, and in general, people in the Western world thinking, oh, we're so great, we're so advanced, so far ahead that we can't get caught up to. I mean, on paper, is the US the greatest military force in the world? Yes. If we had to all of a sudden have a drone incursion, the same level of, you know, what's happening in Ukraine, Russia, you know, coming up from Mexico. And if you take Mexico, I'd just say that they developed, you know, very naive drone technology and et cetera, like on the level of what's happening in Russia, Ukraine and Iran.

• 19:25Speaker 00

  How would we respond to that as a country if we had that coming up across our border? It doesn't matter. Our amazing military might. We built our military to fight the last war. And I think geopolitically, our thinking is, oh, we're the big dog still. And that when it comes to AI technology, there's not the acceptance that export controls and all of the other elements that theoretically would allow us to pace and have people keep paces behind us just aren't good long-term solutions.

• 19:51Harry Stebbings?

  Do you think we should have export controls?

• 19:53Speaker 00

  I am very much a strong believer in free trade and, and free exchange of ideas. The, the exception to that is a little bit, I think China has been a free rider of all of the benefits of, you know, a liberal free trade order for the rest of the world. Will they get to keep everything closed off? I am very, very happy and, and think that any government societies, people that want to embrace free exchange of ideas and trade and everything else. We should have a very vibrant economy and ecosystem, but totalitarian regimes should not be able to participate with that, especially in the case where they get all of the benefits of that and get to export themselves, you know, things that make them better able to have that totalitarian system keep up.

• 20:37Harry Stebbings?

  Can I ask, we, we mentioned, you know, the pacing the frontier and, you know, the different people who supported it. You had Zuck and Jensen say nothing. Well, Zuck actually come out in opposition to it, saying that we should continue as planned. What should we take from those two seemingly silenced and opposing it?

• 20:53Speaker 00

  I, I, I would say, based on my overall beliefs right now, as probably evidenced by the, the conversation so far, I would say I'm overall opposed to, you know, the pace of the frontier direction it's going in. And so I appreciate anyone that is adding to the discussion that is, I think, being realist about the benefits and risks. But you always have to take that with a grain of salt of what are the motives of, of anyone that's, that's discussing in it? And I would say I probably appreciate Zuck or Dario's comments infinitely more than a random politician. And not just random, the, the quote, quote, leading politicians that don't actually understand the technology.

• 21:35Speaker 00

  The scariest thing to me on, on the political spectrum and the way all of this being treated is that it's now become a almost unifying issue on left and right about being anti-data centers. I think that is almost entirely a Chinese psyop.

• 21:48Harry Stebbings?

  Can I ask, why is it a Chinese psyop being anti-data centers? Cause like it does increase, I, I, I'm totally with you in the benefits of them. And, you know, Gavin Baker said how they're the greatest economic kind of needle mover for large parts of the country. I guess they see increased electricity prices, increased water prices and ugly data centers in their backyard. Why is it a Chinese setup? What, what am I not seeing?

• 22:09Speaker 00

  Well, just on the ugly and all that, I, I totally am a supporter of a, uh, we, we, we need to have beautification campaigns and, and really turn them into, uh, centerpieces of, of our society. I think of thousands of years in that from now, you know, a future history looks and they should see these massive data centers, the, the really, really massive, impressive ones. Should have, uh, you know, be like the great pyramids or, or something like, uh, uh, they, we, we need to dress them up to be the, the world wonders that they are technologically. The thing from water usage and like the amount of power they consume, et cetera, like so much of the early information that went out by unsophisticated writers

• 22:47Speaker 00

  or things that are just patently false, like, you know, a single In-N-Out uses, you know, more, more water than, you know, the largest data centers in the United States. And like it's golf courses are orders of magnitude more like it's, it, the meat, like these are closed loop, you know, liquid coolant systems that, that, you know, you don't even want to use water in a lot of these cases even. So from the Chinese psyop perspective, they're not, to your, your point, they're not pacing the frontier. They're adding, you know, gigawatts of, uh, you know, new electricity generation capacity, most of it being dirty. They're building massive new data centers horribly displacing people. Like it just irks me so much that we have the freedom in the Western world

• 23:27Speaker 00

  to criticize companies, governments, everything based on false information. And I love, I love the freedom elements of that. But it is a strategic disadvantage when China can just say, yeah, we're going to just bulldoze all these people's homes and, and do rolling blackouts wherever in order to, uh, serve the greater good of new, new training capacity.

• 23:49Harry Stebbings?

  I mean this with the greatest of respects, but I don't understand how anyone thinks the US or Europe can beat China when they have no regulatory or policy restrictions. And I mean, the UK, you can't, you know, put up a paper airplane without getting a permit. So like, we're, we're fucked. But you are getting there and you're getting, you're becoming a European state in terms of the regulation and policy requirements. Am I wrong? Am I being overly negative? I, I didn't get it.

• 24:15Speaker 00

  I, I, you're right. Um, I think the greatest advantage the US has in that regard is that there's still a lot of land, a lot of places that do not have all the same levels of restrictions. I don't agree with a lot of things of most, you know, administrations of my lifetime. But the, the current administration gets attacked for saying like they're destroying our, our environments and, and, you know, destroying national parks, et cetera. That vast, vast majority, 90 plus percent, I don't know the exact numbers of national federal land is just open, empty desert in, in the West that is not part of a national park or anything. And the fact that there are so many restrictions to utilizing, you know, BLM land for building data centers where it's literally hundreds of miles from, from any, you know, populated area.

• 25:00Speaker 00

  My great state of Nevada has plentiful geothermal, solar, all these green energy technologies. And we could, you know, build, you know, nuclear and other things in the middle of the desert where it won't impact anyone. And that there's restrictions to that is completely absurd to me. And I will say, there, there has been some political will and, and push to solve these things. But literally just in the past year, you have, you know, Republican governors and, and other politicians that at least had part of their platform to be pro-growth and all of these things backing away because they see from their own, you know, political base being anti-data center based on completely false premises. And, and one of the points I want to go back to that you brought up was like that people would have higher electricity costs.

• 25:46Speaker 00

  Like this is the most basic supply and demand. If we increase generation capacity and no one's saying we want to be taking energy from what's reserved for people's homes. Like one of the regulatory problems I see is, is power companies have to have this offer of, of energy availability that is baked into the cost and, and capabilities for everyone. There, there is absolutely zero cases where a data center could be potentially pulling power from anything that's already been allocated. So that's just impossibility. And all these data centers that are getting built right now are coming with generation capacity that covers their own use and beyond that. And we're just not allowing them to hook up to the grid, uh, where they could actually be lowering the prices for everyone.

• 26:27Speaker 00

  And then you've got people on the power company side, like they're lobbying against new generation capacity because that will actually, you know, market forces will, more, more capacity will, will decrease prices, which would be good for consumers. So it's a very wonky market.

• 26:42Harry Stebbings?

  What percentage of data centers that are planned will be completed, do you think?

• 26:48Speaker 00

  From the major providers, I think, I would say that the capacity that they have planned, they may be in different locations. I mean, you, you've had some local communities that have successfully stopped facilities going in there, but then those data centers just move. I don't think a year ago, the major data center builders and operators were thinking that the political problems were as bad as they were. And so there is a lot more effort being put into education in, in those communities now, which I think will, you know, turn the tide a bit. But I mean, it's also just going to mean, mean that those data centers move to locales that aren't going to have those problems as well.

• 27:24Speaker 00

  And like I said, we've got large tracts of land that can support it. So I, I, I'm not too worried that it's going to be like an existential threat in capacity build out. And then of course there's space if Elon's successful.

• 27:37Harry Stebbings?

  Do you believe that space is a viable alternative truly, or is it conference talk and lip service to justify a market cap?

• 27:44Speaker 00

  I think something can start as one thing can turn into something else. Um, I, I would never, ever bet against Elon. I, I mean, I, I primarily bet for Elon. If you asked me a year ago, I just would not have thought that there would be a good reason for it in the near term because it's going to be cheaper, easier, et cetera, to build on land. I also think there's great alternative technologies. A company we're partnered with and I'm good friends with, uh, the, the CEO, um, is, is a company called Pantholassa. That's building ocean-based data centers, basically a very interesting pumped hydro solution in the middle of the ocean. So there are alternatives that don't require going to space, I think, long-term. Part of the, the reason I'm, I'm long-term big believer in space data centers is I just think we're going to need to have a space economy

• 28:28Speaker 00

  for humanity to live up to its long-term potential.

• 28:31Harry Stebbings?

  Love that. Totally agree on ever bet against Elon. If we think about like the cost of intelligence being tied to the cost of energy, how should we think about energy as a bottleneck moving forwards? To what extent is it, we mentioned policy and regulation being a core bottleneck. Is energy a bottleneck moving forwards or less than people consider?

• 28:51Speaker 00

  I think there's two pieces to it. I mean, one positron is trying to deliver more compute, more capabilities per watt, per megawatt. And so, you know, sort of on our base case, we can turn what you would have spent 500 megawatts with Nvidia equipment and do that in 100 megawatt. I don't think that's actually going to mean that you're only going to build 100 megawatt facility. You're still going to build the maximum amount of compute that you can. You're just getting more tokens, more intelligence per jewel. And so if I go back to the long-term thinking, you know, I think assuming humanity continues for hundreds, thousands of years, everything turns into an energy problem and you can go back thousands of years and just look at the progression of mankind.

• 29:33Speaker 00

  Fundamentally, that is a perfect track of our ability to produce and use energy, discovery of fire up to, you know, nuclear power plants. The simple tongue-in-cheek answer to your question is all progress is gated by energy. And even if there's energy available, it may not be economical and so it won't be done. So I actually, I would say the bigger limiter than just saying that energy, like our ability to build and produce energy. We've got plenty of technologies and capability to do it. I would say we have got way more economic limitations. It's like, how much debt is the world willing to take on to build out everything over the next couple of years? Ties into energy, it ties into the infrastructure itself, et cetera. So I think economics is a much easier sort of scapegoat to pet.

• 30:16Harry Stebbings?

  I mean, people are already very concerned by the levels of debt being taken out in the debt cycle. Do you think that concerns are justified and do you share them?

• 30:24Speaker 00

  I think we've got a major sovereign debt problem that masks a huge amount of second and third, third order elements in the financial system, just the inflationary consequences of government that that can print, you know, infinite amounts of its own currency. And the fact that we are, as we're already seeing the treasuries and, you know, the greater bond, bond markets, that there is greater and greater perceived risk of the most quote-unquote risk-free asset. I think will trickle down to all elements of the financial system. And so when people worry about Oracle's debt and credit rating, I'm like, I believe in Oracle's business model and ability to execute and do everything a whole lot more than the United States government.

• 31:08Speaker 00

  It's just the United States government can issue its own currency and also has guns and nukes to take tax revenue. So my biggest economic concern there is that there will be a more acute, specific crisis that that arises out of the compounding of national debt, leading to devaluation of the currency that has all of the consequences downstream, rather than like, I'm really not worried about any of the companies in the AI debt stream not hitting their revenue targets. I'd like if the past three, four years have shown, we're accelerating every aspect of these businesses in terms of revenue profits and how they are improving the productivity and value downstream.

• 31:52Harry Stebbings?

  I'm jumping around, but fuck it. When I was doing the research, I was reading about KV caching and compression as part of this. And I was honestly getting lost, but I was intrigued and digging deeper and deeper. And I was like, why did I not know this before? And so I don't think many will know this. What should we know about KV caching? Why is it important? Can you explain it to me a little bit?

• 32:13Speaker 00

  Yeah. So there's always a, I would say, you know, give and take relationship with innovation. I guess one element I'll just have to explain to make all this this clear is the concept of a sequence. I kind of already talked about a token, you know, a token effectively as part of the training process. When any of these big, you know, model apps are developing a new model, they have a vocabulary that they define. So they take their big giant corpus and they do some statistical work to figure out what is the best encoding method to take all of the text in this and break it into chunks that get reused frequently to have things be more efficient.

• 32:51Speaker 00

  And so what you end up doing is if you took a English dictionary, you'll find that there are common prefixes and suffixes and, you know, gripping supports. And if you just try to think of how would I best compress this, if I just had symbols for these prefixes, suffixes, et cetera, compress this into a thing. And basically what, what ends up happening is a token, you know, if you're using chat GPT and you see text streaming out, if that's going particularly slow or you're quite a keen eye, you'll see that it's portions of words that come out at times. Sometimes a full word, sometimes a small fraction of word. And each of those little flashes that you see is a token.

• 33:28Speaker 00

  And roughly speaking, it's a token is equivalent to a half to like 75% of a word on average and large English corpuses. So that's token, a sequence, or you know, the thing that builds up to be in context in a model is the grouping of all those tokens in N order. And what happens when you're running inference, you've given a prompt, you have, what is the capital of France as your input prompt? That is tokenized, you know, that is four, five, six tokens and that that go into the model. And when you do inference, it's going to say the capital of France is Paris and the city of lights, you know, something after that. And so when you have that entire sequence, when transformers originally came out,

• 34:12Speaker 00

  for every token that was generated, you were doing the computation for generating all of those tokens, including the ones that you've already processed. And, you know, clever, I would say, kind of obvious based on, you know, all of the developments in the past of computer history, but wasn't done initially was that, well, you don't actually have to redo the compute of the things that you've already, you know, had as inputs and what you've already generated in this turn. And so the KV cache was born where within the model, there's these two matrices called K and V, keys and values. And those matrices are fully based on the prompt and whatever is generated during a turn.

• 34:52Speaker 00

  And so by actually storing those two matrices, you can avoid having to redo computation at the cost of now having to store this thing in memory. And that's, you know, the simple example, you know, very, very small, hundreds of kilobytes of data. But the thing is that these things grow with, with the sequence length. But the interesting thing is that for the attention mechanism, the compute per token grows quadratically with the sequence length. So you're having to spend more and more compute quadratically. So that's, you know, an exponential curve as sequence length grows. But when you store that as just your K and V, that's just a linear growth. What KV caching does is means that you don't have to do that compute, which gets very expensive very quickly at the expense of needing to store these things.

• 35:36Speaker 00

  Storing that is a complexity in itself because that's a unique KV cache for every single user that you're serving. And it comes questions of how long do you want to keep that for? And how do you manage all of that in a large system?

• 35:49Harry Stebbings?

  What does it mean then when we hear about compression and uncompression of KV caching and potential entropy within the system?

• 35:57Speaker 00

  So there's two different forms of compression, a couple more than that, but the two main ones. So one is quantization. So if you've got each of your values, be it your weights, your KV caches, activations stored in a particular data type. So before the machine learning revolution, you know, most of the world's computation was done in FP32. So you have 32 bits to represent a floating point number and that's broken into mantissa and exponent. Well, it's pretty quickly realized that having 32 bits of precision was super overkill for the things that you're wanting to represent. And, you know, it's both costs more from a storage and computational perspective than lower precision. So we went to FP16, you know, Google developed BF16, you know, a little rejiggering of those bits.

• 36:42Speaker 00

  Went to FP8, now we're at FP4, you know, in popular systems. And so we've been reducing the precision quite quickly. But that does lead to, you know, for lack of a better word, some brain loss when, when these models run just because you are now trying to encode the same information into fewer bits. And so there's been a lot of interesting schemes to say, okay, I'm going to take this group of FP16 values, BF16 values. And I'm going to quantize those. So I'm going to, you know, use a truncate and rounding that that down to let's say int4 values. So now you, you actually saved 75% of your total size of that group of values. You shrunk that down from 16 bits to four bits.

• 37:27Speaker 00

  But just doing that naively will mean that on a lot of benchmark scores, you'll have them get 20, 30% worse. So you get that 75% savings in space, but you kind of lobotomize the model. But, you know, advanced quantization techniques actually say, okay, these 16 values, I'm able to have a, you know, shared bias and a multiplier for it. Let's say for those 16 now int4 or FP4 values, you store one new FP16 value that gets applied to all of those at compute time. So you get a 75% compression on all those values at the cost of now adding to add one new FP16.

• 38:08Speaker 00

  And basically the state of the art here is you're able to get things compressed from, you know, FP16, 16 bits per value down to like four and a half bits per value. And that can be applied to weights, the, you know, actual parameters in model that could be applied to the KV caches. But there's no such thing as free lunch. You'd still have some lobotomy, but thankfully it's kept within like 1% of a unquantized model.

• 38:31Harry Stebbings?

  Is KV caching the hardest element of building that inference infrastructure? Or is it, you name it, latency SLOs or load spikes or anything else that we could come up with? Is that the hardest? Like, what do we not see that we should see?

• 38:48Speaker 00

  You can run a service and do something without having KV caching at all. You're going to economics and performance and everything else wise is going to be much worse. The dark arts and magic with it is the workloads that the industry so far has found the most valuable happen to be very, very highly cacheable. So semi-analysis has their agent X benchmark and, you know, suite of test data based on taking a whole lot of Claude code sessions and having dozens to hundreds of turns in those Claude sessions with subagents and everything else. And what they found is over these massive number of interactions of these, like real traced code generation, agent encoding sessions, about 96% of all the tokens that go through

• 39:33Speaker 00

  these entire sessions are cached. So if you know your workload is going to have this extremely high caching rate, where you're going to be reusing the same tokens again and again, that drastically shifts the importance of how you can retrieve those caches. Because these things get to be very, very large. Like we, you know, have gone into trillions of parameters. So if we just take, you know, the GPT four, you know, got leaked as, you know, 1.8 trillion parameter model. Now, assuming that that is in for quantized and rounding down a little bit, you know, that's 900 gigabytes of data size for the model weights. If we take like the high expectations of like Claude Fable, you know, that's a 10 trillion parameter model, so around five terabytes of model weights.

• 40:17Speaker 00

  But the crazy thing is at these long context lengths for these size models, you have the individual user sessions being in the, let's say, in the 100 gigabyte range. So with just 50 users on your service, the user context, they're just those individual sessions end up being greater than the model weights that you're trying to store. So that's, you know, Claude and OpenAI have a whole lot more than 50 users. And so it becomes a really interesting trade off of, okay, how much of the accelerator memory do you want to dedicate to weights, which you need to process every single token generated? And you want that to be as fast as possible because that sets your SLO, that sets the token latency. But if you don't have their KV caches persistent,

• 41:03Speaker 00

  you're actually losing a huge amount of efficiency because that was work that you didn't have to actually repeat. So it saves you as an operator money more than anything. At some level, you know, having users KV caches be persistent will give some level of speed improvements that the user perceives. But it's mostly an economics thing for the service provider, where if you can return to them and use those tokens again and again, that saves you money as an operator massively.

• 41:29Harry Stebbings?

  Totally get that. It saves us money because we don't have to use as much compute, but then it's harder from a memory challenge perspective. How do you think about the right logical next step then? If you appreciate the importance of saving on compute, but the challenge of memory with KV caching? What's the answer then? That we just have bigger and bigger memory stacks on chip? What does that look like?

• 41:50Speaker 00

  The most common deployed solution and, you know, the vast majority of inferences out there are taking place on GPUs, you know, followed up by TPUs and a couple of other devices. But the most common paradigm today is you've got your GPU accelerator memory that is primarily responsible for holding the weights. And you will keep some number of user sessions on there, the ones that you're actively processing. But the larger group of users has a tiered hierarchy. So you'll have users that were around, say, in the past couple of seconds, but haven't returned, don't have an active request. That's residing in host memory. And let's say that's on the order of anywhere four to 10x more memory on the host than in the accelerators. So you'll be able to store more of those there. And then if someone hasn't been around in a couple of minutes, maybe a couple of hours,

• 42:35Speaker 00

  that's going to be stored in even further away memory. So that could be in NVMe, so flash storage, so a lot slower, but a lot larger capacity on that host. It could be in flash storage on a network attached, you know, drive. And eventually, like, I bet the chat GPT sessions that I had, you know, six months ago, somewhere residing on a, you know, disk, you know, slow SSD or, you know, somewhere in a data center. But it would be dumb for them to use expensive memory to store that. So that tiering is the norm. But that introduces a huge amount of complexity of how do you decide when and where you're going to store something for your massive number of users? I would say our solution, kind of how we're trying to go about it, you know, both from our expectation that model sizes are going to drastically increase,

• 43:19Speaker 00

  the number of users for all these things are going to drastically increase, and the context themselves. Like two, three years ago, the typical context lengths were on the order of 8,000 to like 64,000 tokens. Then it got up to 128, 256. You know, a million token context lengths are the norm now in terms of what the model supports. But a million token context lengths can only hold, you know, a portion of some of like our internal companies, like largest code repositories. Like it will be a fraction of that. And so if you really want a agent that can, can take over, you know, the capabilities of a whole team of programmers, I think the main limiter today isn't like the model capabilities itself and scaling the model size.

• 43:59Speaker 00

  It's on how much context can that, that model have of all of the data it needs to make smart decisions.

• 44:06Harry Stebbings?

  I just want to break some of the things you said up there. You said that you think model sizes will increase. I thought we were all moving to owning our own intelligence, every enterprise having their own smaller model with proprietary data. Does that go against what you think in terms of model sizes increasing? Can you help me understand?

• 44:23Speaker 00

  Yeah, I think you can kind of break it into two tiers again. So there's going to be the frontier models and capabilities that are being really the forefront of the development by OpenAI, Anthropic, maybe Google, you know, SpaceX AI, et cetera. I still think there is a long road to go in terms of getting to pushing the frontier of model capabilities. And those will continue to grow, continue to get, get better. And there are a lot of workloads where, let's say, internal, just speaking for how Positron uses LLMs, where I don't today care that much about the cost. If I can get 10 times the output value out of a model today, I very gladly pay 10 times more, you know, per token. And I really want the frontier to push that.

• 45:05Speaker 00

  You know, some of it is cost saving, some of it is just truly owning your proprietary data. There is a push from, from enterprises to have inference on site. And it's a lot more difficult to provide that for largest models. And most companies, you know, if they're adapting from open source or developing their own model, don't have the resources to pushing the frontier. And so that is kind of what's gone smaller models.

• 45:27Harry Stebbings?

  And do you not buy that reality?

• 45:29Speaker 00

  I would say like right now, somewhere around 80, 85% of all tokens consumed and produced are done by just the top four model companies. And I would say the next five or 10% is done by the three or four after them. So I can totally buy, believe that 5% of all tokens consumed will be done by things on prem, you know, locked into the big guys. But both from Positron's business perspective and just how I see the world evolving, I'm going to care more about the, the high volume set of things. That being said, I actually think the, the small model stuff is actually much more interesting for everything happening on, on your phone. And, and the amazing thing about that is I think there's this misconception that, oh, if questions could be answered by your phone or, or, you know, any prompts can be,

• 46:14Speaker 00

  can be done locally, that actually is meaning that there's less tokens going to be used with the big guys in the cloud. I think it's the opposite. The, the reality is that if I have an LLM running on my laptop or phone or in my enterprise's, you know, secure on prem cloud, whatever, that is going to be consuming data at such a fast rate of everything coming into it. And it's going to be generating, you know, analysis based on that. And it will decide, okay, what is it that I'm going to actually return to the user, you know, locally? And what is it actually requires more intelligence from a better model that isn't self-hosted? I think for, you know, any of the tokens that are being quote unquote saved by running locally, that's actually going to generate more things.

• 46:56Speaker 00

  Because I mean, in, in some ways, for, for a simple naive use case of a personal user of LLMs, they're only going to prompt chat GPT or Claude ever so often. Like they're, they're sort of limited by their thoughts of when to actually ask an LLM something. But if they have a local LLM that is constantly checking their email, their calendar messages, et cetera, deciding to do these lookups to cloud hosted models frequently, that's now on a per person basis, a massive increase in the number of tokens being consumed and generated by the cloud models, even though there was, the naive view is that, oh, there was the shift to this on-device LLM.

• 47:35Harry Stebbings?

  So can I just understand? So I completely hear you in terms of maybe 5% of them will be in this smaller model, enterprise-owned kind of model landscape. Why are you so bullish then on much larger models and the sizes increasing?

• 47:49Speaker 00

  I'll break into portions. So there's the increase of the model sizes, which I think of the thing that would be shared by a lot of people in the AI space is it's kind of a gut feel where, based on the fact that we have seen these scaling laws, like we, we call them scaling laws, the fact that going from a, you know, 100 million to a billion to 10 billion, 100 billion, 1 trillion parameter models, we've seen this amazing increase in capabilities with that. We see that like still to this day, going from a trillion to five to 10 trillion, that, that the largest end right now. Like we call it a law because we've observed it, that there's no like actual mathematical proof that this will continue. So it's a sort of on vibes that, okay, this has continued scale. There's no sign of it slowing down.

• 48:31Speaker 00

  So is that going to continue to 50 trillion, 100 trillion and, and beyond? And I don't see any indication that that's going to stop. So I'll be bullish on that.

• 48:41Harry Stebbings?

  What did scaling laws look like at three times what it is now? If AGI has been declared now by Jansen, forgive me, but what is three times this?

• 48:51Speaker 00

  I had a good question. I mean, GPT six Astra, my first, you know, 24 hours with it were basically as magical as my first experience with GPT 3.5 in November of 22. I was at the ChatGPT launch at NERIPS in 2022 and it was so funny because, you know, Sam and Ilya were there and they, you know, it was a party in New Orleans for the NERIPS conference and basically at the end, they just said, hey, we launched this little fun, you know, experiments called ChatGPT. Go check it out and like zero fanfare, you know, it was, it was really just a side mention and I don't think anyone really gave it a thought at the event.

• 49:32Speaker 00

  But when I went back to the hotel, I loaded it up and I got back at, you know, 10, 11pm or whatever. And I was up for four or five hours straight, just giving random prompts and that this was the most magical experience that I've ever had with a computer. I would say I got very close when Sora two came out that I had similar experience, short amount of time, but just mind blown by the quality of the videos. And especially the weekend that Sora two launched and there being no restrictions on what you could generate. But yeah, GPT six Astra, I do think is, is AGI. And to, to your question of like, what does that mean going forward? I think my guess is as good as basically anyone's, but.

• 50:12Harry Stebbings?

  Why was GPT Astra so good for you? Why was it comparatively such a breakthrough? Cause I have it and it's great, but honestly kind of the same as before.

• 50:20Speaker 00

  Oh, um, in terms of the things that I've found LLMs to fail the most at in the past. So I'll give a case where it is more linear improvement. So just in terms of general coding capabilities, performance and analyzing problems, et cetera. It is a step function improvement, but not mind bogglingly. So there are a bunch of things that other models have not been able to fix or kind of went in circles and found inelegant solutions. And it's still like a human software architect that was able to come up with a better solution. With Astra, they're just initially giving it a couple of really hard problems that I've not been able to solve with other LLMs. Was able to do it one shot, having it go through codebase and find both performance improvements, bugs, et cetera.

• 51:03Speaker 00

  Basically discovering new spaces that I didn't know existed in our bunch of portions of our codebase. So that's one element, step function, but not mind boggling. The second case that was mind boggling, just from a like, wow, is the, the computer use abilities with a set of generic tools. So like being able to do blender animations, like, you know, it's become, there, there's a bunch of memes online of, of it recreating different videos, et cetera. But just the fidelity of that and where that was basically impossible with GPT 5.6 soul was massive increase capability. And like, I had it design, you know, do interior design of my house just based on a couple of pictures and just like, wow. Like I did not think that at what's fundamentally a text model could, could do that.

• 51:43Speaker 00

  And then finally, like the, the biggest thing for Positron was I've been trying with every single new model release to have these models be able to like, actually take a relatively simple logic design problem, you know, implementing a, a encryption block in this case. And being able to take that through the full RTL to GDS flow. So from basically the specification of do this encryption function, implements the Verilog. So the hardware description language for that. So write that code and then be able to take that code and go through all the way until you have got it shipped design that theoretically you could go tape out. LLMs could do different portions of that and could like write the scripts and, you know, fail a lot of different midpoints on the way.

• 52:26Speaker 00

  But a big problem with the electronic design automation tools, the EDA tools for doing chip design, is that they were designed in the nineties, early 2000s. They're really unintuitive. None of the documentation exists out like in the public web. And so that these models don't have like a real good innate view of them. But GPT six with both combination of computer use and just an ungodly amazing scripting ability. Has been able to take this, this CAC CAC block and implement it with the TSMC and the three PDKs and take that all the way to GDS. And do that in like a little over like 50 something hours and meet timing and, you know, over a gigahertz, et cetera.

• 53:06Speaker 00

  And that as a task, if I was giving to someone similarly new to a, a thing, like getting the flow mostly working, I would say would take on the order of a week. And getting it optimized to the point that Astra is at with, with that design would maybe be one or two additional weeks, depending on the person. So compressing that two to three weeks down to two days and change, it's still mind boggling. It like, it shouldn't be this good at this, as I would naively think about its training sets. But obviously with OpenAI's own chip developments, you know, in-house, they've, I'm, I'm glad that those capabilities are getting added to the, the models they're releasing to the public and not just being kept inside.

• 53:48Harry Stebbings?

  We see jalapeno, terrible name, I think personally, but you know, their own chip development, the anthropic of developing their own chips, deep seekers supposedly developing their own chips. We see the commoditization of the chip player with everyone building their own chips. How should we think about that?

• 54:02Speaker 00

  As a consumer of all these things, if I take my Positron hat shirt off, I would say that that's a great thing for the industry having, um, fundamentally that's going to bring costs down and capabilities up and, and bring it to, to more people. I think it's such an interesting world where when I got started in the semiconductor space, you know, 13 years ago, you know, silicon was a dirty word in Silicon Valley. And now you have all the biggest companies in the world being, you know, somehow connected to semiconductor industry and most interesting, exciting applications. And the company's building them, you know, vertically integrating down to the silicon layer. The interesting thing with all of the, the ones that you mentioned and, and, you know, the broader set is companies have the same macro goals.

• 54:47Speaker 00

  The implementation details are all unique though. And that's just as an engineer and technologist, it's exciting to me that there are a lot of different ways to skin a cat. And people can have their own architectural view and, and go about implementing it and, and get different results. And, you know, Hot Chips, the biggest conference for, for this design space and was where OpenAI unveiled jalapeno last month. I'm happy that the industry is still pretty open and willing to share not as much details as people would have shared, you know, five, six years ago. But still a good amount of in the open discussion of, of things. And so I think how that applies to Positron is, you know, we have our particular architectural views and way that we've decided to do things.

• 55:30Speaker 00

  And that will evolve in, in the future as well. Everyone else's. But, you know, there's still plenty of space to make bets and go in different directions. And the great thing about the market is that the market gets to, to decide what is, is valuable. And those that trade value will receive a reward for that.

• 55:49Harry Stebbings?

  We spoke about context window length earlier and the expansion of it. How much does that expand? Is that infinite expansion capability of context window length? And what does that mean we can do that we can't do today? I'm just fascinated.

• 56:04Speaker 00

  I think with traditional linear or quadratic attention, there is going to be limits of scale, what it came to, what the hardware could provide. Now, one of the big things for Positron is we're trying to massively increase the memory capacity per device. You know, with our, our upcoming generation, we're going to have, you know, eight times more memory capacity than the highest memory skew from Nvidia. And Nvidia is actually decreasing the amount of memory per device, uh, you know, based on the market memory conditions. But I frankly think that context length going from like a million tokens that it is today, going to 10 or even great, much more than that, is really, really hard with that quadratic expansion of memory cost.

• 56:42Speaker 00

  The algorithmic advancements that have happened over the past year have been very, very interesting in terms of being able to further reduce the amount of, of storage and, and compute necessary for that context with linear and sparse attention mechanisms. And those have really, really been innovated by the Chinese model labs. And this is a great example of when you have constraints of, you know, we had export controls on the chips with the highest memory capacity and flops. And so they innovated on not needing that. And so, you know, DeepSeek, beginning of, of 2025 with, uh, DeepSeek V3 had, you know, made a lot of waves because they were able to, to get massive decrease in KV cache size with multi-head latent tension.

• 57:25Speaker 00

  So you were actually spending more flops to be able to have a smaller KV cache. And that's advanced a lot over the past year and a half. And probably the, the most interesting or my, my personal favorite right now is gated delta nets and, and its derivative versions, where you can have like a 75% decrease in the total time, you know, you're spending on, on the attention portion, you know, with, with this mechanism. So.

• 57:50Harry Stebbings?

  How important then is new hardware? If DeepSeek without it, just on architecture innovation alone can cut costs by 80%?

• 57:59Speaker 00

  Yeah, I would say that like there's no such thing as free lunch. So when they have that MLA compression, it does come at cost of model capabilities in some form. And like, there's a reason why the Chinese labs have really heavily embraced, you know, MLA. While none of the US labs have, I should say, based on rumors, but I also have a very good information and belief that, you know, none of the major US companies are, they definitely are not using MLA and not using some of its, you know, brethren. I think that that will evolve and change in the future. But yeah, that basically the sword version of that is it's not just a pure savings on that on that side.

• 58:39Speaker 00

  But the reality is to, and to go back to your previous question of like, everyone does want greater context length. Like if I had 10 million token context length, I think that would be enough for holding multiple of our largest code bases and really have that cross-pollinization happen between them for an agentic coding model. And I guess one, one thing I should have explained, it's not just having the full context. Like there were some early models that advertised a million token context length. But as soon as you went above like 64,000 tokens, you know, it's two years ago. It's recall ability just went garbage. So just saying that something has this maximum context length is one thing. It's can it actually use that context length effectively is entirely different thing.

• 59:23Speaker 00

  And that's actually going back to the Astra thing, amazing thing about it. There, there's a couple of different benchmarks measuring like the long context performance. One of them is called ruler and there's like these find a needle in haystack. So you just flood the, the context window with a bunch of junk, basically, like just passages from books and all this. And you just put somewhere randomly in the middle of all of that, you know, a hash, you know, some value that looks out of place. And you prompt the model, say, what's the secret value? And a lot of models have done really poorly on this. GPT 5.6, which is only like six or seven weeks old, only could do this about 70% of the time. GPT six Astra does it like over 95% of the time correctly.

• 1:00:06Speaker 00

  So that's, there, there's a lot of room for improvement in these things.

• 1:00:10Harry Stebbings?

  Speaking of room for improvement, can I ask you, when I was doing the research for the show, I saw that the silicon data token price index dropped below $1 per million tokens this month. And five years ago, it was $60 per million, $60 to one. What does a million tokens cost in 2028, say, two years from now, do you think?

• 1:00:33Speaker 00

  I think the thing I care a lot more about than just that 60 to one is the fact that a $60 token five years ago, no one would pay a cent for today. Like that was a complete garbage token, relatively speaking, five years ago. And the level of quality of a token that you pay a dollar per million tokens for now is so much astronomically more valuable.

• 1:00:56Harry Stebbings?

  And so that's because of token efficiency and what can be done?

• 1:01:00Speaker 00

  No, I'm saying just in model capabilities. If you, if you say, okay, so, so it's 2026. So the best model in the world in 2021 was GPT three. It's kind of crazy at the rate models get released today that GPT three was the best in the world. Basically from 2020, I think it was August 2020 when it released all the way up to they, they didn't have a new release until ChatGPT in November of 2022. So it was, you know, two, two and a half years between, between model releases. And really GPT 3.5 was just doing reinforcement learning with human feedback on the same base model.

• 1:01:35Speaker 00

  So if you remember how bad GPT 3.5 was and like, what was the value of that in terms of economic productivity value of GPT 3.5 versus GPT six today, or, you know, pick whatever comparison points you want. The value per token in terms of what it can improve a person's life, you know, a company's, you know, business practices, et cetera, is orders of magnitude, I would say a hundred or a thousand fold. I, I think there's actually two points to, to your, your access of going from $60 to $1. Yes, that's a decrease in cost, but that token today is, let's just say, I think conservatively a hundred times more valuable.

• 1:02:15Speaker 00

  So I would actually be saying that there needs to be some multiplier there as well, where like the value per, you know, unit of intelligence is probably closer to a thousand fold, not, not just the 60 fold you're, you're talking about.

• 1:02:27Harry Stebbings?

  What does that mean then, if we extrapolate that out to 2028? What does that mean that, like, does the cost of a token then actually matter? Like, is that the primary unit that we should measure? Cause everyone talks about cost of token. Is there actually a different metric that we should measure?

• 1:02:40Speaker 00

  It is interesting that with the GPT six launch, Greg Brockman had said that he doesn't think that they're going to be pricing things in tokens much longer. And that they, they want to be moving and to, you know, cost per useful result like to that. And I don't think that's where it'll end up because that's really difficult to price and, you know, quality, et cetera. But I think that the price per token is really great because you can easily calculate the cost to generate a token. So determining a margin on that and pricing it in, as in bulk volume to generic customers is really easy. And I think that's going to stick around in large form because that is so easy. We'll see for the largest providers of tokens, how they potentially evolve their business models in terms of

• 1:03:26Speaker 00

  if you have a GPT seven or eight that is superhuman and can fully function as an employee in an amazing capacity. And OpenAI calculates through whatever method that's running at full tilt, et cetera. It's only going to cost them, you know, however many hundreds of thousands of dollars to produce tokens continuously with that. They may decide that it's actually easier and they'll be able to get more adoption if they just charged a million dollars a year. Just, you know, using a random number to have full unlimited usage of that virtual agent worker. So that may be how things evolve.

• 1:03:58Harry Stebbings?

  Can I ask you, I'm always very careful of being like the young naive one. I'm not that young anymore, but like being the naive one who's not seen cycles. But Gavin Baker says it well, when he says, I can't speak to a company that don't have numbers that are parabolic up and to the right. And just everything is better than it's ever been. What would be the first signs of a crack in the chasm? By a shift from frontier models to open weight models, Anthropic and OpenAI are not continuing in the same level, not quite growth rate, because it's impossible to say, but level of growth, missing numbers next year. And then the bubble getting burst a little bit where the two core leaders are having some form of strife.

• 1:04:40Speaker 00

  I, I agree that that's a possibility. The reason I don't think it's likely is I think that the development of open source models and, and things happening locally, et cetera, will actually drive greater token volumes for, for the big guys.

• 1:04:53Harry Stebbings?

  So how does that work? I thought they were competitive.

• 1:04:55Speaker 00

  Yeah, no, I, I think that the smarter and more capable that Siri is on my phone is going to result on in it doing a whole bunch of background tasks and things that remove me from having to be the one that instigates having requests and, and data be processed by even smarter models. I, I really do think that in a lot of AI applications right now, the bottleneck is actually a human making some sort of decision. And different tasks have different levels of autonomy that, that will result in things getting sent to be processed by, by a model, by OpenAI or Anthropic. But I think the, the next really big order of magnitude, couple orders of magnitude increase in token volumes

• 1:05:37Speaker 00

  is going to come when us humans trust a local LLM that has access to all of our data all the time to have it decide to do things on its own that it is not smart enough to do. Right now, I trust Astra a lot more than myself on a whole lot of different things. But I still prompt it to do things and maybe it will go run autonomously for 12 hours or three days. I think the next big leap is when I trust a model running on my laptop or on my phone to prompt the smarter models to do even more wider set of tasks.

• 1:06:11Harry Stebbings?

  When we look at like the data economy that powers the larger models which you believe in, we see Merckor, we see Surge hitting 3 billion in revenue. How big do these companies become? Because Anthropic and OpenAI are four to five trillion dollar businesses, say feasible. It's wholly feasible, isn't it? That Merckor and Surge are $200 billion businesses, which serve both frontier labs and some of the world's biggest enterprises building their own models.

• 1:06:38Speaker 00

  I think my only skepticism there is on there being vertical integration by the frontier labs. I think the reason that hasn't happened is because Anthropic and OpenAI have better uses of their capital and mental power, et cetera, than doing all of the, you know, scale AI, Merckor, et cetera work. But I, I think that won't always exist. And let's say that GPT seven or eight could be an effective replacement for Sam Altman in terms of being able to manage, you know, a, a large business. Why they wouldn't just have agents be taking over those tasks.

• 1:07:12Harry Stebbings?

  Before we leave, I like to finish on a tone of optimism. What are you most excited about today that you think the world does not spend enough time on, that we should spend time on?

• 1:07:21Speaker 00

  I'm very sympathetic to the problems that I think the, the smarter set of the AI alignment, AI safety community are when it, when it comes to thinking about how do you align incentives? Like, a lot of people just talk about AI alignment being a problem. I think there's a key part of that is human alignment. It's like, how do we, society, you know, human race align ourselves to have a good outcome that I think will be empowered by artificial intelligence? And so we discussed all, a bunch of the different problems that we're facing geopolitically and socially and how these different things are handled. And I think a lot of smart people are doing good work on the AI alignment problem and, and thinking through how we solve that.

• 1:08:05Speaker 00

  But they may be gated in what can be done there if we don't get better human alignment on, be it regulatory frameworks and energy production, where we're going to put the data centers, et cetera, et cetera. And I, I think framing it as this being a similar sort of technical problem that smart people can work and reason through will hopefully get more people thinking about it that way. And I think a core element that gets discounted by, I think, a lot of people in that sphere, there, there are a lot of economic factors. And I, I think it's the economic factors that actually will drive real decision-making and, and actions. And if we don't look at it from, you know, a rational, you know, self-interested actors and, and all these different things, you're just not going to make progress.

• 1:08:50Harry Stebbings?

  Thomas, this has been the most varied discussion ever from education on, you know, unbelievable infrastructure evolution to Dario and Sam. You are a star. Thank you so much for joining me today.

• 1:09:01Speaker 00

  Thank you so much, Harry.

• 1:09:04Harry Stebbings?

  But before we leave you today, the best model for your application might not exist yet. The most ambitious AI teams are training open models to beat the frontier in their domain. Now, on November 3 in San Francisco, Forge by Fireworks brings those teams together. People like Jensen Huang, CEO of Nvidia, Michel Catastra, President and Head of AI at Replit, Lin Kao, CEO of Fireworks, and more, to hear how AI leaders are taking control of their differentiation, margins, and roadmap by owning their intelligence. Learn how they're building it with inference, training, and intelligent routing. Forge is free to attend, but space is limited. Apply today at fireworks.ai/forge.

• 1:09:46Harry Stebbings?

  While Fireworks AI powers product intelligence, Asana keeps the work moving. Most companies have tried AI, most aren't seeing results. Not because AI doesn't work, it's because AI hasn't reached the workflows yet. That's the gap Asana is built to close. Asana is the operating system for human agent teams, your easy button for AI productivity across every team. Ready-to-go AI teammates, pre-built for marketing, ops, and IT. No prompt engineering, no setup. They show up where the work is happening, already on-boarded, in your workflows, ready to deliver. With Asana, your whole company can work on the same plan, towards the same goal, whether you're a team of 10 or a team of 10,000. Asana, where humans and agents workflow together.

• 1:10:30Harry Stebbings?

  Try it at asana.com, that's A-S-A-N-A dot com. While Asana organizes the work, Superhuman speeds up the day. Can I be honest with you for a second? Here's what my week actually looks like. Back-to-back meetings all day, and the real pressure builds up in the gaps between them. I'll come out of six founder meetings with a whole stack of follow-ups waiting, and my inbox is hundreds of messages. And somewhere in there is the one thing that really matters, which I always tell my mother is her. I've tried AI tools for this, but they always lived in another tab. I'd have to stop, go find it, paste in the context, write the perfect prompt. It was just enough friction that it never really stuck. Superhuman Go, from the makers of Grammarly, totally different.

• 1:11:13Harry Stebbings?

  It's right there, where you need it, when you need it, in the doc when I'm reading, the email I'm writing. When a thread gets dense, I highlight it and ask Go to pull out what matters without losing my place. And after a run of back-to-backs, I ask what was actually decided, and Go turns it into clear action items, or drafts, the reply I need to send. It's already up to speed, and I'm always the one in control. Try Superhuman Go, from the makers of Grammarly, and find out more at superhuman.com.