(Preview) Inference in the Agentic Future, xAI Is Two Companies in One, Q&A on Elon’s Lawsuit, Intel, Apple

Ben and Andrew discuss the future of computing and its implications for the chip market, including what Cerebras is doing that’s different, why speed may no longer be a top priority for inference, good news for China’s AI ecosystem, the future for Nvidia, and questions on Pat Gelsinger’s role in Intel’s revival. From there: Both sides of the Anthropic-xAI deal, including Anthropic’s compute solution and the triumph of market principles, as well as the market’s message to Elon Musk and xAI, and the implications for SpaceX. At the end: Thoughts on Musk’s OpenAI lawsuit, a theory on Apple’s gross margins and a land grab, and a listener’s wife enters founder mode with Claude.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Andrew Sharp?

  Hello and welcome to a free preview of Sharp Tech.

• 0:08Andrew Sharp?

  Hello and welcome back to another episode of Sharp Tech. I'm Andrew Sharp and on the other line, Ben Thompson. Ben, how you doing?

• 0:19Ben Thompson?

  Well, the question is, how are you doing, Andrew? We unfortunately did not podcast last week. There was no chance it was happening once my travel plans actually worked out. Yeah, good thing, you know, we did the makeup in Taiwan, just reminding everyone. We're very conscientious of your time and attention.

• 0:36Andrew Sharp?

  That's right.

• 0:37Ben Thompson?

  In the meantime, you're Washington Wizards' number one pick in the NBA draft.

• 0:41Andrew Sharp?

  Oh, my God, yeah.

• 0:43Ben Thompson?

  Is this making up for the fact that the NBA let Dallas win last year? So that Luca could go to the Lakers, thus demoting you, so they made it up for you the following year?

• 0:55Andrew Sharp?

  It doesn't quite make up for it because there's no Cooper flag level prospect available this year. But look, I can't get greedy. It's, does it make up for the last 25 years of Wizards fandom? Probably not. It's not making up for that either. However, it's nice to have some hope in Washington DC for the first time in about 10 years or so. I was very, very happy on Sunday afternoon. Now I'm, I would say, pretty anxious about what direction the Wizards are gonna go over the next seven weeks.

• 1:27Ben Thompson?

  I'm glad you had a few moments of happiness. I mean, hopefully that's not the peak level of happiness in like your entire adulthood of being a Wizards fan. But I'm happy that it happened.

• 1:35Andrew Sharp?

  You know what? That's all that matters. There was a window of pure happiness. Now we're back to anxiety. We'll see where we end up by the end of June.

• 1:43Ben Thompson?

  But here's the deal, here's the deal. I hope they choose someone that you don't like. Because my experience of the current Wizards regime is that every time they do something you disagree with, they end up right and you end up wrong.

• 1:56Andrew Sharp?

  So like, we're not going to relitigate the Bradley Beal trade at the top of this Sharp Tech episode.

• 2:01Ben Thompson?

  Hey, you can even become an Alex Sar guy. I mean, like, you wanted a what's his name in Houston? Reed Shepard. I think I'd rather have Sar than Shepard these days.

• 2:09Andrew Sharp?

  I'm not giving up hope on Reed Shepard. I'm not fully in on Alex Sar, but I'm happy to be wrong on that point. I would definitely rather have Sar, and he looked great this year. So things are looking up in our nation's capital.

• 2:22Ben Thompson?

  I'm happy for you. I'm happy for Charles. You know what? I'm a big believer in raise your son to support the hometown team. Like, you don't want your son to end up like Andrew. I guess, no. If you want to be like Andrew and not cheer for the hometown team, just because you're a contrarian, look, potential big history for you in podcasting. So that's fine, as long as you did it yourself. But by and large, either raise your son to support the hometown team, or let him rebel on his own. Um, you know, so we'll see.

• 2:55Andrew Sharp?

  Well, that's one of the reasons I'm excited. I can actually take my son to some Wizards games over the next several years with a clear conscience. I don't have to worry about consigning him to decades of mediocrity, or at least the next decade of mediocrity. But in any event, it's great to see you. You know, I did miss you last week. It's good to see you on the other end of the video call here. And we have a lot to cover. So we're going to begin with your article on Monday that was headlined, "The Inference Shift." And you mentioned that there have been three inflection points over the past three years of AI development. I'll list those three inflection points for anybody who's been asleep for the past few years. ChatGPT demonstrated the utility of token prediction.

• 3:40Andrew Sharp?

  O1 introduced the idea of reasoning, where more tokens meant better answers. And then Opus 4.5 and Claude Code introduced the first usable agents, which could actually accomplish tasks using a combination of reasoning models and a harness that utilized tools, verified work, etc. So, reading your article Monday, it seemed like the kernel of insight that spawned the article was that fast inference for coding is ultimately going to be a temporary use case. Can you explain what you mean by that? Because it was a bit of a lightbulb moment for me that seems obvious, but hasn't really been articulated, at least from what I've seen.

• 4:22Ben Thompson?

  Well, I mean, I don't know. It's when it comes to AI, I think everything has probably been articulated to some extent. You know, you know, this is where the doomers get credit. A lot of stuff they've talked about has come true. But this idea, overall, let me set the stage. When it comes to computing, speed is always important. So I'm going to make some assertions about speed and quickness that some people are like, like, why would you want to sell computer? No, what we're, this entire discussion is about trade offs, scaling, all those sorts of things. So let's have that upfront. If you're coding, of course, you want the computer to be fast. But everything that we've done with computing by and large has been the humans in the loop.

• 5:06Ben Thompson?

  And as long as the human is in the loop, like, we care, like, how fat, like, computers can, like, basically never be fast enough, right? We're always looking for them to, for them to be faster, or at least fast enough for the human interaction speed. If you think about it, think about from like an enterprise perspective, you, why do, why would an enterprise update computers? Like, quite frequently back in the eighties and or in the nineties? Even though they were much more expensive? Because the more expensive assets are the human workers who, if they're waiting around for the computer, you're sort of wasting time and money.

• 5:40Andrew Sharp?

  You're losing productivity. Sure.

• 5:42Ben Thompson?

  That's right. And so your enterprise has always been willing to pay for productivity. And so at least like for like your code or whatever, you typically get a very, a very good computer. And would be updated fairly regularly so that you could work and the computer could respond and you could continue your work. And there's a bit now where more and more of that work is obviously be done by the computer. And the more these agents are capable, there's this, there's a couple weird things going on. One, you're having the agent go and do something and then like, what are you doing in the meantime? Maybe you can spin up another agent to do something else. But are you like, are you losing your own context, mental context? I know for me, I'm certainly have a hard time sort of switching gears sometimes. There's also this weird bit where people have token budgets and like they, or Quadcode will have their limitations of how much usage.

• 6:26Ben Thompson?

  And then they'll use it up. It's like, well, guess I gotta go home for the day.

• 6:30Andrew Sharp?

  What do I do now? This is my new workflow. Sure.

• 6:33Ben Thompson?

  That's right. If I get to like, just go back to working the way I used to for the next three hours? No, of course not. This is ridiculous. Let's wait till tomorrow. But so you have this idea of the, you know, even in today, like there's a measure of how much work these can do. And it's getting longer and longer and longer. I think that's actually one of the more interesting benchmarks of these programs. Is how long can they do an autonomous task before they sort of like lose the thread? And that that's getting longer and longer and longer. And the, but still, it comes back to the human. And then the human has, has to like, tell it what to do next.

• 7:11Andrew Sharp?

  There's human oversight in most of these tasks. Yeah.

• 7:14Ben Thompson?

  That's right. All these sorts of things. And that's obviously, you know, it makes sense. That's where we are. But, and as long as that's the case, I'll get back to the button in a second. As long as that's the case, of course, we want faster and faster inference. And it's worth paying for to get that inference. Cause if you can, you can get that response much more quickly, the better. And I think, you know, I've been focused on the agent bit, but the thinking part is super important. Like the, you know, chat GPT before they got the, the spud model. It was still running on like a GPT four class model. Like the base model was horrible. But their reasoning was so good that it was, you'd still get really good answers. It just took forever.

• 7:57Andrew Sharp?

  Yeah. You just have to wait like 55 seconds for it to come back with an answer.

• 8:01Ben Thompson?

  55 seconds if you're lucky, right? Some of this stuff, it would just like take a good like minutes. But it'd come back with a really good answer. But like, what am I doing here? Like, what, what, what am I waiting, waiting around for? And so you think, oh man, wouldn't it be great if that'd be faster? You see things like cerebrus or grok or whatever. And it's not just amazing in terms of spinning out an answer. But if you're reasoning where the more you reason, the more tokens you use, the smarter it gets. Wow. Wouldn't it be great if that could be faster and faster and faster? And absolutely that is the case.

• 8:34Andrew Sharp?

  And Cerebras and Grok, just for anybody who's not familiar, those two are, those are chip companies that specialize in inference and specialize in speed and sort of responses.

• 8:45Ben Thompson?

  Accidental specializing in inference, both of them started kind of before the LLM moment and sort of retrofitted what they'd been working on to this. There, I think they're like next generation chips for both of them are going to be super interesting in terms of like, how would you change things now that you have that in mind? They're architected a little bit differently. Um, Cerebras is, is actually really interesting architecturally. They, they, you have a wafer, it's 300 millimeters. And usually a wafer, uh, you're limited by the size of the reticle limit. The reticle limit is the lens, basically for your lithography and how much of the chip it covers. Which, uh, you know, it usually that's the size of a chip.

• 9:26Andrew Sharp?

  26 millimeters or something?

• 9:27Ben Thompson?

  I, I don't have the numbers in front of me. They're in my article.

• 9:29Andrew Sharp?

  There was a tiny, tiny number in your article. It's mind boggling how infinitesimal all these measurements are.

• 9:36Ben Thompson?

  Well, so what you have to do though, is if you want a bigger and bigger chip, like Blackwell is actually two chips fused together. And those two chips are defined, the, the size of them is defined by the reticle limit. And then they have to put an interposer to let them communicate and expose themselves to the system as one chip, even though they're actually two chips linked together. Uh, Apple's, I think their Ultra Chips is like sort of something different. And you're limited by, by the, the reticle limit. Uh, and so the idea is on a big wafer, you have, the, the bigger the chips, the more expensive because you have to, you're more likely to have yield problems because if there's a defect on, on the chip, but you're going to have a number of chips that are defined by the reticle limit.

• 10:21Ben Thompson?

  What Cerebras has done is basically they developed a technology to

• 10:27Ben Thompson?

  Now I'm forgetting the name. There's the lines between all the reticle exposures have scribes, I think, a scribe limits. I should have this in front of me. But they they basically run wires across that so that you do a bunch of exposures limited by the reticle limit all over the chip of all the different part of the chips. And then they do this additional step of adding in all these lines across these sort of boundaries so that the entire wafer is one chip. And it's like, and everything's all it's wild stuff. It's it's a really interesting approach to get sort of a lot of compute and a lot of SRAM, like the the super fast on the chip RAM, which is what Grok does also. But Grok is still limited by the reticle limit.

• 11:11Ben Thompson?

  So they're there. Rock is more systemizing, putting different ones together. It's like, no, one wafer, one chip. And that there there.

• 11:19Andrew Sharp?

  And that allows them to serve stuff faster than.

• 11:23Ben Thompson?

  Unbelievably fast. Like, like, this is like, this this solves. So there's different aspects of the inference process. But there's parts of it that are just extremely limited by bandwidth of like, how fast you can get memory into the processor and move on to the next step. And they're unbelievably fast at that, like, like, orders of magnitude faster than than other sort of approaches. But there's limitations, like, you're limited by how much memory you can fit on that chip. And the moment you're going off the chip, like, your performance is totally plummeting, right? So it's definitely a narrow use case. But there's situations where if you want immediate response, and not just immediate response,

• 12:07Ben Thompson?

  but immediate, like, sort of like, thinking through things, sort of response, it makes a lot of sense. But then if you're reasoning and you're doing stuff, it's not just the size of the model fitting on the chip, but the KV cache, which is like, all the context of the conversation, that gets large very quickly. There's lots of limitations, but. The larger the market is, the more room there is for

• 12:29Andrew Sharp?

  Niche use cases.

• 12:30Ben Thompson?

  Approaches. Yeah. So we'll see how it turns out. They're IPO-ing this week or I thought it was gonna be this week. It hasn't come out yet. I don't think maybe it's today. Uh, and of all the times to IPO, like right now is a pretty great one.

• 12:42Andrew Sharp?

  Yeah. Stretch. Sure.

• 12:44Ben Thompson?

  We'll see.

• 12:45Andrew Sharp?

  Well, yeah, you can envision, like, I think you mentioned in your article, if there's voice interactions with AI.

• 12:51Ben Thompson?

  That's the biggest one.

• 12:52Andrew Sharp?

  Is gonna matter. And for the consumer market, speed will always matter. But to the extent that we expect a lot of computing to just be done by the computers, certainly in the enterprise, that will probably be independent of humans and then optimizing for speed just doesn't make as much sense. And the obsession with speed is sort of immaterial to the conversation. Nobody has to care about the speed for the robot users. And that was sort of the secondary implication of the agentic shift that seems inevitable, but was not immediately apparent to me over the last couple of months as we're all sort of obsessing over agents and what they mean in the enterprise.

• 13:36Ben Thompson?

  That's, yep. So if you think about agents are like, what's the upside? They never sleep. They're always working, right? That's like

• 13:43Andrew Sharp?

  24 hour employees. Yeah.

• 13:45Ben Thompson?

  Right. And so we're gonna need all this compute. And I think that's all completely true. But part of the implication of them always being awake and always being available is they can sit around for a compute. It's fine, right?

• 13:59Andrew Sharp?

  Right.

• 13:59Ben Thompson?

  Like, there's no loss in terms of them waiting around. And particularly when for these agentic workflows, at least right now, and I think there will be breakthroughs. There will be algorithm breakthroughs. There will be architectural breakthroughs. But for now, a lot of these agentic workloads, the real limiter is memory. Is it's this KV cache issue. It's pulling in all this context. It's remembering state. And if you want these sorts of things that aren't just useful for a task you define right now, but can be spun up, like suddenly something comes up seven, you know, in a week.

• 14:35Ben Thompson?

  And it spun back up and it has all the right context and it knows what needs to be done and it executes a job and then it goes back to sleep or whatever it might be. All that stuff needs memory and and the memory question is also interesting. Cause everything's been about HBM, high bandwidth memory. The reason we want high bandwidth memory is because we want, what do you think we want from high bandwidth memory?

• 14:57Andrew Sharp?

  Um, is that for training?

• 14:59Ben Thompson?

  We want high bandwidth. Um, is the, is the answer. So.

• 15:03Andrew Sharp?

  Well, I gave it a shot. You got me on the spot there. Okay. So, so. I know that China lacks high bandwidth memory and training is a problem in China. So that's how I landed on that guess.

• 15:15Ben Thompson?

  No, you're totally right, actually. But because the reason we want, we need it for training is we, everything on training is like, it's this highly distributed problem. Where we want the GPUs to do these calculations super fast. We want to keep the GPUs fed. And so it's like this multivariate problem that Nvidia has solved way better than everyone else. It, it, it's not just the fast processors. It's also loading them up with tons and tons of HBM, uh, high bandwidth memory. But then also developing all this crazy networking to tie all this stuff together. So it's not just like you have chips acting as one system or, but you have fleets of them, tens of thousands of them. Acting at, acting as sort of one chip and it's going to be like hundreds of thousands of them.

• 15:59Ben Thompson?

  And, and so this is a, there's a lot about the way development for AI has gone. That has been very focused on this problem. How do we execute stuff quickly, but then keep the executors full? So they're being utilized all the time. And it turns out because they're GPUs and GPUs are fairly flexible, not as flexible as a CPU. But more flexible than like an ASIC, this is also an architecture that works for inference, right? What do we have? What's the inference problem? You, you need to get the model into memory and then you also need to house this KV cache. So if you have all these GPUs linked together, you can solve both problems. You can.

• 16:44Ben Thompson?

  Get large models into like a pod and you can have the KV cache issue. But as it gets larger and larger, actually, the KV cache is is getting to be a problem even for GPUs. So like Nvidia has announced this entire like their, oh, think of Dynamo for inference in general. But they've announced like a whole rack for their systems that's just memory. It's just SSDs and the whole point of that memory is for KV cache. But but even then, like if you fast forward five years, fewer, I don't know. But you have these agents where again, you sort of, there's no limit to how many agents you might want. There is a limit to how much compute you might want for humans. The limiter is how many humans there are, like how much stuff they can come up with.

• 17:30Ben Thompson?

  But at least in theory, the limit for computers doing computing. And especially once they're doing their own programming and spinning up their own sorts of things is effectively infinite. The you're going to need to store all this context. Again, there's lots of innovations around here, whether you're on batching or caching or or like, you know, some prompts have the same sort of context. And so you can put those together. There's all sorts of things that are people are going to figure out and they're going to innovate on these in lots of ways. But as long the the idea of there's a concept called a memory hierarchy. So with traditional computing, like your typical CPU, you have registers, which is like the actual data that's being processed.

• 18:14Ben Thompson?

  Then you have like L one cache, L two cache, maybe L three cache. This is all storage on the chip itself. One of the reasons Apple's M chips are really fast is they have a lot of cache. And so it's a right there on the chip, all the like, a lot of the core foundational like operating system stuff. They they've done a lot of this is where their integration has really paid off to make sure everything is like super available and executes very, very quickly. But then from there you go out to RAM and RAM is. Super fast relative to your hard drive astronomically slow compared to cache, right? And you go past RAM, you go to your SSD. Remember when we got our first SSDs?

• 18:55Ben Thompson?

  Unbelievably fast compared to spinning disks, but SSDs way slower than RAM. Uh, and then, and then you can go out to spinning disks. You can go out, like you could go out to tape machines, like there's still storage using magnetic tape. Uh, huge capacity, very, very slow. In general, there's this capacity speed sort of trade off with memory. And so a part of designing a computer is design the memory hierarchy. Like figuring out in general, what stuff should be super close to the processor and the super fast. But knowing you have a limited amount of space there and what stuff gets bumped down. And this is

• 19:31Andrew Sharp?

  When you say stuff, are we just talking about tasks and different applications on a

• 19:35Ben Thompson?

  No, like literal, like ones and zeros, like the actual like bits that go into calculating this sort of thing, which ultimately is everything, everything is a one and zero at the end of the day. And so this is already happening for inference and it's going to happen even more. So right now mostly everything's in HBM of a bunch of GPUs tied together. There's not much of a memory hierarchy. There, I mean, there is a little bit of what it's, we don't need to over complicate it. But generally speaking, there's been this one size fits all, a bunch of GPUs tied together with a bunch of high bandwidth memory and put everything in there and it'll do everything. But, but we're already seeing this shift to particularly with, if you want a ton of context,

• 20:17Ben Thompson?

  to having other places to put stuff in memory that are slower. But you get way more, uh, just way more room, way more capacity. And as that increases over time, where the memory aspect becomes more important and stuff gets slower, that's okay because

• 20:39Ben Thompson?

  The agents, there's no human in the loop.

• 20:41Andrew Sharp?

  Computing.

• 20:41Ben Thompson?

  That's right. That's right.

• 20:43Andrew Sharp?

  And so, and if that's the world that we live in, who wins in that sort of world? Like, how does that change how AI is served? And what sort of infrastructure is best to serve it? Does the infrastructure get more affordable? Is that bad news for Nvidia in that scenario? What do you think?

• 21:01Ben Thompson?

  Um, I think the biggest winner is China.

• 21:03Andrew Sharp?

  Okay. Why is that?

• 21:04Ben Thompson?

  Well, for a few reasons. Number one, if we, i- i- if a lot of the most important AI is done on, you don't necessarily need HBM. Just regular RAM is fine. And then you need like a lot of, a lot of storage beyond that. And relatively slower chips are okay because they're waiting around a memory anyway. Uh, China can make all that stuff, number one. Uh, number two, they can make all that stuff and start selling it abroad, alleviating the sort of memory sordids that we're facing. Particularly if the, you know, the SK Hynix and Samsung and Micron are all focused on HBM. Uh, so who's gonna make DRAM, right? Like, like the, there's.

• 21:41Andrew Sharp?

  Yeah.

• 21:41Ben Thompson?

  And so it's not just that they can probably self-supply for more AI workloads than you might think in this agentic workload. But also there's gonna be a large market for their companies to expand and sell stuff, even if it's relatively lower end compared, compared to Western companies. Um, overall, the, you know, the hyperscalers in general, uh, you know, buying cheaper, cheaper stuff is always better. And I do think, you know, Nvidia is running as fast as they can. They've, you know, they're, they've completely, this whole dynamo approach, it's basically like an operating system for inference, to balance different loads and adding on like a Grok for the super fast and their regular GPUs for some stuff. And, and the way they tie that together is really interesting.

• 22:24Ben Thompson?

  They have Grok just doing the, the sort of inference, speed-related aspect of inference. Uh, the, the, um, but then also shipping like these new mem- new memory sort of racks. Like, they're certainly out there trying to get ahead of it, but.

• 22:42Ben Thompson?

  It's going to be a challenge in the long run, which we saw with the cloud.

• 22:48Andrew Sharp?

  Mm hmm.

• 22:49Ben Thompson?

  Back in the day, in the dot com era, all the money you raised for a startup went to buy Sun Systems. Like you would buy these incredible servers that was fully integrated, uh, super reliable, best their worth, Spark operating system, and you had to buy that to actually run your website. So you could then build a business. So what happened was, it's funny, actually Hotmail was one of the first ones to do this. Yeah, Hotmail and Yahoo, but Google is the one that really did it at scale. Uh, which was taking commodity hardware, like Intel based systems and just building a ton of them.

• 23:31Ben Thompson?

  And saying like, these are way less reliable, way like flakier, much cheaper. But that's fine because once you get to our scale, even Sun Systems are going to break down. And also, we're not going to pay Sun all that money. And so they, they designed an entire way of computing that assumed stuff was just way slower, way less fragile, but we could, on software, we can work around that. We can have, we can have built in resiliency, built in fault tolerance, all these sorts of things. Amazon took that concept, did it and made it available to everyone. So now if you're a startup, you didn't need your, your, you could start a website right away. Like with, with nothing upfront. And that was a.

• 24:16Andrew Sharp?

  Yeah.

• 24:17Ben Thompson?

  And I, I, you know, that seems like sort of what I'm talking about. We, you start out at the beginning, you have these dedicated, super high end systems that sort of do it all. And in the long run, every piece of that system is going to get disaggregated and become sort of commodity markets in their own right.

• 24:36Andrew Sharp?

  Right. So it goes from integrated to modular in terms of what people are building?

• 24:40Ben Thompson?

  Exactly. And because there's not like a user. I think to hold on to the integration in the long run depends a lot on owning the user to a certain extent. Like, Apple still owns the user. The user cares about the benefits of integration. No one, like, no one cares about how their inference stack is constructed as long as it does the job, right?

• 25:00Andrew Sharp?

  Right. Well, cost will obviously matter in the long run if they're more affordable, more durable options than the Nvidia GPUs. Yeah, well, it'll be very interesting to see where it all goes.

• 25:11Ben Thompson?

  This isn't necessarily totally bearish for Nvidia. Nvidia's approach is still by far the best for training. And it's not like we're gonna suddenly stop training.

• 25:21Andrew Sharp?

  All right, and that is the end of the free preview. If you'd like to hear more from Ben and I, there are links to subscribe in the show notes, or you can also go to sharptech.fm. Either option will get you access to a personalized feed that has all the shows we do every week, plus lots more great content from Certechery and the Certechery Plus bundle. Check it out, and if you've got feedback, please email us at [email protected].