Will inference move to the edge?

Today virtually all AI compute takes place in centralized data centers, driving the demand for massive power infrastructure. But as workloads shift from training to inference, and AI applications become more latency-sensitive (autonomous vehicles, anyone?), there‘s another pathway: migrating a portion of inference from centralized computing to the edge. Instead of a gigawatt-scale data center in a remote location, we might see a fleet of smaller data centers clustered around an urban core. Some inference might even shift to our devices. So how likely is a shift like this, and what would need to happen for it to substantially reshape AI power? In this episode, Shayle talks to Dr. Ben Lee, a professor of electrical engineering and computer science at the University of Pennsylvania, as well as a visiting researcher at Google. Shayle and Ben cover topics like: The three main categories of compute: hyperscale, edge, and on-device Why training is unlikely to move from hyperscale The low latency demands of new applications like autonomous vehicles How generative AI is training us to tolerate longer latencies Why distributed inference doesn‘t face the same technical challenges as distributed training Why consumer devices may limit model capability Resources: ACM SIGMETRICS Performance Evaluation Review: A Case Study of Environmental Footprints for Generative AI Inference: Cloud versus Edge Internet of Things and Cyber-Physical Systems: Edge AI: A survey Credits: Hosted by Shayle Kann. Produced and edited by Daniel Woldorff. Original music and engineering by Sean Marquand. Stephen Lacey is our executive editor. Catalyst is brought to you by EnergyHub. EnergyHub helps utilities build next-generation virtual power plants that unlock reliable flexibility at every level of the grid. See how EnergyHub helps unlock the power of flexibility at scale, and deliver more value through cross-DER dispatch with their leading Edge DERMS platform, by visiting energyhub.com. Catalyst is brought to you by Bloom Energy. AI data centers can’t wait years for grid power—and with Bloom Energy’s fuel cells, they don’t have to. Bloom Energy delivers affordable, always-on, ultra-reliable onsite power, built for chipmakers, hyperscalers, and data center leaders looking to power their operations at AI speed. Learn more by visiting⁠ ⁠⁠BloomEnergy.com⁠. Catalyst is supported by Third Way. Third Way’s new PACE study surveyed over 200 clean energy professionals to pinpoint the non-cost barriers delaying clean energy deployment today and offers practical solutions to help get projects over the finish line. Read Third Way's full report, and learn more about their PACE initiative, at www.thirdway.org/pace.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  A very brief word before we start the show. We've got a survey for listeners of Catalyst and Open Circuit, and we would be so grateful if you could take a few moments to fill it out. As our audience continues to expand, it's an opportunity to understand how and why you listen to our shows, and it helps us continue bringing relevant content on the tech and markets you care about in clean energy. If you fill it out, you'll get a chance to win a $100 gift card from Amazon, and you can find it at latitudemedia.com/survey or just click the survey link in the show notes. Thank you so much.

• 0:34Speaker 01

  Latitude Media, covering the new frontiers of the energy transition.

• 0:39Shayle Kann?

  I'm Shail Khan, and this is Catalyst.

• 0:43Speaker 03

  We could be getting 80% of our compute done locally and leaving 20% of the heavy lifting for the data center cloud. Of the 80%, I would say most of that will be on the edge. I think maybe on the word of 1% ends up being put on your consumer electronics.

• 1:03Shayle Kann?

  Coming up, could the age of edge inference blunt the big data center boom?

• 1:15Speaker 01

  Demand flexibility is now an essential part of the data center speed to power playbook. But how do data centers actually curtail load? GridBeyond is the end-to-end solution for flexible energy management. GridBeyond's advanced DERS and VPP platform is tailored to what data centers care about most: maximizing reliability, intelligently meeting curtailment obligations, reducing costs, and managing all your distributed resources in one place. Ready to transform your distributed energy management? Get started at gridbeyond.com. Catalyst is brought to you by Antenna Group, the communications and marketing partner behind the most influential companies in energy, climate, and infrastructure. For three decades, Antenna has helped ambitious companies stand out,

• 1:58Speaker 01

  build market leadership, and become category-defining brands. From breakthrough startups to global industry leaders, their team brings the expertise, relationships, and track record to accelerate your momentum. Learn more at antennagroup.com.

• 2:13Speaker 00

  The power plant for the modern grid isn't a building. It's a network. Energy Hub's derms platform connects batteries, EVs, thermostats, and commercial and industrial sites into virtual power plants that over 230 utilities dispatch as reliable grid capacity day in, day out. VPPs deploy in 60 to 100 days at 40 to 60% less cost than traditional infrastructure, and they perform like conventional generation from day one. Learn about the flexibility platform for the modern grid at energyhub.com.

• 2:49Shayle Kann?

  I'm Shayle Khan. I lead the early-stage venture strategy at Energy Impact Partners. Welcome. Okay, so here's an energy question disguised as an AI infrastructure question. What proportion of the world's AI compute in 2035 will be cloud, i.e. in large centralized data centers, versus edge versus edge-edge, i.e. on-device? It's an energy question because the answer today is effectively 100% in that first category: cloud. And that's why we have this crazy dynamic in the electricity sector, and actually in the natural gas sector too, where hyperscalers and neo-clouds and developers and real estate speculators and crypto miners turned AI companies and more are hunting for sites that can accommodate hundreds of megawatts

• 3:34Shayle Kann?

  or gigawatts of power. And the whole thing, as we know, is crashing through the electricity sector, affecting generation and transmission distribution, prices, now politics, and so on. But there's a narrative that I've heard a number of times that if borne out would potentially present a very different future from the present. This is one where AI workloads, first of all, shift significantly from training to inference, and then where those inference workloads become highly latency-sensitive and are also able to be executed in a more distributed fashion. And as a result, much of that compute, and thus the power demand, shifts from these big centralized data centers to the edge.

• 4:17Shayle Kann?

  That could mean it shifts to 10-megawatt data centers clustered around an urban core or an autonomous vehicle corridor. Or at the limit, it could mean inference compute happens on-device, and centralized data centers fall back into a pure training position. Any version of this that takes significant share of the market would have profound implications for the energy question and for the grid. So it's worth exploring, which is what I'm doing today with my guest, Dr. Ben Lee. Ben is a professor of electrical engineering and computer science at the University of Pennsylvania. He's also a visiting researcher at Google. By the way, this edge AI infrastructure world and the energy implications thereof is super interesting to me, as you will hear. So if you are building something in the space,

• 5:01Shayle Kann?

  please come get in touch. In the meantime, here's Ben. Ben, welcome.

• 5:07Speaker 03

  Great to be here. Thanks so much.

• 5:09Shayle Kann?

  I'm very excited for this conversation because this is the topic that I, in my energy circles that I travel in, I've heard scuttlebutt about a bunch of times, but have never actually spent the time to like really try to understand. The topic basically being how much of inference compute might move from central cloud infrastructure to the edge? And then how far to the edge, of course, being another question. But I think we should start by actually defining those categories a little bit. How do you think about the categorization of like where compute can occur? And then we'll talk about each of those categories individually.

• 5:47Speaker 03

  Right. So even before we talk about generative AI, there for classical compute, cloud computing in general, all the services we love and change the way we live and work today. There are three levels generally I think about for compute. The first is massive hyperscale data centers, the ones run by Microsoft and Google and Amazon, hundreds of thousands of machines, massive facilities. That's what most people think about when they think about cloud computing. At the other end of the extreme would be personal devices, consumer electronics. So you think about your phone, you think about your tablet, your laptop, plenty of compute can

• 6:29Speaker 03

  happen there as well. There is a perhaps less understood middle layer or intermediate layer called edge computing. And edge computing really means that there are times where you don't want to go all the way to this remote massive facility. And wait for the data to go out to that data center and then come back. You might want to access some compute that's a little bit closer to you, maybe in the same city, maybe the same geographic region. That's edge computing. So they're still going to supply really capable, high-performance machines, these servers. Uh, but you don't suffer those longer communication times or latencies that you might if you um, if you were to go to that remote massive data center.

• 7:11Shayle Kann?

  And my recollection is that there was. I think, okay, so the advent of cloud computing meant the build out of lots of big centralized data centers. There was a fair amount of conversation some number of years ago in the kind of first wave of excitement around autonomous vehicles in particular, that you might see a fair amount of edge infrastructure get built because of the latency tolerance requirement for AVs. I mean, I'm on the outside, so tell me if I've got the kind of narrative wrong here. But then it seems to me that because AVs were generally delayed, or maybe the need wasn't as high, like what we've got today. If you just look at the infrastructure today, it's seems like the vast, vast majority of classical compute even,

• 7:55Shayle Kann?

  um, except for stuff that's sitting in like mainframes that companies is in the cloud and the big centralized data centers. Do I have that right?

• 8:04Speaker 03

  That's right, and this is a decades-long trend. I mean, we've seen this progression, uh, this adoption of cloud computing over the last fifteen to twenty years. And there are a couple of reasons, uh, we are seeing that shift or we have seen that shift. Uh, the first is that, uh, computing in a massive data center run by the hyperscaler companies, these big tech companies, is much more energy efficient. They know how to deploy these facilities. They know how to cool them and build HVAC systems, um, efficiently. So they're incurring very few, uh, very small overheads per watt of compute. There's this industry standard metric called, uh, power usage effectiveness or PUE.

• 8:46Speaker 03

  And that's the ratio of how of the power you're using divided compared to the power that's going to compute. So Google's P U E is close to 1.1, which is to say for every watt going to compute, there's an additional 0.1 watts going to the overheads of power delivery or cooling or whatever. So that's really incredibly efficient. And most mom and pop data center operators, most enterprise data center operators don't get the scale and efficiency that these hyperscalers do. The scale also gives a second key advantage, which is the ability to share hardware. So you buy the hardware once and you have lots of users sharing the same physical hardware. That allows us, again, to drive the costs down,

• 9:29Speaker 03

  allows the hyperscaler operators to drive the costs down. And that essentially gets a massive increase in efficiency. So most compute now is being done in these large data centers and in the cloud.

• 9:42Shayle Kann?

  Okay, so let's talk about the world of AI now, which is where all this growth in compute is happening. You know, AI workloads, of course, divided into two major categories, one being training of models and the other being inference. I think we'll spend most of our time today talking about inference probably, but let's spend one minute on training. Is there any movement or argument that training should take place anywhere other than large centralized data? And it seems very clear to me that the trend right now is just build the largest possible data center to train the largest possible model. So is there anyone who thinks that it might, that might turn in the other direction?

• 10:20Speaker 03

  Some, but that really hasn't gotten much traction. I think the reason why we see most training go happening in massive data centers is because of the scale. You need to communicate large data sets. You need lots of GPUs, all closely coordinated, learning the model parameters. The only scenario that some people have explored for training away from the data center is if you've got private data and somehow you want to refine your model or somehow fine tune your model with that private data. You don't want to share it with the hyperscalers. That has been primarily a research question rather than a production system that people have deployed.

• 10:56Shayle Kann?

  Okay, so let's assume then that that the vast majority of training compute is still going to happen in centralized data centers. As it stands today, I don't know if you know the numbers, but but just high level of all AI workloads, how much is training versus inference? Because I think the other big point people have made is like, over time, the the proportion of workloads going toward inference is going to increase, the proportion of workloads going toward training may decrease as we sort of asymptote the next model or something like that. But like today, it's mostly training still?

• 11:30Speaker 03

  I would agree with that. I think to first order, the training costs are historically what people have cared about the most because the data sets are massive and you're talking about these massive 1,000 megawatt data centers for the training workloads. There was a study we did when I was a visiting research scientist at Meta where we found that energy costs for AI were roughly broken into three categories. There's a data preprocessing aspect as well, and that's about a third. The training is another third, and then the inference or the use of the model is the last third. But clearly, those fractions are evolving rapidly, and I would agree with you when you're saying that the training costs are probably flatlining.

• 12:14Speaker 03

  They were reaching a plateau in how quickly they are growing, perhaps. And if the optimism about AI is to be justified, you're going to have to see inference costs go way up because that will be an indicator that adoption has gone up in a fairly significant way, both among individual users but also among companies and enterprise users. So I think it's true to say that inference costs are large and potentially will grow very rapidly.

• 12:44Shayle Kann?

  Okay, so then we're getting to the sort of crux of our question today, which is inference workloads, inference costs increase over time, usage of the models increases over time. That's the presumption of everything going on in in AI world. And then the question is, will that inference compute predominantly still take place in these big centralized cloud data centers? Or will some or much of it potentially shift either to one of the other two categories you described, sort of edge, localized, or fully localized on device? So let's talk about the edge version first, which is essentially smaller data centers, still data centers, but smaller and more local.

• 13:25Shayle Kann?

  What's the argument for why that might happen? And what are the limitations?

• 13:31Speaker 03

  So the argument in favor of edge computing is mainly the proximity to the end user, right? So when you, we have been conditioned in an era before generative AI that when we access internet-based services like a search engine, we expect the answer to come back on the order of 100 milliseconds. That is the order of magnitude that we're talking about. And as a result, to get those 100 millisecond latencies, oftentimes you require computation closer to the user. So you don't have to travel across the internet. You don't have to travel from the West Coast out to the East Coast and back again, the data, I mean, and get that answer back in a timely way.

• 14:14Speaker 03

  What is interesting with generative AI is that we are being reconditioned to tolerate much longer delays. So if you use something like GPT, or you use something like Claude, or your favorite chatbot, oftentimes it's just sitting there thinking for seconds and seconds, maybe tens of seconds before it gets you the first token. So the question there is to what extent we care about that latency and need that really fast responsive access to the answer.

• 14:38Shayle Kann?

  Yeah, and I think we've been especially trained even further in that direction with the introduction of things like deep research, where, you know, even in the name, you sort of think, well, of course that has to take time. It is deep research that they're doing. So that's an interesting point that maybe we. We are being becoming reconditioned to allowing more latency. The argument that I've heard for why latency is really in a matter apart from just wanting search queries or chat queries to come back quicker is the is the next wave of applications for AI, right? And so maybe we go back to autonomous vehicle world and things like that, where like latency, making decisions in near real time does become really important. Robotics being another category that could be a major user of AI compute, but needs

• 15:22Shayle Kann?

  really, really low latency. Is that part of the argument for shifting some compute to the edge?

• 15:31Speaker 03

  Yes, absolutely. So the cluster compute you mentioned, autonomous vehicles, robotics, fit into what we call cyber physical AI. So cyber physical systems are those that have a cyber component, a computational component, but also interact with the physical world. And once those interactions with the physical world arise, then we care about responsiveness because that underpins safety guarantees and the ability to make sure that your robotic arm is able to respond quickly enough to hazards, your autonomous vehicles are able to do so. So I agree that there will be cases where we will need those really low latencies. And that is going to require edge computing much closer to the user. So we have much shorter internet delays, network delays.

• 16:16Shayle Kann?

  I'm curious to understand the trade offs here, right? Like I know with with model training, there are technical reasons why you want all your compute as clustered together as closely as possible. You want every GPU as close to every other GPU as you can make them minimizing the copper between them or the optics or whatever it is that's communicating between them. And that, for some reason that you can explain to me, makes model training more effective. Is there a similar dynamic in inference? Is there a technical reason why that you are you're paying a penalty if you shift to smaller data centers at the edge? Or is there no technical reason why that's it's suboptimal?

• 17:00Speaker 03

  Right, yeah, let's talk about the training piece first. The reason why we need 1,000 megawatt data centers where we have hundreds of thousands of GPUs connected so closely together is because the data sets are massive and the models are massive. We're trying to learn.

• 17:18Speaker 03

  On the order of a trillion parameters for these machine learning models, these AI models, and we're trying to do it on the wealth of data we find in the internet. There's no way that any single GPU can handle that much data. So what we end up doing is partitioning the data into smaller pieces and then handing each GPU a slice or a partition of this data. And each GPU will churn on its own share, on its own partition of the data, and learn the models that work best for its piece of the data. And all the other GPUs in the data center are doing the same thing on their partitions of the data. Periodically, what they will do is they will compare notes, they will share the weights that they've learned, and this sharing is really, really expensive.

• 18:03Speaker 03

  And some of the people in the energy space may know that there are massive energy fluctuations or power fluctuations we will see in data center usage when the GPUs go from this computational intensive phase where you're learning the model weights to this communication intensive phase where they're comparing notes and sharing their intermediate results with each other. So as a result, that's why we're talking about these massive data centers for training. They all need to communicate frequently to share what they've learned from their own data sets. For inference, we don't see that effect.

• 18:38Shayle Kann?

  Just to add, the craziest thing to me about how model training data centers operate right now, the absolute craziest thing is, as you said, there are these, there are surprisingly large spikes in power demand as a result of how the models are trained. What they do in large part, because those spikes are actually problematic, not just to the grid, but to the equipment inside the data center as well. So what they do at least sometimes to manage that is they create dummy workloads. So they keep the power profile basically flat, but you are literally just wasting energy. On absolutely nothing, nothing is happening during those times. They're dummy workloads. At that scale, the fact that that is happening is wild to me.

• 19:21Speaker 03

  Absolutely, and I think we've seen this in other contexts as well, but not perhaps at this scale. This notion of an electrical engineering, we'll call it the D I D T problem, the change in current divided by a change in time. If if large current swings over very short periods of time, you could imagine building batteries to sort of damp things out or decouple. And certainly, a lot of people are thinking about that. But the easiest thing to do might be to just modulate the software, as you say, because we have very precise control over what the software does. So so that that is an active and ongoing area of research and that needs to further develop.

• 19:59Speaker 01

  For data centers, connecting to the grid quickly means two things: building distributed energy and having a plan for demand flexibility. Grid Beyond helps with both. Grid Beyond's Terms and VPP platform unlocks the potential of distributed assets and dispatches flexibility into the right market at the right time. And this is more than software. Grid Beyond supports developers every step of the way. From modeling your highest ROI energy mix to orchestrating the full set of assets around a data center, the track record speaks for itself. Grid Beyond manages over 1,400 sites and more than 3.7 gigawatts of load across Europe, the UK, and the US. Learn more at gridbeyond.com or click the link in the show notes. If you work in energy or climate, you already know the company's Antenna Group has helped

• 20:44Speaker 01

  build. For three decades, they've been the communications and marketing partner behind the industry's breakout companies, category creators, and market leaders. Today, their team is working across the most consequential areas of the energy economy, from data centers and grid modernization to nuclear, geothermal, storage, critical materials, and next-generation technologies. Antenna helps ambitious companies become the names everyone in the market knows, trusts, and wants to work with. They build reputation, elevate executives, drive demand, and turn momentum into market leadership. If your company is ready to lead its market, Antenna is ready to help you get there. Visit antenna group dot com.

• 21:24Speaker 00

  When you harness the power of millions of DERs in a VPP and put artificial intelligence behind the dispatch, the outcomes exceed conventional generation. Energy Hub's DERMs platform has been trained on years of real dispatch data from over 2.6 million DERs, so utilities can plan with confidence. That means consistent load reduction across a four-hour event window, even during the widest, most uncertain peak periods. Utilities target, shape, and verify capacity down to the meter. Unlock flexible capacity that costs 40% to 60% less to deploy than traditional infrastructure in a fraction of the time it takes you to break ground on a peaker plant. Learn more at energyhub.com.

• 22:08Shayle Kann?

  Okay, so then on to inference. So you're saying inference does not contain that same challenge. So is there any... What is the downside to shifting inference workloads to the edge?

• 22:18Speaker 03

  To my knowledge, there isn't much of a downside. Because the reason why inference is amenable to edge computing is because when you send a prompt for processing by a large language model, that prompt is probably handled by one GPU or maybe eight GPUs inside a single machine. And the reason that is is because the model sits in that machine, the data sits in that machine, and all of your prior conversations with that bot are sitting in that machine. And it's a very localized piece of compute that needs to be done. And you don't need tens or hundreds of GPUs to be coordinating to give you an answer back.

• 23:03Speaker 03

  You've got that one GPU or a tightly coupled GPUs giving you that answer back. And that is amenable. That is great for edge computing, and we can certainly supply that.

• 23:13Shayle Kann?

  So a thought experiment that I've given people recently in thinking about this is, let's just say that you need a gigawatt of inference compute in five years from now or seven years from now, something like that. You think you need a gigawatt wherein the demand for that gigawatt is geographically centralized somewhere. Let's just say you need a gigawatt of inference. You think you're going to need a gigawatt of inference compute to serve the Dallas metropolitan area, whatever it might be.

• 23:46Shayle Kann?

  At that point, a few years from now, then this is back to the the power perspective. Is it going to be easier for you to find and cite a one gigawatt site? Or 110 megawatt sites within that geographic region? Today, I think it is still probably easier to find the gigawatt site. Or at least the past couple of years it has been. But there are that many gigawatt sites out there from a power availability perspective. So at some point, is that going to flip? And is it going to be easier to build 110 megawatt sites? Which is, it sounds really hard to do and indeed is, but these are all hard problems. So if that happens.

• 24:27Shayle Kann?

  Do you think that we are going to see a significant portion of that inference workload move to that type of scale? Is that the right scale? Like, should we be looking at 10 megawatt sites, 100 megawatt sites, 1 megawatt sites? Like, how far to the edge do we want to go?

• 24:41Speaker 03

  Yeah, absolutely. And I agree with the premise of that question 100%. I think that there are two reasons to go to smaller, many smaller data centers. The first is the one you mentioned, power provisioning and connections to the grid. The second is the fact that you don't need massive GPU coordination for an inference workload. I guess the catch might be that if you are thinking about your existing edge data centers, maybe you've got data centers in downtown Los Angeles or something like that already serving workloads. Those workloads may not be configured to handle GPU and AI compute. They may have power delivery infrastructure that was optimized for CPUs.

• 25:24Speaker 03

  They might have HVAC systems optimized for the much lower power density of CPUs. So it's not simply a matter of pulling out your CPUs and replacing them with GPUs. You're going to, you may have to retrofit the the facility itself to support that. But I agree, I think finding capacity there may eventually become easier than finding the next 1,000 megawatts.

• 25:52Shayle Kann?

  Is there any limitation? I can imagine, I'm trying to think of why you wouldn't do that.

• 25:58Shayle Kann?

  You know, you need to sort of house all of the, you need to have a fair amount of memory, you need to house all the model weights and so on in every individual data center, if you're going to do that at the edge, right? So is there, there's got, got to be some minimum viable scale, I assume.

• 26:12Speaker 03

  Right. And maybe to give you a sense of the type of data centers we were talking about in the past.

• 26:19Speaker 03

  Again, in a study that we had done with Meta, we looked at 15 of their data centers before generative AI. And the scale of those facilities were somewhere between 15 to 50 megawatts, right? So less than 100 megawatts. And certainly that is, that was fairly, very fairly conventional, uncontroversial to build those types of data centers in the, in the past. So that, that's the starting point, I think, in terms of the, the, the scale. Now, as you scale down towards, for example, one megawatt, not clear at what point things start making less sense.

• 26:56Shayle Kann?

  I guess the other point here, like the way that the data center build out has gone historically, just like the cloud data center build out. It's been fairly clustered in these regions, right? And there's a reason why Northern Virginia is the data center hub of the world. And there are others as well, Chicago, Dallas, et cetera, Phoenix. Is, and that, that, as I understand it, is largely because the cloud providers needed to offer a certain level of reliability to their customers. And so they could have redundancy within a given region. And that was helpful to them in terms of what they were offering. Do you think that this future world wherein a bunch of inference compute moves to the edge?

• 27:39Shayle Kann?

  Let's call it 15 to 50 megawatt data centers then, instead of, instead of hundreds or, or thousands of megawatt data centers. Does it look similar? Is that you have a bunch of, a, a, a small number of regions that have like a really high concentration of those 15 to 50 megawatt data centers? Or could it be much more dispersed because the whole point of this is really low latency and local and you don't need them to be as clustered?

• 28:06Speaker 03

  I, I think there are lots of different aspects at play in, in terms of data center citing. I, I think the redundancy is definitely one of them. And I have trouble disentangling the role that some of these other factors play as well. Some people talk about tax breaks and incentives from local companies and local states. Some people talk about proximity to internet exchange points. So not only are you talking about congestion-free power movement, but you're also talking about congestion-free data movement into and out of the data centers. Northern Virginia has that. And then, of course, the availability of the power itself. I guess I would say that when you start talking about many of these smaller data centers,

• 28:51Speaker 03

  from a redundancy perspective, it might be okay that they're not all geographically clustered as long as you have a strategy for rolling over the compute or rolling over the workload to spare capacity somewhere within that region that has a similar performance profile or some sort of similar latency or delay characteristic. And so that's really the concern, whether you have robust geographical redundancy and resilience there.

• 29:22Shayle Kann?

  Is this happening? Like, it's interesting, I was thinking, okay, so it sounds like you're saying there's not a big downside. We already have significant inference workloads, so it's not like we're waiting on workloads to show up that could accommodate this. And yet, if you look at everybody, most everybody building data centers, certainly the hyperscalers and I think the colos and folks as well, you know, the focus continues to be on we got to find big sites for big data centers. Why don't we see more development of this small, smaller-scale edge AI inference world?

• 29:59Speaker 03

  I think it really depends on the workload and the application. And we don't know, I would view AI as a more fundamental, basic technology, and we don't necessarily know what application or capability will be layered on top of it. I'd say that we've been talking about edge data centers a lot. There are other words for this type of data center. A content distribution network is one of those examples, a CDN, or a point of presence, a BOP, that these facilities are sometimes called, and they exist in fairly significant numbers. Content distribution networks ensure that when you want to access, for example, newyorktimes.com or wsj.com,

• 30:45Speaker 03

  Your webpage is not being served from the other end of the country. Those webpages are sitting close to you because the content distribution network took those updated webpages and moved them to facilities near you, data centers near you. Likewise, companies like Meta, when they have Instagram or when they have these social media applications, they also have these points of presence that supply data from local points of presence rather than retrieving content for your feed from across the country. So we already see that, but these are application level performance requirements, whether they be for social media or for other sort of news content.

• 31:28Speaker 03

  Once it becomes clear what applications of AI really drive further inference deployments, then we'll know what sort of performance requirements are needed, what sort of what we call caching techniques or strategies might be useful so that we can keep fresher data or more recent, more frequently used models closer to these users and then serve them more quickly. I think I think we'll become clear as we see which models really get traction, which applications will get traction.

• 31:57Shayle Kann?

  Right, so maybe the state of affairs today is, look, the anybody who's developing data centers, we know we need the big centralized data centers because there is currently essentially endless demand to train models, at least relative to the availability of compute today. And so we know we need to build the big centralized ones. We might as well use those big centralized ones that we know we need right now for inference workloads, such as they are today. But we don't have enough certainty yet about what the inference workloads are going to be long term to invest that kind of capital and time expenditure that it would take to build out the network of 110 megawatt data centers in a particular geographic region, something like that.

• 32:39Speaker 03

  That's right, and I would say maybe that my crystal ball is as clear as anyone else's crystal ball, but I feel like there's A huge amount of GPU capacity being discussed in the pipeline in these large data centers. And if it turns out that maybe there are diminishing returns from training larger and larger models, or maybe we run out of data because we've exhausted all the data that's available on the internet. When those things happen, it may be that demand for these GPUs in these largest data centers will flatten out and we're going to have spare capacity. At which point, as you say, they will be used or repurposed to serve inference.

• 33:20Speaker 03

  And then it will be hard to make the case we're building yet more data centers, smaller ones with GPUs closer to the users. I think the catch there will be if one of these model providers or one of these application developers makes performance a distinguishing feature of their offering. Right? If they start competing on performance rather than on capability, then we're going to see, well, I may have a thousand GPUs in the middle of Nebraska that are already deployed, but if I really want to break into the San Francisco market, I've got to build my GPUs right there and have them available.

• 33:56Shayle Kann?

  All right. So speaking of performance, let's transition to the full extreme version of this, which is also, I think, theoretically the most disruptive from an energy perspective, which is shifting any significant portion of these inference workloads all the way onto the device. Skip the, either skip the middle ground of edge, five megawatt data centers or 15 or 50, or include them, but shift workloads that would have gone to a big data center that requires a lot of power, straight onto your iPhone or your iPad or whatever it is. And we've heard some clamors of this as well. Give me the similar sort of like pros and cons of shifting that workload straight onto the device.

• 34:38Speaker 03

  Right. Pros, primarily two things. One is performance, right? You don't have to go across the internet. The model is right there and the compute is right there. Assuming that you get really capable hardware on your device as well, you get really quick responsive answers from your AI. The second is also something we've mentioned earlier, which is the notion of privacy. You don't necessarily need to send your data out into this hyperscale data center where it gets blended with lots of other user data and you have maybe fewer guarantees about what happens to it. Localized compute. It is certainly more private than compute on shared systems. So those are the two key advantages.

• 35:20Speaker 03

  And then I guess the third would be that it gets more tightly integrated with

• 35:27Speaker 03

  the capabilities on a particular platform. So for example, Apple's ecosystem.

• 35:32Shayle Kann?

  Right, and Apple seems like the obvious candidate to do this. Clearly, you mentioned privacy. Apple is particularly focused on privacy. They have the hardware, the device, right? Like, Apple is notoriously, or at least reputationally, behind in the AI race. And so like this, it's not hard to picture that. Like, if somebody's going to move a lot of this inference on device, it's going to be, it's going to be Apple.

• 35:56Shayle Kann?

  Okay, but there is a real trade off here, I assume.

• 35:58Speaker 03

  Yes, and the trade off is primarily, is primarily with respect to the capabilities of the device. So, so if we have a very large model, we're going to have to deploy that model on a much more capable hardware platform than we've got today. This means having some number of gigabytes of memory to hold the model weights, and then also some additional gigabytes of memory to hold the context as you develop this conversation with the model. In addition to the memory, you're also going to need the compute. Now, you're not going to have this high performance GPU sitting inside your phone. So you're going to have to have specialized chips. Those specialized chips on your hand are going to be less powerful, less capable than the ones in the data center.

• 36:42Speaker 03

  So all of this speaks to not getting exactly the same model that you would get into the data center. You would get a shrunk down model. Maybe in the data center, you would have a trillion parameters, this massive GPT-5 model, for example. But on a, on a personal consumer electronics device, you might only have seven billion parameters, so orders of magnitude smaller. And that smaller model will be less capable. It will give you less capable answers. It will be capable of doing fewer tasks. But maybe that's okay, because you've identified only a handful of tasks that you really care about on your, on your personal device. So that is really the trade off. As you go towards the device, you're going to have to shrink the size of the model down.

• 37:27Speaker 03

  You're also going to get less and less capability out of your AI. The final thing, of course, is the power and energy profile. At data center scale, we care primarily about power because power influences infrastructure and power delivery and influences thermal and so on, thermal management. For device level compute, there are two considerations. We care about energy rather than power because that affects battery life. Right? So even if you could deliver a really capable GPU chip onto your phone, the question is, how long would your phone last if you were using that chip on a fairly consistent basis? So that the power, the energy aspect will continue to be challenging.

• 38:11Speaker 03

  And then the thermal aspect will also be challenging. If you have a really powerful device, that's going to be a hot brick inside your pocket. And that's going to be, that's going to be a deal breaker as well.

• 38:23Shayle Kann?

  So when you say deal breaker, do, is there progress toward on-device inference? I mean, to your point on performance, that strikes me as like, okay, this is now, we're now again in the, in the context of like, specific workloads. Certain types of workloads, like a seven billion parameter model might be fine and others it wouldn't be. And so maybe there will be some on-device, an on-device chip and some inference that you could do on-device, but you know, you, you pull up your chat GPT app or whatever and of course it's going to send you back out to the cloud or maybe to the edge. But you know, these other challenges of thermal management, things like that are, are hardware challenges.

• 39:05Shayle Kann?

  Do, where are we in the progression of on-device inference? Is it coming? Is it not coming? Do we not know?

• 39:12Speaker 03

  I, I think the assumption with on-device inference is that you'll be able to shrink the model without loss in performance for, for, for the tasks you care about. That, that is the primary, primary strategy the computer scientists have been taking. On the hardware side, we have.

• 39:30Speaker 03

  Made strides in developing custom chips, custom silicon for the specific types of tensor algebra that are required for for machine learning models. So we know how to build those chips, and that gives us energy efficient compute, higher performance. We know how to build really capable memory systems or solid state disks. So when your phone now has hundreds of gigabytes of memory on it, or hundreds of gigabytes of storage on it. So there's a question of, well, maybe you'll end up using less of it for your photos and more of it for your AI model, something like that. So I think there are fairly significant resource constraints, but I don't think that they are insurmountable in the sense that

• 40:15Speaker 03

  more intelligent hardware design and more intelligent hardware management could go some ways in terms of making these AI models feasible on the device.

• 40:27Shayle Kann?

  Okay, so I'm gonna put you on the spot, and we promise not to hold you to these numbers, but just to give a sense of like where we think things are heading. If I, if we're fast forwarding 10 years, right, let's just say we're in 2035, and imagine there's a total volume of inference compute in the world or whatever, that's, let's just say is 100 megawatts total. What would be your guess of the ranges of how much of that compute is going to take place in large centralized data centers or versus at the edge? Let's, we'll draw a line, let's say, you know, 100 megawatts and above is large centralized, sub 100 megawatts, but not on device is edge, and then the third category, of course,

• 41:08Shayle Kann?

  being on device. Like, how much of it can go anywhere but the centralized data centers?

• 41:14Speaker 03

  So I would go straight to this idea of having a 2080 rule, because we see this all the time in computer systems, where you have 20% of your tasks being extremely popular. Maybe there are 20, 20 things that you always want to do, and you spend 80% of your AI compute doing those things. That could be email processing, that could be photo analysis, that could be. So we can identify what those really compelling applications and tasks are, and we're going to be spending most of our time doing that. And then for the remainder of the long tail, long, heavy tail of other tasks that people might want to do, that would always be backup capabilities residing in the cloud data center.

• 41:56Speaker 03

  So I would say that we could be getting 80% of our compute done locally and leaving 20% of the heavy lifting, or the more esoteric, the more, the more corner case compute for the data center cloud. That is, of course, excluding the training. The training will continue to all reside in the massive facilities. But in terms of the inference, I think there's huge potential.

• 42:18Shayle Kann?

  Right. But that, yeah, that's like actually a very significant shift. If 80% of the, of the inference workload, appreciate that that doesn't include training, but still, if 80% of the inference workload could end up local, that, that's a significant shift and has, has pretty profound implications for the energy picture as well. Are you saying that 80% just to, just to pin you down even a little bit more, is that local in the sense of being at the edge? Or is that local in the sense of being on device? Or like, what do you think the split ends up being there?

• 42:47Speaker 03

  Yeah. So I, I think of the 80%, I would say most of that will be on the edge. Like it, I, like I suspect it is today. I, I think that if you look at what we, what we talked about earlier, content delivery networks, points of presence, they probably identified 20% of the content that 80% of the people will be looking at most of the time, and they're putting it at the edge. I think maybe under the word of 1% ends up being put on your consumer electronics. Actually, even for today's compute, when we set aside AI,

• 43:26Speaker 03

  there is a trend towards consumer electronics hiding that flow of data back and forth between the, the device and the edge for you. Right? So sometimes they'll, like if you use a, a cloud storage service like Dropbox, or if you're using a, a photo storage service, they will let you pretend that you have access to all of your videos, or all of your photos and all of your documents. And they will transparently, behind the scenes, move things back and forth between the data center and your local device. So you may think you have all of it, but maybe you've only got a tiny sliver, less than 1% on your local device.

• 44:05Shayle Kann?

  Right. Certain things open up, in my box instance, certain things open up much faster than others. When I try to open them and, and I've, it's occurred to me that that is why. If I step back then, okay, so it sounds like what you're saying in this scenario, you're painting of the future, roughly 80% of the inference workloads are edge, very little of it actually on device. And then the other 20% or so sitting in cloud, big cloud data centers. So when I think about the energy implications of that, there's, I think, a couple of ways to think about it that are pretty interesting. One is this, okay, so maybe a fair amount of the energy consumption of at least inference

• 44:48Shayle Kann?

  compute is going to shift to these five megawatt, 15 megawatt, 50 megawatt type local sites. That has big implications for the grid in ways that are, I don't know, both good and bad, probably harder to manage in some ways, easier to manage in other ways. But the overall energy consumption of inference compute, I would expect, and you can tell me if I'm wrong, would actually be higher in this scenario than it would be if it was all centralized, because I assume the PUE that you get for these edge data centers isn't quite as good as it is for the large centralized data center. So like on balance, this probably means more overall AI energy consumption. Do you think that's right?

• 45:30Speaker 03

  Yes, yes, I think you get economies of scale when you go to a gigawatt or two gigawatts. You have a single facility, you're managing it in a highly optimized, coordinated way, and you've got hundreds of thousands of these machines all managed very precisely. I think as you shrink the system down, you will lose an efficiency. You will be trying to build these 20-megawatt data centers and maybe footprints of facilities that weren't designed initially for those workloads. So yes, I think total energy costs may go up as a result.

• 46:07Shayle Kann?

  We're talking about inference workloads to some extent as like a monolith. I'm sure they are not. So are there big distinctions in your mind in terms of the different types of inference workloads and how that influences like where they should be housed?

• 46:21Speaker 03

  Right, yes. So that's a really great question, actually. I would say that there are fundamental limits to the number of inference queries a human user can actually produce, because we're ultimately limited by the speed of our typing, the number of tokens we can actually. Produce to query the models. So there is some of that where humans will continue to send requests to agents. But I think increasingly, most of the inference workload will come from other software agents. This could be a search engine retrieving webpages and then asking the large language model to summarize it for a coherent discussion for you.

• 47:06Speaker 03

  This could be, this could be your photo app, learning something about your, your images. And, or this could be your mail app, doing something with your, with, with the mails and helping you compose messages. So all of that is done behind the scenes. And those inference workloads are potentially much larger because of course, software can generate those requests at much, much higher rates. From the perspective of where that computation happens, to the extent that the data center already has servers running your mail workloads, or to the extent that your search engines

• 47:47Speaker 03

  are already running in the same data center, the communication to the model will be a bottleneck. Right? So if you have a data center in Nebraska, running your search engine for you, or doing some of these other big, heavy lifting, heavy software jobs, then potentially they could query and execute inference in these largest hyperscale data centers.

• 48:13Shayle Kann?

  All right, Ben, this was super interesting. Really appreciate your time.

• 48:17Speaker 03

  It was my pleasure. I really enjoyed the conversation. Thanks so much.

• 48:21Shayle Kann?

  Dr. Ben Lee is a professor of electrical engineering and computer science at the University of Pennsylvania. He's also a visiting researcher at Google. This show is a production of Latitude Media. You can head over to latitudemedia.com for links to today's topics. Latitude is supported by Prelude Ventures. This episode was produced by Daniel Waldorf. Mixing and theme song by Sean Marquand. Stephen Lacey is our executive editor. I'm Shayle Khan, and this is Catalyst.