The marketplace for AI compute with Jared Quincy Davis from Foundry

In this episode of No Priors, hosts Sarah and Elad are joined by Jared Quincy Davis, former DeepMind researcher and the Founder and CEO of Foundry, a new AI cloud computing service provider. They discuss the research problems that led him to starting Foundry, the current state of GPU cloud utilization, and Foundry's approach to improving cloud economics for AI workloads. Jared also touches on his predictions for the GPU market and the thinking behind his recent paper on designing compound AI systems. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @jaredq_ Show Notes: (00:00) Introduction (02:42) Foundry background (03:57) GPU utilization for large models (07:29) Systems to run a large model (09:54) Historical value proposition of the cloud (14:45) Sharing cloud compute to increase efficiency (19:17) Foundry’s new releases (23:54) The current state of GPU capacity (29:50) GPU market dynamics (36:28) Compound systems design (40:27) Improving open-ended tasks

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Hi listeners, welcome to No Priors. Today, we're talking to Jared Quincy Davis, the founder and CEO of Foundry. Jared worked at DeepMind and was doing his PhD with Mateo Zahari at Stanford before he began his mission to orchestrate compute with Foundry. We're excited to have him on to talk about GPUs and the future of the cloud. Welcome, Jared.

• 0:23Speaker 01

  Thanks, Sarah, and great to see you. Thanks a lot as well. Yeah, great seeing you.

• 0:27Sarah Guo?

  The mission at Foundry is directly related to some problems that you had seen in research and at DeepMind. Can you talk a little bit about the genesis?

• 0:35Speaker 01

  A couple of the most inspiring events I've witnessed in my career so far were the release of AlphaFold 2 and also ChatGPT.

• 0:45Speaker 01

  I think that one of the things that was so remarkable to me about AlphaFold 2 is initially it was a really small team, you know, three and then later 18 people or so. And they solved what was kind of a 50-year grand challenge in biology, which is a pretty remarkable fact that, you know, every university, every pharma company hadn't solved. And similarly with ChatGPT, a pretty small team, OpenAI was 400 people at the time, released a system that really shook up the entire global business landscape. You know, that's a pretty remarkable thing, and I think it's kind of intriguing to think about what would need to happen for those types of events to be a lot more common in the world. And although those events are really amazing because of the small numbers of people working on them, I think it's not quite the David and Goliath story, neither are quite the David and Goliath story that they appear to be when you double-click.

• 1:29Speaker 01

  In OpenAI's case, you know, they were only 400 people but had $13 billion worth of compute, you know, which is quite a bit of computational scale there. And in DeepMind's case, it was a small team, you know, but obviously they were standing on the shoulders of giants in some sense with Google, right? And the leverage that they had via Google. And so, one thing I think that we thought about is, you know, what can we do to make the type of computational leverage and tools that are currently exclusively the domain of OpenAI and DeepMind kind of available to a much broader class of people? It's also a lot of what we worked on with Foundry, saying, can we build a public cloud, built specifically for AI workloads, where we imagine a lot of the components that constitute the cloud end to end from first principles? And in doing that, can we make things that currently cost a billion dollars, cost 100

• 2:13Speaker 01

  million, then 10 million over time? And that'd be a pretty massive contribution. I think it would increase the frequency of events like AlphaFull 2 by 10x, 100x, or maybe even more, super linearly. And we're already starting to see the early signs of that, but quite a lot of room left to push this agenda. So really exciting. So that's kind of maybe an initial introduction preamble to how we thought about it. And I can trace that line of reasoning a bit more, but that's kind of part of what we've done.

• 2:41Sarah Guo?

  Jared, for anybody who hasn't heard of Foundry yet, what is the product offering?

• 2:47Speaker 01

  Yeah, so Foundry, we're essentially a public cloud built specifically for AI. And what we've tried to do is really reimagine all of the systems undergirding what we call the cloud, end to end, from first principles for AI workloads. And we've tried to do this a bit of a new way. I think the AI offerings from the existing major public clouds and kind of some new GPU clouds haven't really re-envisioned things. And by thinking about a lot of these things a bit of new, we've been able to improve the economics by, in many cases, 12 to 20x over lower tech GPU clouds and existing public clouds. And we'll, partially based on some of these products that we'll talk about today, that we're releasing, and a lot of new things that we're working on, we think we can push

• 3:32Speaker 01

  that quite a bit further as well. And so our primary products are essentially infrastructure as a service. So our customers come to us for elastic and really economically viable access to state of their systems, and also a lot of tools to make leveraging those systems really seamless and easy. And we've invested quite a bit in things like reliability, security, velocity and just the core price performance.

• 3:56Speaker 02

  How underutilized are most GPU clouds today? And I think there's almost three versions of that. There's things on hyperscalers like AWS or Azure. There's large clusters or clouds that people who are doing large scale model training or inference run for themselves. And then there's more just like everything else. It could be a hobbyist, it could be a research lab, it could be somebody with just, you know, some GPUs that they're messing around with. I'm sort of curious for each one of those types of or categories of users, like what is the likely utilization rate and how much more do you think could be optimized? Is it 10%? Is it 50%? Like, I'm just very curious.

• 4:27Speaker 01

  One of the most, I'd say, positive cases with the highest utilization, which is the case where you're running kind of an in to end pre-training job. Right? And so that's a case where you've done a lot of work upfront. You've designated a time that you're going to run this pre-training workload for, and you're really trying to get the most utilization out of it. And for a lot of companies, utilization even during this phase, you know, is sub 80%. So why? One reason is actually that these GPUs, particularly the newer ones, actually do fail a lot. As, you know, black practitioners would know. And so one of the consequences of that is that it's very common now to hold aside 10 to 20% minimum of the GPUs that a team has as buffer, as healing buffer in case of a

• 5:12Speaker 01

  failure, so you can slot something else in to keep the training workload running. Right? And so even for a lot of the more sophisticated orgs running large pre-trainings that scale, the utilization is sub 80%, sometimes less than 50% actually, depending on how bad of a batch they have and the frequency of failure in their cluster. And so even in that case, now, there are often though also large gaps and intermissions between training workloads, even if you have the GPUs are dedicated to a specific entity. You know, and so even in those most conservative cases, which all come back to the less conservative extreme cases, utilization really can be really a

• 5:46Speaker 01

  Quite a bit lower than people would imagine. So we can pull on that case a bit more because I think it's actually quite counterintuitive and really interesting. I think there's a really fundamental disconnect between people's mental image of what GPR, GPUs are today and what they actually are. I think that in most people's minds, GPUs are chips, right? And we talk about them as chips, but actually the H100 systems are truly systems. You know, they're 70 to 80 pounds, 35,000 plus individual components. They're really kind of monstrosities in some sense. And the remarkable thing that I think Jensen and Nvidia have done, one of many, is they've basically taken an entire data center's worth of infrastructure and compressed it down into a single box. And so when you look at it from that perspective, the fact that 70 pounds isn't quite as alarming.

• 6:31Speaker 01

  But it is, these are really gnarly systems. And when you end up composing these individual systems, these DGXs or HGXs, into large supercomputers, what you're often doing is you're interconnecting thousands, tens of thousands, hundreds of thousands of them. And so the failure probability kind of multiplies. And so because you have millions perhaps of individual components in this supercomputer, the probability that it will run for weeks on end, and this is basically a verbatim quote from Jensen's keynote, is basically zero. That's a little bit of a challenge. And funnily enough, I think the AI infrastructure world is still somewhat iniquity and so doesn't have perfect tooling, broadly speaking, to deal with these types of things. And so one of the compensatory measures I think people take is reserving this healing buffer, for example.

• 7:17Speaker 01

  I think that disconnect maybe helps explain why these things fail. And it's actually funny enough, the newer, more advanced systems, in a totally fail a lot more than historical systems that were worse in some ways.

• 7:29Speaker 02

  And do you think that's just like a quality and quality control issue for those systems? Or do you think it's just some form of complexity with some failure rate per component that's rising? Or like, what do you think is sort of the driver of that?

• 7:41Speaker 01

  I think it's more the complexity has grown, right? And we're in a different regime now. I think that it's fair to say, so maybe stepping back again to definitions. We throw the term large around a lot in the ecosystem. I guess one question is what does large mean? And one useful definition of large that I think roughly corresponds to what people mean when they invoke the term is that a large language model, you enter the large regime when the, essentially the amount of compute necessary to contain even just the model weights starts to exceed. The capacity of even a state of the art, single GPU or single node. I think it's fair to say you're in the large regime when you need multiple, you know,

• 8:24Speaker 01

  state of the art servers from Nvidia or from someone else to even just contain the model, you know, just run the, basically run the training or definitely even just to contain the model. That's definitely the large regime. And so the key characteristic of the large regime is that you have to somehow orchestrate a cluster of GPUs to perform a single synchronized calculation. Right? And so that becomes a bit of a distributed systems problem. I think that's one way of characterizing the large regime. Now, a consequence of that is that you have many components that are all kind of collaborating to perform a single calculation. And so any one of these components failing can actually potentially, you know, lead

• 9:06Speaker 01

  to some degradation or challenge downstream and stop the entire workflow. Right? You know, you've probably heard, people have talked a lot about InfiniBand and the fact that Nvidia, part of Nvidia's advantage comes from the fact that they do build systems that are also state of the art from networking perspective. Right? And their acquisition of Mellanox was one of the better of all time arguably from a, you know, market cap creation perspective. And the reason they did this is because they realized that it would be really valuable to connect many, many machines into a single, almost contiguous supercomputer that almost acts as one unit. Yeah. And the challenge of that though, is that there now are many, many more components and many more points of failure. And these things kind of, you know, the point of failure kind of multiplies, so to speak.

• 9:47Sarah Guo?

  You implied that like GPUs, well, you described GPUs as this unique asset that is more CapEx than OpEx and that the hyperscalers like an Amazon are making a certain assumption about what that depreciation cycle is. Like, where do you think the assumptions for Foundry versus those hyperscalers or versus, let's say, like a CoreWeave are different?

• 10:12Speaker 01

  This opens up a pretty interesting conversation around like What is cloud? I think we've kind of forgotten in this current moment what cloud was originally supposed to be, and it was value proposition was intended to be. I think current AI cloud is not cloud in the originally intended sense by any means. So we should pull on that thread. But I'd say right now, it's basically colocation. Yeah, it's basically colocation, right? It's not really cloud. Yeah. Maybe, yeah, it's definitely worth pulling on that thread a little bit.

• 10:36Speaker 02

  Yeah, do you want to break that down for sort of our listeners in terms of what you'd use the differences?

• 10:40Speaker 01

  First, I guess there's a little bit of context for people. The cloud as we currently know it is arguably one of the most important business categories in the world. That's, I think, pretty clear. The biggest companies in the world are either clouds, Azure, AWS, GCP, you know, core component of the biggest companies in the world, or Nvidia, who sells to the clouds, obviously. So it's clearly an important category. AWS is arguably a trillion dollars plus of market cap if you burke it out from Amazon. So, you know, at one point relatively recently, it was all of Amazon's profit and more. So it's an important category, to say the least. Cloud, as we know today, really started in 2003, which is when Amazon, you know, vacated 50 people initially, and I believe to start working on AWS. And they worked on this for three years within Amazon, launched in 2006 in March with S3.

• 11:25Speaker 01

  Later that year, in September, they launched EC2, and that kind of was the beginning of the cloud. It took quite a while for this model to catch on, and people were not quite clear on why this would be useful, even until quite recently. And so in 2009, 2010, you know, Mate, Sarah, who you mentioned, wrote this paper called Above the Clouds with some collaborators at Berkeley, a Berkeley-driven cloud computing. And they talked about why cloud would be a big deal. And I think it was not, to say the least, not appreciated, that the points they were making were valid at the time. I think it was kind of, you know, not clear to people, to say the least. You know, fast forward a few years, Bezos in 2015 said AWS was, for all intents and purposes, market size unconstrained. And he was kind of laughed at by many. That was seen as a really ludicrous statement to make.

• 12:08Speaker 01

  You know, fast forward and no one's laughing. You know, in 2019, I remember people saying like, it's not clear if cloud would be a big deal. Snowflake hadn't yet scaled, you know, Databricks hadn't yet scaled.

• 12:19Speaker 02

  I wonder if it's worth making a distinction on these things because, you know, I agree with parts of what you're saying, but, you know, I was building startups in that era. And at the time, there was a pretty strong belief that at least for startups, these clouds were incredibly useful because it used to be you'd spend a lot of money and effort on setting up your server, your server racks and wiring everything up and all the rest. And then, you know, the emergence of things like AWS and nobody expected it to be Amazon, right? It just didn't fit what people thought of as basically an e-commerce marketplace company, right? They were suddenly building infrastructure and providing it. It felt very natural for somebody like Google to provide that. But I think for startups, because I was doing a startup in 2007, you know, we thought that it was magic, right? Because suddenly, and not all the services were there yet and everything else, but suddenly

• 13:04Speaker 02

  you could, you didn't have to deal with all the infra. And there was a number of companies that started before that, like Twitter and others, who kind of ended up having to continue to build and maintain their own clouds. And it was really brutal. And then to your point, I think the big transition was degree to which enterprises, particularly in regulated services or financial industries, kind of fought it initially and then they started opting it. So I agree with the law here saying, I wouldn't say that it was one of those things that nobody believed in though, you know, like actually felt there was a lot of buy-in and a lot of belief and, you know, adoption.

• 13:33Speaker 01

  Definitely not nobody. I think, you know, proficient people and a lot of builders really saw the value really early, particularly startups and the VC committee. You know, I think Berkeley and some of the researchers there really got it early, like 2009 and earlier, you know. By 2015, I think it was already around three billion a run rate. So it was not small. It was very far from what it is today, but it was not small. They didn't break it out yet, but it was, you know, not small at all. It was a meaningful thing. I think that people didn't recognize it would become anything like it is today. And it still wasn't clear to people what the value proposition was. You may recall that there was, you know, Dropbox, for example, famously exited the cloud and went back on-prem and that story, they published why they did that and the economics of it. And that caught a lot of attention and people were saying, ah, I'm not sure if cloud actually makes sense anymore, et cetera. And I think people kind of lost track of that.

• 14:15Speaker 01

  Yeah, I think one of the insightful things that some of the early cloud systems companies like Databricks and Snowflake recognized was why the value proposition of the cloud is really special. I'll actually use one of the quotes from, well, Philip Saunders to kind of illustrate this was the thing, they were trying to think about what was unique about the cloud, what you could do in the cloud that you couldn't do anywhere else. And the killer idea that they converged on, fundamentally, the cloud made fast free, quote unquote, fast was free in the cloud, is how they put it explicitly. And the idea was if you have a workload that was designed to run for 10 days on 10 machines in the cloud, you could theoretically run it on 100 machines for one day. Yeah. Or 10,000 machines for 15 minutes. And that would cost the exact same.

• 14:59Speaker 01

  And so you could run it 1,000 times faster for the same cost theoretically. And it's like, well, that's kind of a big deal, right? You can run something 1,000 times faster for the same cost, you know, and then give the compute back. Now, what that would require, though, a number of things to make that actually work, one being that you'd have to be able to kind of reshape a workload that was designed to run on 10 machines to run on 10,000 machines, not trivial. And another is that you actually have to have the 10,000 machines worth of capacity in the cloud and make sure that was utilized to make the economics kind of work, you know? But yeah, if you could do those two things, it'd be a really, really big deal. That has to be free in the cloud. And so one of the key ideas was this elasticity, right? And I think that's one of the things that's really absent in AI cloud today. In AI cloud today, you're kind of forced to get really long-term reservations, often three years for a fixed amount of capacity.

• 15:44Speaker 01

  No one really wants 64 GPUs for three years or 1,000 GPUs for three years. You want 10,000 for, you know, a couple of months, maybe nothing for a little while. Maybe you don't know how much you need because you're launching a new product and you're not sure how much demand there will be for it, how much inference capacity you'll need, et cetera. It's very, very challenging if you have to reserve, you know, the total amount that you may need upfront for a long duration.

• 16:08Sarah Guo?

  I want to go back to this idea of like, it's actually hard for a lot of engineering teams, especially younger ones, to picture like a pre-cloud world because, you know, their experience with it is like... You know, I have virtual machines on Amazon. They're limitless. Like, maybe it's serverless. And like, if you go all the way back to what Alad was talking about, you gave us a great historical view. But if you look at like, functionally, like, I had, you know, I had my servers in my closet on-prem, right? And then I had colocation, which is like, I have my servers, they're still my servers. I control them and I manage them. But I, they physically live in somebody's data center where

• 16:50Sarah Guo?

  they're offering me like, real estate, like cooling and power, right? And then you had hosting, which is like, I'm buying a machine in a data center, like reservations for a long time, essentially, or for the lifecycle of the machine. And then you had virtualization and containerization and all of these services that came out of this, like, you know, separation into cloud services where you have higher level functions with scheduling orchestration. And, you know, serverless is like, I'm just gonna write, like, the logic, you deal with it, and place that workload. And like that, you know, we haven't obviously gone to an endpoint in non-AI computing. But I feel like the engineering world is so used to being over here, whereas

• 17:33Sarah Guo?

  the AI, the AI hardware resource world, it's like, we're somewhere between colo and, you know, colo and hosting, still.

• 17:40Speaker 01

  That's right. Yeah. And that's challenging. That means that you have to raise all the capital you may need upfront, you know, means that, you know, that's, and that's a challenging model. And perpetuity means also that, you know, you can't quite, you know, grow the product elastically. As demand and interest in it grows, you're kind of ball-necked by the supply chains in a way that I think developers haven't experienced in quite a while. So it's a pretty challenging state, I think, of affairs. And it's also a very challenging, from a risk management perspective, for these companies, because they're making these big commitments, you know, that are potentially, you know, if they don't work out, are pretty catastrophic for them. On this hardware, paying all upfront, paying for these long duration contracts, etc. It's a very challenging thing. And there's no analog yet. The markets aren't returning

• 18:23Speaker 01

  off that there's any analog. So what we have in other domains, like in commodities markets, like wheat, oil, etc., where you can buy options and futures and hedge and sell back and things like that. It's kind of, we're pretty, still pretty in quid in terms of where the market's at. And yeah, I think that's leading to a challenged state of affairs that's going to, you know, continue to bring a lot of pain for people. And so, you know, there are several things I think we've employed to do this better. Some are, Mixes kind of business model innovations and technical innovations at the same time. You know, but I think we're making a pretty substantial dent in this, but also in a way that's really viable economically for us. And, you know, does it involve buying all GPUs and taking undue risks on them per se? And so that's kind of a lot of what we've

• 19:05Speaker 01

  tried to do is can we do something a lot more efficient? You know, can we find some leverage, some points of leverage to address this problem?

• 19:12Sarah Guo?

  You guys have some new releases as of, I think, a day or two ago. Can you describe what's just come out from Foundry?

• 19:21Speaker 01

  So I think right now, AI Cloud is, the AI Cloud business is very much like a parking lot business. And that sounds really funny because cloud is supposed to be high tech and you can hardly conceive of a less sophisticated business, at least on the surface, than parking lots. And what do I mean by that? Well, there's fundamentally, there are two models in the parking lot business. One is pay as you go. For pay as you go, the race is curious. You know, you may or may not find a space. But you, I'm sure many of us have the experience of driving through SF and seeing a lot, a lot full sign, you know, for a lot after lot as we drive around trying to park. And if you do get a spot, you might pay, you know, the $12 an hour, you know, rate or something like that. I'm choosing that rate because it's the rate of AWS, you know, for on demand. On the other hand, if you want

• 20:06Speaker 01

  to kind of guarantee that you'll have a spot and also have a better rate, you can basically buy a spot reserved. And so you can have your own reserved parking spot in your building. Maybe you pay, you know, $4 an hour. So you're getting a massive discount, but it's 3K a month, effectively, right? Which is actually pretty substantial. And if you're only using it 40 hours a week when you're in the office as a typical worker, it actually might be effectively $16 an hour as opposed to 12. So it's actually worse. And so I think one kind of funny analogy for one thing that we want to do with a couple of these products is kind of create the, enable the equivalent of allowing people to park, pay as you go in someone else's reserved spot. And that sounds kind of funny, but you can imagine that, okay, that'd be actually an interesting thing. And if you could do that, then depending

• 20:51Speaker 01

  on how much, how the percentage of the spots that are typically reserved, you might have 10x the effective capacity in a lot.

• 20:59Speaker 01

  And then also, it can be a win-win. Instead of the pay-as-you-go person paying 12, they can pay something much lower. In this case, I'll say seven. I'm betting it should be a lot lower. The person who owns the spot, instead of paying four, can make five equivalent. And then the lot can also make a couple of bucks. And so it was a win-win-win for everyone. And you're double picking the lot and it's really, really efficient. Now, that sounds great, but it wouldn't quite work if you showed up to your reserved spot and there was someone parked there. That might be a little bit aggravating. It also might not work if you're forced to call two hours in advance and say, "Hey, I'm coming." And then the person who was parked in your spot had to leave their dinner reservation to move their car. That wouldn't be a fun model. And so I think one thing that we had to do was create the analog of a system to make this all really convenient.

• 21:42Speaker 01

  And so maybe the V1 of this system was you came into the lot and a sensor was triggered saying, "You're here to go to your reserved spot." And then some valet ran to the car that was parked there and moved it. And they maybe moved it from the second floor to the 10th floor. And then the person who had parked there previously now comes to the counter, asks the valet where their car is, and gets some ticket saying it's on the 10th floor. But there are no stairs, so they have to walk up the stairs and get it. I'm stretching this analogy, but the idea would be inconvenient. And so part of what we did was added more and more convenience features, which we broadly call spot usability. And these are a lot of things that we're keen to add. And so the scenario that we're now at is basically you letting someone else's user spot show up, the sensor kind of knows, "Okay, you're here."

• 22:24Speaker 01

  And then the car in your spot is automatically moved via conveyor to another spot. We're managing the spaces to ensure we can move it somewhere. And then when the person comes to pick up their car, it's brought to them. Now, yeah, so it's all really convenient, it's seamless for everyone involved, and creates a ton more effective space, allows us to get much better economics out of the machines, and is really, really helpful for companies. That's one really powerful, I think, thing that we've done. And this is offered in the context of a spot product. I think people are somewhat familiar with spot usage in the typical cloud context. It's a lot more challenging to do with GPUs for a few different reasons, which is why there are not very many GPUs available on spot, definitely not at scale, and definitely not with interconnect. And so one of the things we want to do is enable this. And it makes a lot of other things possible that are pretty neat. And so it's a mechanism we're employing in a few different ways.

• 23:08Speaker 01

  So that's, I think, one thing, and I'll give some more analogies to explain why that's powerful. Well, we found that companies are using this type of mechanism quite a bit for everything from training, which is classically seen as a workload that's difficult for Spot. But also especially for things like inference, batch inference especially. And that actually opens up another interesting conversation about the different classes of workloads and what each workload needs and cares about and how this might evolve over time. Actually, that ties a little bit to the Compound AI Systems concept and also to the Llama 3.1 release in a funny tangential way. But yeah, that's kind of one analogy for the product that we launched on the Foundry Cloud platform around Spot.

• 23:45Sarah Guo?

  Yeah, I think that Spot usability increasingly deep and flexible and automated is like a really powerful primitive. Changing tack a little bit to just something I think the entire industry, like the tech industry is very interested in. We did a little bit of work together a while back, just understanding like where is the GPU capacity in the world today, right? Of all of the different types, how much is it? How consolidated is it? And obviously, this is near and dear to your business. Can you just describe a little bit like where you think we are and then like what caused the shortage sort of last year?

• 24:26Speaker 01

  One kind of funny bit of trivia that I've posed to a few people that I think reveals how off-base a lot of our priors are is kind of what percentage of the world's GPU Petaf capacity or XFO capacity is kind of owned by the major public clouds. And I've asked many people and I typically have gotten guesses in the high tens of percents. And the only time I got a lower guess was from Satya at Microsoft who guessed basis points, which actually is correct. It's a very small amount. And maybe as one anecdote to maybe illustrate how this looked at least a couple of years ago, this is an evolving thing, but how it's looked, the example of GPT-3 and its training is to have an interesting one.

• 25:08Speaker 01

  It's a bit dated, but I'll use it just because the numbers of the types of machines and numbers there are public, as is not the case for some of these other systems. And so GPT-3 was trained on 10,000 v100 GPUs in an interconnected cluster in Azure for about 14.6 days. To put that in perspective, it was a state-of-the-art system. It was kind of, by many estimates, eight figures for a single run at the time. So it's a pretty substantial investment by OpenAI.

• 25:36Speaker 01

  That tells you that 10,300 run continuously for 40 minutes of days is quite a bit of compute. I think one interesting kind of maybe trivia question then is how many equivalent GPUs normalized in terms of the number of flops, you know, they weren't fully interconnected, but just interesting processing measure anyway, were there in the Ethereum network at the peak of Ethereum? And so I kind of asked people this question and it's fun to just people guess. Can I solicit a guess actually, a lot, can you guess, Sarah, you might know. So, because we've talked about this.

• 26:04Sarah Guo?

  We had this conversation, so I'm not allowed to guess anymore.

• 26:06Speaker 01

  Yeah. So, do you know by chance?

• 26:08Speaker 02

  For Ethereum?

• 26:09Speaker 01

  Yeah, for Ethereum and don't, don't think too hard, just make a guess based on priors.

• 26:13Speaker 02

  When? So when it first launched or?

• 26:15Speaker 01

  The very peak, the tippy top of Ethereum. How many V100 equivalents were there, given there were 10,000 for two weeks for GPT-3? How many were there in Ethereum? Noting, by the way, that these were running 24/7 in Ethereum, so you can modulate your guess based on that.

• 26:28Speaker 02

  I would guess a few hundred thousand, a few million.

• 26:32Speaker 01

  That's an aggressive guess. Yeah, that's an aggressive guess and you're actually very correct. It was about 10 to 20 million.

• 26:37Speaker 02

  Yeah.

• 26:38Speaker 01

  Which is quite a substantial scale. You can, and you can, by the way, structure this really easily by looking at the, you know, basically the hash power in Ethereum at the peak, which was around 900 teraflops per second, I believe, about a terahashes per second, about a petash per second, so quite a bit of peak power. And a typical V100 will give you between, I believe, 45 to 120 megahashes per second, if you really know what you're doing. So that's kind of one guess, there, there are tens of millions to compute.

• 27:01Speaker 02

  Yeah, because it's funny, I remember Bitcoin even years ago, all the CPU dedicated to it at the time was like larger than all of Google's data centers.

• 27:12Speaker 01

  Yeah, Bitcoin though, used a lot of ASICs in particular. Ethereum actually had a higher relative ratio of GPUs. And so the larger GPU, sorry, Ethereum mining providers like, like Hive, for example, had about, you know, had less than 1% of global hash power. You know, and so you can actually start to extrapolate, you know, and they had had quite a few GPUs, like tens of thousands of Nvidia GPUs. Yeah, it starts to give you a number of the scale of capacity, and then a lot of other, you know, mining, my equipment for way less than 1% of the total hash power, about 0.1%. So quite a bit of hash power Ethereum, and I think that's kind of one proxy, but to give you one more anecdotal on that line, actually, an iPhone 15 Pro now is actually stronger than a V100 as a funny, as a funny example, has about 35 teraflops in every 16,

• 27:57Speaker 01

  I believe, where a V100 is around 30. And so there's actually quite a bit of compute in the world, broadly speaking, is I guess the point I'm making. Now, it's not all useful, it's not all interconnected, it's not accessible, it's not all secure, but this is one point to make, that there's a lot of compute in the world, and even for the high-end GPUs, there's a lot more than people think. You know, by many measures, utilization of these, even H100 systems, which are kind of state-of-the-art, the most viable, the most precious, et cetera, is in many cases 20%, 25% or lower, according to some, you know, pretty high-quality data I've seen from some great sources here. Yeah, so quite, quite low. And as I mentioned, even during these pre-training rounds, it's often 80% lower because of the healing buffer partially. That's actually ties to another product that we've launched, which is this product that we built actually largely for ourselves called Mars. It's kind of a funny name, but it's monitoring, alerting, resiliency and security.

• 28:42Speaker 01

  It's basically a suite of tools that we've messed a lot of IP in to really boost and magnify the availability and uptime of GPUs for our own platform. It was actually something that we planned to make available to other people just to use in their own clusters as well. Actually, one of the reasons why we invested in SPOT is because we reserve very aggressively healing buffer ourselves, so that if a GPU failure, we can automatically swap in another GPU, and a user won't perceive a disruption. And so we actually, we maintain buffer for that reason, and so actually being able to pack that buffer with preemptable nodes is actually a really useful thing. But now we're allowing other people to do this, including third-party partners, who want to, for example, make their healing buffer available to others.

• 29:26Speaker 01

  The foundry is really offsetting their economics and the cost of their cluster for them. So it's a really, really powerful thing. And so between Mars and SPOT, you can kind of see how these things are really interconnected and in a nice way. But yeah, the number of GPUs available, there's quite a few, particularly if you look at it more broadly, in terms of total AI compute capacity. The percentage of accessible, useful, and used is a pretty diminished fraction of the total.

• 29:50Sarah Guo?

  How does the GPU market dynamics and your prediction of them factor into Foundry's strategy going forward, right? Because there are, especially for anybody doing large-scale training jobs, there is definitely a, you know, a significant effort to be at the leading edge, right? Access to B100s and beyond is at a premium. And then access in, you know, the largest possible interconnected cluster with sufficient power is also a fight now. It sounds like you, you know, see the opportunity differently, or you feel like there are resources that like can be used that don't require just building new data centers.

• 30:28Speaker 01

  I think it's a little bit of all of the above, to be clear. I think two things can be true at once, and that's definitely the case here. I think there'll be many workloads and use cases for which having state-of-the-art, extremely large interconnected clusters is a really valuable thing. You know, part of something, though, that we're noticing and also trying to promulgate further is, are basically paradigms that don't require this, though, as well. And so here's why I'd say you can have two things to be true at once. And so I think there is a massive shortage of both power, space, and interconnect for the kind of largest of clusters. It's actually very hard to come by and to construct or to find a really large interconnected cluster. You know, it's kind of starts to vanish the larger the cluster gets. Like, there are a lot more 1K clusters than 10K clusters and 20K clusters, and you can keep going.

• 31:12Speaker 01

  Now, I think one thing that, and it'll get harder and harder to keep the scaling going from there, you know, I think there's one question, which is... How will we continue to push the scaling laws? You know, one fact about scaling law curves is that they're all plotted on logarithmic house and, you know, things get better predictably, but it requires a continued two X-ing or ten X-ing to get that next bump in performance. It's but it's quite a bit harder to get the next ten X-ing. And so it starts to come kind of intractable pretty quickly. And so it's already prompted, I think, a lot of innovation. So Google, for example, has been doing a lot with, you know, training across facilities, for example, across data centers, interconnecting them for these models like Palm two, something that previously would have seemed to be inconceivable, or things like Deepako or Deepoco, these models they released that are half-trained across facilities.

• 31:57Speaker 01

  And that's one innovation, but I think actually an even slightly more radical thing is we're starting to see a shift towards a pretty different paradigm. I think myself and a number of my collaborators and Mate and others have kind of termed this Palm on AI Systems. And I think you actually see it with these most recent models like Alpha Geometry, Alpha Code, and Lama three. And so I think this actually points the way towards like what the AI infrastructure future might look like. And I think it looks a lot less like everything requiring these big clusters. And it's a little bit more interesting. And so maybe I'll use five, three as an example. With five, three, they take a little bit of a different approach where they trained a really high, Microsoft trained a really high-quality small model on high-quality data.

• 32:41Speaker 01

  You know, and the small model did not need the kind of big interconnected cluster. You can train it on a pretty small cluster. However, it was still, you know, a non-trivial endeavor because they had to curate and get and obtain this kind of high-quality data. And so one of the things I think you're seeing is for these models like some of the like the Lama three, point one, eight B and seventy B variants. Those models are really small, but they're extremely smart. They're smarter than, you know, much, much larger systems, you know, like the prior generation for OpenAI. And the way that they trained Lama eight B and seventy B looks a little bit different. So what they did was they did, they generated a ton of synthetic data. It seems with Lama 3.145B. And they distilled that larger model into the 70B and 8B variant.

• 33:26Speaker 01

  Right. And so they got a very, very high quality, small variant. Another example is with AlphaCode 2 was able to achieve extremely high code proficiency and win competitions with really, really small language models. But what they did was they called the model a million times for every question. That's an immensely paralysable workload. You can scale it horizontally infinitely. They called it a million times per query and then had a nice, kind of pretty elegant regime to filter down to the top 10 responses, which they didn't try it one by one. And so this is basically what they did. They generated a million candidate responses and basically filtered down to the best one as a way to solve coding, so to speak. So that's pretty interesting. And I think you're seeing that type of regime a bit more and more.

• 34:11Speaker 01

  You know, same with AlphaGeometry, really a powerful system was just announced recently. You know, one, you know, the silver medal level in the IMO broadly, not just geometry for broader class of problems. And, you know, these are kind of compound systems with major kind of synthetic data generation pieces. And so I think you're seeing people kind of move computation around, interpolate between training and inference, for example, to make the best use of the infer they have. And this is actually a kind of a funny, I think, reframing of the scaling laws that we hold in your gear as an ecosystem. I think that, you know, the Chinchilla paper scaling laws that DeepMind uncovered have fueled a lot of the scaling effort. But actually, one kind of funny way of looking at those results is they show that if you want to make a model, I think, model as smart as possible, there is an ideal way to distribute

• 34:54Speaker 01

  parameters, compute, and basically training iterations. One, the funny thing I think some people have done, like Mistral, is actually choose to maybe inefficiently train a small model to be smarter than it should be, wasting money. But then that small model is actually really cheap to inference because it's small. And it's way smarter than it should be for its size. And so I think people are getting more sophisticated at thinking about cost in a more of a lifecycle way. And that's actually leading to the workload shifting from large pre-training more and more towards things like batch inference, which is actually a really, really horizontally scalable workload that you can paralyze. You don't need interconnect in the same way. You don't even need state of the art systems in the same way.

• 35:38Speaker 01

  And I think you're seeing that type of workload maybe grow in prominence. And so just to give one, just to unpack one more statement on that. One thing you can do that people are doing sometimes is you unroll the current state of the art model many, many times, basically doing chain of thought on the current state of the art model. And then you take what required six steps with the previous day of the art and that becomes your training data for the next day of the art. Right, that type of approach to generating data that then you'll that and then filtering that down to high quality examples and then training malls on it looks very different than just throwing more poor quality data into a massive supercomputer to get the next generation.

• 36:17Speaker 02

  Yeah, it seems like that bootstrap up is really sort of under discussed relative to as you hit a certain threshold of model, the rate at which you can increase for the next model just kind of accelerates. One other thing that'd be great to cover, I know we only have a couple of minutes left of your time, is the recent paper that you authored, which I thought was super interesting around compound AI system design and sort of related topics to that. So would you mind telling us a little bit about that paper and sort of what you all showed?

• 36:47Speaker 01

  So I think it's kind of in this regime that we were just talking about where more and more often to go beyond the capabilities on frontier accessible to today's state of the art models and kind of get GPT five or GPT six early practitioners are starting to do these things oftentimes implicitly where they'll call the current state of the art model many, many times. There are many scenarios where maybe you're willing to expend a bit of a higher budget. Maybe it's code or something and if I said that I can give you a 10% better model, you know, for code, many developers might pay 10x for that access to that instead of $20 a month, they might be very willing to pay $200 a month, right, for obvious reasons. And so there's a question of what do you do in that setting? And so people are, you know, if you're willing to call the model many times, you can compose

• 37:30Speaker 01

  those many calls into almost a network of network calls. Right, and you know, I guess one of the questions is how then should you compose these networks of networks or principles should guide their architecture? We kind of know how to construct neural networks, but we haven't yet elucidated the principles for how to construct networks of networks, so to speak, these compound AI systems, so to speak, where you have many, many calls, maybe external components. And so one principle that we start to explore was maybe one thing you can probe to figure out how to compose these calls or whether composing many calls will help you is you can look at how verifiable the problem is. And if it's verifiable, you can actually bootstrap your way to really high performance. So what does this mean? Well, verifiable means that it's kind of easier to check an answer than it is to generate an answer. And there are a lot of cases where this is true.

• 38:14Speaker 01

  Most software engineering computing tasks kind of classically have this property. You know, we looked at things like time factorization or a lot of math tasks. Classically, it can take someone years of suffering to write a proof. And you can like read the proof in a couple of hours. Right. I think we've all had that experience with some training. So there are many examples like this. And so one thing you can you can do is you can have models. You can embarrassingly parallel, you know, you can horizontally scale out and generate many, many candidate responses. And then relatively cheaply check those candidate responses and kind of do a best of K type of approach. And turns out the judge model or the verifier choosing the best candidate response might actually have a lot higher accuracy at selecting the best candidate response from the set.

• 38:57Speaker 01

  Because you can kind of repeat this as a as a procedure to actually bootstrap your way to really with a high performance. I mean, many cases. And so, you know, we did kind of some preliminary investigations here. And we were able to, in one case of prime factorization, you know, kind of connects the performance go from 3.7% to 36.6%. On prime factorization, which is pretty hard, kind of factor, you know, taking a number that's a composite composite of two primes, two three digit primes and factorizing it, factoring it into the constituent primes. Kind of a classic problem that pops up a lot in cryptography. And then also we looked at subjects in MMLU and found that for kind of the subjects you would expect, math, physics, electroengineering, this type of approach was really helpful. Now, we use language models. It doesn't have to be a language model.

• 39:41Speaker 01

  It could be a simulator. These could be unit tests as your verifier, et cetera. But we think this type of approach kind of points towards maybe a very different paradigm for getting better performance. Then just kind of scaling the models and doing a whole new pre-training from scratch. MMLU performance bump was about 3%. And to put that in perspective, the gap between some of the previous best models is often less than 1%. Between, for example, Gemini, 1.5 and Lama 3.1 and things like that. So actually 2.8% or 3% is actually a pretty major gap on MMLU. So pretty intriguing and Yeah, I think a lot of practitioners are hopefully going to explore this setting a lot more.

• 40:16Sarah Guo?

  It's a it's a super cool paper. Do you have any creative ideas about how you could apply some of the ideas here to improve performance on more open-ended tasks?

• 40:27Speaker 01

  In many ways, we're not originating these. I think some of these are baked largely into systems like Alpha Code, Alpha Geometry already. You know, I was pretty inspired to see the Alpha Geometry results recently as well. Yeah, I think that what we'll see people doing is kind of composing, this sounds funny, but massive networks where maybe each stage in the network will basically be maybe some best of A, best of K component with many, many calls to different language models, you know, Claude, Gemini, GPT-4, each with their own, you know, spikes in terms of capabilities. You know, kind of throw multiple of them at questions in many cases and then kind of choose the best response. You might, you know, also ensemble that with other components

• 41:07Speaker 01

  like classical heuristic-based systems and, you know, simulators, et cetera, and kind of compose large networks that may make millions of calls to answer a question. And I think that type of approach, it sounds kind of farcical right now, but I think it'll seem common sense pretty soon. You know, we think that's, we think it's a fairly interesting approach. We've seen a lot of interesting evidence that we'll speak more about pretty soon for things like code generation and agitative tasks in the code regime, you know, for things like design, chip design, you know, for things like actual neural network design or network of network design even, funny enough, in recursive ways. So it's actually a really good, it turns out a lot of these problems that we care about have that property was verifiable and you can compose these systems and bootstrap your way

• 41:52Speaker 01

  to, you know, much higher performance than people might have imagined. So it seems pretty applicable downstream, but there's a lot of open questions, a lot of work to do further. And I think, you know, part of our hope is that the community will explore this more and that these types of workloads that are a bit more parallelizable will become more and more common. There'll be a lot more batch inference, a lot more synthetic data generation. And you won't necessarily need the big interconnected cluster that maybe only OpenAI can afford to do kind of cutting edge work in the future.

• 42:16Sarah Guo?

  Yeah, really cool set of ideas and overall a great conversation. Thanks so much for doing this, Jared.

• 42:21Speaker 01

  No, thank you, Sarah, and thank you a lot. Great to see you. Yeah, great to see you too.

• 42:25Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.