Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend

The current LLM era is the result of scaling the size of models in successive waves (and the compute to train them). It is also the result of better-than-Moore’s-Law price vs performance ratios in each new generation of Nvidia GPUs. The largest platform companies are continuing to invest in scaling as the prime driver of AI innovation. Are they right, or will marginal returns level off soon, leaving hyperscalers with too much hardware and too few customer use cases? To find out, we talk to Microsoft CTO Kevin Scott who has led their AI strategy for the past seven years. Scott describes himself as a “short-term pessimist, long-term optimist” and he sees the scaling trend as durable for the industry and critical for the establishment of Microsoft’s AI platform. Scott believes there will be a shift across the compute ecosystem from training to inference as the frontier models continue to improve, serving wider and more reliable use cases. He also discusses the coming business models for training data, and even what ad units might look like for autonomous agents. Hosted by: Pat Grady and Bill Coughran, Sequoia Capital Mentioned: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, the 2018 Google paper that convinced Kevin that Microsoft wasn’t moving fast enough on AI. Dennard scaling: The scaling law that describes the proportional relationship between transistor size and power use; has not held since 2012 and is often confused with Moore’s Law. Textbooks Are All You Need: Microsoft paper that introduces a new large language model for code, phi-1, that achieves smaller size by using higher quality “textbook” data. GPQA and MMLU: Benchmarks for reasoning Copilot: Microsoft product line of GPT consumer assistants from general productivity to design, vacation planning, cooking and fitness. Devin: Autonomous AI code agent from Cognition Labs that Microsoft recently announced a partnership with. Ray Solomonoff: Participant in the 1956 Dartmouth Summer Research Project on Artificial Intelligence that named the field; Kevin admires his prescience about the importance of probabilistic methods decades before anyone else. 00:00 - Introduction 01:20 - Kevin’s backstory 06:56 - The role of PhDs in AI engineering 09:56 - Microsoft’s AI strategy 12:40 - Highlights and lowlights 16:28 - Accelerating investments 18:38 - The OpenAI partnership 22:46 - Soon inference will dwarf training 27:56 - Will the demand/supply balance change? 30:51 - Business models for data 36:54 - The value function 39:58 - Copilots 44:47 - The 98/2 rule 49:34 - Solving zero-sum games 57:13 - Lightning round

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The things that are brittle right now where you're like, oh my God, like this is a little too expensive or it's a little too fragile for me to use. Like, all of that gets better. Like, it'll get cheaper and, like, you know, things will become less fragile. And then, like, more complicated things will become possible. Like, that is the story of each generation in these models as we've scaled up.

• 0:37Pat Grady?

  On any given day, Microsoft may be the most valuable company in the world. And arguably, no one has been more ambitious, more strategic, or more effective in its AI strategy than Microsoft. The key architect behind that strategy? Kevin Scott, CTO of Microsoft. We've had the pleasure of knowing Kevin for a couple of decades now, dating back to his time at Google, when he overlapped with our partner, Bill Corn. Bill will join us today for a very special episode of Training Data. We hope you enjoy.

• 1:14Pat Grady?

  Kevin, thank you for being here on Training Data.

• 1:17Speaker 00

  Yeah, glad to be here.

• 1:19Pat Grady?

  So just to start, I know you've talked about this before, but for our listeners who might not be familiar with your story, how does a kid from rural Virginia end up becoming the CTO of Microsoft?

• 1:33Speaker 00

  Who knows?

• 1:36Speaker 00

  It's certainly not a, not a, not a repeatable plan, I don't think. I don't know. Like, it is, um... The thing, when I reflect back on my, uh, yeah, my story is, it's just a lot of being at the right place at the right time. So I'm 52 years old, so I was, uh, yeah, I, uh...

• 2:02Speaker 00

  Ten, eleven, twelve years old when the personal computing revolution started to hit full steam. And so like right at that moment when you're a kid trying to figure out what you're about and like what you're gonna latch on to, like I had this like really convenient thing that captured my interest and was a good place for me to ground my curiosity on. And, you know, and I think that's, you know, one of the object lessons in general is if you happen to be interested and like really motivated to learn more and do more with something that is at the same time growing really, really quickly, like you probably

• 2:45Speaker 00

  are gonna end up in a reasonable place. And so, you know, I was interested in computers. I, yeah, I was the first kid in my, or the first person in my family. Neither my mom nor my dad went to university. So I was the first one to graduate with a bachelor's degree. Like I majored in computer science and minored in English literature. Like I had this moment when I was, at this moment when I was trying to decide what I was gonna go do after I got my undergraduate degree where my two advisors were arguing about

• 3:25Speaker 00

  whether it should be PhD in computer science or PhD in literature. And I was very seriously considering both, but I was so broke and just so tired of being busted all the time that I picked the pragmatic path. You know, not, not that, you know, like I still imagine what my life would have been like as a person with a PhD in English literature. And I think it would have been just fine. But like I chose, you know, one of my two equal, you know, interests.

• 4:02Speaker 00

  And then, you know, for a while I thought I was going to be a computer science professor. And at the last minute, this is where Bill and I intersected. Like, I decided I was a compiler optimization and programming languages person through years and years in grad school. And I got almost all the way to the end and I was like, I don't think I want to be a professor anymore. Like, I would work on these things where it was, you know, six months of effort to write a paper and you make some synthetic benchmark 3% better. And I was like, this doesn't feel to me like the way to have a lot of impact in the world.

• 4:43Speaker 00

  And like, I don't want to do this over and over and over again for the next 25 years of my life. And so I sent my resume cold into Google in 2003. And I got an email from this guy Craig Nebel Manning, who had just gone off to New York to open up Google's first engineer, remote engineering office. And like, I had an amazing interview at Google. I don't know whether this was on purpose or not, or like, I just got luck of the draw. But like, it seemed like every compiler person who was working at Google was on my interview slate.

• 5:28Speaker 00

  And I was like, this is amazing. Like, all these people know all of this stuff that I know. And, you know, we can have easy conversations. Now, I worked on nothing that was even remotely close to compilers at Google. And I was confused why all of these people were there. But it was a great interview and I was super stoked. And I joined Google. And Google is yet another one of those things, just like the PC and just like the internet. Like, it was a phenomenon that was growing crazy fast with a bunch of smart people working there. And that resulted in this opportunity. I had to go join this startup, AdMob, when it was very early on, like, right at this pivotal moment when mobile was taking off.

• 6:13Speaker 00

  And you needed things in mobile, like advertising infrastructure. And I helped build. Yeah, the the seminal company, I think, in in mobile advertising and then was back at Google. And then I helped LinkedIn go public, running its engineering and operations team. And then I was at LinkedIn when we got acquired by Microsoft. And so like, yeah, none of that, I think you can plan. It's just like a lot of right place at right time. And, you know, trying at every point you can to do the most interesting thing you can do on the thing that's growing really fast.

• 6:55Speaker 02

  You know, when you talk about your personal history, Kevin, I guess, you know, the focus nowadays is on AI, machine learning. A lot of the practitioners are people with PhDs. How do you think about sort of practical teams for AI? Since you're obviously doing a lot of that work at Microsoft and involved with partnerships with OpenAI and others.

• 7:25Speaker 00

  Yeah, I mean, I think if you are building the really complicated platform pieces of AI, so like the big distributed systems for training and inference, the big like networking and silicon and, you know, system software components or the algorithms that you're using to do training and inference. I think a PhD is super helpful. Like, there's just a huge amount of prior knowledge that you need to have in order to jump into the problem space and be able to like go quickly.

• 8:08Speaker 00

  And like, you know, you need to be clever, but like you don't PhD is, I like, yep, I know you have a PhD bill and are far cleverer than I am, but like usually folks with PhDs are clever, but like they're not the only people in the universe who are clever. So like, I think it's mostly helpful in the sense that you've gone through like a pretty rigorous training regimen where you get a whole bunch of prior art stuffed into your skull and like you, you know, demonstrably can do a very complicated project. And, you know, the PhD projects, you know, look kind of like.

• 8:47Speaker 00

  AI platform, systems, projects, except the AI platform and systems projects are lots and lots of people working together, whereas, you know, when you're getting your PhD, you often are like working in relative isolation on your particular thing. So like, that's the, you know, one of the things people have to learn is like how to get yourself docked into a group and to be able to collaborate effectively with a bunch of other people like yourself. So useful, but you know, there's so much else in AI that needs to be done other than building the platform. Um, you know, and for those things, a PhD is helpful,

• 9:27Speaker 00

  but certainly not necessary. Um, you know, like figuring out how do I apply this to education? How do I apply this to healthcare? How do I, uh, like, how do I build developer tools around this? How do I, you know, do all of the million things that happen when new platforms emerge that you, you know, sort of complete the whole platform into like a portfolio of products and a portfolio of middleware and a portfolio of like all of the other stuff you need?

• 9:55Pat Grady?

  Well, speaking of which, Microsoft seems like it has about the most sort of far-reaching or ambitious AI strategy of anybody out there. Can, can you just kind of say in a couple of words, what is the AI strategy for Microsoft? And then just for fun, if you're going to grade yourselves, what have you done particularly well? What have you done, maybe not as well as you could?

• 10:18Speaker 00

  Yeah. So, I mean, we've been sort of talking about this strategy. Microsoft is a platform company. Like we, I think, have participated or like helped drive a handful of the big platform waves in computing. Uh, like we were certainly one of the pillar companies in the personal computing revolution. Like we had a important part to play in the internet revolution, although I think that one was a far more diversely contributed to revolution than, uh, yeah, than personal computing. Um, yeah, we kind of miss the mobile, uh, computing revolution.

• 10:59Speaker 00

  But like each one of those things, like we have thought about, uh, how do you go build a technology platform for this particular era of technology that allows other people to go build on top of that platform to make useful things for other people? And so that is our AI strategy. It is, uh, like, how do you from frontier models to small language models to, uh, like, highly optimized inference infrastructure, uh, you know, like hyperscale on both training and inference, uh, like

• 11:35Speaker 00

  economies of scale, like making the entire platform more accessible because it's cheaper and more powerful, uh, with every turn of the crank. And like all of the developer tools and safety infrastructure and testing and everything that has to be there in order to have robustly built AI applications, like go build that, uh, and like listen to developers and listen to people building, uh, building AI is as intently as you possibly can so that you are filling in all of the gaps that you can for them as they are encountering problems

• 12:16Speaker 00

  deploying this technology to, to users. Um, so that, that is, that is our strategy. Um, and so, yeah, I think we're doing a reasonable job of it. Um,

• 12:34Speaker 00

  I don't know, like I hate to grade myself. It seems a little bit disingenuous, right? I want to lowlights. Well, so, so, but maybe before I do that, like, you know, let me describe something about my own psychology. So I, I, uh, like I am an engineer and I think most engineers are, uh, like,

• 12:54Speaker 00

  short-term pessimists, long-term optimists. Uh, and so the short-term pessimism is like, you come in every day and you're like, oh my God, this is like a bag of crap. Like, I just don't like any of this and like, everything's broken and like, I gotta, like, I got, I got so much stuff to fix and I'm so frustrated. But you work on all of those things anyway, because you're optimistic that all of the problems can be fixed and that, uh, they're going to be worth fixing at the end of the day. And so, yeah, I mean, the-

• 13:25Speaker 00

  There's a bunch of stuff that I think we're doing really well. Like, I think we have absolutely, along with OpenAI, made very powerful AI dramatically more accessible than it otherwise would have been to a larger group of people. I think you, because of that work that we've been doing alongside OpenAI, we have, we're just seeing lots and lots of customers who otherwise wouldn't be building powerful AI applications. And so, like, I feel like we're doing a good job in the way that we're partnering. I think we're doing a good job in, like, having a really particular point of view.

• 14:07Speaker 00

  And it's not an immutable point of view, but like, it's a point of view about like, what an AI platform ought to look like, and we're trying to, like, make it as complete as we can. Yeah, low lights is, I think, we were a little bit late to, like, some of the basic AI stuff. So it wasn't that we were not investing in AI at all, and like, you can sort of look at some of the work that Microsoft Research had done over the years, and like, MSR was an early AI leader. And I think, you know, the, yeah, I mean, Bill knows this just as well as I did, just

• 14:50Speaker 00

  from his time at, time at Google, and, you know, where we overlapped for a number of years, but many, you know, maybe most of the really important advancements in AI over the past 20 years have been a function of some kind of scale, you know, and it's usually, you got data scale and compute scaling combination, let you, like, do things that weren't possible at lower scale points. And, yeah, at some point, like, that, that scaling of data and compute is so exponential that you get past the point where you can have fractal

• 15:34Speaker 00

  Fragmented bets where you could literally it's it just becomes economically impossible to bet on 10 different things that are all exponentially scaling or have the ambition or the need to exponentially scale simultaneously. And so I think one of the things that we were a little bit late to is like we we didn't put all of our eggs into the right basket soon enough like we just yeah, we were spending a lot on AI, but it was fragmented across a whole bunch of different things. And because we didn't want to hurt any of the feelings of smart people or yeah, whatever, right? Like, I don't even know what the diagnosis was because a lot of that was before I was at Microsoft. We

• 16:15Speaker 00

  like, we we just weren't as quick as we should have been at like saying, nope, scale is what matters. And like, here's how we're gonna focus focus our investments on scale in a principal way.

• 16:27Pat Grady?

  When did that? When did you get religion that scale is what matters? Was there a particular event or a moment that really crystallized that for you?

• 16:35Speaker 00

  Yeah, I mean, I was so I've been at Microsoft for about seven and a half years now and like my

• 16:45Speaker 00

  when I when I became CTO, my

• 16:49Speaker 00

  my job was like, take a take a scan left to right across both Microsoft and the entire industry and try to see where yeah, we we just had holes in execution where like we we were not doing things at that point in time, which was I guess, 2017 early. Where all right, like, what are we not doing today that we're going to deeply regret in 2019 or 2020? So like, two, three years out. And like, the biggest thing on the list was like, you know, our rate of progress on AI was not fast enough. So I'd say mid 2017, like, I had

• 17:34Speaker 00

  religion that that was going to be a big part of my job was like helping us figure out what the strategy was going to be. And then in 2018, like, if anything, the

• 17:49Speaker 00

  BERT, the publication of the BERT, BERT paper from, from Google was like a real crystallization of that belief. So like everything that I had, that was in my analysis, I was like, this, this is as fine an example as anything of like, why we have to really, really accelerate on getting more serious here. And so like, you know, very shortly after that, like I restructured a whole bunch of stuff inside of Microsoft to get us more focused on AI. And then about a year later, we did that first deal with OpenAI. And yeah, we have been accelerating

• 18:31Speaker 00

  our investments and like trying to get more focused, more crisp, more purposeful since then.

• 18:38Pat Grady?

  When you were very early to appreciating the potential of OpenAI, what did you see in them at that time when that first partnership was structured?

• 18:47Speaker 00

  Well, we, we had like, or at least I had this like real belief that what was happening with these models as they scaled, as they actually became a basis for building a platform that like the big shift that was happening is it wasn't just, you know, I ran one of these teams at, at Google where you had a pool of data and a bunch of machines and an algorithm and you were like training a model and like the model was for a specific thing. Like in the case of the thing that I was doing at Google, it was like click through rate prediction for advertising and you know, like a handful of other things.

• 19:30Speaker 00

  And like just outrageously effective, right? But most of the work before this, before GPT.

• 19:43Speaker 00

  Was about those sort of narrow use cases, like you were purpose building models for narrow things and it was just tough to scale like you couldn't you'd invest a bunch of compute and like you couldn't amortize the cost of the compute across anything more than just the narrow thing that you were building the model for and you had to have a lot of expertise that you know if you wanted to replicate all of this, it's like you had to have different data and like different you know AI PhDs and you know different processes every time you wanted to go build AI into an application and what was happening was like yeah, you had these big, large language models that were useful for lots of different

• 20:28Speaker 00

  things. So you didn't have to, you know, have a separate model for machine translation and sentiment analysis and yeah, like all of the different text things that you were doing. And I was like, okay, this is extraordinary. And like they were also becoming more platform like as a function of scale. So transfer learning was working better as things scaled up. And yeah, and this is still the general pattern. So like everything, everything that we understand that large language models can do, plus or minus, like will get better when you get to the next scale point. And on top of that, they will become slightly

• 21:11Speaker 00

  or maybe dramatically more general in the sense that their capability set broadens. And OpenAI had that same belief and they also had like a very principled analysis of like how the those platform characteristics emerged over time as a function of scale. Had a bunch of experimental validation that said that their forecasts were right. And so like it just, you know, you sort of like look at what the forecast says and you're like, this is how much money it costs to run the experiment to see if you're going to be on forecast for the next turn of the crank. And like, you know, it was felt like

• 21:55Speaker 00

  a big number at the time, like it was billion dollars. But like relative to what was happening, it just wasn't a large amount of money. And then

• 22:09Speaker 00

  Yeah, GPT-3 was on forecast and GPT-4 was on forecast. So like it, it, it just was like finding a partner that had the same platform belief that you did and like a track record of, you know, being able to execute through, uh, through these scale points. Like it was. It, it didn't, like I, I've done a bunch of things before that I have way more reservation about, uh, in the past, like just in terms of investments like this one didn't, you know, like there was a bunch of people who didn't agree with me, but like I, I had pretty high conviction.

• 22:41Speaker 02

  You touch on investment. I guess, you know, there's a lot of, um, trade publications now speculating about the cost of doing training and so forth. And, you know, rumors of billions and billions of dollars being spent and so forth. Uh, and I guess based on my own background, I, I think training is going to get dwarfed by inference here pretty soon. Uh, how, how do you see the?

• 23:07Speaker 00

  I should hope so.

• 23:09Speaker 02

  Well, yeah, otherwise we're building models that nobody knows what to do with. That might not be a great investment, but.

• 23:15Speaker 00

  Yep.

• 23:16Speaker 02

  How, how do you see kind of computing landscape evolving and where's it going? You know, the, I think people are joking that all the money is going to Nvidia at the moment.

• 23:27Speaker 00

  Well, look, I think Nvidia is doing a, Nvidia is doing a good job. Uh, so like the, you know, the, the two interesting things that are happening with these models, just in terms of, um, the efficiency of the scale up is each hardware generation's better price performance wise, uh, usually by an extent greater than Moore's law used to work for a general purpose, uh, computing. So, um, yeah, A100 was about, you know, three, three and a half times, uh, better price performance than V100, uh, you know, H100, like not quite that much, but you know, close. Uh,

• 24:08Speaker 00

  you know, the, on paper, the next generation looks, uh, very good as well. Uh, and so like you've got hardware for a variety of reasons, uh, like, you know, part of its, you know, process technology, but part of its architecture and like a, a lot of it is like being able to leverage, uh,

• 24:29Speaker 00

  narrower word size in the computation. So like, you know, instead of needing 64 bit arithmetic, like you're, you know, you're doing arithmetic with much less precision right now. And so like that, you know, there's just an embarrassing amount of parallelism there. Then like we're getting better and better at extracting that architecturally in the hardware. And, um, you know, there's a bunch of innovative stuff happening with networking as well. Like we're well past the point for the frontier models, at least where you can do anything interesting on a single GPU. So for years and years now, like both training and inference have been multi GPU, multi compute node, um, problems.

• 25:13Speaker 00

  Uh, and so like, there's a bunch of innovation happening on the network side as well, which allows you to strap all of the compute together, uh, like at the chassis level, the rack level, the row level of, you know, data center level more effectively, um, which is great because, um, you know, for the nerds listening, like we, we haven't had effective, um, power scaling, uh, uh, or Dard scaling since 2012 or so. Uh, so, yeah, we're not, we're, we're getting more transistors, but like they are not, uh, they're not getting cooler.

• 25:51Speaker 00

  Uh, um, yeah, and like we, we just, we have a lot of, uh, a lot of density issues just with power dissipation that we have to go, uh, go deal with.

• 26:05Speaker 02

  Do you, do you see inference is driving different data center, uh, architecture?

• 26:11Speaker 00

  Yeah, I mean, look, we, we already architect our training environments and our inference environments differently. They just need different things. And like I, I think, you know, all the way down to, you know, silicon and, uh, like through the network hierarchy, you need different things for inference. And like inference is, inference is kinda easier than training. Like training, the way that we're doing it now is like, we go build big environments that take, you know, a few years to build. And.

• 26:46Speaker 00

  Yeah, with inference, like if somebody came along with a like a better, a better silicon architecture, a better network architecture, like a better cooling technology, like it's a much easier experiment to go run. You just go swap some racks out. I mean, like I'm, like my data center people will yell at me, like it's not quite that easy, but it is, it is easier than like having to go like do a big capital project like a, like a training environment looks like. And so like I, you know, intuitively, you would think that that is going to result in more diversity in the inferencing environment and more competition

• 27:25Speaker 00

  and more like a faster rate of improvement. Like that's, and like on the software side, that's certainly what we see, like the inference stack, just because it's such a large fraction of the overall compute footprint and it's constrained because we have more demand than supply at the moment. Like you just have very, very powerful incentives to go optimize the software stack to squeeze more performance out of it.

• 27:55Pat Grady?

  Do you think we'll be in an environment anytime soon where that demand-supply balance changes? Not necessarily at Microsoft, but it feels like we're seeing that at the market level as well.

• 28:06Speaker 00

  Yeah, I don't know. I mean, if we, if we continue to see the platform continue to expand capability-wise and it just becomes more useful, like I think demand increases, if anything. Now, the shape of the demand is probably going to move around. Like I think you're already seeing a little bit of that. Yeah, building, building a frontier model is like a very, very... Resource intensive thing and as long as people are like building frontier models and making them accessible. And like, maybe they're not acceptable quite the way that people want, you know, like they're

• 28:50Speaker 00

  only API accessible and like there isn't an open source thing you can go instantiate and you know, muck around with. But like, it's way more accessible than it was, uh, you know, six or seven years ago where like the only way to access some of this stuff is you had to go work for like, you know, two, two or three tech companies. Uh, and it's so anyway, like, but I think, you know, you do have to ask yourself, and somebody else should do the asking, right? Cause I'm like all kinds of biased, right? Uh, but like, I, I don't know how many frontier models, you actually need.

• 29:29Speaker 00

  If, you know, they're all roughly speaking in the same tier of capability. Yeah, that's an awful lot of money to spend for, you know, things that are, you know, roughly equivalent. It, you know, it's, it's sort of like, you know, if, if you're starting a company right now and you believe that you have to build your very own frontier model in order to go, deliver an application to someone, that's almost the same thing as saying like, I gotta, I gotta go build my own smartphone, uh, hardware and operating system in order to deliver this mobile app. Like, maybe you need it, but like, probably you don't. Um, and so,

• 30:16Speaker 00

  I mean, to Bill's point, like, I think the thing that makes sense, you know, for the market is like, you just, you, you're going to want to see lots of people doing lots of inference, because that means you've got lots of products that have found product market fit and those things are scaling.

• 30:33Speaker 00

  But like, lots of speculative dollars flowing into infrastructure, uh, R and D, like probably ends the same way that, yeah, the, yeah, uh, uh, many speculative infrastructure booms have ended.

• 30:51Speaker 02

  On this, excuse me, on the scaling front, you know, Microsoft published a paper some time ago, pointing out that the quality of training data is maybe at least as important as volume. And I think one of the speculations you see now in the industry is that we're running out of sources of high quality training data. And you're reading at least some articles claiming that various partnerships are being struck to get access to training data and it might be behind paywalls and so forth. How do you, how do you see that evolving?

• 31:32Speaker 02

  Because it feels like we have more and more computation, but we may not have more and more training data.

• 31:41Speaker 00

  Yeah, I mean, I think that was almost inevitable. It is, you know, in my opinion, a good thing that quality of data matters more than quantity of data because it gives you an economic framework to go do the, do the partnerships that you need to go do to, you know, make sure that you're feeding your AI training algorithm or curriculum that, you know, is going to result in smarter models. And like honestly, like not wasting a whole bunch of compute, feeding it a bunch of things that are not. And I think one of the, you know, from an infrastructure perspective, one of the things people have been very confused

• 32:21Speaker 00

  about is like a large language model is not a database, it's not a repository of facts. You know, it's like important for it to, you know, quote unquote, know some factual things, but it is like the world's crappiest database, you know, honestly, like if you need it to be your retrieval engine. And so like, you just shouldn't think about it as like, hey, I got this thing and like, it has to have everything baked into it, like into the model weights themselves so that you can, you know, recall a bunch of stuff. You know, like, as you've seen, like, the, the recall is.

• 33:04Speaker 00

  Imprecise in the same way that human recall is imprecise. So they're just much more efficient ways to do recall.

• 33:12Speaker 00

  Um, yeah, so look, I, I think you, you are. You, I mean, the way that we say, see things developing is like you have data that is valuable for training models. And then you have data that you need to have access to for an application in order for the model to reason over. And like, those are two different things. And I think there are probably two different business models around those things. So, and like, at the end of the day, this is all just, this is about business models, right? Like, people who produce data

• 33:47Speaker 00

  want to be compensative for, uh, you know,

• 33:53Speaker 00

  use of that data.

• 33:57Speaker 00

  And so, yeah, we, we have all of this data sitting inside of search engines right now. Like not, not in, you know, randomized weights, but like quite explicitly, it's like sitting at a, you know, sitting in indices and, you know, being in Google and whatnot, just waiting to be retrieved. And like,

• 34:17Speaker 00

  you know, plus or minus, everybody's okay with that because there's a business model there that makes sense. Uh, so you like, you enter a query and like, you're either sending traffic or, you know, like there's, you know, SEO and like advertising, like a whole bunch of like business model that surrounds that. I think we'll figure out a business model for that, um, referral data so that like, you know, when an agent or an AI application needs to retrieve some information from someone so that it can, you know, reason over it and give the user an answer. Like we will figure out the business model for that. Like it'll either be subscription rev share, it'll be licensing, it'll be like some new flavor of advertising.

• 35:01Speaker 00

  Uh, yeah, I was just telling someone the other day, like if I was in my twenties right now, like, you know, for, for all of your entrepreneur, like somebody ought to be out right now figuring out what the new ad unit is for agents. And like just building the company. Because like it, it, it will have the same characteristics and qualities as, uh,

• 35:25Speaker 00

  Yeah, previous ad units, like you have people with information and products and services who are going to want to get to the attention of someone who might want those data and products and services and like, quality is gonna matter and the relevance is gonna matter and a bunch of other things. And I would be shocked if there isn't a auction model for, you know, that's gonna be the right way to value everything and, you know, like, yeah, maybe there's referrals and like referrals will have some economic value. I think retraining is just a little bit different because it's very, very hard when you're

• 36:05Speaker 00

  building that model to, at the time that you're doing the building to really be able to ascribe monetary value, value to a particular token of input. Just because it's contribution to the model, like the same way that a word from, you know, Moby Dick has like a very diffuse contribution to like your own human intelligence, you know, even though you definitely read it at some point in your career or your life. Like, how valuable that is to like forming, you know, Bill Corin or Kevin Scott's, you know,

• 36:50Speaker 00

  useful intelligence, like who knows?

• 36:53Pat Grady?

  Speaking of which, this, one of the things that we hear a lot of times is the, the value function is in some ways the bottleneck to broader reasoning capabilities. It's easy enough to construct a value function when you're playing a game with a, you know, with a, with a known winner and loser like Go or Chess or Poker or Diplomacy. But it becomes a lot harder to construct the value function when you're going into broader domains and it's, and it's things like assigning the value of Moby Dick to Kevin Scott's life, you know, that sort of thing. Are there, are there practical solutions to this? Are there practical implications to this? I guess the broader question would be, where do you see the overall field of reasoning in LLMs going?

• 37:36Speaker 00

  Well, look, I think people are trying to get at this. So you've got a bunch of benchmarks, like- Yeah, GPQA and MMLU and yeah, like we're just sort of rolling through a bunch of benchmarking paradigms to like try to come up with scores of performance for these models, like whether it's reasoning capability or yeah, and yeah, I think yeah, one of the interesting things we've seen over the past handful of years is like, we just are very quickly saturating these benchmarks like where you like one emerges and then

• 38:13Speaker 00

  yeah, within a model generation, like you'll completely or like get very close to saturating the particular benchmark and then you gotta go find something else to help be your guiding light. So but like, let's just sort of assume that like, you know, you'll have some interesting benchmarks that are correlated with the reasoning capabilities you want models to have. Um, yeah, then the question, it's just an expensive experiment to run. Like you could run an experiment where it's like, okay, like I'm going to train a model like, you know, with this information in and or out and like, does it get better or worse at um, you know, performance on these reasoning benchmarks?

• 38:56Speaker 00

  And you know, like, I think all of us have done different versions of those experiments, like they're just extraordinarily expensive to run at the most granular scale you can imagine running them.

• 39:11Speaker 00

  Part of that paper that Bill referenced, like textbooks are all you need is like a

• 39:17Speaker 00

  it's not the full story, but it's like part of a story that is like, you know, just sort of evaluating like token contribution quality to model performance. So yeah, I like, I think it's

• 39:31Speaker 00

  and everybody's got every incentive in the world right now to try to figure out like what that is. Uh, if for no other reason than you, uh, like in a world where you're synthesizing data, you're literally spending compute to generate synthetic tokens for training. You, you really want to make sure that the tokens that you're generating are actually useful or not.

• 39:57Speaker 02

  Where do you think the models are at the moment? I, you know, I think Microsoft has introduced a whole bunch of co-pilots to try to help end users with their, with your products and so forth. On the other hand, I see lots and lots of companies trying to build agents that can be kind of autonomous actors now. That's a wide spectrum of kind of expected performance of what the models can do. Where do you think we are? Where do you think we'll be in a couple of years?

• 40:29Speaker 00

  Well, yeah, so I think that's a super good question. And there's, you know, there's even a philosophical thing there about, you know, what it is we should want.

• 40:41Speaker 02

  Well, there is the, the specter of everybody's job getting replaced, right? So.

• 40:46Speaker 00

  Yeah, you know, somebody, look, we, we chose the, the name co-pilot for the things that we were doing relatively deliberately because we, we want to, at the very least, encourage everyone who's building these things inside of Microsoft to think about, how can I help augment someone who's doing some form of cognitive work? So like, we want to build, you know, assistive, not substantive tech.

• 41:21Speaker 00

  And, and, you know, the good news is it's also easier to think about how to go from, you know, sort of rough frontier model capability to useful tool when you're narrowing it down to a domain.

• 41:36Speaker 00

  And so like, I, I think that's been a reasonable deployment path. And like, we've got a handful of our co-pilots that have real market traction right now and are like, you know, in daily use by like, a lot of people doing, you know, real, non-trivial cognitive work. And, you know, and I think that will expand over time.

• 42:00Pat Grady?

  What are, just on that, what are some examples of co-pilots that have really like hit the bullseye already versus maybe co-pilot where the technology's not quite ready?

• 42:10Pat Grady?

  In terms of like jobs to be done.

• 42:12Speaker 00

  Yeah, like I, I think GitHub, GitHub Copilot is like probably the, you know, the thing we've talked most about and like there's, you know, the most, uh, public conversation around, uh, you know, like it, it's, it's been a, been a hit. Um, you know, it is genuinely useful. Um, yeah, we, we've got some other co-pilots that are like that, that are, um, you know, that are, that are super useful. Um, but you know, I think the thing that Bill was getting at, like the more general, the more general the co-pilot is, like the, you know, the, the harder

• 42:52Speaker 00

  it is to have it actually take, uh, very high precision action on your behalf autonomously.

• 43:04Speaker 00

  Um, yeah, particularly if, you know, it's doing something where it's representing you,

• 43:11Speaker 00

  where, yeah, there are stakes and, you know, consequences and accountability back to you if this agent makes a mistake. And we're trying to be very deliberate there because I think one of the things that you don't want to do is, uh, introduce a thing that's going to make a whole bunch of, uh, these sort of errors where the user's first reaction is like, this doesn't work and like, I'm not going to try it again for a good, long while. Uh, so we'd rather have it be very good before we introduce it. Uh, you know, which again, you have means you're, you're sort of optimizing for use cases, not for like, you know, super,

• 43:55Speaker 00

  super broad things. I mean, there's a good, like we, um, we did a partnership recently with, uh, you know, with Devin, uh, um, which I think is another one of these like very interesting, like, you know, use case specific things where it's like, you know, frontier model plus a whole bunch of other stuff that is like optimized for like giving humans high quality recommendations for actions that they can take. And then when you click accept on the action, uh, like you, uh, you have reasonably high confidence that, you know, it's going to work and you haven't made another set of problems for yourself. Inside like I'm, I'm guessing too, you all see this in your portfolio companies.

• 44:39Speaker 00

  Like there, there seem to be a bunch of companies out there right now that are like doing exactly this and, and that it's, you know, useful and working.

• 44:46Pat Grady?

  Well, it's interesting because we hear, one of the things that we hear fairly consistently from the companies that are further on in their AI journey. You know, everybody kind of starts in the same way where they start playing with OpenAI and then maybe they start using some of the other proprietary foundation models. And then incorporate some open source models and maybe they have some of their own stuff. There's a vector database in there somewhere.

• 45:07Speaker 00

  Yep.

• 45:07Pat Grady?

  From an architectural standpoint, it feels like people tend to go on a, not quite the same journey, but journeys that sort of rhyme. But then what we hear from them when they're 12 or 18 months down the road is there's kind of this massive 80 20 rule at play. And maybe it's a 98 2 rule, but you can automate most of a task pretty quickly and pretty effectively. But getting it to the point where it's actually end to end, running autonomously in a way that is compelling and consistent enough that you can actually trust it. Kind of that last mile, that last couple percent that makes you really trust it. Yeah, it seems like that's been pretty elusive for a lot of tasks.

• 45:47Pat Grady?

  And so I'm, one of the things that we're really curious about is, okay, well, when do the foundation models themselves, you know, get good enough to knock out that last 2%? Or is that a domain specific thing? And that's really the job of the software vendor who, who lives on top of the platform to figure out that last 2%.

• 46:05Speaker 00

  I, I, look, I think it's going to be both for a while. Like the, the two things that I think you can trust. Yeah, I, I know you, you guys are probably going to ask this question at some point, but um, like we're, we're, despite what other people think, we're, we're not diminishing marginal returns on scale up. And like I, I try to, I try to, you know, help people understand like, you know, there, there is an exponential here. And like the unfortunate thing is you only get to sample it every couple of years because it just takes a while to build supercomputers and then train models on top of them. And, and so the, the next sample is coming.

• 46:47Speaker 00

  And like I, I, you know, I can't tell you when and I can't predict exactly how good it's going to be, but it, it will almost certainly be.

• 46:58Speaker 00

  Better at like the things that are brittle right now where you're like, oh my God, like this is a little too expensive or it's a little too fragile for me to use. Like all of that gets better. Like it'll get cheaper and like, you know, things will become less fragile. And then like more complicated things will become possible. Like that, that is the story of each generation in these models as we've scaled up. And so like, you know, we even think about this inside of Microsoft and like one of the category errors that our own developers who are building these AI products can make is like get too convinced that the only way to solve my problem is like, I have to go take the current frontier and supplement it with a whole bunch of things.

• 47:40Speaker 00

  But what you do have to do, but like you want to be very careful architecturally when you're doing that, that it doesn't prevent you from taking the next sample when it arrives. So you just want to architect these, these applications where when the new goodness comes, you can go plug it in. And like, you'll have to go optimize that as well. Uh, like, I think that's just sort of the grind that we're on. But like you just, like the, the thing that was killing us internally is I, I would have teams inside of the company who would look at a frontier model and say, oh my God, there's no way that we can ever deploy a product on top of this because like this is fragile and this is too expensive.

• 48:20Speaker 00

  And so, you know, like, please give me like giant pools of GPUs and like, let me go spin up a big team doing, uh, you know, like a very, you know, tailored version of this. And like, we're going to build a specific model and like, and, and yeah, they, they would go off and like spend a whole bunch of money and like the thing would be, you know, a little bit better cost wise at the same level of performance as the current frontier. And then the frontier would snap to the new point and it would be just doomed. And, and so like you, you just architecturally don't want to get trapped by that. I don't think. I mean, like that'd be my advice I give to everyone is like, just give yourself the flexibility to snap to the new,

• 49:05Speaker 00

  you know, to the new frontier when it emerges. And like that lets you preserve all of your skepticism. You can believe, uh, you know, believe all you want that the new frontier is not coming. Like go, you know, like, you read your favorite Twitter troll that says it's all over in a sham and like, and like, but, but just give yourself the option that, you know, maybe what's been happening for six years now is going to continue.

• 49:31Pat Grady?

  That well, hearing, hearing that, hearing that we are not at diminishing returns to scale, I'm going to count that as good news. And so let's stay on the theme of good news. I know that you're a short-term pessimist, long-term optimist. Can you give us some of the optimistic point of view for where, where we're heading in this world of AI? Like, what are the, some of the things that you're most excited to see in the world in five or 10 or 15 years, or whatever you count as a longer-term horizon?

• 49:58Speaker 00

  Well, look, I think the thing that everybody ought to like spend some time thinking about is where are the gnarliest zero-sum problems that we have in society? Like where, where are the things where like we just are fighting with one another? Or, you know, like we are immiserating people because whatever it is that people need, there doesn't appear to be enough of it. Uh, and I think for a good number of those things, like what you have to have to, to, to turn them into non-zero-sum games to, yeah, to create abundance, uh, and to relax some of these constraints is you have to have technological breakthroughs.

• 50:42Speaker 00

  It's, it's like the only thing reliably that's ever turned zero-sum to non-zero-sum in, in human history. It's like some tech has to come along that like, you know, lets us, lets us have more. Um, you know, like, you know, whenever tech comes along and like creates more, it doesn't mean that the more gets, uh, you know, equitably and uniformly distributed. Um, and, and like, I think there are real conversations to go have about that. But like, what you do want is the more, uh, and you want it to, you know, be directed at things where like we, we're just having a tough time right now. Um, yeah, I like, I'll, I'll tell this, uh, you know, story.

• 51:25Speaker 00

  That I've told a couple of other times recently, but you know, my mom, like I grew up in rural central Virginia, my mom's 74 years old and like she's suffered from this thyroid condition called Graves' disease for 26 years. And so, you know, when you have Graves' disease, your thyroid is hyperactive, you're generating too much thyroid hormone. And so they go in and like irradiate your thyroid gland to reduce its activity level. And then you take hormone replacement therapy to like upregulate your hormones for the rest of your life. And so she was having some blood pressure issues and her doctor, you know, dorked around with

• 52:05Speaker 00

  her dosage of this hormone medicine. And then like she just had like some serious health issues as a consequence of that, that landed her in the ER in rural central Virginia like six times, like in a pretty, you know, short number of weeks. And yeah, the interesting thing there was the first time she went to the ER, like she was presenting all of these cardiac symptoms and it was pretty clear they hadn't even read her chart. Like it hadn't registered on the that she had Graves' disease and like the thing that they needed to do was like go order a TSH panel to see what her like thyroid hormone levels were. Like if

• 52:47Speaker 00

  they'd done that right away, like they would have said, okay, like we gotta go, you know, adjust your medicine. Yeah. And like I'm not even ascribing ill intent, like, you know, this is a healthcare system that is egregiously overburdened. Like this is not a place where, you know, like you've got this influx of, you know, talent like coming into, you know, this part of the country. And like, it's a lovely part of the country. Like I, I like, love it. Yeah. So like, I'm not criticizing anything. It's just, you know, they, they have an aging population and they don't have enough like young people coming in to be

• 53:30Speaker 00

  things like doctors to like help this healthcare system keep up with all of the challenges that they have.

• 53:38Speaker 00

  If some of those doctors had had access to GPT-4 and like it was an approved, you know, product use, all they would have, what it needed to do was like put the symptoms that she was presenting in and her medical record and it would have said, hey, she needs a TSH test. And if you would put the TSH test result in, the recommendation it would add was like, look at the dosage of her hormone replacement therapy that she's on. And like if, like this isn't theoretical, like I did this. Like it could have helped alleviate a massive amount of her suffering right now.

• 54:22Speaker 00

  And like, I think the only reason like that she got out of this tough situation that she was in was I had to intervene. Like I, I've sent her to a specialist that was, you know, 400 miles away that she couldn't have gotten into without, you know, special, you know, and like it's, it's ridiculous. Like, yeah, there, there are so many 74 year old, old Southern ladies or old Midwestern ladies or like folks who are going through similar sorts of things who do not have someone who's going to go in and intervene on their behalf, who are suffering unnecessarily because we're not even deploying the technology that we've got right now. And like, it's just going to get better.

• 55:07Speaker 00

  And so like, that's the thing that I'm excited about. It's like, let's, let's go, let's go give kids a leg up in education. Let's go fix some of these crazy problems we've got in the healthcare system that is just, you know, absent technological intervention is just going to get more strained over time. You know, let's equip our scientists with better tools so that they can, you know, find better carbon capture catalysts so that, uh, you know, like we can design safer modes of transportation so we can more quickly get to a, you know, like post-carbon economy. Like it's just so many things we can go do with this stuff.

• 55:51Speaker 00

  Like I'm super, super optimistic about it. And so, you know, like it just kills me like.

• 56:01Speaker 00

  What we don't want to do is get distracted on, you know, like things that are

• 56:11Speaker 00

  just, you know, effectively noise in the ecosystem right now. And, you know, we're, we're getting so sideways sometimes with,

• 56:25Speaker 00

  you know, like this model like said something that hurt my feelings or, you know, and like, and I don't, I don't want to dismiss like, you know, people's feelings matter and like, you know, I'm, I'm not trying to be a jerk here, but I, I do want to make sure that as we're thinking about how we develop and deploy the technology that we are always remembering like what the cost of not deploying the good is. Because that, that is a high, high cost. Mm hmm. Very well put.

• 56:59Pat Grady?

  Yeah. Yeah, I'm sure.

• 57:03Pat Grady?

  Probably a good note to end on, I think.

• 57:07Speaker 00

  Well, we have, we have one other question that we like to ask people and it's a quick one. So I'm going to ask you, and I'm going to ask you this one. Who do you admire most in the world of AI? You know, I, I was, I was thinking about this. So I, I think it's Ray Solomonoff, who was one of the, one of the folks who was at that Dartmouth workshop in the, in the 50s where, you know,

• 57:36Speaker 00

  Marvin Minsky and

• 57:40Speaker 00

  Simon and, you know, like a whole bunch of folks convened that summer and like, they were all interested in machine intelligence and they, they coined the, the term artificial intelligence at that workshop. And like, the reason that Solomonoff is so interesting and like, not, not many people I think know who he is outside of computer science is like, he was the one from the very beginning who was pushing on this whole notion that probabilistic methods were going to be very important for the development of AI. And, you know, when, when I was in grad school in the 90s,

• 58:21Speaker 00

  The prevailing academic theories about how we were going to get to AI were all about like, okay, well, you know, like there's some magic, you know, minimalist calculus about human intelligence and like, we've just got to figure it out and it's got to be, you know, rule-based systems and ontologies and, you know, symbolic reasoning and like a bunch of like, stuff where, you know, we like, we do in physics, we were going to have to like, you know, divine, you know, the inherent simplicity in the system, like, figure out what the rules are. And as soon as we understood the rules, like, we'd be able to make software emulate human intelligence. And Slomnov is like, no, no, no, like, this

• 59:05Speaker 00

  is just a, intelligence is an extraordinarily complicated phenomenon. And like, the only way that we're ever going to really get there is modeling it with probabilistic methods. And he was right. And like, he was, he was judged wrong for a very long time. And so I really admire his contrariness. Yeah, all the way back in the 1950s and like, he stuck with his, he stuck with his beliefs his entire career. And I don't know whether Ray actually like, lived to see how right he actually was.

• 59:49Speaker 00

  That's a great answer and a great story. Thank you, Kevin. Yeah, you're very welcome. Thank you.