#324 Sharon Zhou: Inside AMD's Plan to Build Self-Improving AI

AI is not just getting smarter. It is getting faster by learning how to optimize the hardware it runs on. In this episode, Sharon Zhou, VP of AI at AMD and former Stanford AI researcher, explains how language models are beginning to write and optimize their own GPU kernel code. We explore what self improving AI actually means, how reinforcement learning is used in post training, and why kernel optimization could be one of the most overlooked scaling levers in modern AI. Sharon breaks down how GPU efficiency impacts the cost of training and inference, why catastrophic forgetting remains a challenge in continual learning, and how verifiable rewards from hardware profiling can help models improve themselves. The conversation also dives into compute economics, synthetic data, RLHF, and why infrastructure may define the next phase of AI progress. If you want to understand where AI scaling is really happening beyond bigger models and more data, this episode goes under the hood. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Preview and Intro (00:25) Sharon Zhou's Background and Transition to AMD (02:00) What Is Self-Improving AI? (04:16) What Is a GPU Kernel and Why It Matters (07:01) Using AI Agents and Evolutionary Strategies to Write Kernels (11:31) Just-In-Time Optimization and Continual Learning (13:59) Self-Improving AI at the Infrastructure Layer (16:15) Synthetic Data and Models Generating Their Own Training Data (20:48) AMD's AI Strategy: Research Meets Product (23:22) Inside the NeurIPS Tutorial on AI-Generated Kernels (30:59) Reinforcement Learning Beyond RLHF (39:09) 10x Faster Kernels vs 10x More Compute (41:50) Will Efficiency Reduce Chip Demand? (42:18) Beyond Language Models: Diffusion, JEPA, and Robotics (45:34) Educating the Next Generation of AI Builders

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Catastrophic forgetting is definitely a problem, especially in post-training and when you don't have access to the original pre-training data.

• 0:08Craig S. Smith?

  How much of that is focused on improving the design of AMD hardware? And how much of it is focused on general model development, which AMD is not doing, right? AMD does not develop its own models.

• 0:23Speaker 00

  I think people want infinite chips, Craig, so no, it doesn't relieve the pressure. I am Sharon. I'm the VP of AI at AMD. And I think about self-improving AI, self-improving LLMs, which we'll get into later. But my background comes from AI research. So I used to be an AI researcher at Stanford where I did my PhD with Andrew Ng. I taught there as adjunct in generative AI back before all this ChatGPT stuff. And after Stanford, I started a startup, an AI infrastructure startup, doing post-training of language models, actually on AMD GPUs. This was started a couple of months before ChatGPT launched.

• 1:06Speaker 00

  And most recently, over last several months, we have transitioned now to AMD. So my team and I are there now and very excited to enable more people to use compute and to get access to compute. Because that really is the limiting factor, one of the big limiting factors for developing AI and being able to enable more people to steer these models. So that's what I'm really excited about. And yeah, that's why I'm here.

• 1:36Craig S. Smith?

  Yeah, and I do want to talk about self-improving AI. Can you start by defining what we're talking about when we talk about self-improvement? Are we talking about models that rewrite their own code or something like refining their own training data?

• 2:00Speaker 00

  Yeah, I think that's exactly it. It is a broad category, but essentially it's the idea of these models being able to edit any part of themselves to improve themselves. So whether that be the data, whether that be the actual model architecture, whether that be how they evaluate themselves. And actually, the part that I'm working on is below all of that, and it's actually how fast they actually run on the GPUs themselves. So they are writing the kernel code that underlies these models to run faster on these GPUs, and to run effectively on them. Yeah, and on new hardware, too. So that's been really exciting to see.

• 2:37Craig S. Smith?

  Yeah, I just read about kernel evolve.

• 2:41Speaker 00

  Yes.

• 2:42Craig S. Smith?

  Who is doing kernel evolve? I've forgotten. Is that Google?

• 2:47Speaker 00

  Meta.

• 2:48Craig S. Smith?

  Is that your guys' work?

• 2:51Speaker 00

  So there's a lot of different pieces of work around kernel generation and being able to use LLMs to generate these kernels. So we're doing some of that, but I think that work might have been from Meta. But there's work across... Yeah, I think there's work across the board, across the industry, that is very important, I think, towards this end, because it enables more people to get on different types of compute. What we did most recently was, in collaboration with actually a bunch of different institutions like Meta, Google DeepMind, ML Common, Stanford, Nvidia, et cetera, was a NeurIPS tutorial

• 3:32Speaker 00

  on generating kernels using AI. And so we presented that, and basically goes through how we're using AI agents to generate these kernels, and how we're thinking about, you know, post-training these models to generate kernels more effectively. Because what's really exciting about kernel generation and kernel development is actually, we have the profiler. So we have the ability to actually see how fast the generated kernels are on the chip themselves. And so that's really exciting. My team were also working on a kind of more robust production-level benchmark to share with the community as well, as well as different techniques to modify the models to do better on this task.

• 4:16Craig S. Smith?

  Okay, the, yeah, and for, again, I'm trying to make my, uh, my podcast a little more accessible.

• 4:24Speaker 00

  Oh, yeah, of course.

• 4:25Craig S. Smith?

  Audience. So kernel in this sense, because kernel is used in.

• 4:29Speaker 00

  Everywhere, too many places.

• 4:32Craig S. Smith?

  But kernel, in this sense, you're talking about a small, uh, piece of software, uh, that, that lives on, uh, the, uh, the processing unit, uh, that performs a specific task. Is that right? And, and what are the, the kernels that you're, uh, that you're generating?

• 4:55Speaker 00

  Yeah.

• 4:56Craig S. Smith?

  From AI?

• 4:57Speaker 00

  So, um, so there are many layers of the stack for AI. There is the model layer where we kind of build out the model architecture. This is where people talk about transformers and attention, um, and kind of write that out in PyTorch, uh, or JAX or TensorFlow. Um, and so people are like building out models there. And then, uh, they're using maybe hugging face on top, or they're using different tools to leverage those models. Now underneath all of that, um, are ways where you're actually, the model is running on the GPU. And there are many, many layers, but one of the layers is making those models run really fast on the GPU. And, um, it's not, it, it, sometimes you can like break that down into small, different pieces,

• 5:41Speaker 00

  because it's a lot of matrix multiplications, for example. It's a lot of different operations happening on the GPU. And so for each of those operations, you can have a way of basically optimizing, um, the speed of that on the GPU itself. And so, uh, for a given piece of hardware, it may have been designed to be of a certain speed. Um, but you do have to write the software to connect the models of today and of tomorrow ideally, um, to that hardware. And that's kind of what that layer is doing. It's, it's improving the efficiency. So it's utilizing the, um, GPU effectively, both the memory, so like what, you know, the GPU has as memory and storage, um, as well as the actual raw compute power.

• 6:25Speaker 00

  So being able to schedule all of that and and use that effectively and maximally is is what kernels do. And this is really important because it's expensive, right? Like GPUs over a long time and over just a large capacity when you're parallelizing, it becomes expensive. And so you want to be able to eke out as much as you can on on a on each GPU.

• 6:50Craig S. Smith?

  Yeah, and and I I understand how and and you're using evolutionary strategies to to develop these kernels.

• 7:01Speaker 00

  Yes, I would say it's a combination of evolutionary strategies, agentic strategies, as well as different types of post-training strategies to be able to get these models to actually be able to write that code effectively, or at least assist our internal kernel engineers to do so. And I think one of the most important things is to do so on like useful kernels as well, like the most important kernels for for language models.

• 7:29Craig S. Smith?

  Yeah, well, again, for listeners, what are the most useful? What's what's an example?

• 7:34Speaker 00

  Well, I would say matrix multiplication. So when you're doing, when you're learning about the math of a neural network, you'll see that there's a lot of matrix multiplications and a lot of those need to be optimized. And you know, at first you might think, oh, isn't there just one, you know?

• 7:51Craig S. Smith?

  Yeah.

• 7:51Speaker 00

  These multiplying, but actually, no, you can optimize different sized matrices that multiply together. And you can get like even better performance for something of a different size, like you can optimize that. So there are thousands, possibly hundreds of thousands of of those kernels that you could optimize for for language models. And of course, for a particular customer, a particular like foundation model company, they may have a certain set that they particularly care about for their architecture. And those are ones that are are of high priority.

• 8:24Craig S. Smith?

  Yeah, and and the these kernels in the past have been written manually, is is that right?

• 8:32Speaker 00

  Yeah.

• 8:33Craig S. Smith?

  So having there's a tremendous productivity gain in having AI either do it autonomously or do it as an assistant. What what sort of impact does that have on AMD?

• 8:51Speaker 00

  I think it's having larger and larger impact. So I would say that, yeah, so historically, they've been written manually. And just to give you a sense of how much knowledge this kernel engineer needs to have in their head. They need to know about the GPU architecture. So not just general GPU architecture, not just your average architecture class, but like that generation of the GPU. And what changes have been made, you know, what would is actually, you know, how to actually write the code to run the software on it. And all the different, you know, possible new things that have been invented at the hardware layer. They have to understand all those things. And then they also need to understand at least enough about what this matrix multiplication is doing, right?

• 9:35Speaker 00

  And to actually then write that optimally in the code. And I would say that having those two areas of expertise in one person's head is quite rare. And as a result, it's very, very valuable, but also it is bottlenecked at, at a lot of hardware companies. Or at actually a lot of different companies, including Frontier Labs. They also have people writing kernels to speed up their models as well, since they have more visibility on their own models. That may not be fully visible to everyone else. And so I think it's a rare skill. And if you're listening, I encourage you to go learn about it. But of course, we're also teaching the models about this, right? And so what's exciting is for AMD, we have the Rockem stack, which is open source.

• 10:20Speaker 00

  That's the equivalent of Nvidia's CUDA stack, which is not as open. And so that being open actually helps us from a language model perspective, because the models can kind of read that data, right? And train from that data and use that data to then learn about the Rockem stack and then write those kernels for us. And so that's, that's what we're doing here. And that's, the impact is massive from both a, I would say, like direct customer lens of what it could do to improve that. But also, I would say, and the long tail. So there are researchers working on various different models that are not just chat GPT or, you know, just Grok or just, you know, just a few of these frontier models.

• 11:05Speaker 00

  But they're working on a bunch of different other models. And they're creating, they're inventing the next model, right? And those also need to be optimized. And I think having something that is Almost just in time, ideally just in time, but you know, like almost automatically give you a optimized kernel for is is very essential to enabling just the entire ecosystem to be able to move over.

• 11:31Craig S. Smith?

  Yeah, and when you say almost just in time, is this is being done on the fly? Or?

• 11:42Speaker 00

  That is my goal, Craig. That is my goal.

• 11:46Craig S. Smith?

  Yeah, because you talk about self improving AI. That suggests that there are models that are evolving over time or improving over time, and that leads to the question of continual learning, which is kind of the holy grail. But there is catastrophic forgetting and expandability of neural nets and all those problems. Can you talk about where the the research stands on continual learning?

• 12:21Speaker 00

  Yeah, so catastrophic forgetting is definitely a problem, especially in post training and when you don't have access to the original pre training data. Because I think we found in the literature at least that even if you include, originally it was like if you include 20%, but actually if you include only 1% of the pre training data back in during post training, you can actually prevent catastrophic forgetting, basically enabling the model to actually like connect back to its representations back in pre training, or at least help significantly for for that. And I think it again depends on data access in terms of who is doing some of that post

• 13:04Speaker 00

  training, but I think that's what the literature points to. So it's the extent to which you can access some of that data or use some of those online data sets to just bring back some of those data points from pre training that could actually help your model significantly in preventing catastrophic, something like catastrophic forgetting. But of course, I think this is something that needs to be continually monitored. So I think the frontier labs maybe don't have this problem as much, but as an individual who is probably doing some type of post training, that is something that you do have to consider, especially as you get your, especially as your workload gets heavier and heavier. Small little bits of fine tuning that doesn't really change as much, but.

• 13:46Craig S. Smith?

  So self improving, are you, yeah, define self improving for me. I mean, what are you improving in or attempting to improve in a model?

• 13:59Speaker 00

  So the part of self improving that we've been working on is using these language models to write low level kernel code. So this is code that makes the language model itself run really fast on AMD GPUs. So get the model to actually write its own code to run even faster on the GPU so that it can learn even faster. And so this is the self improving loop that we're enabling the models to do here. That's really exciting because then that'll enable more people to access this compute and it'll label more powerful models within a shorter amount of time.

• 14:33Craig S. Smith?

  And is that something then that you deploy with AMD hardware? That people can use with their models? Or is it a separate model that you make available to users of the hardware?

• 14:51Speaker 00

  Yeah, this is such a good question. Right now it's being used internally and we're releasing different data sets and evaluation benchmarks and methods out into the world. So we did a NERIP tutorial that presented a lot of this work. And we're preparing for ways to actually package this and to make it make it possible for our customers to use it as well.

• 15:13Craig S. Smith?

  Yeah, and then the the idea of editing a model editing its own training data. Or why would you want to do that? And can you give me an example and how is is that being done?

• 15:33Speaker 00

  Yeah, so we're editing the kernel kernel code for the model. In terms of editing training data, what that looks like is usually this is like a synthetic data setup. So the model is generating data for the next round of training. So yeah, self improving AI is more general than just having the models generate kernels and write these low level kernels to make the models really fast on GPUs. It could also touch on, you know, generating data for itself as well as generating evaluation for itself. And also like generating different architectures for itself as well. So basically using the models to improve itself at any stage of the pipeline.

• 16:17Speaker 00

  I think data generation is really interesting, just, you know, synthetic data generation overall. And there's been a lot of work and research and speculation on it. And I think, you know, we have found that using synthetic data to train these models is very helpful. Especially if it's actually not a frontier model and you're distilling from a frontier model. It's not quite distillation, sorry, but it is using synthetic samples from a frontier model to then supervise a smaller model that you might be training. That's very effective.

• 16:49Speaker 00

  And I think like there's also debate around synthetic data generation and how much that might may or may not lead to the, you know, collapse as the model trains, since it's looking at its own kind of data. But I do think it is an important space to continue monitoring and to continue understanding. Yeah, so I think what's really interesting is this holy grail that everyone's looking towards of having the model fully improve itself and create its own next generation. I think we're both further than we think and closer than we think. And so I think we're further than we think in the sense that there are, there is still a lot of human expertise that goes into, you know, how these things are laid out.

• 17:34Speaker 00

  And what we should try next, for example, of what the model, you know, what we should add into the model, whether it be new data source or not. But I do think there's more than we think. As well in terms of the use of AI to help us code in general. So the amount of code that AI is writing is pretty enormous. And it is supporting a lot of engineers that are putting these together and creating the next generation of that AI. So I think it's both, we're further and closer than we think. Basically, we're further than we think from the like ultimate like, okay, there's only one person works at OpenAI now, it's only Sam there.

• 18:17Speaker 00

  And it's just the end with ChatGPT versus we're, we're closer than we think because actually every, everyone is actually using ChatGPT quite significantly to write that code. So I, I think it's, it's in, in, in that state right now. But it's, it's not exactly like autonomously improving itself fully. There are pieces that you can launch that feel more autonomous, where you could have it like reflect on its own output and then improve that. It can look at the profiler information, so it can look at how fast a kernel ran on the GPU and improve that. But it's not quite like a, I'm gonna let this go and

• 19:01Speaker 00

  come back in a month and it will be fully solved.

• 19:05Craig S. Smith?

  Yeah, uh, yeah, well, let's, can, can we back up a little bit? As I said at the beginning, AMD is a hardware company. You're in charge of AI at, at AMD.

• 19:18Speaker 00

  Part of it, yeah.

• 19:19Craig S. Smith?

  You're part of it, yeah. Is, is that

• 19:24Craig S. Smith?

  is, is that a research function? Because, and, and how much of that is focused on improving the, the design of, of AMD hardware and how much of it is focused on

• 19:44Craig S. Smith?

  on general model development? Which, which AMD is not doing, right? AMD does not develop its own models.

• 19:53Speaker 00

  Oh, we have actually pre-trained some models and released them, but they are small in the single-digit billion parameter range. And yeah, we're probably like not developing models in the sense of like not chat GPT. But, uh, but yeah, we, we do have some, um, that are, uh, open source models. Um, yeah, so I focus on, I, I almost call it like product research. I think there is a research element to it that we don't know how good, um, the models are at solving this full task end to end. Um, but there is a product component in that this is part of the loop with what becomes customer facing. Right? So then this becomes part of the loop of what we, um, want to make available to our customers.

• 20:38Speaker 00

  And so, uh, it's, it's shippable and there is a production component to it. And so I would say it kind of sits at that intersection, which I know a lot of things today, um, sit at. So, so yeah, I think that's, that's one of my teams works on that. And the other team works on, um, I would say like research and, uh, education. Um, so we're thinking about how to, um, stay, um, at the forefront of research and engage with researchers everywhere, um, but also educate the world and evangelize, um, uh, you know, educate the world about AI and evangelize AMD GPUs that are able to support, um, all of those new AI workloads.

• 21:17Speaker 00

  And so, um, so yeah, that's, that's the, a couple of teams that, um, I've been, um, uh, very grateful to, to be leading.

• 21:26Craig S. Smith?

  Yeah. Uh, and I, we tried to connect at NeurIPS and I'm sorry that we weren't able to. And I missed the, the, the talk. Can you go through what the presentation that you gave at NeurIPS?

• 21:39Speaker 00

  Yeah, so it was a tutorial, um, two and a half hours. Uh, and, uh, my team did a huge part of the presentation. So definitely not just me here. Um, and it was a collaboration across a lot of different institutions. So like, you know, Stanford, uh, Google DeepMind, Meta, Nvidia, um, ARM, et cetera. Um, and so, uh, it was, yeah, a really, a really great group of people. Um, and what we presented in our tutorial, um, was a few different things. First, we, because it's a NeurIPS audience, AR researchers, um, they might not be actually as familiar, um, on, uh, GPUs and hardware and how the GPU actually works.

• 22:19Speaker 00

  And so we kind of stepped through in, in hopefully very easy to understand ways of the actual GPU architecture and what pieces that an AI researcher might find interesting, like the memory and like how things are, you know, scheduled in. Um, like something I find interesting is, you know, when you change your batch size, why suddenly like your, your job takes forever, right? Your training job takes forever. Oh, it's because of, you know, X, Y, Z happening in, in the actual GPU. Um, and so, so that's what we go through first and then we go through, um, uh, kind of different kernels. So like showing you what a kernel actually is in code, um, and a very simple kernel. Um, and then showing you what it looks like when it's optimized and unoptimized.

• 23:04Speaker 00

  Um, so you can understand like, oh, when a kernel runs fast or slow, what it looks like. Um, so if something runs, uh, really slowly, that means like chat GPT could answer you in like minutes as opposed to seconds, right? So that, that's ultimately what it feels like. But then when you go down into like just a single matrix multiplication, um, what does that look like? Or a single addition, you know, like X plus Y, um, what does that look like, um, in terms of time? And how do you profile it? And then we go into, okay, what are AI agents and methods that we're using for the self-improving AI essentially, but essentially what are some of these methods? Um, that we're using, um, and sharing, sharing a lot of that. So like, what kind of, what kind of things

• 23:46Speaker 00

  can you do with agents? Um, you can have them like, you can write, uh, write, write me a more optimized kernel. Okay. It, it does it. Um, it maybe is or isn't optimized. You have to check the correctness of it. Maybe it's cheating. Um, and then when you profile it, you can actually get the answer back. And you're like, actually it was only 1.2X faster. I want it to be even faster. And then you can continually evolve it. That was based on, uh, Google's Alpha Evolve paper, but essentially like continually improve it after multiple calls. Um, the profiling information also provides a really interesting. Way of collecting or creating an RL environment so that these models can learn

• 24:26Speaker 00

  from essentially a verifiable reward from the profiler itself. So it's able to give this number back as, okay, this is how fast your kernel was that you generated and tells the model in a very verifiable way. Kind of like when ChatGPT, they were training on math tasks, it was very verifiable, whether the math was correct or not. And this helps the model improve in post-training. And so, I'm sorry, I'm trying to make this accessible, but like, so basically, we go through all of those different techniques as well, and we share that with the community.

• 25:03Craig S. Smith?

  Yeah. And this is done at what point in the process? Someone, a customer is using AMD hardware and they have a model that they want to run on it. And so they're optimizing the kernel before deploying the model? Or is it after deployment and they want to improve it? Yeah.

• 25:27Speaker 00

  Yeah. So I think it's a combination of those things. So I think, you know, you could use it definitely before you deploy something. So before you deploy something, you're like, okay, this isn't fast enough, given my budget with this many GPUs, and we need it to hit these markers, right? And so, because that basically translates to TCO, translates to the total cost of ownership. And so, let me reduce the speed of all, sorry, yeah, let me make these, all of these faster inside of the model so that the full model runs really quickly. So that could be before deploying something. It'd also happen afterwards, too. A lot of different kernels are written afterwards as well, to just speed things

• 26:09Speaker 00

  up even further. So it could be something you've already deployed, and now you want to speed things up to make use of the hardware you have even more. So I think it could occur in both of those settings.

• 26:21Craig S. Smith?

  Yeah. And this is, in my mind, this is more automation than self-improvement, right? You're automating the kernel writing process, but, and that improves the model. But where does self-improvement come in?

• 26:42Speaker 00

  Yeah, so the self-improvement is, so I view it as self-improvement, not automation, but I view it as self-improvement because the kernel itself is what is, what the model is running on. So it is the model's own code and it enables the model to run even faster on the GPU. But I guess when broken up, it could be seen as just like, oh, automation. And I do think, I honestly think everything is technically automation, but if we want to view it through a lens that's like more exciting, where it feels like, you know, autonomous, that is just a different perspective on it.

• 27:20Craig S. Smith?

  Yeah, but can the day come where a model will autonomously go in and rewrite its kernel

• 27:29Speaker 00

  Yeah.

• 27:30Craig S. Smith?

  to speed things up?

• 27:31Speaker 00

  Yeah, I definitely think so. I think more likely it will be a different model that does it. Yeah.

• 27:40Craig S. Smith?

  An external model other than the inference model, so what you mean?

• 27:44Speaker 00

  Yeah, or, I mean, it still could be that model, but I guess it depends on how you view what that model is, whether it's the same instance of that model or not, or a different prompt. But yeah, it will, it will be optimized whether by that own model or by another model.

• 28:03Craig S. Smith?

  Yeah. Are there things other than kernel optimization that you're working on? I mean, you mentioned a few things.

• 28:11Speaker 00

  Yeah, so another area has been RL research, so reinforcement learning inside of post-training. And so that's been really exciting to track and to work on. It is related to kernel generation since a lot of those techniques can be ported over, but more generally, we've been exploring those internally for different use cases, as well as for ways to actually make sure that all of this does run on AMD's hardware. The type of RL research that we're looking at includes

• 28:43Speaker 00

  Uh, kind of the techniques that started with ChatGPT. So um, the techniques that got us uh ChatGPT included something called RLHF. RL with human feed um reinforcement learning from human feedback. And that used an algorithm called PPO. And so it used a different, a certain type of algorithm that now has um, I think, found successors um as well as, you know, complements to um like GRPO, um which came from DeepSeek earlier last year. Um but basically, there's a hu- huge set of research um in this area to more effectively um use reinforcement learning in the post-training of these language models. Um and what reinforcement learning really does um is a couple of things.

• 29:28Speaker 00

  Uh, one area that's really been exciting is that human feedback piece, right? So that you could actually take human preferences, um whether it be like rankings of things or you know, pairwise comparisons of things, of your own preferences, um and you just give a lot of examples of those. You can actually teach the model to mimic those preferences. Um and those preferences could turn into um a more helpful model. Um it could turn into a safer model so that it doesn't like say harmful things. Um and uh, so yeah, and and and I think another, you know, application of reinforcement learning has been uh around reasoning. So if you're familiar with thinking in all these models that you're using, um that's what reasoning is um effectively doing using far more tokens.

• 30:12Speaker 00

  The model's basically writing down its thoughts um before it responds to you. And um that process is done um uh using, using, well, using post-training. Um definitely part of it being RL. Um and I would say like that extended um to, let's say, uh math or coding uh tasks has become really exciting because they're verifiable, is what we say um in the research world. And what that means is that these tasks um, you can verify like a math proof or you can verify the code. And that becomes really interesting because then you don't need to collect that human feedback, which might take time, um and it might be. It might be brittle because you can't cover every possible use case.

• 30:55Speaker 00

  But instead, you have something that checks it that can cover everything. And that provides feedback back to the model. And that's called, you know, RL with verifiable rewards instead of RLHF from human feedback. And so that's been a really exciting direction as well. And been a direction that we've been looking into a lot because we were getting these models to write code. And we have verifiable rewards from the profiler of of our GPUs that tell us how fast these kernels are actually running on the GPU. And that's a number and that's verifiable and it's true. It's not subjective. And so that can easily go back into the model as it's learning.

• 31:35Speaker 00

  So yeah, it's a lot of different algorithms going in, you know, in that direction to improve that process and make it more effective.

• 31:43Craig S. Smith?

  And how the kernel writing takes place at the customer on the customer side, I mean, when someone is deploying a model onto AMD hardware or any hardware and they want to optimize the kernel.

• 32:04Craig S. Smith?

  Or is it taking place? Do you do this work at AMD and then make those optimized kernels available to people depending on the model that they're running?

• 32:16Speaker 00

  Yeah.

• 32:18Speaker 00

  It's the latter right now. So we are basically doing this, a lot of this internally and then making it available. Open source, all our kernels are open. And we're, that's what we're doing. I think the the hope is to actually get it out there. Hopefully sometime this year we'll see. But the goal is to get it out there and make it possible for our customers or users or any user really to be able to leverage that very easily. Now, today, I think what's really effective is you can go into cursor or can you, you can go into any of these AI coding agent environments and you can do some of this on your own as well. And there are companies doing this as well. Yeah.

• 33:02Craig S. Smith?

  Yeah, and you're presumably also working on code assistance for use within AMD beyond writing kernels. Are there other parts of the stack that you're focused on?

• 33:20Speaker 00

  Oh, there's so many parts of the stack that need code assistance. So I mainly focus on kernels and the actual models. But I think there's so many layers of the stack, like we have to integrate with open source libraries so that things just work by default on AMD when a developer in one of those higher level stacks is using us. So there's a lot of code at so many different layers of the stack, which I think I honestly failed to appreciate before joining AMD, like that there are so many layers, even underneath the shiny model layer that everyone knows about.

• 34:03Craig S. Smith?

  Yeah, but you said you do work on models as well.

• 34:08Speaker 00

  Yes, it's for the kernel generation.

• 34:10Craig S. Smith?

  Right, right. Yeah. And can you give me a sense of how

• 34:18Craig S. Smith?

  how much time is saved or how many man hours are saved by having a kernel generation of software like this?

• 34:30Speaker 00

  I probably can't give you direct stats, but what I can share is that

• 34:38Speaker 00

  that these kernels could take a very long time to write. A very, very complex kernel can take at least, you know, let's say a non-expert, but someone who might still be tasked to do it, months to write. Like an expert would take a couple of weeks, but that's still a substantial amount of time. It's not like it's a, you know, a couple of minutes, a couple of hours, right, type of task. So it is a more complex task than throwing up a website, and certainly more niche and therefore not as represented in pre-training data so that the models don't Just as by default, know everything about it. Um, but it's a highly valuable task, um, and one that is both rare.

• 35:20Speaker 00

  Um, and there's like not enough expertise in any company, I think, um, to do it. Um, and, uh, and very just necessary and urgent today. Because if you can even shave off like, uh, if you shave off like a tiny amount of time, it takes for one matrix multiplication that occurs billions, trillions of times inside of a model. And that can incur billions, hundreds of billions of dollars to a company. And so that's, that's a lot of money. If you're doing a frontier model, of course, but like, even if you're not, it could, it could save, um, a lot of money. And I think, um, what people don't realize is behind a lot of these APIs, for example, like the Together AI or Fireworks AI or Base10, like they are writing kernels,

• 36:03Speaker 00

  actually, to make these models run even faster than if you were to try to do it on your own. And they're, they're writing a lot of those, right? Make them faster. Um, so we're doing that. OpenAI, Frontier Labs are doing that. A lot of people are doing that so that they could get mu- far more out of their compute. Um, because the equivalent is if you can 10x the speed of your kernel, um, that's the equivalent of buying 10x more compute. And as we've seen, like buying 10x more compute is interesting for the scaling laws for scaling these models, um, to get to the next level of intelligence. Um, so this is, I think, actually a very ripe layer to continually optimize, um, and actually be able to get scaling even from optimizing this existing layer.

• 36:45Craig S. Smith?

  That, that's, that's interesting. Uh, yeah, it's something that, that people don't think about.

• 36:51Speaker 00

  Yeah, I didn't think about before either. Yeah.

• 36:55Craig S. Smith?

  Yeah, and, and, um, uh, the, the kernels that you're writing, uh, are they, uh, optimized for AMD hardware? Or could you then take them for other to other hardware?

• 37:08Speaker 00

  Yeah, excellent question. So it is optimized for AMD hardware. And that's because our hardware has, you know, specific specific things in it so that we need to optimize the kernel to run, you know, specifically use all of our HBM, for example. We have higher HPMs, more memory than the other GPU. And, and as a result, how do we utilize that effectively, right, for certain tasks? And so I think that's wise. We need to be very specific to this GPU and what we're eking out. I think there are people working on kind of more like general kernels. For example, flash attention was a famous one.

• 37:54Speaker 00

  It basically takes the usual attention mechanism inside of all transformers. And they do some math to make it so that like scheduling it on the GPU doesn't require as as many trips to, you know, grab from, from, from memory from storage and as a result speeds things up significantly. And I think that is an algorithm. A kernel type algorithm that has now been used across across the different hardware providers. So that's like more of a general invention, which I think kernel generation could get to, which is very exciting. But yeah, but today we're definitely very focused on making sure that these kernels can run very effectively on AMD GPUs.

• 38:38Craig S. Smith?

  Yeah, you know, there's been a lot written recently about

• 38:45Craig S. Smith?

  overcapacity of, or looming overcapacity of hardware, both chips and data centers. And as you speed up the compute

• 39:01Craig S. Smith?

  using these strategies like kernel optimization, is it, is it going to relieve some of that pressure right now for chips?

• 39:16Speaker 00

  Good question, yeah.

• 39:17Craig S. Smith?

  Yeah, I mean, do you have any sense of that? Impact the macro

• 39:25Craig S. Smith?

  landscape?

• 39:26Speaker 00

  I think people want infinite chips, Craig, so no, it doesn't relieve the pressure.

• 39:33Speaker 00

  I don't think they found a plateau where they are like, we're, we're done. Like if you just hundred x this, we're done. I, I don't see that right now.

• 39:45Craig S. Smith?

  Yeah, that's right. And are, are AMD chips, are you focused on a, a particular use case?

• 39:54Speaker 00

  Uh, so I guess the use case is language models. Um, what's, uh, I guess crazy is that, um, that's a vertical. Um, cause people are like, what else could there be? So what else could there be? Uh, so there could be, you know, computer vision models. Um, there could be, uh, these GPUs. Um, now they're all optimized for AI, but before they, um, were originally developed for, um, high-performance computing, HBC, which includes like weather modeling and prediction, right? They're like a lot of different other tasks. Um, and even within, um, what, or like adjacent to language models. There's a bunch of other types of models as well. Um, so I'd say that we're very focused on, um, or I'm very focused on the AI side of things.

• 40:38Speaker 00

  Um, and the language model thrust, um, and kind of what people are developing there. Um, what kind of divert, so it includes for sure the, what the frontier labs are doing, but also what leading startups are doing. And some of those startups, like I just met with Yan Le Kuhn, um, regarding, uh, me and, um, they're developing Jepa. And that might look, you know, a little bit different in certain ways. Um, I might need, you know, it might not care about that, like low latency, um, token inference, uh, that, you know, autoregressive language models like ChatGPT care about, but something else, right? And so that might look different from a hardware perspective. And so we also care about leading AI startups and what they're doing.

• 41:19Speaker 00

  Um, and making sure that all of those workloads do work very effectively on our chips and that we're very knowledgeable about them. So like the, you know, latest type of diffusion models, um, for example, like those run smoothly on, on AMD. And so I would say that like, that's, that's something that's really important. So that's kind of where the focus is. And I know it doesn't feel like a focus because everyone's focused on it, but, um, it, it, it actually is a focus because GPU is fairly general type of compute.

• 41:45Craig S. Smith?

  Yeah. Uh, yeah, that's interesting. Uh, Yan Le Kuhn's Jepa. So you, you're, you were talking with him about optimizing.

• 41:56Craig S. Smith?

  Colonels to run Jepa on AMD hardware?

• 42:01Speaker 00

  Uh, we were talking more generally than that, probably. But, um, but understanding what Jepa does is really important for us to optimize.

• 42:09Craig S. Smith?

  Yeah.

• 42:09Speaker 00

  We can, yeah, we can take some of their, um, uh, models today and be able to actually run it on, on AMD hardware. Uh, a lot of it's open source. Yeah.

• 42:19Craig S. Smith?

  Yeah. I just, um, had Sergey Levine or Levin, he pronounces it. On, uh, just before that was.

• 42:27Speaker 00

  Yeah, hi, right?

• 42:29Craig S. Smith?

  And, uh, are you working with robotic, uh, foundation models at all?

• 42:36Speaker 00

  AMD definitely is. AMD definitely is. I, I'm personally, I'm not working directly with some of those startups. Um, though I've chatted with some of them, um, but we, we have, uh, folks very focused on that. Uh, so I, I teach online. I teach about a million people online, including developers, but also executives, um, and different professionals. And, uh, lately, um, we have a part, we, you know, built up a partnership with Deep Learning AI and Ring's, um, platform to teach, uh, different courses. And we launched a post-training class on, you know, reinforcement learning and fine tuning of these models. Um, but actually the first module I think is this is accessible to everyone. Um, and enables everyone to just like double click and get one level deeper on how these models actually learn how to behave,

• 43:20Speaker 00

  how to chat with us, how to, um, act safely and add guardrails, how to, you know, hallucinate a little bit less. Uh, and how to stay more focused in a long conversation. Um, and so I think, uh, that's, you know, that's something that's important and I'm, um, actually working on a book, um, uh, that matches that class too. And then, um, the other courses that, um, I've been preparing, uh, kind of jointly between AMD and, um, And deep learning AI also are around like why compute matters and a more general course on transformers and understanding that. And before a more general audience, I've actually been working and spending time at Harvard University to produce courses as well.

• 44:03Speaker 00

  So this is a collaboration where we are, where I'm, I'm teaching a few fundamentals around AI for a professional audience. So that's anything ranging from vibe coding, so basically enabling more people to write code. Whether they, especially people who can't write code today and be able to build things that might help them in their professional careers. And whether that be building a board deck or like analyzing, you know, something for M and A. And so these are things that are not like, you're not, you, these people don't, if you don't view yourselves as a software engineer, this is like the right class for you.

• 44:41Speaker 00

  And then, of course, just generally like AI for leaders and leadership and any kind of like manager who's thinking about how to change behavior within their organization with AI and understand things a little bit more fundamentally. And, and how to use AI that's more practical, for example, if, you know, the models are producing different outputs and you find that frustrating and you find it not trustworthy, maybe we can, you know, flip the script a little bit and think about it from the other side of, okay, actually, this is why they act this way. This is why they produce variation. And now that you understand why, you can actually take advantage of this and you can, you know, run the model multiple times or run multiple models

• 45:24Speaker 00

  and be able to then take that analysis with you and, and use that in your day to day more as a tool. And so it's doing that, but like 10, 20 times and examples. And, and be able to just disseminate that knowledge that I know I have as an AI researcher, but want more people to have when they're using this technology so that they can view it more as a tool that's useful to them. Other than like something that they're a little bit worried about and don't trust. And so. So yeah, that's that's where um the other kind of stuff I'm working on as well. That might be helpful to your audience.

• 46:00Craig S. Smith?

  Yeah, where where would people access those those courses?

• 46:04Speaker 00

  Yeah, so the deep learning AI courses are available for free on deep learning AI. Um, if uh if you want a certificate, I think you do have to pay, but without that, they're free. Um, and then uh the Harvard ones have not been released yet, but um uh we we've been working on them, so.

• 46:22Craig S. Smith?

  Wow, that's exciting. Yeah, yeah.