Generative AI 101: Tokens, Pre-training, Fine-tuning, Reasoning — With Dylan Patel

Dylan Patel is the founder and CEO of SemiAnalysis. He joins Big Technology Podcast to explain how generative AI work, explaining the inner workings of tokens, pre-training, fine-tuning, open source, and reasoning. We also cover DeepSeek’s efficiency breakthrough, the race to build colossal AI data centers, and what GPT-5’s hybrid training approach could unlock. Hit play for a masterclass you’ll want to send to every friend puzzled (or excited) about the future of AI. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Want a discount for Big Technology on Substack? Here’s 40% off for the first year: https://tinyurl.com/bigtechnology Questions? Feedback? Write to: [email protected] Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  If you've ever wondered how generative AI works and where the technology is heading, this episode is for you. We're going to explain the basics of the technology and then catch up with modern day advances like reasoning to help you understand exactly how it does, what it does, and where it might advance in the future. That's coming up with Semi Analysis founder and chief analyst Dylan Patel, right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience, and what companies need to get right in the AI era. Then I sat down with Adam Mitchell,

• 0:43Alex Kantrowitz?

  head of enterprise business solutions at Voya Financial, to go inside a large scale agentic CX transformation, including the business decisions, trade offs, and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 1:26Alex Kantrowitz?

  Welcome to Big Technology Podcast, a show for cool headed and nuanced conversation of the tech world and beyond. We're joined today by Semi Analysis founder and chief analyst Dylan Patel, a leading expert in semiconductor and generative AI research, and someone I've been looking forward to speaking with for a long time. Now, I want this to be an episode that A helps people learn how generative AI works, and B is an episode that people will send to their friends to explain to them how generative AI works. I've had a couple of those that I've been sending to my friends and colleagues and counterparts about what is going on with in generative AI. That includes one, this three and a half hour long video from Andre Karpathy, explaining everything about training large language models. And the second one is

• 2:11Alex Kantrowitz?

  a great episode that Dylan and Nathan Lambert from the Allen Institute of AI did with Lex Friedman. Both of those, three hours plus, so I want to do ours in an hour. And I'm very excited to begin. So Dylan, it's great to see you and welcome to the show. Thank you for having me. Great to have you here. Let's just start with tokens. Can you explain how AI researchers basically take words and then give them numerical representations? And parts of words and give them numerical representations? So what are tokens?

• 2:42Speaker 01

  Tokens are, in fact, like chunks of words, right? In the human way, you can think of like syllables, right? Syllables are often viewed as like chunks of word. They have some meaning. It's the base level of speaking, right? Is syllables, right? Now, for models, tokens are the base level of output. They're all about like, sort of compressing, you know, sort of this is the most efficient representation of language.

• 3:09Alex Kantrowitz?

  From my understanding, AI models are very good at predicting patterns. So if you give it 1, 3, 7, 9, it might know the next number is going to be 11. And so what it's doing in, with tokens, is taking words, breaking them down to their component parts, assigning them a numerical value, and then basically in its own word, in its own language, learning to predict what number comes next because computers are better at numbers, and then converting that number back to text. And that's what we see come out. Is that accurate?

• 3:39Speaker 01

  Yeah, and each individual token is actually, it's not just like one number, right? It's multiple vectors. You could think of like, well, the tokenizer needs to learn, king and queen are actually extremely similar on on most, in terms of like the English language, extremely similar, right? Except there is like one vector in which they're super different, right? Because a king is a male and a queen is a female, right? And then from there, like, you know, in language, oftentimes kings are considered conquerors and, you know, all these are the things, and like, these are just like historical things, right? So a lot of the text around them, while they're both like royal, regal, right? Like, you know, monarchy, etc. There are many vectors in which they differ. So like, it's not just like converting a word into one number, right?

• 4:22Speaker 01

  It's like converting it into multiple vectors. And each of these vectors, the model learns what it means, right? You don't initialize the model with like, hey, you know, king means male, monarch, and it's associated with like war and conquering, because that's all the writing about kings is on, you know, in history and all that, right? Like, people don't talk about the daily lives of kings that much. Or they mostly talk about like their wars and conquests and stuff. And so like, There will be, each of these numbers in this embedding space, right, will be assigned over time as the model reads the internet's text and trains on it, it'll start to realize, oh, king and queen are exactly similar on these vectors, but very different on these vectors. And these vectors aren't, you don't explicitly tell the model, hey, this is what

• 5:06Speaker 01

  this vector is for, but it could be like, you know, it could be as much as like, one vector could be like, is it a building or not, right? And it doesn't actually know, you don't know that ahead of time, it just happens to in the latent space. And then all these vectors sort of relate to each other. But yeah, these numbers are an efficient representation of words, because you can do math on them, right? You can multiply them, you can divide them, you can run them through an entire model, whereas, and your brain does something similar, right? When it hears something, it converts that into a frequency in your ears, and then that gets converted to frequencies that should go through your brain, right? This is the same thing as a tokenizer, right? Although it's like, obviously a very different medium of compute, right? Ones and zeros for computers versus, you know, binary and

• 5:49Speaker 01

  multiplication, etc. being more efficient. Whereas humans' brains are more like analog in nature and, you know, think more in waves and patterns in different ways. Well, they are very different. It is a tokenizer, right? Like, language is not actually how our brain thinks, it's just a representation for which it to, you know, reason over.

• 6:10Alex Kantrowitz?

  Yeah, so that's crazy. So the tokens are the sufficient representation of words, but more than that, the models are also learning the way that they are, all these words are connected. And that brings us to pre-training. From my understanding, pre-training is when you take the entire, basically the entire internet worth of text, and you use that to teach the model these representations between each token. So therefore, like we talked about, if you gave a model, the sky is, and the next word is typically blue, in the pre-training, which is basically all of the English language, all of language on the internet, it should know that the next token is blue. So what you do is, you

• 6:54Alex Kantrowitz?

  want to make sure that when the model is outputting information, it's closely tied to what that next value should be. Is that a proper description of what happens in pre-training?

• 7:06Speaker 01

  Yeah, I think that's the objective function, which is just to reduce loss, i.e. how often is the token predicted incorrectly versus correctly?

• 7:15Alex Kantrowitz?

  Right, so if you said the sky is red,

• 7:20Alex Kantrowitz?

  that's not the most probable outcome, so that would be wrong.

• 7:22Speaker 01

  But that text is on the internet, right? Like, because the Martian sky is red, and there's all these books about Mars and sci-fi.

• 7:27Alex Kantrowitz?

  Right, so how does the model then learn how to figure this out? And in what context is it accurate to say "blue" and "red"?

• 7:33Speaker 01

  Right, so, I mean, first of all, the model doesn't just output one token, right? It outputs a distribution. It turns out, the way most people take it is they take the top K, i.e. the most high probability. So yes, "blue" is obviously the right answer if you give it to anyone on this planet. But there are situations and contexts where "the sky is red" is the appropriate sentence, but that's not just in isolation, right? It's like, if the prior passage is all about Mars and all this, and then all of a sudden, it's like, and that's like a quote from a Martian settler, and it's like, "the sky is..." And then the correct token is actually "red", right? The correct word. And so it has to know this through the attention mechanism, right? If it was just "the sky is blue", always you're going to output "blue",

• 8:15Speaker 01

  because "blue" is, let's say, 80%, 90%, 99% likely to be the right option. But as you start to add context about Mars, or any other planet, right? Other planets have different colored atmospheres, I presume. You end up with this, um, distribution starts to shift, right? If I add, "we're on Mars, the sky is..." You know, then, then all of a sudden, "blue" goes from 99%, you know, in the prior context window, right? The text that you sent to the model, the attention of it. All of a sudden, it realizes "the sky is blue", preceded by the stuff about Mars. Now, "blue" rockets down to like, you know, let's call it 20% probability, and "red" rockets up to 80% probability, right? Now, the model outputs that,

• 8:58Speaker 01

  and then, most people just end up taking the top probability and outputting it to the user. And that's sort of like, how does the model learn that? Is the attention mechanism, right? And this is sort of the beauty... Yeah, the attention mechanism is the beauty of modern, sort of large language models. It takes the relational value, you know, in this vector space, between every single token, right? So, you know, "the sky is blue", right, like, when I think about it, yes, "blue" is the next token after "the sky is", but... In a lot of older style models, you would just predict the exact next word. So after "sky", obviously, it could be many things. It could be "blue", but it could also be like "scraper",

• 9:42Speaker 01

  skyscrapers, that makes sense. But what attention does is it is taking all of these various values, the query, the key, the value, which represents what you're looking for, where you're looking, and what that value is across the attention. And you're calculating mathematically what the relationship is between all of these tokens. And so going back to the king-queen representation, the way these two words interact is now calculated. And the way that every word in the entire passage you sent is calculated is tied together, which is why models have challenges with how many documents can you send them? Because if you're sending them

• 10:26Speaker 01

  just to question like "what color is the sky", OK, it only has to calculate the attention between those words. But if you're sending it like 30 books with insurance claims and all these other things, and you're like, OK, figure out what's going on here. Is this a claim or not? And in the insurance context, all of a sudden, it's like, OK, I've got to calculate the attention of not just the last five words to each other, but I have to calculate every 50,000 words to each other, which then ends up being a ton of math. Back in the day, actually, the best language models were a different architecture entirely. But then at some point, transformers, large language models, large language models which are based on transformers primarily, rocketed past in capabilities because they were over the scale and because the hardware got there. And then we were able to scale them so much

• 11:11Speaker 01

  that we were not to just put some text in them and not just a lot of text or a lot of books, but the entire internet, which one could view the internet oftentimes as a microcosm of all human culture and learnings and knowledge to many extents because most books are on the internet, most papers are on the internet. Obviously, there's a lot of things missing on the internet. Um, but this is sort of, this is the sort of modern, you know, magic of like what, what it was sort of like three different things like coming all together at once, right? An efficient way for models to relate every word to each other, the compute necessary uh to scale the data large enough. And then someone actually like pulling the trigger to do that, right? At the scale that was, you know, got to the point where it was useful, right? Which was sort of like GPT three point five uh level or four level, right?

• 11:56Speaker 01

  Where it became extremely useful for normal humans to use uh, you know, chat, chat models.

• 12:02Alex Kantrowitz?

  Okay, and so why is it called pre-training?

• 12:04Speaker 01

  So, so pre-training is, is, is sort of called that because it is what happens, you know, before the actual uh training of the model, right? Uh, the objective function in pre-training is to just predict the next token. Uh, but predicting the next token is not what humans want to use AIs for, right? I want it to ask a question and answer it. Uh, but in, in most cases, asking a question does not necessarily mean that the next most likely token is, is the answer, right? Oftentimes it is another question, right? Uh, for example, if I ingested the entire SAT, um, you know, and I asked a question, the next like five answer, and then all the next tokens would be like, A is this, B is this, C is this, D is this. Like, no, I just want the answer, right?

• 12:47Speaker 01

  Um, and so pre-training is, the reason it's called pre-training is because you're ingesting humongous volumes of text, no matter the use case, right? Um, and you're learning the general patterns across all of language, right? I don't actually know that king and queen relate to each other in this way. And I don't know that king and queen are opposites in these ways, right? Um, and so this is why it's called pre-training is because you must get a broad, general understanding of the entire sort of world of text. Before you're able to then do post-training or fine-tuning, which is let me train it on more specific data that is specifically useful for what I want it to do.

• 13:25Speaker 01

  Whether it's, hey, um, in chat style applications, um, you know, go and, go and, you know, when I ask a question, give me the answer. Or um, in, in other applications like teach me how to build a bomb, well, obviously, no, I'm not going to help you teach, build a bomb because that's what, I don't want the model to teach me how to build a bomb. So, you know, it's sort of got a Do this, and it's not like you're teaching it, you know, when you're doing this pre-training, you're filtering out all this data. Because in fact, there's a lot of good useful data on how to build bombs. Because there's a lot of useful information on like, hey, like C4 chemistry and like, you know, people want to use it for chemistry, right? So you don't want to just filter out everything, so that the model doesn't know anything about it. But at the same time, you don't want it to output, you know, how to build a bomb.

• 14:07Speaker 01

  So there's like a fine balance here, and that's why pre-training is defined as pre, because you're still letting it do things and teaching it things and inputting things into the model that are theoretically like quite bad, right? For example, books about like killing, or war tactics, or what have you, right? Like things that like plausibly you could see like, oh, well, maybe that's not okay. Or wild descriptions of like really grotesque things all over the internet, but you want the model to learn these things, right? Because first you build the general understanding before you say, okay, now that you've got a general framework of the world, let's align you so that you, with this general understanding of the world, can figure out what is useful for people, what is not useful for people, what should I respond on, what should I not respond on?

• 14:51Alex Kantrowitz?

  So what happens then in the training process? So the, is the training process that the model is then attempting to make the next prediction? And then just trying to minimize loss as it goes?

• 15:02Speaker 01

  Right, right, I mean, like basically, you have, you have, loss is, is how often you're wrong versus right, in the most simple terms, right? You will, you will run through passages, right, through the model, and you'll see, how often did the model get it right? When it got it right, great, reinforce that. When it got it wrong, let's figure out which, which neurons, neurons in the model, you know, quote unquote, neurons in the model, you can tweak to then fix the answer, so that when you go through it again, it actually outputs the correct answer. And then you move the, move the model slightly in that direction. Now, obviously, the challenge with this is, if I, if I first, you know, I can, I can come up with a simplistic way where all the neurons will just output the skies blue. Every single time says the skies.

• 15:47Speaker 01

  But then, when it goes to, you know, Hey, the color blue is commonly used on walls because it's soothing, right? And it's like, oh, what's the next word? Is soothing, right? Soothing, you know? And so like that, that is a completely different representation. And to understand that blue is soothing and that the sky is blue and those things aren't actually related, but they are related to blue is like very important. And so, you know, over oftentimes you'll run through the training data set multiple times, right? Uh, because the first time you see it, oh, great, maybe you memorized that the sky is blue. Um, and you memorize the wall is blue and that, uh, and when people describe art and oftentimes use the color blue, it can, it can be representations of art or the wall, right?

• 16:31Speaker 01

  And so over time, as you go through all this text in pre-training, yes, you're minimizing loss initially by just memorizing, but over time, because you're constantly overwriting the model, it starts to learn the generalization, right? I.e. blue is a soothing color, also represents the sky, also used in art for either of those two motifs, right? And so that's sort of the, the goal of pre-training is, is you don't want to memorize, right? Because that's, you know, in school, you memorize all the time, um, and that's not useful because you forget everything you memorize. Uh, but if you get tested on it then, and then you get tested on it six months later, um, and then again, six months later after that, or however you do it, ends up being, oh, you don't, you don't actually like memorize that anymore. You just know it innately, um, and you've generalized on it.

• 17:13Speaker 01

  And that's the real goal, uh, that you want out of the model, uh, but that's not necessarily something you can just measure, right? And therefore loss is something you can measure, uh, i.e. for this group of, this group of text, right? Cause you train the model in, in steps. Every step you're inputting a bunch of text, you're trying to, uh, you're trying to see what's predict the right token, where you didn't predict the right token, let's adjust the neurons. Okay, on to the next batch of text. And you'll do this, these batches over and over and over again, um, across, you know, trillions of words of text. Right? And as you step through and then you're like, oh, well, I'm done. But I bet if I go back to the first first group of texts, which is all about the sky being blue, it's going to get the answer wrong because maybe later on in the training, it discovered it saw some passages about sci fi

• 17:58Speaker 01

  and how Martian skies red. So like it'll it'll overwrite, but then over time, as you go through the data multiple times, as you see it on the internet multiple times, you see it in different books multiple times, whether it be scientific, sci fi, whatever it is, you start to realize and it starts to learn that that representation of like, oh, when it's on Mars, it's red because the sky Mars is red because the atmospheric makeup is this way, whereas the atmospheric makeup in on Earth is a different way. And so that's sort of like, the whole point of pre-training is to minimize loss. But the nice side effect is that the model initially memorizes, but then it stops memorizing and it generalizes. And that's the useful pattern that we want.

• 18:33Alex Kantrowitz?

  Okay, that's fascinating. We've touched on post-training for a bit, but just to recap, post-training is so you have a model that's good at predicting the next word and in post-training, you sort of give it a personality by inputting sample conversations to make the model want to emulate the certain values that you want it to take on?

• 18:55Speaker 01

  Yeah, so post-training can be a number of different things. The most simple way of doing it is, is yeah, pay for humans to label a bunch of data, take a bunch of example conversations, et cetera, and input that data and train on that at the end, right? And so that that example data is is useful, but this is not scalable, right? Like using humans to train models is just so expensive, right? So then there's the magic of sort of reinforcement learning and and other synthetic data technologies, right? Where the model is helping teach the model, right? So you have many models in a sort of in a post-training where, yes, you have some example human data, but human data does not scale that fast, right?

• 19:37Speaker 01

  Because because the internet is trillions and trillions of words out there. Whereas, you know, even if you had, you know, Alex and I write words all day long for our whole lives, we would have millions or, you know, hundreds of millions of words written, right? It's nothing. It's like orders of magnitude off in terms of the number of words required. So then you have The model, you know, take some of this example data. Um, and you have various models that are surrounding the main model that you're training. Right. And these can be policy models, right? Teaching it, hey, is this, is this what you want or that what you want? Uh, reward models, right? Like, is that good? Is that a good response or is that a bad response? You have value models like, hey, grade this output. Right. And you have all these different models working in conjunction to say, um,

• 20:22Speaker 01

  uh, you know, different companies have different objective functions, right? In the case of Anthropic, they're, they want their model to be helpful, harmful, harmless, and safe, right? So be helpful, but also don't harm people or anyone or anything. And then, you know, uh, you know, safe, right? In other cases, like Grok, right? Uh, Elon's model from XAI, um, it actually just wants to be helpful. Um, and maybe it has like a little bit of a right leaning to it, right? And, and for other folks, right? Like, you know, you, you mean, most AI models are made in the Bay Area. So they tend to just be left leaning, right? Um, but also the internet in general is a little bit left leaning. Um, because it skews younger than older. Um, and so like all these things like sort of affect models. Um, but like, it's not just around politics, right? Post-training is also just about teaching the model.

• 21:05Speaker 01

  If I, if I say like, uh, the movie where, uh, the princess has a slipper and it doesn't fit. It's like, well, on, if I said that into a base model that was just pre-training, like, the answer wouldn't be, oh, the movie you're looking for Cinderella. You know, it, it would only realize that once it goes, you know, once it goes through post-training. Right. Because a lot of times people just throw garbage into the model. And then the model still figures out what you want. Right. And this is part of what post-training is. Like, you can just do stream of consciousness into models. And oftentimes it'll figure out what you want. Like, you know, if it's a movie that you're looking for, or if it's help answering a question, or if you throw a bunch of like unstructured data into it and then ask it to make it into a table, it does this, right? And that's because of all these different aspects of post-training. Right. Example data, but also.

• 21:49Speaker 01

  Uh, you know, generating a bunch of data and grading it and seeing if it's good or not. Um, and whether it matches the various policies you want, is it help, you know? A lot of times grading can be based on multiple factors, right? There could be a model that says, hey, is this helpful? Hey, is this, is this safe and what is safe, right? So then that model for safety needs to be tuned on human data, right? So there's, it is a quite complex thing, but the end goal is to be able to get the model to output in a certain way. Models aren't always about just humans using them either, right? There can be models that are just focused on like, hey, like, you know, if it doesn't output code, you know, yes, it was trained on the whole internet because the person's going to talk to the model using text, but if it doesn't output code, you know, penalize it, right? Now, all of a sudden, the model will never output like text ever again. It'll only output code.

• 22:32Speaker 01

  Um, and, and so like these sorts of like models exist too. So post-training is not just a uni-variable thing, right? It's what variables do you want to target? Um, and so that's why models have different personalities from different companies. It's why they target different use cases and why, you know, it's not just like one model that rules them all, but actually many.

• 22:50Alex Kantrowitz?

  That's fascinating. So that's why we've seen so many different models with different personalities is because it all happens in the post-training moment. And this is when, when you talk about giving the models examples to follow this, that's what reinforcement learning with human feedback is, is the humans give some examples and then the model learns to emulate what the human is interested in, what the human trainer is interested in having them embody. Is that right?

• 23:18Speaker 01

  Yeah, exactly.

• 23:19Alex Kantrowitz?

  Okay, great. All right. So first half, we've covered what training is, what tokens are, what loss is, what post-training is, post-training is, post-training, by the way, also called fine-tuning. We've also covered reinforcement learning with human feedback. We're going to take a quick break and then we're going to talk about reasoning. We'll be back right after this. This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data and most teams have no single way to see it, secure it and prove it's under control. That's exactly what App AVPT does. For 25 years, they've been the trusted layer beneath the world's most demanding

• 24:04Alex Kantrowitz?

  data, now extended across your entire AI estate, your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AVPT, the unifying trust layer for AI. Head to avpt.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal

• 24:49Alex Kantrowitz?

  details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwell.com/bigtechnology. Learn how to stop online threats before they

• 25:32Alex Kantrowitz?

  become real-world attacks. The link's in the show notes.

• 25:40Alex Kantrowitz?

  And we're back here on Big Technology Podcast with Dylan Patel. He's the founder and chief analyst at Semi Analysis. He actually has great analysis on Nvidia's recent GTC conference, which we just covered recently on a recent episode. You can find Semi Analysis at semianalyses.com. It is both content and consulting. So definitely check in with Dylan for all of those needs. And now we're going to talk a little bit about reasoning. Because a couple of months ago, and Dylan, this is really where I entered the picture of watching your conversation with Flex, with Nathan Lambert, about what the difference is between reasoning.

• 26:22Alex Kantrowitz?

  And your traditional LLMs, large language models. If I gathered it right from your conversation, what reasoning is, is basically instead of the model going, basically predicting the next word based off of its training, it uses the tokens to spend more time basically figuring out what the right answer is, and then coming out with its, with a new prediction. I think Karpathy does a very interesting job in the YouTube video talking about how models think with tokens. The more tokens there are, the more computes they, compute they use because they're running these predictions through the transformer model, which we discussed, and therefore they can come to better answers. Is that the right way to think about reasoning?

• 27:05Speaker 01

  So, so I think that humans are also fantastic at fat, pattern matching, right? We're really good at like recognizing things. But a lot of tasks, it's not like an immediate response, right? We are thinking. Whether that's thinking through words out loud, thinking through words in an inner monologue in our head, or it's just like processing somehow and then we know the answer, right? And this is the same for models, right? Models are horrendous at math, right? Historically have been, right? You could ask it, you know, what is 9.11 bigger than 9.9? And it would say, yes, it's bigger, even though like everyone knows that 9.11 is, is way smaller than 9.9, right? And that's just like a thing that happened in models because they didn't think or reason, right?

• 27:50Speaker 01

  And, and it's the same for you, Alex, right? Like, you know, or myself, right? Like, if someone asked me, you know, 17 times 34, I'd be like, I don't know, like right off the top of my head, but you know, give me, give me a little bit of time, I can do some long form multiplication and I can get the answer, right? And that's because I'm thinking about it. And this is the same thing with reasoning for models, is, you know, when, when you look at a transformer, every word is this, every token output, it has the same amount of compute behind it, right? I.e., you know, when I'm saying the versus sky is blue, the blue and the the have this, or the, the is and the blue have the same amount of compute to generate. Right, and this is not exactly what you want to do, right? You want to actually spend more time on the hard things and not on the easy things.

• 28:35Speaker 01

  And so reasoning models are effectively teaching, you know, large pre-trained models to do this, right? Hey, think through the problem. Hey, output a lot of tokens. Think about it, generate all this text. And then when you're done, you know, start answering the question. But now you have all this stuff you generated in your context, right? And that stuff you generated is helpful, right? It could be like, you know, all sorts of, you know, just like any human's thought patterns are, right? And so this, this is the sort of like new paradigm that we've entered maybe six months ago, where models now will think for some time before they answer. And this enables much better performance on all sorts of tasks,

• 29:19Speaker 01

  whether it be coding, or math, or understanding science, or understanding complex social dilemmas, right? All sorts of different topics they're much, much better at. And this is done through post-training, similar to the reinforcement learning by human feedback that we mentioned earlier. But also, there's other forms of post-training. And that's, that's what makes these reasoning models.

• 29:41Alex Kantrowitz?

  Before we head out, I want to hit on a couple of things. First of all, the growing efficiency of these models. So I think one of the things that people focused on with DeepSeek was that it was just able to be much more efficient in the way that it generates answers. And there was this, obviously, this big reaction to Nvidia's stock, where it fell 18% the day, or at the Monday after DeepSeek weekend, because people thought we wouldn't need as much compute. So can you talk a little bit about how models are becoming more efficient, how they're doing it?

• 30:13Speaker 01

  Yeah, so there's a variety of, the beauty of these, of AI is not just that we continue to build new capabilities. Right, because those new capabilities are going to be able to benefit the world in many ways. And there's a lot of focus on those. But there's also a lot of, there's a lot of focus on, well, to get to that next level of capabilities is, is the scaling laws. I E. The more compute and data I spend, the better the model gets. But then the other vector is, well, can I get to the same level with less compute and data? Right? And, and those two things are hand in hand, because if I can get to the same level with the less compute and data, then I can spend that more compute and data and get to a new level. Right? And so A I researchers are constantly looking for ways to make models more efficient,

• 30:55Speaker 01

  whether it be through algorithmic tweaks, data tweaks, tweaks in, you know, how you do reinforcement learning, so on and so forth. Right? And so when we look at models across history, they've constantly gotten cheaper and cheaper and cheaper. Right? At a stupendous rate. Right? And so one easy example is G P T three. Right? Because there's G P T three, three point five turbo, Llama two, seven B, Llama three, Llama three point one, Llama three point two, right? As these models have gotten bigger, we've gone from, hey, it costs sixty dollars for a million tokens to it costs less than, it's cost like five cents now for the same quality of model. Now, and the model has shrank dramatically in size as well. And that's because of better algorithms, better data, et cetera.

• 31:40Speaker 01

  And now what happened with DeepSeek was similar. You know, OpenAI had G P T four, then they had four turbo, which was half the cost. Then they had four O, which was again half the cost. And then Meta released Llama four oh five B, open source. And so the open source community was able to run that. And that was again, like roughly like half the cost, but or or five X lower cost than four O, which was lower than four turbo and four. But DeepSeek came out with another tier, right? So when we looked at G P T three, the cost fell twelve hundred X from G P T three's initial cost to what you can get Llama three point two, three B today. Right? And likewise, when we look at from G P T four to DeepSeek V three, it's fallen roughly six hundred X in cost. Right?

• 32:24Speaker 01

  So we're not quite at that twelve hundred X, but it has fallen six hundred X in cost from sixty dollars to less than, you know, to about a dollar. Right? Or to less than a dollar, sorry, sixty X. And so you've got this massive cost decrease, but it's not necessarily out of bounds. Right? We've already seen, I think what was really surprising. Was that it was a Chinese company for the first time, right? Because Google and OpenAI and Anthropic and Meta have all traded blows, right? You know, whether it be OpenAI always being on the leading edge or Anthropic always being on the leading edge or, you know, Google and Meta, you know, being close followers, but oftentimes, sometimes with a new feature and sometimes just being much cheaper. We have not seen this from any Chinese company, right? And now we have a Chinese company releasing a model that's cheap.

• 33:09Speaker 01

  It's not unexpected, right? Like, this is actually within the trend line of what happened with GPT-3. It's happening to GPT-4 level quality with DeepSeek. It's more so surprising that it's a Chinese company. And that's, I think, why everyone freaked out. And then there was a lot of things that, like, you know, from there became a thing, right? Like, if Meta had done this, I don't think people would have freaked out, right? And Meta's gonna release their new Llama soon enough, right? And that one is gonna be, you know, a similar level of cost decrease. Probably similar area as DeepSeek V3, right? It's just not, people aren't gonna freak out because it's an American company and it was sort of expected.

• 33:46Alex Kantrowitz?

  All right, Dylan, let me ask you the last question, which is that you mentioned, I think you mentioned the bitter lesson, which is basically that there's, I mean, I'm gonna just be kind of facetious in summing it up, but the answer to all questions in machine learning is just to make bigger, bigger models. And scale solves almost all problems. So it's interesting that we have this moment where models are becoming way more efficient. But we also have massive, massive data center build-outs. I think it would be great to hear you kind of recap the size of these data center build-outs. And then answer this question. If we are getting more efficient, why are these data centers getting so much bigger? And what might that added scale get in the world of generative AI for the companies building them?

• 34:28Speaker 01

  Yeah, so when we look across the ecosystem at data center build-outs, you know, we track all the build-outs and server purchases and supply chains here. And the pace of construction is incredible, right? You can just, you can pick a state and you can see new data centers going up all across the US and around the world, right? And so you see things like capacity in, you know, for example, of the largest scale training supercomputers goes from, hey, it's a few hundred million dollars. It's not even a few hundred million dollars years ago, but like, you know, hey, for GPT-4, it was a few hundred million dollars. And it's one building full of GPUs too.

• 35:07Speaker 01

  GPT four point five and the reasoning models like oh one oh three were done in a in three buildings on the same site and you know a billions of dollars to hey these next generation things that people are making are tens of billions of dollars like OpenAI's data center in Texas called Stargate right with Crusoe and Oracle and etcetera right and likewise applies to Elon Musk who is building these data centers in old in an old factory where he's got like a bunch of like gas generation you know outside and he's doing all these crazy things to get the data center up as fast as possible right and and and you can go to just basically every company and they have like these humongous

• 35:48Speaker 01

  build outs and this sort of like and and and because of the scaling laws right you know 10x more compute for linear like improvement gains right like it's sort of like or it's log log sorry but you you end up with this like very confusing thing which is like hey models keep getting better as we spend more but also the model that we had a year ago is now done for way way cheaper right oftentimes 10x cheaper or more right just a year later so then the question is like why are we spending all this money to scale and and and there's a few things here right A you can't actually make that cheaper model without making the better model bigger model so you can generate data to help you make the cheaper model right

• 36:32Speaker 01

  like that's part of it but also another part of it is that you know if we were to freeze AI capabilities where we were basically in what was it March 2023 right like two years ago when GPT four released and only made them cheaper right like DeepSeek is like much cheaper it's much more efficient but it's roughly the same capabilities as GPT four that would not

• 36:58Speaker 01

  Pay for all of these build outs, right? AI is useful today, but it's not capable of doing a lot of things, right? But if we make the model way more efficient and then continue to scale and we have this like stair step, right, where we like increase capabilities massively, make them way more efficient, increase capabilities massively, make them way more efficient. We do the stair step, then you end up with creating all these new capabilities that could in fact pay for, you know, these massive AI build outs. So no one is trying to make with these, you know, with these $10 billion data centers, they're not trying to make chat models, right? They're not trying to make models that people chat with, just to be clear, right? They're trying to solve things like software engineering and make it automated, um, which is like a trillion dollar plus industry, right?

• 37:40Speaker 01

  Um, so these are very different like sort of, um, use cases and targets. And so it's the bitter lesson because yes, you can make, you can spend a lot of time and effort making clever specialized methods, um, you know, based on intuition, um, and you should, right? But these things should also just have a lot more compute thrown behind them because if you make it more efficient, uh, as you follow the scaling laws up, it'll also just get better and you can then unlock new capabilities, right? And so today, you know, a lot of AI models, the best ones from Anthropic are now useful for like coding, uh, as an assistant with you, right? You're, you're going back and forth, you know, as time goes forward, as you make them more efficient and continue to scale them, the possibility is that, hey, it can code for like 10 minutes at a time and I can just review the work and it'll make me 5X more efficient, right?

• 38:25Speaker 01

  Um, you know, and so on and so forth. And this is sort of like where, where reasoning models and sort of the scaling sort of argument comes in is like, yes, we can make it more efficient, but we also just, you know, that's not going to solve the problems that we have today, right? The earth is still going to run out of resources. Uh, we're going to run out of nickel because we can't make enough batteries and we can't make enough batteries. So then we can't, with current technology that we can't replace all of, you know, uh, gas, you know, gas and coal with renewables, right? All of these things are going to happen unless like you continue to improve AI and invent and or just generally research new things and AI helps us research new things.

• 39:03Alex Kantrowitz?

  Okay, this is really the last one. Where is GPT five?

• 39:07Speaker 01

  So OpenAI released GPT-4.5 recently with what they called Training Run Orion. There were hopes that Orion could be used for GPT-5, but its improvement was not enough to be really a GPT-5. Furthermore, it was trained on the classical method, which is a ton of pre-training and then some reinforcement learning with human feedback and some other reinforcement learning like PPO and DPO and stuff like that. But then, along the way, this model was trained last year. Along the way, another team at OpenAI made the big breakthrough of reasoning, strawberry training, and they released O1, and then they released O3. And these models are rapidly getting better with reinforcement learning with verifiable

• 39:51Speaker 01

  rewards. And so now, GPT-5, as Sam calls it, is going to be a model that has huge pre-training scale, like GPT-4.5, but also huge post-training scale like O1 and O3, and continuing to scale that up. This will be the first time we see a model that was a step up in both at the same time. And so that's what OpenAI says is coming. They say it's coming this year, hopefully in the next three to six months, maybe sooner. I've heard sooner, but we'll see. But this path of scaling both pre-training and post-training with reinforcement learning with verifiable rewards massively should yield much better models that are capable of much more things, and we'll see what those things are.

• 40:36Alex Kantrowitz?

  Very cool. All right, Dylan, do you want to give a quick shout out to those who are interested in potentially working with semi-analysis, who you work with and where they can learn more?

• 40:45Speaker 01

  Sure. So at semianalysis.com, we have the public stuff, which is all these reports that are pseudo-free. But then, most of our work is done directly for clients. There's these data sets that we sell around every data set in the world, servers, all the compute where it's manufactured, how many, where, what's the cost, and who's doing it. And then we also do a lot of consulting. We've got people who have worked all the way from ASML, which makes lithography tools, all the way up to Microsoft and Nvidia, making models and doing infrastructure.

• 41:16Speaker 01

  And so we've got this whole gambit of, you know, folks, there's, there's, uh, there's roughly 30 of us across the world in US, Taiwan, Singapore, Japan, uh, France, Germany, um, Canada. So, you know, there's a lot of, uh, engagement points, but if you want to reach out, just go to the website, um, you know, go to one of those specialized pages of models or, uh, sales, um, and, and, and, and reach out and that, that'd be the best way to sort of interact and engage with us. But for most people, just read the blog, right? Like I, I think like, unless you have specialized like needs, unless you're a company in the space, um, or you're an investor in the space, like, you know, you just want to be informed, just read the blog and it's free, right? Um, I think that's, that's the best option for most people.

• 41:55Alex Kantrowitz?

  Yeah. Well, I, I will attest, the blog is magnificent and Dylan, it's really a thrill to get a chance to meet you and talk through these topics with you. So thanks so much for coming on the show.

• 42:03Speaker 01

  Thank you so much, Alex.

• 42:05Alex Kantrowitz?

  All right, everybody. Thanks for listening. We'll be back on Friday to break down the week's news. Until then, we'll see you next time on Big Technology Podcast.

• 42:13Speaker 02

  Now more than ever, we need someone on our side, but Rob Whitman, Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC.com, not authorized by any candidate or candidate's committee.