LLMs in 2026: What’s Real, What’s Hype, and What’s Coming Next

Is AI actually going to replace developers? Or is the hype getting ahead of reality? On this episode of Digital Disruption, we’re joined by Sebastian Raschka, AI Research Engineer and author. Sebastian Raschka sits down with Geoff Nielson to unpack the real state of Large Language Models (LLMs) in 2026. As an LLM research engineer, Sebastian bridges deep technical expertise with practical, real-world AI implementation. In this conversation, he cuts through AI hype to focus on what’s actually achievable with modern LLMs, reasoning models, reinforcement learning, and inference scaling and where the limitations still exist. Sebastian explains why most companies should not build a large language model from scratch, but also why understanding the fundamentals may be one of the most important investments technology leaders can make. This conversation breaks down: ◼️Why coding is currently the strongest LLM use case ◼️Why “reasoning” models still fail simple tasks like counting letters in “strawberry” ◼️The reality behind Math Olympiad gold-level AI claims ◼️The true cost of training large models (millions in GPU compute) ◼️The privacy risks of uploading proprietary data into APIs ◼️How enterprises should think about fine-tuning vs API-based prompting ◼️Why benchmarks and leaderboards can be misleading Sebastian Raschka has over a decade of experience in artificial intelligence and machine learning. His work bridges academia and industry, serving as a Senior Engineer at Lightning AI and as a faculty member at the University of Wisconsin–Madison. He is the author of Build a Large Language Model from Scratch and is widely recognized for his practical, code-driven approach to AI education and research. His expertise lies in LLM research, transformer architectures, reinforcement learning, and the development of high-performance AI systems, with a strong focus on real-world implementation. In this video: 00:00 Intro 01:23 The Rise of “Reasoning” and Thinking Models 03:06 Inference scaling vs training scaling 06:17 What LLMs are actually good (and bad) at 07:09 The “Strawberry” Problem and Reasoning Limits 09:00 Tool use and why LLMs don’t need to count letters 10:20 Math Olympiads & self-refinement techniques 12:01 Why coding is the killer use case 13:28 Does AI make developers obsolete? 18:02 The Reality of 10x developer productivity claims 21:43 Generalist vs specialized models 23:53 Build from scratch vs fine-tune vs API prompting 25:01The true cost of training an LLM 27:33 API customization vs owning your model 29:12 Who should build an LLM from scratch? 33:16 Data requirements & why you need terabytes 34:28 Enterprise data challenges 35:40 Retrieval-Augmented Generation (RAG) explained 46:05 Multi-agent systems & tool calling 49:48 The problem with LLM benchmarks 55:43 Using LLMs as judges 58:00 Biggest misconceptions about LLMs 1:04:19 Reinforcement learning with verifiable rewards 1:06:32 Advice for technology leaders 1:11:48 Escaping AI hype through fundamentals Connect with Sebastian: LinkedIn: https://www.linkedin.com/in/sebastianraschka/ X: https://x.com/rasbt Connect with Sebastian: LinkedIn: https://www.linkedin.com/in/sebastianraschka/ X: https://x.com/rasbt Our links:Visit our website: https://www.infotech.com/?utm_source=youtube&utm_medium=social&utm_campaign=podcastFollow us on YouTube: https://www.youtube.com/@InfoTechRG

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Geoff Nielson?

  Hey everyone, I'm super excited to be sitting down with Sebastian Raschka. He's the author of the Build a Large Language Model From Scratch and Build a Large Reasoning Model From Scratch book and video series. As an LLM research engineer, he's bridged academic teaching at the University of Wisconsin-Madison with the hyper-practical, working for the AI development platform Lightning AI. But what I love about Sebastian is that he has zero appetite for AI hype and can dive right into what you can actually do with these tools. I want to ask him what impact AI is having on coders. Is it really going to make them obsolete? What real capabilities can we expect AI to develop from here? And who should be actually building an LLM from scratch? Let's find out.

• 0:46Geoff Nielson?

  Sebastian, thanks so much for joining today. For those who don't know, Sebastian is the author of Building an LLM From Scratch book as well as video series on YouTube. Sebastian really gets deep into the technical aspects of LLMs and how we can actually, you know, create these and build our own and get beyond the hype sometimes we hear about of AI. But just before we get into, you know, who should be doing that, what that looks like, I wanted to zoom out a little bit. And Sebastian, maybe you can tell me a little bit about, you know, in your view, the state of LLMs in 2026. How are the capabilities advancing? What do you see on the horizon for this technology?

• 1:23Speaker 01

  Yeah, thanks for first, thanks for inviting me on the podcast to talk about LLMs. It's yeah, one of my favorite topics, so I think we will have a lot of fun in this episode. So, but yeah, you began with a very broad question here on like the state of LLMs in 2026. I would say 2025 was particularly interesting because there was at the beginning of 2025, DeepSeek and then this new paradigm. We can maybe get into this in more detail later, but the reinforcement learning with verifiable rewards, which is like a technique to develop, yeah, reasoning capabilities in LLMs. Reasoning is also in quotation marks. It's a broad topic.

• 2:05Speaker 01

  It's like the reasoning in LLMs, I would say we shouldn't take it too literal, like how humans reason. But it is like a set of techniques that make LLMs better at solving complex tasks. And 2025 was pretty much. Like dominated by this idea of developing these so-called reasoning, sometimes called thinking models. So everyone like from OpenAI to Google Claude, Grok, all the overweight LLMs, they all have now like different variants of LLMs, like the regular instruct or the regular variant and then the thinking variant. We can maybe talk also later a bit about the trade-offs here. But

• 2:45Speaker 01

  um so I think this was like 2025, we will see this kind of like continue in 2026 because these techniques, they are still relatively new and people are currently, I would say in the the first iteration or version of these techniques where um okay, this works and now it's like let's hone in on that and let's um make that even better, uh add some tips and tricks to that and like really like exploit that type of mechanism. So we will see more of that. At the same time, I think also, I would say a lot of progress came from the inference scaling side. And so inference, so there are basically two paradigms for LLMs. One is like the training, um and then is the usage,

• 3:27Speaker 01

  the inference. And um there's also like a trade-off. You can spend a lot of money in training, uh which is very expensive. And then you get that one model and it gets maybe used for a few months and then it gets replaced by the next model. Or we may also even update this. Um but yeah, training is very expensive and it gives you, you know, longer training, like the scaling loss, training on more data, um gives you better uh models, essentially better performance. And it is expensive though. So you can also spend money, uh extra compute, and I mean extra compute usually costs extra money because you need more uh resources during inference. That means after the training when you use that LLM. Um so you can say, okay, instead of, let's say I have a

• 4:11Speaker 01

  user and the user uh has a query, instead of just giving the first answer, you can have maybe three answers. Like if it's a math question, you have the LLM with different settings uh running three times. And then you take um like the highest scoring one or the majority vote and that but that will be three times as expensive. So that's one one version of inference scaling. There are other techniques, for example, generating longer outputs. And that sometimes helps the L L M also to think through a problem like in quotation marks. Think coming back to the reasoning. And so I would say to answer your question before we go into technical details.

• 4:51Speaker 01

  Two thousand twenty six, I would say it is still on that trajectory of we can still make a lot of progress in training, especially. Maybe not so much the pre-training, because that's not where the low hanging fruit is anymore, but more like the reinforcement learning for the reasoning capabilities. And at the same time, more clever inference scaling techniques, like more more of that. So there are like these two things, and I think this is going to continue. I don't see anything right now surprising on the horizon, but this is always like that in A I. If someone knew a new architecture that is much better than the current status quo, that's like a trillion dollar idea. Basically, we would.

• 5:33Speaker 01

  So if someone has something like that, it wouldn't be something someone had had shared already. So it's like, it's always like gonna be a surprise if something like that lands. But I don't see any indicators or any anything that, you know, gives me like, like the confidence that there will be something really, really different. It will be more like honing in on these things, basically.

• 5:57Geoff Nielson?

  I think that's really interesting and and so much of, you know, your outlook, Sebastian, comes back to this notion of reasoning or inference. And, you know, use the word thinking in quotation marks. But the reason this is so interesting to me is it sounds like, you know, there's not going to be, in your view, a huge, you know, step up in two thousand twenty six from what we see right now. We're going to continue to see performance improvements. But the reason I find the reasoning piece so interesting is. So much of what you hear in the media and so much of the noise coming out of Silicon Valley and from CEOs in any industry is these sort of grandiose statements about what they will

• 6:41Geoff Nielson?

  use AI for, what AI can do and how transformative it is. I'm curious in your mind if you can share a little bit more about what are some of the better use cases for this and maybe debunk some of the things that this technology is just still not going to be able to do at the end of 2026. And I think specifically, when you mentioned reasoning about, I call it the strawberry problem, and you may have heard of this example, but the issue right now that if you ask a lot of the leading LLMs, how many Rs are there in strawberry? It can't accurately answer that question. It says, oh, there's one and there's two Rs.

• 7:22Geoff Nielson?

  There's one in straw and one in berry, which is obviously not right and sort of lays bare some of the reasoning limitations here. So what will this be good for at the end of the year? What will it still not be good for at the end of the year?

• 7:36Speaker 01

  Yeah, so I should also, well, yeah, it's also a broad question, like not a super broad question, but like I wanted to preface this with another point to add on on my previous answer with the reasoning capabilities. It's also a spectrum. Like, well, what I'm trying to say is the same LLM with, let's say, different inference getting can have different capabilities. I mean, or so basically you can have an LLM and you can, so coming also back to the strawberry problem, you can use it in the high-power reasoning mode, but it might make mistakes on simple tasks like that. It's like almost, it's called like overthinking, but it's also some, I mean, again, I don't

• 8:19Speaker 01

  want to say LLMs think like humans, but it is also something humans suffer from. Like, you know, I know a lot of things, I, you know, I'm a researcher, I can do a lot of things, but I don't know, ask me at 11:00 PM what, like a simple math question, like 21 times, I don't know, 11 or something like that, and I will give you a wrong number maybe because I, I'm tired, my brain doesn't work anymore or in other ways, like I sometimes make really dumb, stupid mistakes, but it doesn't mean I can't do these other things. And I think for LLMs, that's also true, where the counting the R in strawberries, it's almost like. Well, it's not like you're not evaluating it in the real use case you care about, I would say. Like, this is something where you would not use the L L M. And um.

• 9:05Speaker 01

  You can actually now, I I think that's also one of the big drivers of progress in two thousand twenty five and going to be in two thousand twenty six. Is a tool use, like the L L Ms can use tools and so they don't have to. Do everything from memory, like. Counting the hours in strawberry instead of the L L M trying to do that. And I think the limitation here is usually around the tokenization and um like. Basically, how L L Ms work. But well, instead of having the L L M tell you based on its own internal compute. The L L M can do like a tool call, it could use a Python interpreter, like read this as a string. And then just use um string finding, like the letter finding in the string, and then it gives you the accurate answer.

• 9:50Speaker 01

  And so I think we also have to think about it like this, that um there are different modes how we use L L Ms and um, you know. Your mileage may vary and um. Problems, you know. Uh that, well, I I so coming back also again to two thousand twenty five again. Um, there was OpenAI, Google and some others that participated in these math Olympiad type of competitions. And they got really, really, really good results. Um, I think, well, what they call gold level, gold level performance, but. Um, at least for ChatGPT, OpenAI, they didn't use a model that is publicly available. So they used like.

• 10:32Speaker 01

  Some custom version, and this usually also in involves inference scaling. So for example, the same is true for DeepSick Math version two, they had like a paper. And they had that same L L M and they cranked up the self refinement steps where the L L M. Uh evaluates its own outputs and refines those and has multiple tries at each problem. And it boosted the performance significantly, so.

• 10:56Speaker 01

  What I'm also here trying to say is like it depends a bit on how you use the L L M. It's like a small L L M can be really good at being, you know, efficient and cheap at a certain problem, but it's not going to solve all your tasks. You can specialize the L L M more for complicated things, but then it might fail at another task. And right now, I think, well, when you go to chat G P D dot com, for example, this is like a general purpose model and has some modes where it has this auto mode like thinking, non-thinking, deciding what is the right one. But it is still trying to be, you know, like a jack of all trades in that sense that some people use it for summarizing emails, some people, I mean, you ask medical questions, some people use it for coding.

• 11:40Speaker 01

  And so, well, it's just like at that, it's pretty good. It does a little bit of everything, but it's not super specialized. And I think with L L Ms right now, the most, the biggest use case and the most promising or utility wise, the biggest use case right now is coding, for example, it's really good at coding. And we'll, we'll see it, how well it performs at other things. And, but yeah, I think I'm diverging here. There was like a question to, at the end of the two thousand twenty six year, what is like some of the tasks? And I would say a coding for sure. It's maybe the boring answer, but it's a text problem. It's pretty easy, not pretty easy, but it's pretty approachable for L L Ms. Yeah.

• 12:23Geoff Nielson?

  If you work in I T, InfoTech Research Group is a name you need to know. No matter what your needs are, InfoTech has you covered. A I strategy? Covered. Disaster recovery? Covered. Vendor negotiation? Covered. InfoTech supports you with the best practice research and a team of analysts standing by, ready to help you tackle your toughest challenges. Check it out at the link below and don't forget to like and subscribe.

• 12:51Geoff Nielson?

  I completely agree with you. From everything I've seen, this seems to be one of the low-hanging fruit areas where it seems like we can dramatically improve productivity with developers here. You had a quote I came across in your blog where you said that I still write most of the code I care about by myself without AI. Is that still true? And what do you think the implications are in terms of how developers and development teams use LLMs or don't use LLMs for what they're trying to accomplish?

• 13:28Speaker 01

  This is, to a large extent, still true. But I also use LLMs for coding in different, I would say, in different ways. The other week, I wrote a program. I can do Python coding. I'm a scientific coder. I can use PyTorch, Python, some other languages to, for scientific computing. But I'm not really a web designer and I can't really build apps. If you ask me how to build an iPhone app or macOS app, I have no idea. I've never done that before. But for myself, I automated things all my life, like 15, 20 years ago. I usually write scripts that do something to help me rename files, these types of things.

• 14:12Speaker 01

  For example, for my blog post, I have a workflow where I have all these images in one PDF. And then I usually export it and I have a script that converts it to different crops automatically and converts to different file formats. But it's still a bit tedious. I have to go to the location of my script, type the commands and everything. So I thought, okay, I can just make my life easier here and develop a macOS app, like a native app, where I can just drag and drop the file in and it performs the cropping and the conversion and everything for me. And that is something I used an LLM for. It took a few hours, maybe one or two hours, to just get it right the way I wanted it. But it's something I would have not been able to do otherwise. It's kind of like magic. I have now a native macOS app that does that for me.

• 14:57Speaker 01

  And I can do that for a lot of things, like everyday things. On my computer. Um. For that, I would like just use the L L M. I don't really care about how it does it. I mean, I can see, okay, it works. Um. And if it doesn't work, I can give it some more prompts. I have no idea how, you know, Swift UI works. I could probably figure it out, but I always wanted to learn it. But it's something where I don't have time to learn it. There are so many other things that are more important each day than just learning how to do that. Because it's not, you know, my main job essentially. But then for things I care about, for my, let's say, um, research experiments, there I usually write most of the code myself, just to figure out, like, just to think also through the problem and to do through the shortcomings, just to

• 15:40Speaker 01

  get an idea like, okay, this is, I have a very good idea what I want to do. But I, I don't often, let's say, or there are cases where I make mistakes. Um. And I usually use an L L M to like, just get a second opinion, like, hey, does it look okay? Like, it's something where in, back in the day, I had a colleague or something, or you do P R's on GitHub open source projects where other people chiming in. And it is like another layer between that. Before you share it with other people, you do like a sanity check with L L Ms to to make your work better in a sense, like to, you know, just have a, like, kind of like a proofreader. Um. Second pair of eyes in that sense. It's really good at that. Um. Sometimes also it has suggestions to make things more stable. Sometimes I, um,

• 16:23Speaker 01

  also I have multiple experiments. They have, let's say, different settings. And I have written some code here and then I could, I fixed it here. I now want to apply to all the other scripts, all the other experiments. And then I would use an L M and say, hey, uh, look at this, what I've done here. Now basically copy that over to all the other ones. I could do that manually, but sometimes it's tedious. And this is something that I can easily review because I know the code and I can see what changes it made. And then, oh, that looks okay. I just, you know, check, okay, next, next, next. And in that sense, um, I do use L L Ms for my coding workflow, but depending on the context. Well, I don't want it to, like, do everything for me in a sense, because that is, then I have no idea what's going on anymore.

• 17:05Speaker 01

  I want to do it more like in a controlled, um, type of way for, for the work I care about. That was basically what the quote was also about, yeah.

• 17:13Geoff Nielson?

  It, it makes complete sense to me. And, and the reason I bring it up is, you know, in the context of, you know, there, there's certainly some alarmist narratives out there that you may have heard, you know, the, the most extreme one, basically saying that, well, you know, Computer sciences or development as a whole discipline is going to be obsolete because we won't need to hire these people anymore. Everybody can vibe code and the machines can do it by themselves. And there's, there's, you know, a more balanced version of that as well. That just says developer productivity is going to be so radically changed that, you know, one developer can have the same, uh, you know, throughput as, you know, maybe 10 developers could a year ago. Do you see that as holding any water and holding any water in the next, you know,

• 17:58Geoff Nielson?

  two years or is that just sort of fanciful thinking?

• 18:02Speaker 01

  I think there is a kernel of truth in that in a sense that, uh, it is true, like I noticed that myself, like the use cases I just described, that it just goes faster if I tell, let's say the L M like apply my patch that I have here to the other files or something like that. So in that sense, a hundred percent. Also, for example, uh, for my own website, I added a dark mode button that I otherwise would have, it would have taken me like weeks or months. I mean, I had it on my to do list for like years and I never got to it because I knew it's going to take a lot, a lot of work. And then I did that in one day, basically. So in that sense, yeah, it is true. It kind of like makes things faster, but, um, it still work. You know, it's like, even like with this macOS app I described,

• 18:45Speaker 01

  it still took me a few hours to do that and it's a very basic app. And so what I'm trying to say, it's not making people developing code or, um, designing apps or building apps obsolete because it's still work. You can't just say, I mean, maybe one day, but I don't even think that's true. Where you can say, okay, build X, Y, Z and it will build a version of that. But usually the first version is not the final version. So there are iterations, you have to test it, you have to use it, you have to tweak it and that is going to be still work. I think what will change is, um, That the, well, what I'm hoping is with L L M's apps like that or websites, they get better than they used to be.

• 19:31Speaker 01

  I'm hoping it's like people use L L M's to improve, but they would build otherwise, but not to just have more low quality work, you know, like I think that that's what I'm hoping for the future. But everything is still work. I also noticed that for, I mean, my own experiments, it's not just the code. It's just, it's also running the experiments, doing the comparisons, thinking of additional things to compare. And, and that's all still working. I think the same is true, like there are some people on the internet saying like, uh, software is basically free now. Or like, yeah, free in a sense that, um, an L L M can do it. There is no value in, let's say, open source projects anymore.

• 20:12Speaker 01

  And I don't think that's true because, well, I think I would always take something that has been developed over many years and tested over something that the L L M gives me as a one-shot solution, basically. Because, um, yeah, people spend, I mean, I think the best of both worlds is if people use L L M's to improve things that are already there and build new things, but then iterate over it and just make it better than they would be otherwise, like adding more tests, making it more robust, patching bugs and that type of thing. And, um, it's going to be still a lot of work to do all these things, even if you have L L M's.

• 20:47Geoff Nielson?

  And it feels, um, that, that makes complete sense to me. And it, it, it feels even more reasonable, given the conversation we were having earlier about the fact that if we really want to push any of these L L M's to their limits, that probably won't be done through the generalizable, you know, just chat G P T standard model. It's looking at, you know, some of these, you know, unique use models for unique use cases, really. And as soon as you're starting to build out some of those, you need someone who actually understands what they're doing to be able to set those up appropriately. And so I, I mean, first of all, I want to feed that back to you because that was something I took away from our earlier conversation,

• 21:27Geoff Nielson?

  that it, it sounds like just trusting a singular L L M to be able to push the frontier in every given area is going to be less effective than being able to, you know, have more specific ones for specific tasks. I, i- is that fair?

• 21:43Speaker 01

  Yeah, that is a good characterization and it comes back to older problem in deep learning. Like, uh, deep learning is the field of training neural networks, artificial neural networks. Because, yeah, L L M is essentially, at the end of the day, uh, deep neural network. And the one problem is basically if you train it on one thing, it will forget other things. It's like, but it's the same also again, L L M's work different from humans, but it's the same like for us, right? I mean, if I just solve math problems every day, I will get really good at math. Um, if I don't do math, then for a few years, like, uh, do something else, you you forget things. You know, like, it's, uh, because you get new information, you don't, let's say, hone in on your skills.

• 22:27Speaker 01

  And then, uh, it's kind of like that, and the same is true for L L M's. Um, so people, when they develop or pre-train L L M's, they're very careful what goes into the pre-training mix and also in which order. And then, once you have that base model, you fine-tune it usually. And then, uh, in addition, you have also often domain-specific fine-tuning. It's like an older paper, but, um, I think it was Code Llama, um, which had like a nice graphic on that. How they developed the model in in these different stages. And usually at the end, um, the last stage is more like what you really care about. I mean, you, if you want to develop, for example, a coding L L M, you always always have to have the coding data already in the pre-training. You have to carry it through.

• 23:09Speaker 01

  Um, but at the end, you will have like a specific phase where you just fine-tune it on coding problems. But then it will probably get worse at math or Spanish or something like that. And that's the trade-off. So you you start off still with a generalist L L M, but then you kind of, um, specialize it in that trades of other skills, so it becomes worse at other things, basically. So, yeah, that that's basically how it works.

• 23:34Geoff Nielson?

  Right, and I I can wrap my head around that and it it it. Makes the case really nicely for building some of these more specialized models. So I'm curious, Sebastian, you know, we've got the proprietary generalized models sort of on one side of the spectrum. We've got, you know, completely building your own L L M on the other side of the spectrum. And then in the middle, and you know, feel free to reject my characterization here. We've got sort of the, you know, custom G P Ts or finding ways to customize some of their proprietary models. When in your mind, does it make sense to be customizing a proprietary model versus building your own? And, you know, who are the, the types of people? And what, what are the use cases that make the most sense when we start to talking about building your own?

• 24:18Speaker 01

  Yeah, so there are different levels of that. You can essentially start from scratch, like just pre-training, fine-tuning your own, own model. Like that, that's like the most work. And that I would say I would not recommend anyone doing, except you are a company who, well, whose goal it is to build L L Ms essentially. Or you are a big company who has a lot of money and really want to do something specialized. I remember it's a few years ago now, I think Bloomberg had like pre-trained a model from scratch and just focusing on their news headlines. And writing news articles or something like that, like at that scale, maybe it makes sense, but it's still, it's going to cost millions of dollars. So it's, you know, it's not cheap.

• 25:01Speaker 01

  And some, I think, well, I think what we are going to see is big, like fields, like, you know, finance, law. There, I think it does make sense if you want to develop it because like lawyers can't just use chat G P T for data privacy reasons and other reasons. And so I think like, if like there is some, like people get together in that field, there's a lot of money also in that field. They could spend a few dozens or millions, hundreds of millions of dollars to develop an L L M, like a base model for law type of things. But then it's like this general law model, and then you still have to maybe fine tune it on your internal data at your company or something like that. But so, yeah, one thing would be completely from scratch.

• 25:45Speaker 01

  And like I said, I would not recommend it because it's very expensive unless, you know, a select few players might want to do that.

• 25:53Speaker 01

  The second use case would be, or the second variant would be you take an existing pre-trained model and then you specialize it. And I think that is more feasible because there are a lot of L L M's out there in all different sizes. Like all the open weight models, for example, DeepSeek, really big model, Qwen-3 is a very popular model, or even from OpenAI, the G P T O S S model. An open weight model, it's free to use and you can then fine tune it. But it's again not trivial, so you will still end up spending tens to hundreds of thousands of dollars if you want to have a really good model there. Depending on the size, it might vary, but it's also something as a hobbyist, I think, that totally out of reach. Like unless you really, really are passionate about something.

• 26:38Speaker 01

  But the problem is, like, if you want to really build something competitive like that, or you have to have a big user, customer or use case for that, because it's going to cost and it will also at some point become obsolete. For example, if I, I don't know when Qwen-4 comes out, but like I'd say, I today use a Qwen-3 base model. That's from summer 2025 and I spent a lot of money and time to make it really good. I don't know, a few months, there's Qwen-4 or some other model is much, much better. And then my model is completely obsolete and I have to start over again. But yeah, it does make sense still for certain things. And then so taking a base model, fine tuning it.

• 27:20Speaker 01

  And the third use case would be taking an existing model and essentially just customizing it with a prompt. And I think that's a lot of what a lot of people do, they use an API where they don't even host the model. And then with a prompt, you can steer it in a certain way. It's not going to be perfect, but. For a lot of use cases, it gets you a lot of like bang for the buck because you don't have to train your own LLM, basically. But again, there are limitations. For example, if you use an API, you have restrictions. You can't use your private data or you shouldn't at least use your private data because data is public. There were a few instances on the news in 2026 where prominent people did that data leaked.

• 28:04Speaker 01

  So I think, yeah, it really, really depends on what your goal is. But if I, for example, today need something to translate articles from one language into the other, I wouldn't go out there and train my own LLM. I would just use, like, pick one of the popular ones like ChatGPT Gemini, use a prompt for that and see how far it gets me, basically.

• 28:27Geoff Nielson?

  Right. And I'm, you know, chuckling a little bit and I have to ask, I want to come back to something you said, which is, you know, you're sort of, you're sort of steering people away from using, from building an LLM from scratch, which I'm chuckling at because, you know, it's something that you've obviously invested a lot of time teaching. And so for the people that, you know, you're teaching this to, is it mostly people who are interested in learning this either as hobbyists or they're interested in learning the fundamentals of how this works? So that then when they use LLMs, you know, in a more, you know, commercial and setting, maybe it's more proprietary. They understand, you know, the basic building blocks. Who's typically interested in this?

• 29:12Speaker 01

  Yeah, you bring up a good point because it sounds paradoxical. On the one hand, I'm building these things from scratch and then I tell people not to use them. So I don't, I would say I'm not trying to steer people away from it. It's more, I want to set the expectations, right? You know, like, like just what you said, like, who are these people who should be doing it? And I think, so I also know from my experience how much work it is to build something from scratch. It is actually better than something out there. So.

• 29:42Speaker 01

  So if there is someone, like, there were some readers, for example, they stumbled upon my book. And they are not coders, they are like from different fields and the expectation is, oh, I read the book and I will be able to. Let's say there was actually a case with language translation, uh, an L L M that translates documents for me better than ChatGPT. And that's not gonna happen, basically, unless you spend, like, unless you, I mean, L L M's also, usually they are not trained by a single person. They are usually trained by a large team. And so to answer your question, we can use an analogy, for example, like something else. For example, let's say you are interested in cars, um, and you wanna learn, uh, how, you know, you just, you're passionate about cars.

• 30:24Speaker 01

  You wanna understand how the cars work, the motor works, everything works, the steering and everything. Um, but you would not build a Ferrari, like, you know, it would be very expensive as a single person. I mean, that's even, it's not even in reach. You would need a team, you need the factory, you need the design documents, you need a lot of time and money and millions of, hundreds of millions of dollars to develop a Ferrari. Instead, you would maybe develop, uh, you know, like a simpler, I don't know, car that would resemble a car from the ninety eighties or something like that. Something you can build in your garage. But building that car, you will understand how the Ferrari works. The Ferrari is essentially just a fancier version of that. And so the book is in a similar way. The, the goal is kind of like to understand how things work. It's for people.

• 31:08Speaker 01

  Well, I mean, people who maybe want to build these large ChatGPT types of models, uh, education wise, because right now, I mean, how would you learn that, you know, like, how would you, to get hired at the company, you have to show usually that you have some skills already. And so you have to start somewhere and that, that could be an entry point to learn how to build these L L M's. But it's also for people who don't even wanna build L L M's, they just wanna understand what are the limitations, why, why does L L M's struggle with strawberry, the number of letters there, like what goes into the L L M? How, how does the whole workflow work? And One way would be you could explain everything conceptually, you can, you know, explain everything in words and like, yeah, the LLM

• 31:51Speaker 01

  has text, it tokenizes it, it converts it into numbers and they go in and then there's some computation. But that's all very vague and could be misunderstood. Because a lot of like things, details you glance over. And I think the best way to really see how it works is, yeah, by actually doing it, by going through the actual steps. And these steps also don't lie, you, at the end, you have the working LLM. And so you know, okay, this is, this is actually working, this is not made up, it's not like fantasy explanations, it's really concrete. It works and that is essentially also part of the goal. At the end, you can develop your own LLM, but then the disclaimer is, it is a lot of work to get something that is really competitive, basically, yeah.

• 32:33Geoff Nielson?

  Right, because the, setting up the LLM in some ways is not even the, the hard part, there's the training, there's the pre-training, there's the data set you have. That like, all of that is what, you know, separates your car you made in your garage versus the Ferrari, right?

• 32:49Speaker 01

  Yeah, and it's also a good point in the, the data set you mentioned, in the book, I'm only using public domain data from Project Gutenberg, like a simple example book, that is a public domain hundreds of years old. Where because of copyright concerns, like to not just, you know, do that, but if you want to build a real big LLM, you need, you know, trillions of tokens, like you need terabytes of data, basically, and that would be impossible for a human who buys the book to do, because he would have to buy all the hard drives, you have to buy or rent hundreds of GPUs and everything, and that would be really, yeah, not feasible.

• 33:30Speaker 01

  And in that case, the book is also data-wise focusing only on a very small data set, and the model will then learn how to write text that is similar to this book, basically, but it's not like gonna be, you know, your next chat GPT, because that, that's impossible, it's like, I mean, not impossible, but you would, for a single person, it's kind of not feasible, you would spend a lot of money, a lot of time, and, you know. And so the goal is really helping explaining things, yeah.

• 33:59Geoff Nielson?

  Right, let me, um. Let me take this in maybe a slightly different direction. But the the dataset conversation got my wheels turning a little bit. One of the challenges for a lot of organizations trying to do this themselves is the basically marrying capabilities of an LLM with the the quality of their current data that they may have sitting in their enterprise applications. And, you know, I think it's easy when we talk about, you know, a book, for example. You ingest a book and you, you know, kind of tokenize the words and, you know, all the characters and all of that.

• 34:40Geoff Nielson?

  When we think about, you know, whether it's data that's structured in a database, whether we think about it as, you know, the unstructured data that maybe people have around that or metadata. How capable are LLMs right now at making sense of that? And is that something that you see changing in the next year or two? Or is that an inherent limitation of the, you know, of the structure of the LLM? And and basically, the reason I'm asking that is for organizations that are worried about the the quality of their data. Is that is that a problem that's being misunderstood? Is it going to go away soon? Or is that just an inherent limit of this model that they're going to have to come to terms with sooner or later?

• 35:26Speaker 01

  So let me just try to rephrase to see if I understood your question correctly. Like, the limitation is like working with your personal data, like, like data you have

• 35:35Geoff Nielson?

  That's right, with organizational proprietary data. Call it customer data or employee data or something like that.

• 35:40Speaker 01

  Yeah. And so there are different ways you can work with that data. So usually, the limitation of LLMs, I mean, it's still a limitation, but there are so many tricks now that this becomes more feasible. And the limitation was usually that you can only fit so much into the context of LLMs. And see, this is where a book, like, from Scratchbook would come in handy because you would see or understand, like, what is the context? How does the LLM process the data? And there's a limitation to the size. If I crank that up, it's going to be really expensive, or it's sometimes, at some point, exceeded, so I can't just put everything into the context. I have to be smart about it. And so.

• 36:19Speaker 01

  Traditionally, I mean, in an ideal world, you would put everything into the context, but that that doesn't work because it's too expensive. So people developed something called a RAG, R A G, which stands, I think, for Retrieval Augmented Generation. And so that's like an application layer around the L L M, where you take the documents you have that you care about, and you chunk them up, put them into a database, and then the L L M, you query the L L M. It produces like a vector embedding, like, let's say, a compressed version of your thing, like the query, and it looks for more like, simple, like, let's say, dot product math. You look for similarity to other chunks in your database, and then you retrieve that chunk,

• 37:03Speaker 01

  and you hope that this is gonna be relevant. So it's essentially like a smart lookup. But you could also, in simpler ways, think about it like that. You have a query, and then you chunk up your document into smaller parts, and then you go through it iteratively and try to find what is the most similar one. And then can the L L M use that to answer the question? So for example, let's say you are in a law firm, and you say, what was the case in 1983 where X, Y, Z happened? And you you try to pull that out, and then the L L M can use that as a part of the answer, basically.

• 37:42Speaker 01

  It's not perfect, because you are chunking the document, and it's not the full context. You have always like little chunks. So but one of the, I wouldn't say breakthroughs, but one of the, it's because it's more like a continuous development. But one of the progress, parts of the progress we've seen in 2025 was that contexts are no longer, the supported context sizes are longer. So we are now, it really depends, but there are like, even like, open weight L L Ms like Nvidia Nematron, they can do up to 1 million tokens. Of course, it's gonna be more expensive. You need more GPU power for that. But I think even like, you know, ChatGPT Online, the version, I think it can do 100,000, 200,000 tokens. And I think that's about the size. I mean, I might be wrong,

• 38:26Speaker 01

  but I think it's about the size of like one of the Harry Potter books, the first one or something. Like, it's a long context. And so for many people, this is actually sufficient. So you don't need any specific fancy application around.

• 38:41Speaker 01

  The L L M to process that you just put it in there. And there is the problem of it's called like the needle in the haystack problem where what people found though, I mean, there are multiple problems, but there's also something related to attention things where the L L M kind of focuses more on the beginning of the text that you put in. But then also the needle in the haystack problem is when people develop these long context supporting L L Ms. They have, let's say you have a question like, I don't know, like some factual data you want to retrieve, it's kind of buried in these hundred thousand words. The L M should find it and the longer the context, the harder it is for the L M to answer correctly because there's a lot of noise, it sometimes gets distracted.

• 39:26Speaker 01

  But I mean, it's similar for us humans to like the more stuff you throw it, the more complicated it is to figure it out. And so that's kind of like, like where people, I would say, where companies made a lot of progress last year to make that better. So it kind of, most of the time works and I would recommend for most people just try that first instead of building something. It's also like how I approach problems, you do the simplest thing first, you write down what performance you get and then you try to iterate and tweak it and try other things and see if it's better. But before using the most complicated thing, always try to, you know, do the obvious, simple thing and then maybe that gets you already most of the way there and then you can iterate later and see if it's worth the effort to iterate there.

• 40:11Speaker 01

  If it's worth spending three months to build something around it that can do it maybe one percent better. One limitation though is that the case you described, you may not want to do that with chat G P T because well, your data will be online. I think beginning of two thousand twenty six, there was like a case where, I mean, based on the news I read that a government employee uploaded some sensitive documents that got leaked. I think they found out because it appeared in answers from other people and it was like a confidential, high security type of data. And so, yeah, I think as far as I know, I mean, I don't want to say anything to like anything that's wrong, but I think. ChatGPT, for example, does use your data for training their models.

• 40:56Speaker 01

  They don't really like, you know, I don't think they specifically single out specific data and like publicize it or something. But it's implicitly used for the training. They try to anonymize everything. But well, if you upload it to ChatGPT, you have to be aware it might be part of the training data. And so you can't do it for everything. There are laws for certain fields where you can't just, you know, can't just share patient data, like sensitive data. And but then again, you can use a local L L M that runs locally. For example, I mean, most L L Ms that run locally support up to one hundred sixty thousand tokens, which is like, again, the Harry Potter book. It's a lot of data. And those, I mean, there's special ones like Nimtron three from Nvidia has one million tokens.

• 41:39Speaker 01

  So there's always like something you can do locally that gets you most of the way there. And at the beginning, one more thing that you interject. There's like a paper that I found really interesting at the beginning of two thousand twenty six. Let me see, I think it was called recursive language models. So the title of the paper. So what they do is it's kind of like a clever trick. They have like the query and they want to answer, let's say, a question like, like similar to a rack setup where you wanna, you have a lot of data to process. A whole document base, a whole folder of, let's say, a lot of data, basically, that can't fit into the context. And so what they do is they parse.

• 42:21Speaker 01

  So they, instead of letting the L L M do everything, they parse the input into like a string in Python in a coding environment. And then let the L L M come up with ways to like to chunk it up into sub problems. So for example, if your problem is, let's say, summarize, I know, let's, let's say summarize all the chapters in this gigantic book or something. You have twelve chapters. Instead of feeding the whole book into the L L M and trying to have all the summaries for the twelve chapters. What you say, what the L M decides to do, oh, maybe I can just have one chapter each and process it in twelve different parallel.

• 43:03Speaker 01

  Let's say execution loops and then I just pull together each summary from all the 12 chapters and write an overall summary or something like that. So like just chunking it, it's not really like rocket science is really just using a coding environment where the L M can use a tool to chunk up everything itself. But that gets you all already um most of the way there and they did that also with chat G P T. It doesn't have to be a local L M. You can do it with A P I's with tool calls. And so there is a lot of, I would say in the field, there are a lot of workarounds where you have limitations in the L L M itself. But you solve the limitations by doing clever tricks in in the surrounding A P I layer or like the application, basically.

• 43:43Geoff Nielson?

  And does that fall under the bucket in your mind of, you know, reasoning enhancements or is that something else?

• 43:49Speaker 01

  Uh, I would say it's something else. It can be reasoning related. So if it's like a problem that requires good reasoning capabilities. But I would almost, if I had to group it into something, it's more like general inference scaling, for example, where um. I mean, it's not even inference scaling in that sense that it makes it more expensive. You're just chunking it up basically into separate um into a separate subcalls. But like you said, I mean, it could be related to reasoning if your query is a reasoning query. But here I think it's also a bit tricky um because if you have like, let's say your task is to solve a math proof.

• 44:28Speaker 01

  Like something really complicated where you have a lot of sequential steps where you derive all the quest, uh, like the individual intermediate steps. I think that would not be a good case for the method because the method runs things in parallel, like subcalls and like parallel. In kind of like independent of each other and um reasoning models, they usually benefit from this um so-called chain of thought. Where they think through a problem, which is more like sequential.

• 44:55Geoff Nielson?

  Right. And and the the reason I'm asking, and it probably gets a little bit outside of your field of expertise, Sebastian, but I'll ask anyway. Is when we think probably.

• 45:05Geoff Nielson?

  In other, you know, quote AI or automation applications that start to get outside of LLMs and the transformer model, but maybe have some overlap with them. You know, I'm thinking about, you know, agentic AI or some of these AI use cases where, you know, these different kind of, uh, you know, AI systems work with each other to understand what a task, you know, what the outcome of a task should be and actually, you know, orchestrate to get it done. It seems like there's some overlap in terms of the, the processing here and being able to understand, you know, what's more likely and chunk it up. Is that, you know, does that come into play or is that completely off base?

• 45:45Speaker 01

  I do think it's a very good point. I do think it's kind of like related in a sense like to, um, understand the input, process it and present it in a way that can then be processed. In, in this particular case, instead of interacting with other agents, it's kind of like interacting with itself, but it could be other agents too. I mean, it's basically how can I, you know, divide and conquer my problem here, like in a sense, but then it, it calls itself on, on the chunked problem, but it could be as well, like delegating it to other types of, um, models. And, um, I would say that is one of the biggest progress drivers in the recent months that,

• 46:26Speaker 01

  um, yeah, the tool calling, like using different tools for different things and self, trying to do everything itself. And, um, I think it's also where a bit of, let's say, the magic, uh, comes in when you use something like Gemini or ChatGPT or Claude. It's not just the LLM. I, I mean, it's just a hypothesis, but I do think if you take something like DeepSeek or some other open weight model that is really good, and you would put that into whatever like framework they have behind the scenes, and let's say Gemini or ChatGPT, I think it would be almost like identically or similarly good. Like, so what I'm trying to say is I don't think the LLM itself is necessarily the differentiating factor anymore.

• 47:08Speaker 01

  They're all kind of like similarly good, but what is really important is how you like format things, like how to deal with context and how to the history and then how like the previous, if in a conversation, the previous, um, Yeah, like the previous back and forths, how you process that, how you use tools and that stuff. Where there's a lot of work that goes into making it really, you know, robust. I mean, even I don't know exactly because it's proprietary, how they process input at, let's say, ChatGPT. But when if I sometimes type something, I make a typo. And it's sometimes even like a relatively technical term, I have a typo in my prompt. But often I don't even care about fixing it. Because I know, okay, it's kind of dealing with that already.

• 47:53Speaker 01

  So it's just instead of all delete, delete and fix that word, I just have it with a typo in there. And I can see based on the response, because often the response involves repeating part of the answer. I can see it fix the spelling of my word. So I don't know if it's necessarily like the LLM itself, because it's like has all the different spellings or tokens or subtokens. Or there could even be like, you know, like a processing layer, like fix obvious typos, because that enhances the performance of the LLM. So because then instead of having to have a huge vocabulary or subtokens for all the different ways someone can misspell a certain word. You can just have like a dictionary fix, like, you know, like a simple fix and make it then that's work for the LLM itself.

• 48:37Speaker 01

  So I think there's a lot of magic like that happening behind the scenes to improve the performance. And I think that's why you see that the performance of something like ChatGPT or Gemini is better than something you would run locally, where most of the tools that run LLMs locally, they kind of like run it bare bones without much stuff around it, basically.

• 48:58Geoff Nielson?

  And, and, you know, I'm glad you brought that up because it still feels like there's so much tied to like the quality and the clarity of the prompt. And there's some, you know, cosmetic fixes it can do for you. But, you know, I, I'm curious on your, on your thoughts on this, Sebastian, around performance benchmarking. Because if we're talking about, you know, variable output based on the clarity of what you're asking for.

• 49:22Geoff Nielson?

  A lot of performance benchmarks are like, there's just, they seem like they're pretty clear. They're trying to do something quite clear. And, you know, to bring back a point you made earlier. It also feels like, you said, like, some of these, you know, organizations are not using their publicly available models, or there's some inference scaling that's happening. It's, that's, you know, not really there behind the scenes. You know, how, how much weight do you put into performance benchmarks at all right now? And how much should people be looking at those as a measure of the capability of these tools going forward?

• 49:56Speaker 01

  Yeah, that's a good question. I think that's one of the biggest problems in the field, how to evaluate models, let's say, fairly. And so, well, there are different ways you, like benchmarks, wise, there are different types of benchmarks. Top of my head, I would say there are three or four ones. So let's see if I can come up with what I have in mind. So that one is basically more like the classic MMLU. So that is like a multiple choice benchmark. And so that one is basically, you know, like a trivia question almost, or like a, you know, who wants to be a millionaire type of question? It's like a question, and then there are A, B, C, and D. And the model has to select one of these answers. And people use that usually to test the knowledge.

• 50:39Speaker 01

  So does the, does the LLM know about world knowledge and math and like basic things? But it's ultimately not how you use an LLM. It's basically, you would never give it all the solutions and say, give me A, B, and C, and D. You would, you know, ask free form, for example. But then the free form is really hard to evaluate programmatically because there are different ways to spell it. You can have it as a word, as a sentence. And so that's why they do A, B, and C, and D. And so like multiple choice, but then it has a lot of limitations. And so, yeah, I mean, I think that comes also like, it's like a minimum threshold. Like, I think an LLM should have a minimum score on these benchmarks to be okay.

• 51:17Speaker 01

  But then at some point, if it's passing a minimal threshold, I don't think it matters if it's 90 or 95% or that, basically. Also. We have to keep in mind also, some people run these benchmarks with and without tool use and GPT OSS model, Openweight model had a nice chart about that. When you have a model and you allow it to use tools, then it gets much better performance than not. Like, for example, if you ask a model, um, who won, let's say, the Soccer World Cup in 1998? It can try to remember it, but it can also use a tool and look it up on the internet. Let's say on the official website. And then you increase the accuracy on these types of things.

• 52:01Speaker 01

  So that is one type of benchmark. And I think, well, it is like a minimum threshold. It should be able to do these things and answer these correctly. But it doesn't really tell you how, how the LLM performs when I actually use it and query it and prompt it in different ways. Because they can be also sensitive to the prompt format. So another way is like these so-called leaderboards where, um, they have like a website and then you can use different LLMs for the same prompt. And then you can compare the answers. And you say, oh, I prefer this answer over this answer. So that sounds like actually more related to what we would care about. Like, let's say I have Gemini and ChatGPT side by side and answers a question. Okay, oh, I actually prefer this answer.

• 52:46Speaker 01

  And if you do that a lot with a lot of people, with a lot of pairwise comparisons, you can use a statistical model, like a Bradley-Terry model, to convert it into a ranking, like into numbers like 1, 2, 3, 4, 5, 6, you know, like a, so you can say, oh, this LLM is on top one. But this also has limitations because people really prefer also a certain style. Like, people are sensitive to the answer's style and not necessarily the correctness. So, because if I ask a question to an LLM, I usually don't know the answer. Like, if I have like a challenging math problem, otherwise I wouldn't ask really, right? And so, um, then you get different answers that say, and you say, oh, I prefer this one because it's maybe nicer explained or I like the language better. But it doesn't mean it's more correct or something.

• 53:31Speaker 01

  So, so with leaderboards, it's kind of like sensitive to a certain style. And there was like an incidence also, or not incidence, but like a thing with Lama 4 last summer, where, I mean, I don't know the full story because I'm not affiliated with the companies. I don't know the behind the scenes, but what was reported was that they used a different model on the leaderboards and it got really high. Either what scores, but in reality it was not a good model because it it it's all like, you know, what is it called like? It's like there's no substance, let's say, always behind it. It's more like the the glamour. Like it looks better than it really is behind the scenes than when you actually use it on hard tasks. And so it's always a bit challenging. Another way to evaluate models would be verification.

• 54:14Speaker 01

  For example, you can have math or code or something like that, like something that you can verify. Uh, let's say math and you have a correct answer, like it's a numeric answer. And there are tools where you can compare two numeric answers. You can usually how it works is you tell the L L M. Hey, I have this problem, you know, and then explain and then write the intermediate. Uh, sorry, write out whatever you need and then put the final answer in a box. Like the answer box, it's usually a latex boxed format. And then you can programmatically retrieve this answer and compare it to the reference answer. And then you can use a calculator, like you can use a calculator to say, oh, um, these numbers are the same or different.

• 54:59Speaker 01

  Like there are different tools, like, you know, Wolf from Alpha, I use Simpi, it's like an open source program. Or a library for Python where you can symbolically compare solutions. And that is really, I mean, this is accurate, right? I mean, if the model follows the instructions and puts the the answer in the prompt, I can really with a lot of certainty. There may be some parsing errors, but like ninety nine percent of the time you have a fair comparison. But the problem is it's kind of limited to um, get to math or code with the code compiled. It's it's harder to evaluate the whole answer at its whole. You can only um, yeah, evaluate the last answer point. And so I think I listed three, so one more that comes to mind, one way to evaluate is uh, using uh, judges, L L M judges.

• 55:44Speaker 01

  And so it's basically using another L L M and then you provide a rubric like evaluate. I don't know, uh, if the answer is correct, evaluate the intermediate steps if they make sense. And you give a different criteria and you say, uh, given these criteria, evaluate this answer given that other reference. And gives you, you can say, give me the answer, like the quality of the answer on a level between zero and 10, where 10 is best. And then you can do that for different L L Ms and average the numbers and say, oh, this model gets eight point five on average. This model gets nine point five on average. That is a numeric way you can kind of evaluate free form answers. But then again, there's always the catch.

• 56:22Speaker 01

  The catch is while you're using another, a different L L M for that and the L L M might not correctly always evaluate things unfairly. It might have a bias towards a certain answer style. And so, a long story short, each of these different methods to evaluate L L Ms has its shortcomings and the best way is to look at all of them together in context, not like a single one. And try to see, well, what are the weaknesses and strengths of the different L L Ms on these benchmarks? And that's really hard. And in the end, they all look kind of similar when you saw, see like releases and really you have to use it basically and see, oh, that works for me, that doesn't work for me. You know, it's like, um, it's really hard.

• 57:06Speaker 01

  It's one of the biggest problems right now to have a fair comparison, basically.

• 57:11Geoff Nielson?

  It makes sense and it's, you know, it's interesting because there's, at the frontier, there's, you know, these small differences and okay, do I do I prefer this style slightly more than this other style versus is it getting the answer fundamentally wrong? And, and, you know, do you use, you know, an L L M judge or like, how do you, how do you best answer that question? I, I am curious, Sebastian, I have to imagine, you know, you, you've been doing this for a while. And I have to imagine the interest in this from, you know, non-technical people has, has increased in the last couple of years. What, what do you see as, like, what do you personally see as some of the biggest misconceptions people have

• 57:52Geoff Nielson?

  about L L Ms and how they function and how they can, you know, get, get value and use out of them?

• 57:59Speaker 01

  It's a good question, actually. I think, top of my head, well, I don't think there's a huge misconception, but it comes more like down to expectations again, like what we mentioned earlier, that, well, it's really hard, it's really expensive to train L L M's. I think that's like, um, the misconception is, oh, I just have to, you know, X, Y, Z, and I'm, I can do that on a weekend, basically. I think that's, I think it's also, a person can understand everything on a basic level, that's not a problem, but I think the challenge compared to previous, let's say, problems in the field, is that you need a whole team to do it, like, you need an expert in GPU

• 58:44Speaker 01

  infrastructure, you need the person, the researcher who implements the core architecture, you need to run experiments, it's like, it's not usually something someone can do by themselves or on a weekend. I mean, understanding, yes, but doing this whole thing is a lot of work, um, looking, for example, at, I mean, the, most people don't share these types of details, so, but there's a, I think it was a Lama, two or three paper, they had like a very nice section on what it took to train that model back then, and, um, I forgot all the, let's say, nitty gritty numbers, but they had like, they reported, okay, we, we trained it on so many thousands GPU of GPUs, but we had so and so many failures each day, because it's also like hardware fails, like,

• 59:28Speaker 01

  um, crashes, GPU crashes, and then you might lose your whole model, and so you have to checkpoint it, or you have to build in robustness to when one GPU fails, it doesn't crash your whole million dollar run, right? So it's like, there's a lot of that, and this usually requires, um, a whole team, basically. It's not like a single person, because you have to monitor everything all the time, and, I mean, yeah, you can have notifications coming up and everything, but it's like a full time job for a lot of people to develop an L L M, um, yeah, so, so I think that's thing that has changed compared to machine learning or deep neural network training before, and that's also why you see not that many academic L L M's anymore. For example, there are a few, but for example, back in the day,

• 1:00:12Speaker 01

  I mean, with convolutional networks, image classifiers, everyone was able to do that by themselves in a small lab because university labs are usually, you know, like a handful of people. But now, yeah, the budget is like, you have to have a huge budget, a whole team, a lot of time, a lot of expertise, it's a lot of things. And that's why this is now mostly restricted to companies, basically, that have resources, basically.

• 1:00:37Geoff Nielson?

  So I'm gonna play that back to you and let me know if I'm getting this right. But it sounds like just based on how much this field has evolved, how many resources, you know, the biggest players have, the, you know, amount of staff required, the amount of compute required, the amount of, you know, just cost required and just raw data required to do this, that, you know, if you're looking to do this as a small shopper, as a hobbyist, it is valuable to learn how to build an L L M from scratch. That's something that will help you personally or professionally, but it's probably not something in most cases that you're going to then implement, you know, and try and do at scale because it's just not as practical as it may have been five or 10 years ago. Is that fair?

• 1:01:22Speaker 01

  Yes, that is fair, also with a caveat. So I agree with you.

• 1:01:28Speaker 01

  But if you fine tune, so there's like the pre-training and the fine-tuning. If you use an open weight L L M that is already out there and you build on top of that, then it becomes easier. So also just to put in some numbers, if, so taking the DeepSeek model again, because it's just such a popular model, the version three and R1 that came out December 2024 and January 2025, because they had some nice numbers in there. If you would rent the GPU they used, they put in the price for like, I think $2 per hour, it would have cost them $5 million to train the DeepSeek version three model. And this is not including any cost for the like staff, like the salary and everything,

• 1:02:13Speaker 01

  or like the building and that stuff, like the physical building, like the rent for the building to have people there. And it also doesn't include the fail runs because when you do something like that, you will fail a lot of times until you find the right configuration. A lot of trial and error also. But if you only take the solution that worked and you run it, the pure GPU cost would be around $5 million.

• 1:02:36Speaker 01

  Which is a lot of money. But then if you look at the fine tuning, the reasoning training they had, it was more like on the order, I forgot, but like hundred, two hundred thousand dollars, something like that. Uh, much, much, much lower, and that is much more approachable. Now, this is for um. Six hundred seventy one billion parameter model, it's a very, very big model. Now, if you go down in size um from six hundred to, let's say, twenty billion, you could probably do something really good with a few thousand of dollars. And then it again would make sense, but again, it's um. Not gonna be something on the weekend, it it usually requires a few weeks or a few months to get really good results. But once you are confident and can do it, you can then

• 1:03:20Speaker 01

  later on swap out the L M and can repeat the procedures. So once you you learn the workflow, it's actually easy to, yeah. Once you get going, it's kind of easy to apply your skills in a sense on other L M's. And with that, I wanted to say is, yeah, you can can do interesting things. And um there are also. A P I's now, I mean, again, not affiliated with any of the companies, but there is um. Uh, let me think, think, I think it's because thinking labs, thinking machines. Uh, there's like a company from one of the OpenAI co-founders um that has an A P I where you can fine tune and customize L M. So you don't have to have the G P U's yourself, you can use um.

• 1:04:04Speaker 01

  Uh, like like a chat G P T A P I, you can use an A P I they have, but for fine tuning, you can give it the data and the settings and then run it on cloud machines without having to worry about managing, let's say, G P U failures and that type of thing. But again, I think here it really, really helps to learn how to build it from scratch first. Um, to understand what you're doing with these different settings. For example, in my book, uh, in the reason, build a reasoning model from scratch book, I have a chapter on reinforcement learning with verifiable rewards, which is essentially the uh deep sick um reinforcement learning. And I'm only running it on a data set, it's called math five hundred. Where there are uh, so my my sorry, math five hundred is a test set that is popular for benchmarking.

• 1:04:47Speaker 01

  But I have like the training set, it is twelve thousand math problems that are not overlapping with this, and I'm training on that. And it, just to give you some numbers, it takes about, uh, so there are. Twelve thousand, about twelve thousand problems. It takes about a day to train for five hundred steps. So on one GPU, there are only. But if you would do twelve thousand, it would be ten, about twenty days or something to just train it. And it's a small model. But doing that on a few steps, you understand what is the reliable rewards? What is happening there? What am I comparing against? What are the reference answers? What is it checking? What are the different settings? There's something like number of rollouts, batch size.

• 1:05:28Speaker 01

  There are different settings in the GTRPO, like clip ratios and everything. And like the epsilon clip ratio, a lot of little tweaks and knobs. And by building it from scratch, you know what all these types of things mean. And then, once you understood, oh, that's what I'm doing. That's what's happening here. Then you can, for example, go to an API and say, oh, I'm actually confident. I know what the setting is. I used it before. I understand it's just a knob I have to tune. I will try this setting and this setting. And you know, like where it helps a lot to build from scratch to just get that intuition before you have a production system.

• 1:06:02Geoff Nielson?

  Yeah. That's great. So just, you know, as we start to wind down the conversation, I did want to ask you, Sebastian, you know, what's, what's your kind of takeaway advice for technology leaders who, you know, may be interested in learning more about LLMs or ensuring their teams better understand LLMs? Like, what's kind of the, the main takeaways you would give them in terms of how they should move forward into this space?

• 1:06:32Speaker 01

  Yeah, well, shameless plug here. I would do, I would say, doing like, well, coding an LLM from scratch. I mean, not from like scratch, scratch without any template. But for example, you know, my book or something where you do have like a guide that guides you through it. And if you are comfortable with Python, PyTorch, this, this is something you could technically do on a weekend or maybe four or five days. And then that gives you like the foundation. And then there's a lot of jargon out there, mixture of experts and like different attention mechanisms, group query attention, multi-atletin attention. But it's all derived from the original GPT model. So once you understand the core building block,

• 1:07:13Speaker 01

  it kind of like demystifies all the other things. They are all basically uh built on top of that or like flavors of that. And I think, like, I I think it is important. I mean, that's what I also like to do is to build a foundation of something. And then it's always easier to look something up that is and and you you can see how it evolves from there compared to starting. Okay, I've no idea how anything works. What is this mixture of experts? And I think uh so that is a good point because I think I had actually, uh, there were some prominent people asking me, for example, about advice on mixture of experts. Would that be something worth investing in like more like a big picture investment as a person who, let's say, is not building L L Ms. And

• 1:07:54Speaker 01

  for example, the misconception would be, oh, a mixture of experts, I can have, I can train different L L Ms, and then I can uh combine them together. Uh, I can have a math L L M and I have a Spanish L M, and then I don't have to train all everything together. And I think that sounds very plausible, but then if you uh it's not how mixture of experts works, though. It's a very different thing. And I think by building the foundation, it helps you really demystify these misconceptions, basically, like by uh so mixture of experts is essentially a module in the L L M that is like a feed, it's called a feed forward uh layer. It's just like

• 1:08:33Speaker 01

  basically a weight matrix. Uh, it's like a classic multi-layer perceptron. And if you have that connection, you know, you can't just swap anything in there. You have, it has to be trained end to end. And uh the experts are also more implicit. You you can't tell, okay, this is just doing math and this is doing uh Spanish. It's more like, yeah, it's maybe this one is stronger at math because it gets more activated when there's a math problem, but it's not like this discrete distinction. It's more fuzzy. And I think these are things that are really hard to understand by. Not looking at the fundamental architecture building blocks. It's really like even right now, I'm trying to explain it. In a sense where it's more like big picture, but big picture doesn't capture it.

• 1:09:18Speaker 01

  I think that's maybe my message is like, I think there are a lot of visualizations and big pictures that are very, they're not incorrect, but they're very fuzzy, they're very vague, and then they can lead to misunderstandings because they don't show the full picture. They try to do big picture, I think. Really doing like you, you don't have to learn about every nitty gritty detail, like GPU optimizations, it's like a,

• 1:09:44Speaker 01

  I guess it's like a detail if you don't really train on L L M's, you don't have to necessarily understand M V P for like floating point for position and how that's implemented, but having a big picture, you you would understand, okay, four bit position is less than sixteen bits. So in that sense, it's cheaper, but it also approximates because you can't store as much information and like understanding and appreciating these type of nuances, I would say, yeah.

• 1:10:11Geoff Nielson?

  Right, and and there's just no substitute to your point for like getting your hands a little bit dirty and seeing how it actually operates in an environment.

• 1:10:19Speaker 01

  Yeah, because I think that's like the, it doesn't lie, it's the truth. It's like, it's not hand wavy, it's really concrete. And this way, you, you don't, you don't have then these types of, I wouldn't say knowledge gaps, because or even if you build something from scratch, you don't build everything in all different directions from scratch, you focus on the core. But the core itself, it is the the true core, it's not, you know, like a vague concept anymore. And so it does, it's like almost like self verifying, you know, it's, yeah. It's like when you have math equations and math, sorry, but like when you derive things from first principles, you like, there are certain formulas, you can just use them and memorize them, or you can derive them from from first principles.

• 1:11:05Speaker 01

  And then you can see if you do that from first principles, oh, this formula, I mean, you don't have to do it all the time, it's just you do it once, but then you know, okay, this is actually rooted in something and these are the parts and this is why it is, because I derived it this way. It's not just a fantasy formula that someone came up with. There is like a reasoning or like a, yeah, process behind it basically. And then, yeah, I think that answers a lot of questions that people would have.

• 1:11:30Geoff Nielson?

  Well, and it feels like this is a space where there's so much misinformation or so much hype and there's so much opportunity for people to misunderstand how it works. That there's a lot of benefit to being able to just actually go to the source and see it for yourself. Yeah.

• 1:11:49Speaker 01

  Yeah, one example that just came to mind when you were saying things about like the fundamental like misunderstanding of certain things. For example, there was this model, I think it's very cool research. It's called the hierarchical reasoning model. And there was also something called the tiny reasoning models. And that came out last year in 2025. And they, I think they even won the ARC benchmark. ARC is like a logic puzzle type of benchmarks. And it got a lot of hype, it was huge in every, like in the media and everywhere, basically. But I think it would have helped if people like build something like that or like follow the paper. Because it is a really, really cool model, but it's not an LLM. They compared it to ChatGPT.

• 1:12:34Speaker 01

  It's a tiny like transformer model and it is only working on this particular task. You can have, you can train it to do, let's say Sudoku. You can train it on these ARC benchmark puzzles, but you can't say, translate my sentence from Spanish to English. Because it can only do that one thing. And I think here it would help if people, yeah, they understand a bit like what is the architecture? How is it trained? And then you see these limitations, basically. It doesn't mean it's a bad model. It's actually pretty impressive for its size, but then it wouldn't be fair to compare it to an LLM. Because LLM can do a lot of things and it is a general model where this is very, very, very specific. And I think these are things where if you understand the fundamentals, you can kind of like escape this type of hype.

• 1:13:20Speaker 01

  You can kind of, oh, this is, yeah, this is clearly just a news headline. They're trying to get attention with something. And I think there's a lot of that, like you said, there's a lot of this type of hype where, you know, it sounds good. Gets like, I guess hype and clicks and everything, but often they had it. It's if it's too good to be true, it's often too good to be true. There's usually some catch and it's easy to find that catch if um if you know the fundamentals essentially.

• 1:13:50Geoff Nielson?

  Well said. Sebastian, I wanted to say a big thank you today for coming on to the show. It's been a really interesting and insightful conversation. And yeah, I was excited how deep you could take us on some of these topics.

• 1:14:01Speaker 01

  Yeah, thanks so much for inviting me. I had a lot of fun talking about all these technical things. You know, that's what I do for work, that I do what I do for a hobby and I had a lot of fun. And so, yeah, thanks for inviting me here and yeah, it was fun.

• 1:14:16Geoff Nielson?

  If you work in IT, Infotech Research Group is a name you need to know. No matter what your needs are, Infotech has you covered. AI strategy, covered. Disaster recovery, covered. Vendor negotiation, covered. Infotech supports you with the best practice research and a team of analysts standing by, ready to help you tackle your toughest challenges. Check it out at the link below and don't forget to like and subscribe.