#335 Sriram Raghavan: Why IBM Is Betting Everything on Small AI Models

Why IBM Is Betting Everything on Small AI Models In this episode of Eye on AI, Craig Smith sits down with Sriram Raghavan, Vice President of AI at IBM Research, to explore one of the most important debates in enterprise AI right now. Do you actually need a massive model to get world class results? IBM's answer is no, and Sriram breaks down exactly why. Sriram explains why IBM chose to train its Granite models directly using reinforcement learning rather than distilling from larger models like most of the industry. The reason goes beyond performance. It comes down to data lineage, safety alignment, and a belief that small, efficient models are the only sustainable path for enterprises running AI across hybrid cloud environments. We get into the full technical stack behind that bet. How data quality has replaced model size as the real competitive advantage. Why parameter count is becoming the wrong metric entirely. How IBM's inference time scaling techniques allow an 8 billion parameter model to match the performance of GPT-4o and Claude 3.5 on code and math benchmarks. And why IBM is pioneering a new concept called Generative Computing, which treats AI models not as prompt receivers but as programmable computing elements with runtimes, modular LoRA adapters, and proper programming abstractions. Sriram also shares where IBM Research is headed next, including breakthroughs in continuous learning, agent orchestration, and making unstructured enterprise data actually usable at scale. Subscribe for more conversations with the people building the future of AI and emerging technology. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Why IBM Skips Distillation and Trains Small Models Directly (04:50) Did We Even Need Giant AI Models in the First Place? (08:12) How Data Quality Became the New Competitive Moat (11:54) Why Parameter Count Is the Wrong Way to Measure a Model (15:36) Reinforcement Learning Without Losing Broad Capabilities (22:05) Inference Time Scaling: Getting Big Model Results From Small Models (28:12) Generative Computing: Treating AI as a Programming Element (36:40) Why IBM Open Sources and How Small Models Make It Sustainable (41:25) The Path to Continuous Learning Without Rewriting Weights (51:00) IBM's Full Roadmap: Models, Data, and Agents

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Craig S. Smith?

  Wonderful to talk to you again. We've spoken on the phone. I met you here at Yorktown Heights a few months ago. At that time, you were getting ready to release Granite 3.2. And I wanted to follow up on that, but I was interested in what you guys were doing. In particular, and I think it's still the case, you have a 2 billion parameter model and an 8 billion parameter model in the 3.2 series. And what was unique about those models, from what I understood, is that you trained them directly. They're not distilled from a larger model.

• 0:42Craig S. Smith?

  So can you talk a little bit about why that's an advantage?

• 0:47Speaker 01

  Yep, yep, absolutely. So Craig, first of all, good to see you again. And welcome to Yorktown. So we'll maybe wind back a little bit to our conversation. I think that was around the time when with Sun Sein, the DeepSeek models released. And distilling, while distilling is an old technique, had come back to the fore. Because DeepSeek, in addition to releasing their big model, had also shown this idea that you could take smaller models. I think they experimented with Lama and Mistral smaller models. And said that using the big model, I could distill those Lama and Mistral models down and get their performance on specific benchmarks, like code and mat benchmarks, to really shine well beyond the original capabilities of those models. So that's why distilling was in four.

• 1:31Speaker 01

  So that's when we had the discussion, and that was with 3.2. And we have continued that actually and released 3.3 now, in the same approach. So there were two reasons why we wanted to explore an alternate approach. So one is, while distilling allows you to optimize, especially if you, because when you're distill, you're basically essentially having this bigger model teach that smaller model to do better at something. That tends to do make that smaller model do better at that that thing. But it does break some of the base training and alignment of that model. So what we had shown was, yes, absolutely.

• 2:11Speaker 01

  Lama and Mistral models do better at the code and match benchmarks than they were originally doing on their own. I believe they they showed it with a Lama 8B, I think similar Mistral models. But you do then lose the broader capabilities of the model. So their bench performance on broader benchmarks goes down. And more importantly, it also often, and we validate with measurements, takes away some of the safety alignment of the models. So what we were doing was to say, we certainly see the logic of that. But as you know, our overall strategy with Grenade has been really focused around small language models. And what can you do with smaller models and how far can you push them?

• 2:52Speaker 01

  Because the more you can do, you open up AI to run more cheaply in wider environments. And as you know, we've been doubling down on 2 billion and 8 billion dense models. And we started that with our 3.2 series. So we said, in this particular case, can we do RL, reinforcement learning, on our 2 and 8 billion models directly? And furthermore, on top of that, how much can we push with inference scaling or runtime scaling? Now, the reason to do that is, we can do that without, by avoiding the two pitfalls that I said before. I can do it by saying, can I do this in a way that, yes, it gets better at what I'm teaching it, but it still holds on to the overall base performance?

• 3:37Speaker 01

  And we were able, we showed that you could do that with 3.2, and we continued that journey with 3.3. We further, and I don't know whether we talked about it at that time, we had also been innovating on inference scaling techniques on top. So as a very concrete example, our Granite 3.3 model with a combination of the RL on top plus inference scaling using some new techniques, and we can talk about those techniques, can match the performance in code and Mac benchmarks of GPT-4O Claude 3.5 models that are

• 4:09Speaker 01

  Two orders of magnitude bigger in size. So that's sort of the journey that we are on. And the reason we like that is A, it addresses those two disadvantages. Second, we control the origins of the data that went into this. When you're distilling from some other model, then you're always beholden to okay, the origins of the data. And as you know, we are very focused at IBM on making sure our granite models have lineage data that we control. We have curated and we have gone through the vetting process. So this path gives us an organic way. Using data that we can fully speak to or aware of have curated to produce models with very, very, very high quality performance.

• 4:50Craig S. Smith?

  This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data and the ability to understand risk clearly. That's one of the reasons I like what Tasty Trade is building. With Tasty Trade, you can trade stocks, options, futures and crypto all in one platform with low commissions, including zero commissions on stocks and crypto.

• 5:34Craig S. Smith?

  So you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like Active Trader Mode, One Click Trading and Smart Order Tracking. And if you're still learning, Tasty Trade offers dozens of free educational courses

• 6:18Craig S. Smith?

  plus live support from their trade desk reps during trading hours.

• 6:24Craig S. Smith?

  If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker dealer and member of FINRA, NFA, and SIPC. Yeah, and that idea of, of direct training smaller models, just in the evolution of these models, is it the case that if you knew what you know now,

• 7:03Craig S. Smith?

  uh, back before OpenAI train GPT two and three and four, that you wouldn't need to spend that much money on training that larger model? I mean, not that there aren't some advantages to having a large model, but it just took building those really big models to realize that you can do it.

• 7:26Speaker 01

  Yeah, sometimes it's the arc of technology.

• 7:28Craig S. Smith?

  Yeah.

• 7:28Speaker 01

  Uh, or you know, you can call it hindsight 2020. But, but I do think that it was, it, it, the field needed to go through at some level. Now, it was intuitively clear to a number of us and we had been talking about efficiency, cost and price from the get-go. And I'm talking 2023 when we released Watson X. Even at that time, we were talking about that being going to be important. Um, it was intuitively, it was clear that if you know what you're going to do, and it doesn't have to be one thing, it could be a collection of tasks that an enterprise needs. Versus, I'm going to give you a model that's going to do many, many, many things, including things I have no idea what you're going to ask it to do. Then certainly possible to do it with much, much smaller model.

• 8:13Speaker 01

  But I do think that that intuition played out. And in fact, it played out, maybe I would say even faster than you could have imagined. But it played out because of a combination of factors. So one is, A, we certainly got better as a field, as a community around data quality.

• 8:30Craig S. Smith?

  Right.

• 8:30Speaker 01

  So that allowed us to say, you know what? I can now do today, I mean, today in our training process, Granite 3.3 halfway into its training was way better than Granite 3.0. Halfway into training, why? Because we had done so much data quality work. That now when you are digesting tokens to put into the model, you get better. So data quality has certainly improved as a community. Second, there have certainly been architecture innovations. So now we know how to more efficiently package what you can in smaller number of parameters. And I think the third to your point is, I think we have gotten better at understanding what are the real requirements for enterprises.

• 9:09Craig S. Smith?

  Yeah.

• 9:09Speaker 01

  And since our focus is on enterprise AI, I think these three combination of factors has led to to where we are today. And as you rightly point out, there are always scenarios where larger models are needed. We know, for example, for extended reasoning tasks, we know larger models do better. But that's where this inference scaling idea we like, because the advantage with inference scaling is it lets you take a small model, spend compute to make it behave like a larger model.

• 9:39Craig S. Smith?

  Yeah.

• 9:39Speaker 01

  And I can turn that on demand. But if I have built a larger model, it's not easy to shrink it. I'm always paying that cost. So we like this idea of continue to push the smaller model, make it better and better and better. Then always you have the luxury to spend more compute time to make it do even better when it really, really matters and it's an important question worth that kind of money.

• 10:01Craig S. Smith?

  And that size, the two billion or eight billion, that is a direct correlate to the size of the training data. Is that right?

• 10:11Speaker 01

  No, actually, the training data has remained quite. So in fact, I would say the other way, which is we had bigger models that were sometimes consuming smaller data early on. We have managed to even pack more data into small models. Now, what has happened is, in fact, I think the data has gone to occur. So early on, it was about, the whole conversation was about two things. Size of the model and the amount of pre-training data. Now it's become architecture of the model and the quality of the data. So that's sort of the shift that has happened. Now, you do still put a fair amount, I mean, these models still take six, eight, ten trillion tokens worth of data, but they are certainly much, much more high quality aware. I think the second is the focus has also shifted a lot

• 10:56Speaker 01

  from pure pre-training to all of these other stages that have gone. So if you sort of look at what has happened with training. We had this very simplistic view of you do pre-training, which is the self-supervised learning. And then you do post-training. And post-training typically used to be basically supervised fine-tuning or instruction tuning. Now, if you look at the pipelines of, you know, what us and others are doing, there's obviously pre-training. But then there's like this mid-training, where you do some things like prepare the model, make it ready to receive RL.

• 11:27Craig S. Smith?

  Yeah.

• 11:27Speaker 01

  That involves some amount of, you know, supervised fine-tuning. Then you do specific actions to make the model good at chain of thought. Then you do the RL phase. Then you do supervised fine-tuning. You might do a couple of rounds of RL. So the stages have become more sophisticated. And almost to the point where you could argue that base pre-training, which used to be the biggest focus early on, is almost, I'm going to use the word commoditized. I'm going to use that, you know, in this context. Yeah, everybody does pre-training.

• 11:57Craig S. Smith?

  Yeah. And to improve the quality of the training data, what, how difficult a process is that? Or is it?

• 12:07Speaker 01

  It is. It is probably the one where a lot of work has happened again in the open community. We've contributed, others have contributed. In fact, we've created open source projects, a project called Data Prep Kit, DPK. It's born out of something we use to allow the community to create and share recipes for cleaning pipelines. Because there isn't necessarily one way to do it. I may make a certain set of decisions in my model training pipeline because I know what I'm doing. Content that I am actually discarding may actually be important for you. Imagine you're building a model explicitly to detect poisonous content or, you know, hateful content. I'm going to throw it away because I don't want my model to see. You might actually want it because that's what you want to build your detectors, right?

• 12:52Speaker 01

  So we know that data cleansing requirements, processing requirements are different. So we're starting to share the infrastructure and the pipelines to do that. And then websites, you know, communities are releasing data sets. So I think that has been an important focus. In fact, interestingly, we even released from IBM, and we can send you the information, a data set we call Nice Web. And what we tried to do is to show, hey, here is the cleaning recipe we used. Have at it, you can improve the cleaning recipe, take it, adapt it. So that's now become a currency of sharing in the open environment. So we went from just sharing data sets to sharing recipes that now you can adapt the recipe. So I think that's a fascinating, fascinating journey.

• 13:35Speaker 01

  And also remember, because the training stages have become more, there is now data sets at every stage. A lot of the focus used to be on just pre-training data.

• 13:44Craig S. Smith?

  Yeah.

• 13:44Speaker 01

  Now, instruction tuning data, RL data, all of that is starting to become important. And it's a very, very vibrant community. That's what's so fascinating about it.

• 13:52Craig S. Smith?

  Yeah. And, and do you intend then to scale up the eight billion? Even though you, you know, you don't want to go presumably to four hundred billion or something. But is there an advantage once you've worked out this, uh, this training strategy or pipeline? That then you can start building bigger models? I mean, is there any advantage to that?

• 14:16Speaker 01

  We can, but I think one of the things we're making is we're also now doubling down on mixture of experts architectures. And actually, let me talk about sort of two architecture innovations. I almost think that, I think we're at a point where the number of parameters as a model starts to not become the right descriptive element. Because what happens now with these architecture innovations is that simple number that we used to track sizes probably is now underserves the purpose of figuring out, you know, what model choices are, right? So one is with mixture of experts now, you have two numbers. You have the overall parameters used and then the actual size of the individual active parameters.

• 14:58Speaker 01

  We're also now, and we actually worked with Princeton and CMU on, on bringing state space models. Now, state space models are models that try to address the, the quadratic computation bottleneck of, of attention, right? With state space approaches. So we actually worked with the researchers across CMU and Princeton, including some of the researchers who had originally invented space-based model to look at hybrid architectures where you can combine. Classical attention or not classical, the traditional attention and state space together. So you get the efficiency benefits of state space while preserving the overall accuracy quality. Right? We released that in the open.

• 15:38Speaker 01

  So now what's happening is I think model parameters are going to not become the most important currency. A better metric might be, what is the memory requirement of the model? And you characterize that by asking, okay, on average, let's say I have this many user sessions with this context length. I want to run against the model, how much memory do you need? I think that is a much more useful functional characteristic. Because you might have, I don't know, 50 billion parameters. I might have 38 billion parameters. But the choice you made with how much state space you're using, what mixture, architecture, number of exports you used.

• 16:20Speaker 01

  Your 50 billion with your choices might actually take lesser memory than my 34 billion with a different set of choices. So I think very soon, we will recognize that, yes, there is a simplicity to using number of parameters to talk. It's just easy to understand. But what's going to matter when you start making model choices is really going to be memory requirement. Because more and more, that's what's determining your latency, performance. Do I need a H200? Can I get away with a H100? Can I work on an A100? Can I run on an L40S? That all of them start to become important questions.

• 16:55Craig S. Smith?

  And on state based, is that Mamba that you're talking about?

• 16:59Speaker 01

  So we released a model called actually Bamba 2. So it's in that family.

• 17:03Craig S. Smith?

  Right.

• 17:03Speaker 01

  Inspired by Mamba, and we'll, we'll send you the, the information around it. But Bamba 2 was jointly done across CMU, Princeton, IBM Research. And we released that in the open. We are bringing some of that learning into the next generation of Granite 4 that, that we are working on. Which will be both mixture of experts and have some of that, some of that architecture. So lots of innovation.

• 17:23Craig S. Smith?

  Yeah.

• 17:23Speaker 01

  And the advantage is that, look, the currency, as I said, is memory. And so the question you're asking yourself is what's the size of the KV cache when your model is running? That's like, you know, the most important performance question.

• 17:35Craig S. Smith?

  Yeah. Uh, in, in the, uh, in enhancing, uh, the reasoning through direct training.

• 17:42Speaker 01

  Right.

• 17:43Craig S. Smith?

  And I'm reading a, a question now.

• 17:47Craig S. Smith?

  How did you avoid overfitting to logic-heavy tasks like math and coding at the expense of?

• 17:55Speaker 01

  Yeah, so it's always the usual trick in AI.

• 17:58Craig S. Smith?

  And actually, in that, can you talk about thought preference optimization?

• 18:02Speaker 01

  Right, right, sure. So I think the, so the game is always balancing the task you want with constantly reminding the model about the other things you have to do. So there's a, there's a training procedure that constantly credits the model both for what you want it to do, but keeps reminding the model for all of the other things you want to do. There's a use of a replay buffer from past data. That's where some of the balancing act comes. Also, remember that, that the, this idea of training for code and math, the reason we all go there first is those are tasks for which you can write verifiers easily.

• 18:44Craig S. Smith?

  Right.

• 18:45Speaker 01

  It continues, it continues to remain an interesting research question, which is,

• 18:50Speaker 01

  how are we going to scale this to other domains? Where verifiers will take work to write, possible to write, we can certainly do that. Are there scenarios in which you deploy models? And then learn those chains of thought by actually observing humans in action? Like, imagine I'm, I'm looking to debug IT systems, which is an area I'm interested in. We, you know, we're interested in this at the company. Now, the chain of thought, the sequence of steps that an SRE uses to look at a problem. Go, let me go look here. Oh, that isn't it. Let me go look there. Well, that isn't it. If this and this are not true, let me go look there. That's a chain of thought. That's backtracking. So teaching that to a model is a different task.

• 19:31Speaker 01

  Now, it will, the model will understand how to do this broad idea of exploration and backtracking using code and math. But how to backtrack, how to, how to explore is a very domain-specific thing. And that is something we are going to have to continue to teach the model. So back to your question, this balancing act is usually the usual trick in AI, which is create your objective function, your data, so that you are not overfitting to the task you are constantly balancing. And then, I think this whole space of RL, I would characterize it as there are sort of two variables that you're going to see a lot of innovation on. One variable is how am I getting the model to do the exploration?

• 20:14Speaker 01

  Am I doing it by generating a lot of synthetic data? In some domains, that's the best way. Which means I know I can synthetically generate chains of thought. And I do training of the model data-driven. In certain other cases, I will need an external program, like a trained reward model. Yeah. To teach the model how to explore. So that's going to be one vector of innovation. The second vector of innovation is this whole verifier. So how you do exploration and how you do verification, I think are going to be the game. And you're going to see thought preference optimization, a process reward model. All of these things, I think, are going to evolve very, very, in very interesting ways.

• 20:57Speaker 01

  One example of work, actually, our innovation team in Red Hat, jointly with IBM Research, is this exploration called particle filtering. Oh, I'm sorry. Called particle filtering. It was an inference scaling technique. And the intuition for the idea is, look,

• 21:15Speaker 01

  can I think of the task that I'm trying to teach the model as a probabilistic inference? Borrow ideas from probabilistic inference? And what you end up doing is you keep a lot of paths active. So instead of always picking the best at any given time, you keep a lot of, and that's why it's called particles, because it's like you're exploring. At any given time, there is a frontier of particles. You keep a number of them active at any given point of time. Periodically filter some, explore the remaining particles. And by formulating it by taking ideas from, you know, well-understood probabilistic inference, you get a very efficient scaling technique, right? Inference scaling technique.

• 21:58Speaker 01

  So I just think that this is going to be one of those rich spaces. What we have seen is, for things like code and math, The data side, right, because you're able to now, there's lots of data sets out there, you're able to generate data. Even simple majority voting pushes a long way. Now, it works very well when you want to hit benchmarks.

• 22:17Craig S. Smith?

  Yeah.

• 22:17Speaker 01

  It's not clear that majority voting will necessarily be sufficient for every other domain. But in domains like code and math, the result that I showed you where our 3.3 model went up to Poro, it were able to do that with just majority voting. Why? Because we had trained the model in our RL training direct on top of the model with enough high-quality, long-chain-of-thought data that it had a very good understanding of code and math. Last comment I will make is, I did say that, yeah, pre-training is getting commoditized, right? And all the interesting innovation is downstream. But the amount of tokens of certain kinds you put in pre-training still matters. What do I mean by that?

• 22:59Speaker 01

  Is if I want to build, using all of these kinds of innovations, a high-quality code or math model, I better have put a non-trivial amount of code and math data to pre-train. If I don't do that, then this ain't going to work. So there are some dependencies. But I think this is getting to a point where a lot of the data mixtures for pre-training are sort of getting standardized in the community. So I think everybody is adopting, you know, roughly the.

• 23:27Craig S. Smith?

  Yeah. Can you talk about inference time scaling? In particular, as I recall, I'm not sure in 3.3, but you allow developers to turn it on and off.

• 23:43Speaker 01

  Correct.

• 23:45Craig S. Smith?

  In that, you know, when, as a consumer, when you use these models on your desktop, you can choose whether you want reasoning or not. So what's the difference? I mean, this is through an API, it's switching it on and off in the code, but.

• 24:04Speaker 01

  Right, so the backend implementation, right? So is the differences between, do I have two different models? And when you select, I'm actually routing it to two different models? Or my same model can turn it on or off on the.

• 24:16Craig S. Smith?

  I see.

• 24:17Speaker 01

  That's the real difference. And it makes a difference because running two versus one is cost, right? And I have to train. So that's the difference.

• 24:22Craig S. Smith?

  Okay.

• 24:23Speaker 01

  But back to inference scaling. So I give you one example of this particle filtering approach. There's lots of other approaches getting implemented. I think that inference scaling, depending on, I like to think of it as the, my simple mapping is how much compute are you spending at runtime? Now, you could spend compute by taking one model, asking it to run that many different times and doing something. You can take one model, do this kind of exploration. I also put inference scaling under the category of, I want to answer a user question. I'm going to take multiple models, combine their results.

• 25:06Speaker 01

  That also in some respects you're scaling because to answer that one user query, you're now spending more compute. So, I think of inference scaling as this broader category that says, if your model is primed well, then you can get stuff out of the model that are much better than simply asking the model to give its best answer. When the model gives the best answer, it's making a set of decisions throughout the network. And you're basically saying, okay, that's your best guess. Now try harder, evaluate more possibilities. I think that is the general idea of inference scaling. And the reason I think there's going to be a fertile area of research is,

• 25:48Speaker 01

  A, small models have an advantage here because it's possible to make a small model behave like a big model by inference scaling. And I pay the cost only when I need to, whereas it isn't possible to go the other way around. So, everything being equal, if I told you, you have a complicated task to do. You want to do that complicated task once every, you know, one out of 10 task you want to do. Nine out of the 10 task is not a complicated task. And if I can make sure that I can get you the expected accuracy of that complicated task. And I can do that with a eight billion model or a ten billion model with inference scaling.

• 26:30Speaker 01

  I can optimize this to the point where its cost is as good or cheaper. Way cheaper in practice than a big model that you end up having to host for all of the nine tasks where you don't need the model, right? So that's why I believe that this this cost and optimization is going to drive this move towards. Small model asked many times better than a big model asked once. The big model probably has the capacity to give the correct answer right off the bat. But it's not important if I can figure it out because you're all hidden behind an API get the answer back. The second point is the small versus big and this is a different dimension, not just an inference scaling.

• 27:12Speaker 01

  One of the reasons sometimes we need a bigger model is especially when you build more complicated applications like agents. Your prompts become really, really big. Now, what happens is if you look under those prompts. When you take a big complicated agent prompt. You you peek under the covers and sometimes these things run into pages and pages like like an English essay. You will see on the prompt instructions to the model, requirements, security. Thou shall not do this, thou shall do this. It will have formatting. It will insist that this must come before this, this must come after this. So it's really a program. So it's a program written in English.

• 27:56Speaker 01

  And now you can see why you really, if you want the model to parse through all of that. You need a big model to have seen a large amount of data to be able to do that. But if you take that simple leap of saying, look, this one is not something an end user is going to do. This is a developer task. End users working with a chat G P data interface want to use natural language, I get it. This is a developer task. So we see a shift from massive prompt engineering behaving as if it's a software engineering task to. Let me actually turn those into actual programming tasks. When you start to break it down. Hey, I'm going to declare some instructions. I'm going to declare the requirements for the model. Here are dos and don'ts for the model.

• 28:41Speaker 01

  You start to decompose it. You can get away with a much, much smaller language model. So with a little bit of procedural logic on top of a small language model, you can get as good performance as a large model. So I see two vectors pushing towards the fact that small languages model can get you what you want in many, many cases. One is this vector of inference scaling, which is a built in. The developer doesn't see it, the model does it. The other is where the developer is involved in saying, okay, you know what? I know what I want you to do. I'm just going to break it down. And in fact, this is where you can even do checks outside the model.

• 29:19Speaker 01

  You can say, hey, as a developer, I know I wanted the last piece of the document to always have my name in the email signature. It's a very simple check to do. If it doesn't show up, go back to the model, try again. These things can be done. You will see now order of magnitude reduction in model performance. So we certainly see these two vectors continuing to push the needle on how small models are going to do. Today, because of the way the field has evolved, prompts have become sort of this only lingua franca. But we're at a point where we are building more and more sophisticated applications. If you're only building a Q and A application, prompt is okay. If you're building a full-blown agent, talking to tools, making planning calls, iterating, reifying,

• 30:03Speaker 01

  I think you're going to see programming obstructions. And we're very actively working on some of those programming.

• 30:09Craig S. Smith?

  Yeah, and that's done by the developer in the code of the application.

• 30:14Speaker 01

  At the application, but we are going to give them now programming frameworks. So we have started to use the word, by the way, generative computing. So we keep talking generative AI. We really believe we are on this cusp and you will hear more about this, I think, when you're there. You're going to see us talk about generative computing because I think we are at the shift where. We went from, hey, this cool self-supervised model L L M, cool, what can it do? Then we went from that to, okay, you know what? A model is part of an entire platform. So that's why you started having platforms from us and others. We built Watson X. Where models are important, but you need all this plumbing around the models. We talked a lot about models as a data representation. We said it's important to get enterprise content into those models.

• 30:59Speaker 01

  That's why techniques like Instrac Lab, you use RAG to give the model enterprise. That started appearing. I think we are at a point where we are now, all of those are true. It's a powerful self-supervised representation. It's core to an A I platform. It is a data representation. But I think we are at the point of saying a model is now a new type of computing element. If it's a new type of computing element, a computing element has abstractions. It has a runtime. It has waste you speak to it. And I think that is what we see starting to emerge. I think people are very quickly realizing. So we think that just as we talk about classical computing and quantum computing,

• 31:40Speaker 01

  we are very comfortable now starting to use the word generative computing. Now, only difference is that unlike quantum, it's not an entirely new hardware off on the side. It's a hardware we understand. But I think treating the model as a computing element now allows you to ask, how do I program this computing element? A programming a compute element doesn't have to mean writing five pages of English. Programming a computing element can be, here are the abstractions. So that's where we are pulling and it's motivated by what we just discussed. If I can do that, I can do very, very well with much, much smaller models and that drives efficiency.

• 32:17Craig S. Smith?

  Yeah. And when you say generative computing, is there a model that's generating the program? I mean, actual code to do these things?

• 32:30Speaker 01

  No. So we are using generative computing to what we described, which is a program that you use to work with the model. But not as if you just have to give it a prompt, but you can actually use instruction. So as a concrete example, let me, let me give you a flavor of what a generative computing program could look like. So we released along, actually with Granite 3.3. Instead of what we call LoRa adapters, right? These do one thing very, very well. So they, you know, there was a LoRa adapter that does hallucination detection. There's a LoRa adapter that can rewrite queries better for better, better search performance. There is an adapter that does uncertainty quantification, which says, these are all trained adapters for those specific things, right?

• 33:14Speaker 01

  So you can think of these LoRa adapters almost like plugins stuck on top of Granite 3.3. So you take Granite 3.3, you can have three, four, five adapters, you know, enabled. Now, a generative computing program will actually, where a program, you say, what? I'm building a RAG application and I want to turn on or off one or more of these LoRa adapters. So that's what we mean by generative. Now, why is this interesting? This is interesting because I could do this whole thing without these individual LoRa adapters with a big model and in English say, I want you to do hallucination check. But now that big model has to be trained to get all of this done. It has to rewrite the query, it has to do hallucination detection, it has to do uncertainty quantification.

• 33:57Speaker 01

  But now by decomposing it into a grid-based model and individual LoRa adapters, the programmer controls, they stitch a workflow together, turn it on and off when needed. I'm working in an environment where uncertainty score is not important. I don't need that, that LoRa adapter. So that's what we mean by a generative computing program. So the model is really wrapped by a runtime. So you no longer are talking raw to the model. You're talking to a runtime that wraps the model. The runtime exposes abstraction that lets you on and off functionality. And then the runtime enables or disables these LoRa adapters. Eventually you're running an inference through. Finally, that's the operation.

• 34:39Speaker 01

  But you are setting up that inference based on these flags and other control parameters.

• 34:44Craig S. Smith?

  And turning on and off these elements, turning on and off inference time compute, for example, is that automatic within the model? Does it decide?

• 34:57Speaker 01

  The runtime, now the runtime associated with the model does it.

• 34:59Craig S. Smith?

  Right. It's not, it's not the developer in advance.

• 35:04Speaker 01

  No, no. So you just set up that environment as the developer comes in and uses this abstractions. So for example, Take this one, you have a requirement to, you have a requirement that your output looks a certain way. The developer can say, here is a little function that's going to check that you meet the requirement, right? Can be a little piece of a program. So, think about it, if you didn't do it this way, then you have to teach that model how to do this requirement. That's why you end up writing these laborious English, using examples that show this is the right, this is wrong. Because that's the way for the model to learn. On the other hand, all of that is useful if the model is directly exposed to an end consumer in a consumer use case.

• 35:47Speaker 01

  We're talking about enterprise applications that developers are writing. So, if there is a simple check that says, "This document's output must always end with date and IBM confidential at the end." That's a simple check that any programmer in high school can write. You declare a little function. Okay, you write your generative computing program. You can iterate through and you will achieve the same effect. So, that's what we mean by the generative computing program. What's interesting is, there are now more and more efficient ways to do this. So, we talked about LoRa adapters, right? One of the challenges with LoRa adapters is, yes, they allow you to build these modular pieces of functionality.

• 36:29Speaker 01

  But anytime you invoke a LoRa adapter, it's like you're processing a set of tokens. Then you invoke a LoRa adapter. While processing the set of tokens, you would have generated a KV cache for the base model. Soon as you invoke the LoRa adapter, you'll have to throw out that KV cache because you'll have to redo the KV cache because the LoRa adapter has been trained on its own, right? So, you'll have to repopulate that. So, you've been developing an interesting technology called activated LoRa's. And the goal of the activated LoRa is this, can I share KV cache with the base model? So, you can turn me on and I can actually reuse the work that was done by the base model in processing some amount of tokens.

• 37:11Speaker 01

  Then just do my thing that I need to do, move on, and the base model can continue to move on. So, this is now squeezing out even more. And we have seen activated LoRa's can give two, three X advantages between dedicated LoRa's. So, you see where I'm going with all of these things. So, they're all about finding powerful ways to bring down. So, we are, we are, we are very...

• 37:33Speaker 01

  I would say very convinced that abstractions around models are going to emerge. We're experimenting with a set of abstractions. But these are going to emerge and they are going to be supported by both the programming model and then these kinds of innovations, LoRa has activated LoRa, I'm sure there are other innovations waiting that will allow us to write really complicated things. And if you step back and look at what's happening, things like what people call agents, already agents contain some form of classical procedural imperative code. They always do.

• 38:09Craig S. Smith?

  Sure.

• 38:09Speaker 01

  Right? And what people are figuring out is what should I do and what can the model figure out on its own? But there's a little bit of a contract on what I should do. Certain things today we are asking the model to do. We may be en route of just doing it outside because then you can still achieve a lot of the same functionality but get it done with a small model. So I think that's the journey we are on and that's what's exciting to us. That's why we think we are at a point where it's no longer, we're not just doing AI. We actually have a new kind of computing element that we're going to know how to interact with.

• 38:37Craig S. Smith?

  And actually that leads me to a question I wasn't going to get into to the end. But IBM's commitment to open sourcing these models. Meta has just asked, you know, people to help raise money for training future Llama models. And that's something I've kind of been used to ask Yama Kuhn about. I mean, how long are they going to stay committed? This is such an expensive enterprise if you're just giving the models away. If you're doing it with capable but smaller models,

• 39:17Craig S. Smith?

  you can do many more than Meta can with Lamas. Is that part of the strategy with these small models?

• 39:24Speaker 01

  Yeah, I think, look, our strategy was about small focus models from the get-go.

• 39:29Craig S. Smith?

  Yeah.

• 39:29Speaker 01

  Right? And I think that was motivated less necessarily by whether that's the only sustainable way to put it in the open or not. We think that's the only sustainable way. To get enterprises to at scale adopt AI because finally they have to run these models. Right, I, yes, I could spend the capital and train the biggest model. But if that isn't necessarily that going to get deployed and particularly with IBM strategy. We are looking for it to run across a hybrid environment. So we wanted to work on multiple clouds, on prem infrastructure, diverse set of hardware. We've been very public about that too. Run on Nvidia or run on AMD hardware. We have our Spire chip where we run on our own mainframe. So hardware agnostic, multiple environments.

• 40:12Speaker 01

  It's very important and that means that being able to optimize the model to get it running very, very efficiently is really important. So if I deal with a huge object that isn't easy to manipulate. I'm basically shooting myself in the foot with kind of a strategy. So we were on small models anyway. So I don't think it was necessarily one way or other informed by this. And you can see that even in the philosophy I've taken, Granite 3.3, for example, included a speech model, a vision model, a two billion parameter document processing model. Because eventually what's going to happen is you're going to solve a problem. To solve a problem or build an application, you're going to have AI models work together. Some cases they are orchestrated by an agent. Other cases you're stitching together. I have a document processing pipeline.

• 40:56Speaker 01

  I use a two billion parameter model on the runtime. I use an eight billion parameter model. If I have speech input, I use a speech model. I do need one model to do all of these things. I can run these models differently. If that speech model needs to only run in a certain environment, I only need GPUs to run the speech model in that environment. If my document processing needs, I'm a hybrid cloud company. I'm using Watson X on Azure to run my document processing. That model needs to run on Azure. My application could be running on the mainframe. Okay, that model needs to run on Spire chip on the IBM mainframe. So this unique philosophy of fit for purpose decomposition. We think is the right approach to build AI. So we've been on that path.

• 41:40Speaker 01

  Yes, it does make it therefore much easier for us to iterate, rapidly release. Look at Granite 3, we released Granite 3 in October, 3.1, end of the year. 3.2 in January, 3.3 in February. We're going to be releasing Granite 4 by 2Q. That pace is possible because you have made this choice.

• 41:59Craig S. Smith?

  Sure.

• 42:00Speaker 01

  And you can compare that to people who have made a different choice whose release cycles tend to be six, nine months a year. It's just the way, the way things are.

• 42:07Craig S. Smith?

  Yeah, yeah, I wanted to switch to more forward looking and not that this isn't forward looking, but continuous learning and the search for, I mean, right now, although you've managed with these small models to,

• 42:28Craig S. Smith?

  to train them at inference time, that doesn't adjust the weights in the underlying model. So there's no learning in the underlying model during inference.

• 42:43Speaker 01

  Correct, correct.

• 42:44Craig S. Smith?

  Or after.

• 42:45Speaker 01

  Inference scaling is on the top, correct. The model remains the same. The only change is the LoRaS. So you can drop in and take out LoRaS. So that does give you runtime adaptability.

• 42:53Craig S. Smith?

  Right, I see. Are you pursuing research? And I know that it's an active area of how you could

• 43:03Craig S. Smith?

  update the weights of the underlying model during inference time to, to, to create a continuous learning loop.

• 43:13Speaker 01

  Right, so I think maybe to, to pass that a little bit. So I think first is the, this notion of LoRaS and activated LoRaS already give us a way to, to change notionally the weights of this composite thing. Instead of calling it a model, I want to call this this composite thing, which consists of a base model and multiple adaptable LoRaS, right? So already I get more flexibility than just having the one base model. I can turn it on and off. I can add a LoRa, I can remove a LoRa out of the system. So that already gives me modularity. So that's number one, which means that I can deploy a system with five LoRaS. I'm collecting telemetry data of usage of the model.

• 43:57Speaker 01

  I can use that to go off and train another LoRa. Pluck it back into this model. So I don't necessarily have, it doesn't necessarily mean that every online learning I need to do has to go back and touch the base model. Now, there are certain environments in which maybe I need to do that. So for example, I'm trying to collect chain of thought data or backtracking logic in an entirely new domain. I may want to, I may need to seed some of that early on in the training of the base model, maybe in mid-training or something like that. So I may deploy a system instrumented to understand how humans are doing that, collect the data, curate it, generate synthetic data and go back to it. That I think is a, is a longer loop.

• 44:41Speaker 01

  But there are certain scenarios where I may be able to connect trajectory information like this. Just train a LoRa adapter that is trained on that and simply helps the model do that backtracking. Or I may be able to adapt a reward model that sits outside the base model that's trained on, oh, I see, this is how humans backtrack in this domain. Let me get a reward model that's going to now teach the main model how to backtrack. And those reward models, because they are separate, I can update them and refine them on their own. So I think the way we are going to do online learning, until we figure out maybe a methodological innovation, is essentially by making the runtime that is actually serving AI highly modular.

• 45:26Speaker 01

  So reward models, LoRa's, I see as the mechanism to do that. Because every time you are to fall back on updating the base model, you're right, that's a long loop. Are there, have there been ideas around in real time updating the watch of the base model? Yes, but I haven't seen a practical deployable answer. Right now, the base model is still a retraining activity.

• 45:50Craig S. Smith?

  Right.

• 45:50Speaker 01

  And their small is easier to update, that's about it. I haven't seen anything fundamentally moving the needle. But I do think modularity gives you a good way to address that.

• 46:01Craig S. Smith?

  Yeah, although what I'm talking about with continuous learning is that the, the model would

• 46:08Craig S. Smith?

  Would increase its knowledge base without getting larger, maybe would have to get larger. I don't know. And so, so once you, you're, you're, you're working with it on, on inference, it gets smarter. I mean, this is, you know.

• 46:27Speaker 01

  The original, yeah, the learn smarter. Yes, yes.

• 46:30Craig S. Smith?

  Intelligence, yeah.

• 46:31Speaker 01

  Correct. But I think the, I think, I think what we are discussing is the mechanism for that to happen, right? So the long loop of that is let the system run, collect the data, then you come back and retrain the base model. That's the long loop. It's not automatic. It's the longest loop. The second faster loop is I can update the, some of these components quicker. And so I learn faster. I notice that the model is not doing well for certain classes of queries. Quickly collect the data, build an adapter that addresses that, build a better reward model. That's the second way to do that. I think there is a third way to do, which is at a application level. You deploy a model, look at where it's not doing well, and then use that data to train a second model and route.

• 47:14Speaker 01

  So I think at systems level, you can make the system more and more adaptive.

• 47:19Craig S. Smith?

  I see.

• 47:19Speaker 01

  That's, so I think those techniques, I think are going to be very, very practical because they allow you to react. Now, I do think that what you are asking for, which is this vision that the model discovers what it is not good at on its own versus a human being looking at it and deciding to train. I think the big piece there is this whole area of uncertainty quantification, right? So that's why we are, we are working a lot on that because the first step to doing that is for the model to have a principled way to know what I'm good at, what I'm not good at. And if there is a region of space in this knowledge that I'm consistently having low uncertainty, then you can figure out what action to take.

• 48:02Speaker 01

  Today, all the actions are human driven. The human decides when to come back and retain a Laura. You're talking about a system that automatically recognizes it's, it's, it's false and goes there. I still think that is still in the realm of, you know, innovation and research right now.

• 48:16Craig S. Smith?

  But possible.

• 48:17Speaker 01

  But possible, but certainly possible. I view uncertainty quantification as a, as a significant element because that's the first step. You've got to know what you're not good at. If you continue to hallucinate your way and you don't know, then you don't know you have a problem. So uncertainty quantification, because remember that. If the model produces, if a system produces an output and the model does not get a judgment, it doesn't even know whether it did well or not.

• 48:41Craig S. Smith?

  Yeah.

• 48:42Speaker 01

  Now, today, some other external feedback signal says, I'm getting customer complaints. I'm not resolving these tickets, whatever they are. And a human looks at it and comes back and fixes it. So for the model to, to learn, the model has to automatically know what it is not good at. And that's why I think uncertainty of quantification is actually a really, really important piece of the puzzle for self-improvement.

• 49:03Craig S. Smith?

  Okay.

• 49:07Craig S. Smith?

  Yeah, I'm just, yeah, this was a question off of that. If inference time weight adjustment is possible,

• 49:21Craig S. Smith?

  would it happen at layer-specific updates? How fine-grained would that weight adjustment be? Or, or, or would it be, I mean, the problem with right now with catastrophic forgetting is, is it may update, overwrite previous knowledge.

• 49:43Speaker 01

  Right.

• 49:44Craig S. Smith?

  But would it be done at, at a very granular level, like different nodes or different layers or?

• 49:53Speaker 01

  I, I think, I think the reason I keep going back to LoRa is that's actually a principal way to do that because.

• 49:58Craig S. Smith?

  I see.

• 49:59Speaker 01

  Because if you, if you draw the box, not around the base model, but around the base model with the LoRa, you are adjusting weights.

• 50:07Craig S. Smith?

  Yeah.

• 50:07Speaker 01

  But you're adjusting weights by moving from a certain computation into activating a LoRa, which does something different. It's sort of like inserting something at that point. So that's really what you're doing. So at a system level, LoRa's do give you an opportunity to update the effective network that is giving you the answer. That's different from, and the reason I think that is, that is, that is useful is, as you said. Changing the weights of a base model is complicated because you took something that was created by a massive optimization, which is balancing many, many things. You're making some local changes whose implications you don't know. That's what leads to all sorts of challenges.

• 50:51Speaker 01

  The philosophy with LoRa is let's decompose it to specific pieces of functionality. I want to teach the model a piece of knowledge, like put a knowledge LoRa. I want the model to be good at hallucination. So when you do it that way, you are isolating these things. And so you have less worry about conflicting with what was taught to the base model. You have individually optimized those LoRa's to do what they're done. And you're putting this together almost like a workflow. I think that's a more viable way to do what you're thinking about online learning. Because now it allows you to reason about this in a meaningful way. It allows you to separate functionality.

• 51:33Speaker 01

  So you know that if performance comes down when a LoRa was enabled, then you can go debug that LoRa. If I did this update of the base model, I don't even know how, but let's say I did that. Debug-ability, traceability, all that becomes extremely hard. So I keep going back to, I think the way we are going to get to online changes is really by not treating the AI system as one big network, but as really a base network with lots of plugins. And we are probably early in the innovation associated with it. And so I think that's a more viable way in which we will get online learning. Otherwise, it becomes impossible to reason about it. These optimizations, because training is effectively a big optimization problem.

• 52:18Speaker 01

  Over all the data that you send, minimize this loss function. So it's very hard to reason about. LoRa's have the advantage that I optimize them for one thing. And I plug it in. I trained another LoRa, I optimize for something, I plug it in. It's much easier for me to reason about it, swap it in and out. So I feel like that, again, this is just me picking intuitively.

• 52:41Craig S. Smith?

  Yeah.

• 52:41Speaker 01

  Maybe there is a brainwave way to do this that I'm not thinking about, but I view this as a very pragmatic way to get the effect that you're thinking about.

• 52:49Craig S. Smith?

  It is, the Granite series, is that the main thrust on generative AI right now?

• 52:57Speaker 01

  So at IBM Research, it's bigger. So maybe I'll step back to talk about our overall focus, right? So one is certainly obviously the models themselves and everything we have talked about. A second big area for focus for us is on the data side.

• 53:11Craig S. Smith?

  Yep.

• 53:12Speaker 01

  From an enterprise context, unstructured data is becoming important now because of Gen AI. So a huge focus for us is making our Watson X dot data lake house really best of breed for unstructured data. Which means thinking about all aspects of data management, from ingestion to processing to ETL pipelines to security, access control, governance. So that's some big, big focus here with a lot of innovation from research. So you'll see again announcements around agentic RAG, announcements around unstructured data processing pipelines and document understanding, that's driven by research innovation. That's a big thrust, which is how do you make enterprise data management ready for unstructured data?

• 53:53Speaker 01

  There's a second side to the data side, which is, there's a lot of work in enterprises that data engineers or data stewards or, you know, people, and we call this broadly this term data ops, right?

• 54:04Craig S. Smith?

  Yeah.

• 54:05Speaker 01

  Running, this is even with structured data, forget unstructured data, today, even before Gen AI, there's a lot of time spent in creating pipelines of structured data, cleaning them, morphing them, transforming them, moving it from one place to the other.

• 54:21Speaker 01

  So data ops is a very expensive piece. So one of the areas we are working on is how does agentic AI help those personas who are data stewards and data engineers do their job better, faster, cheaper? So that's applying AI to data management. So on the data side, we sort of located as two particles, right? Preparing data for Gen AI and then using Gen AI to make data management more efficient and effective. So that's the second thrust. The third thrust for us is everything to do with agents and agent middleware. So we believe where the world is going, very much just like our philosophy around models, right? Yeah. We are about fit-for-purpose models that you can pull together. Same for agents.

• 55:05Speaker 01

  We think the way the world is going to be built is multiple agents, you have to orchestrate to get your work done. Because it's much easier to securely build an effective agent that does one thing as opposed to one humongous agent that does everything under the planet. Which means managing agents, the life cycle of agents, governing agents, the security of agents, orchestration of agents, that's going to become an important focus. So we have from our product portfolio, Watson X orchestrated as our flagship product for agentic middleware. And a lot of what we are doing from research is looking at problems like observability of agents. Because you have to look at agents in action, just as you observe IT systems. How do you do agent ops and observability?

• 55:49Speaker 01

  How do you securely make tools available to agents? How do you look at security and authorization of agents? This is a big issue. Because today, if you're going to have agents, so imagine Craig, you're an IBM employee. I give, put in front of you an agent that lets you do HR tasks. Now that HR agent is going to be equipped with tool to talk to HR systems, make operations changes. Those tools have to only have your authorization so that you are only allowed to do. So securely delegating authorizations and permissions is a big area that touches on the overall security fabric. So that's a big focus for us. Security, governance and operational lifecycle of agents and agent orchestration is an area.

• 56:34Speaker 01

  And we are building also agents in three specific domains. Those are where we have IBM businesses. So we have a big focus on agents for software development. But not general purpose software development. We have doubled down on areas where we have a big IBM business. We have a huge franchise on Cobol on the mainframe.

• 56:52Craig S. Smith?

  Sure.

• 56:52Speaker 01

  So we have Watson Code Assistant for Cobol, Watson Code Assistant for Java. So that building agents for software development in those domains. A second big focus for us is building agents for IT automation. We have had a rich portfolio now of IT automation, combination of organic and inorganic acquisitions, right? So across our AppTO, Insta, and our Turbonomic portfolio, we address the entire lifecycle of IT. From cost management to operations to security and compliance. So we're building agents to help those personas, whether it's a CISO, whether it's an SRE, whether it's the IT analyst managing costs, agents to help their work do better.

• 57:32Speaker 01

  And then the third area we are building agents is for our Maximo portfolio, which is used to manage physical assets. So Maximo today is used to manage physical assets like in factories and utility companies. And so there, the equivalent of the SREs, there is equivalent of maintenance personnel who have to keep these systems up and running, look at reliability, manage faults, building agents. So those three domains, software engineering, IT automation and physical asset management, we're building agents. And as we build those agents, we understand what agent middleware is needed, so that drives our innovation in middleware. So models, data, agents and three domain specific agents. That's basically the.

• 58:13Craig S. Smith?

  I should probably leave it there, but this has been fascinating for me.

• 58:17Speaker 01

  Thank you. It's been a great conversation. Yeah, it's been a great conversation.

• 58:20Craig S. Smith?

  This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns, and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tasty Trade is building. With Tasty Trade, you can trade stocks, options, futures, and crypto, all in one platform with low commissions, including zero commissions on stocks and crypto,

• 59:05Craig S. Smith?

  so you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection, and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like Active Trader Mode, One-Click Trading, and Smart Order Tracking. And if you're still learning, Tasty Trade offers dozens of free educational courses

• 59:49Craig S. Smith?

  plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker-dealer and member of FINRA, NFA, and SIPC.