Optimizing for efficiency with IBM’s Granite

We often judge AI models by leaderboard scores, but what if efficiency matters more? Kate Soule from IBM joins us to discuss how Granite AI is rethinking AI at the edge—breaking tasks into smaller, efficient components and co-designing models with hardware. She also shares why AI should prioritize efficiency frontiers over incremental benchmark gains and how seamless model routing can optimize performance. Featuring: Kate Soule – LinkedIn https://www.linkedin.com/in/katesoule/Chris Benson – Website https://chrisbenson.com/, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: IBM Granite https://www.ibm.com/graniteIBM Granite on Hugging Face https://huggingface.co/ibm-graniteIBM Expands Granite Model Family with New Multi-Modal and Reasoning AI Built for the Enterprise https://newsroom.ibm.com/2025-02-26-ibm-expands-granite-model-family-with-new-multi-modal-and-reasoning-ai-built-for-the-enterprise

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep, technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at fly.io. Launch your AI apps in five minutes or less. Learn how at fly.io.

• 0:44Chris Benson?

  Welcome to another episode of the Practical AI Podcast. This is Chris Benson. I am your co-host. Normally, Daniel Whitenack is joining me as the other co-host, but he's not able to today. I am a Principal AI Research Engineer at Lockheed Martin. Daniel is the CEO of Prediction Guard. And with us today, we have Kate Soul, who is Director of Technical Product Management at Granite for IBM. Welcome to the show, Kate.

• 1:13Speaker 02

  Hey, Chris. Thanks for having me.

• 1:15Chris Benson?

  So I wanted to, I know we're going to dive shortly into what Granite is, and some of our listeners are probably already familiar with it. Some may not be. But before we dive into that, wondering, we're talking about AI models, that's what Granite is, and the, and the, the world of LLMs, generative AI. Wondering if you could start off talking a little bit about your own background, how you arrived at this, and we'll get into a little bit about, you know, what IBM is doing and why it's interested in, and how it fits into the landscape here for those who are not already familiar with it.

• 1:51Speaker 02

  Perfect. Yeah, thanks, Chris. So I lead the Technical Product Management for Granite, which is IBM's large family of large language models that is produced by IBM Research. And so I actually joined IBM and IBM Research a number of years ago, before large language models were really became popular. You know, they had a bit of a Netscape moment, right, back in November 2022. So I've been working at the lab for a little while. I am A little bit of a odd duck, so to speak, in that I don't have a research background. I don't have a PhD. I come from a business background. I worked in consulting for a number of years, went to business school

• 2:31Speaker 02

  and joined IBM Research and the AI Lab here in order to get more involved in technology. You know, I've always kind of had one foot in the tech space. I was a data scientist for most of my tenure as a consultant and always, you know, thought that there was a lot of exciting things going on in AI. And so I joined the lab and basically got to work with a lot of generative AI researchers before large language models really kind of became big. And, you know, about two and a half years ago, a lot of the technology we're working on, all of a sudden, we started to find and see that there were tremendous business applications. You know, OpenAI really demonstrated what could happen if you took this type of technology

• 3:13Speaker 02

  and force fed it enough compute to make it powerful, it could do some really cool things. So from there, we worked as a team really to spin up a program and offering at IBM for our own family of large language models that we could offer our customers and the broader open source ecosystem.

• 3:31Chris Benson?

  Do you, I'm curious, with, one of the things that I've, you know, we've noticed over time is different organizations kind of are positioning the, these large language models within their product offerings in, in very unique ways. And, and we've, you know, we could go through some of your competitors and say they do this way, the, how do you guys see that in terms of, you know, how large language models fit into your product offering? Is there, is there a vision that IBM has for that?

• 3:59Speaker 02

  Yeah, I, you know, I think the fundamental premise of large language models are that they are a building block that you get to build on and reuse in many different ways, right? Where one model can drive a number of different use cases. So, you know, from IBM's perspective, that value proposition resonates really clearly. We see a lot of our customers, our own internal offerings where, you know, there's a lot of effort on data curation and collection and kind of creating and training bespoke models for a specific task. And now with large language models, we get to kind of use one model and with very little labeled data, all of a sudden, you know, the world's your oyster, there's a lot you can do.

• 4:41Speaker 02

  And so that's a bit of the reason why we have centralized the development of our language models within IBM Research, not a specific product. It's one offering that then feeds into many of our different products and downstream applications. And it allows us to kind of create this building block that we can then also offer customers to be able to build on top of as well. And open source ecosystem developers, you know, we think there's a lot of different applications for that one offering. And so, you know, that's a little bit kind of from the organizational side, why we're, why it's kind of exciting, right? That we get to do this all within Research. We don't have a P and L, so to speak. We're doing this to create ultimately a tool that can support any number of different

• 5:26Speaker 02

  use cases and downstream applications.

• 5:28Chris Benson?

  Very cool. And, and you mentioned open source. So I want to ask you, because that's always a, a big topic, uh, uh, among organizations is, uh, if I remember correctly, uh, Granite is an under an Apache two license. Is that, is that correct?

• 5:41Speaker 02

  That's correct.

• 5:42Chris Benson?

  Why, I'm just curious, because we've seen, uh, strong arguments on both sides. Why, why is Granite, uh, an, an open source license like that? Why, what was the decision, uh, from IBM to, to go that direction?

• 5:55Speaker 02

  Yeah, well, there was kind of two levels of decision making that we had to make when we talked about how to license Granite. One was open or closed. So are we going to release this model, release the weights, uh, out into the world so that anyone can use it regardless if they spend a dime with IBM? Uh, and ultimately IBM, you know, believes strongly in the power of open source ecosystems. A huge part of our business is built around Red Hat and being able to provide open source software to our customers with enterprise guarantees. And we felt that OpenAI was a far more responsible environment to develop and to incubate this technology as a whole.

• 6:34Chris Benson?

  And when you say OpenAI, you mean open source AI?

• 6:37Speaker 02

  Open source AI.

• 6:38Chris Benson?

  Just making sure.

• 6:39Speaker 02

  Very important clarification. Very important clarification. So that was why we released our models out into the open. And then the question was under what license? Because there are a lot of models, there are a lot of licenses and a bit of the like, Moment that everyone's seeing is you have a gamma license for a gamma model. You've got a llama license for llama model. Everyone's coming up with their own license and you know, it kind of, in some ways it makes sense. Models are a bit of a weird artifact. They're not code. You can't execute them like on their own. They're not software. They're not data per se, but they are kind of like a big bag of numbers at the end of the day. So like, you know, some of the traditional licenses, I think some people didn't see a clear fit.

• 7:24Speaker 02

  And so they came up with their own. There are also all these different kind of potential risks that you might want to solve for with a license with a large language model that are different than risks that you look at with software or data. But at the end of the day, IBM really wanted just to keep this simple, like a no nonsense license that we felt would be able to promote the broadest, broadest use from the ecosystem without any restrictions. So we went with Apache two, uh, cause that's probably the most widely used and just easy to understand licenses that's out there. Uh, and you know, I, I think it really speaks also to where we see models being important building blocks that are further customized.

• 8:03Speaker 02

  So we really believe the true value in generative AIs being able to take some of these smaller open source models and, and build on top of it and even start to customize it. And if you're doing all that work and you know, building on top of something, you want to make sure there are no restrictions on all that IP you've just created. Uh, and so that's ultimately why we went with Apache two point O.

• 8:24Chris Benson?

  Understood. And one last follow up on licensing and then I'll move on. Uh, and it's more, it's partially just a comment. IBM has a really strong legacy, uh, as, as someone in the AI world and, and decades of software development along with that. I know both Red Hat with the acquisition some years back, uh, being strong on open source and IBM, uh, both before and after has, was it, was it, I'm just curious, did that make it any easier, do you think to go with open source? Like, hey, we've done this so much that we're going to do that with this thing too, even though it's a little bit newer, you know, in context. Culturally, did it seem easier to get there than some companies that, that possibly really struggle with that? They don't have such a legacy in open source?

• 9:08Speaker 02

  I think it did make it easier. I think there are always going to be like any, any company going down this journey has to take a look at, wait, we're spending how much on what? And you're going to give it away for free? And come up, you know, with their own kind of equations on how this starts to make sense. And I think we've just experienced as a company that the software and offerings we create are so much stronger when we're creating them as part of an open source ecosystem than something that we just keep close to the best. So, you know, it was a much easier business case, so to speak, to make and to get the sign off that we needed. Ultimately, our leadership was very supportive in order to encourage this kind of open ecosystem.

• 9:49Chris Benson?

  Fantastic. Turning a little bit, as IBM was diving into this realm and starting, you know, and obviously, like, you have a history with Granite, that, you know, you guys are on 3.2 at this point. That means that you've been working on this for a period of time. But as you're diving into this very competitive ecosystem of building out these open source models that are big, they are expensive to make, and you're, you know, looking for an outsized impact in the world. How do you decide how to proceed with what kind of architecture you want? You know, how did you guys think about, like, you're looking at competitors? Some of them are closed source, like OpenAI is. Some of them, like Meta AI, you know, has

• 10:34Chris Benson?

  Llama and, you know, that series. As you're looking at what's out there, how do you make a choice about what is right for what you guys are about to go build? You know, because that's one heck of an investment to make. And I'm kind of curious, when you're looking at that landscape, how you make sense of that in terms of where to invest?

• 10:52Speaker 02

  Yeah, absolutely. So, you know, I think it's all about trying to make educated bets that kind of match your constraints that you're operating with and your broader strategy. So, you know, early on into our generative AI journey, when we were kind of getting the program up and running, you know, we wanted to take fewer risks. We wanted to learn how to do, you know, common architectures, common patterns before we started to get more, quote unquote, innovative in coming up with net new. Add additions on top. So early on the gen and you know, also you have to keep in mind this field has been just been like changing so quickly over the past couple of years. Yeah.

• 11:32Speaker 02

  So no one really knew what they were doing. Like if we look at how models were trained two years ago and the decisions that were made, the game was all about as many parameters as possible and having as little data as possible to keep your training costs down. And now we've totally switched. The, you know, general wisdom is as much data as possible in a few parameters as possible to keep your inference costs down once the models finally deployed. So the whole, whole field's been going through a learning curve, but I think early on, you know, our goal was really working on trying to replicate some of the architectures that were already out there, but innovate on the data. So really focus in on how do we create versions of these models that are being released

• 12:18Speaker 02

  that deliver the same type of functionality, but that were trained by IBM as a trusted partner, working very closely with all of our teams to have a very clear and ethical data curation and sourcing pipeline to train the models. So that was kind of the, the first major innovation aim that we had was actually not on the architecture side. Then as we started to get more confident, as the field started, I don't want to say mature because we're still very, again, very early innings. But you know, we started to coalesce to some shared understandings of how these models should be trained and what works or doesn't. You know, then our goal really has started to focus on from an architecture side, how can we be as efficient as possible?

• 13:03Speaker 02

  How can we train models that are going to be economical for our customers to run? And so that's where you've seen us focus a lot on smaller models for right now. And we're working on new architectures. So for example, mixture of experts. There's all sorts of things that we are really focusing in really with kind of the, the mantra of how do we make this as efficient as possible for people to, to further customize and to run on in their own environments.

• 13:32Chris Benson?

  So that was a fantastic start to as we dive into Granite itself, kind of laying it out. You know, your last comments, you talked about kind of the smaller, more economical models so that you're getting efficient inference on the customer side. You mentioned a phrase, which some people may know, some people may not, mixture of experts. Maybe talk as we dive into, you know, what Granite is and its versions going forward here. Maybe start with mixture of experts and what you mean by that.

• 14:04Speaker 02

  Absolutely. So if we think of how these models are being built, they're essentially billions of parameters that are representing small little numbers that basically are encoding information. And, you know, to like draw a really simple explanation, if you have a, you know, a linear regression, like you've got a scatter point and you're fitting a line, y equals mx plus b, like m is a parameter in that equation, right? So this, that, except on the scale of billions. With mixture of experts, what we're looking at is, do I really need all 1 billion parameters every single time I run inference? Can I use a subset? Can I have kind of little expert groupings of parameters within my large language model?

• 14:47Speaker 02

  So that at inference time, I'm being far more selective and smart about which parameters get called. Because if I'm not using all, you know, 8 billion or 120 billion parameters, I can run that inference far faster. So it's much more efficient. And so really, it's just getting a little bit more nuanced of instead of like, I think a lot of early days of generative AI is just throw more compute at it and hope the problem goes away. We're now trying to like, figure out how can we be far more efficient in how we build these models?

• 15:19Chris Benson?

  So I appreciate the explanation on mixture of experts. And that makes a lot of sense in terms of trying to use the model efficiently for an inference by reducing the number of parameters. I believe you're, right now, you guys have, is it 8 billion and 2 billion are the, are the model sizes in terms of the parameters? Or have I gotten that wrong?

• 15:41Speaker 02

  We got actually a couple of sizes.

• 15:43Chris Benson?

  Okay.

• 15:43Speaker 02

  You're right, we've got eight billion and two billion. But speaking of those mixture of expert models, we actually have a couple of tiny MOE models. MOE stands for mixture of experts. So we've got a MOE model with only a billion parameters and a MOE model with three billion parameters. But they respectively use far fewer parameters at inference time. So they run really, really quick, designed for more local applications like running on a CPU.

• 16:09Chris Benson?

  So and when you make the decision to have different size models in terms of the number of parameters and stuff, do you have different use cases in mind of how those models might be used? And is there one set of scenarios that you would put your eight billion and another one that would be that three billion that you mentioned?

• 16:30Speaker 02

  Yeah, absolutely. So if we think about it, when we're kind of designing the model sizes that we want to train, a huge question that we're trying to solve for is, you know, what are the environments these models going to be run on? And how do I, you know, maximize performance without forcing someone to have to like buy another GPU to host it? So, you know, there are models like the small MOE models that were actually designed much more for running on the edge locally or on the computer, like just a local laptop. We've got models that are designed to run on a single GPU, which is like our two billion and eight billion models. Those are standard architecture, not MOE.

• 17:09Speaker 02

  And we've got models on our roadmap that are looking at how can we kind of max out what a single GPU can run? And then how can we max out what a box of GPUs could run? So if you got eight GPUs stitched together. So, you know, we are definitely thinking about those different kind of tranches of compute availability that customers might have, and each of those tranches could relate to different use cases. Like obviously, if you're thinking about something that is local, you know, there's all sorts of IOT type of use cases that that could target. If you are looking at something that has to be run on, you know, a box of eight GPUs, you know, you're looking at something that you have to be okay with having a little bit more latency, you know, time it takes for the model to respond.

• 17:52Speaker 02

  But where the use cases also probably needs to be a little bit higher value because it costs more to run that big model. And so you're not going to run like a really simple, like, you know, help me summarize this email task hitting, you know, eight GPUs at once.

• 18:09Chris Benson?

  So as you talk about kind of the segmentation of these of the of the family of models and how you're doing that, I know one of the things you guys have a white paper, which we'll be linking in on the show notes for folks to go and take a look at either during or after they listen here. And you talk about some of the models being experimental chain of thought reasoning capabilities. And I was wondering if you could talk a little bit about what that means.

• 18:34Speaker 02

  Yeah, so really excited with the latest release of our Granite models. Just the end of February, we released Granite three dot two, which is an update to our two billion parameter model and our eight billion parameter model. And one of the kind of superpowers we give this model, the new release is we bring in an experimental feature for reasoning. And so what we mean by that is there's this new concept, relatively new concept in generative AI called inference time compute, where if you what that really equates to, just to put in plain language, if you think longer and harder about a prompt about a question, you can get a better response. I mean, this works for humans. This is how you and I think.

• 19:16Speaker 02

  But it's the same is true for large language models and thinking here, you know, is a bit of a risk of anthropomorphizing the term, but it's where where we've landed as a field. So I'll run with it for now, is really saying generate more tokens. So have the model think through what's called a chain of thought, you know, generate logical thought processes and sequences of how the model might approach answering before triggering the model to then respond. And so we've trained Granite eight B three point two in order to be able to do that chain of thought reasoning natively, take advantage of this new inference time compute area of innovation. And what we've done is we've made it selective.

• 19:59Speaker 02

  So if you don't need to think long and hard about, you know, what is two plus two? You turn it off and the model responds faster just with the answer. If you are giving it a more difficult question, you know, and pondering the meaning of life, you might turn thinking on and it's going to think through a little bit first before answering and answer with a much, in general, a longer, kind of more chain of thought style approach where it's explaining kind of step by step why it's responding the way it is.

• 20:25Chris Benson?

  Do you anticipate kind of, and I've seen this done from different organizations in different ways, do you anticipate that your inference time compute capability is going to be kind of there on all the models and you're turning it on and off? Or do you, do you anticipate that some of the models in your family are more specializing in that and that's always on versus others? Which way, you kind of mentioned the on and off, so it sounded like you might have it in all of the above.

• 20:51Speaker 02

  Yeah, you know, I, right now, it's marked as an experimental feature. I think we're still learning a lot about how this is useful and what it's going to be used for, and that might dictate what makes sense moving forward. But what we're seeing is kind of universally, it's, it's useful, one, to try and improve the quality of the answers, but two, as an explainability feature. Like if the model is going through and explaining more how it came up with a response, that helps a human better understand the, the response. So, you know, I think it is something we're heavily considering just baking into, to the models moving forward, which is a different approach, right? Than some models which are just focused on reasoning. I don't think we're going to see that very long. You know, I think more and more, we're going to see more selective reasoning.

• 21:34Speaker 02

  So like Claude 3.7 came out, they're actually doing a really nice job of this, where you can think longer or harder about something, or just think for a short amount of time. So I think we're going to see increasingly more and more folks move in that direction. But, you know, well, I, there's still, again, early innings, I'll say it again. So we're going to learn a lot over the next couple of months about where this is having the most impact, and I think that could have some structural implications of how we design our roadmap moving forward.

• 22:01Chris Benson?

  Gotcha. With, there's been a larger push in the industry toward, towards smaller models. So, you know, kind of going back over the, the recent history of, of LLMs and, you know, you saw initially, you know, the, just the number of parameters exploding and the models becoming huge. And obviously, you know, we talked a little bit about the fact that that's very expensive on inference.

• 22:25Speaker 02

  Yeah.

• 22:25Chris Benson?

  To, to run these things. And over the last, especially over the last, I don't know, year, year and a half, there's been a much stronger push, especially with open source models. We've seen a lot of them on hugging face, pushing to smaller. Do you anticipate as, as you're thinking about this capability of being able to reason that that's going to drive smaller model use toward models like what you guys are creating, where you're saying, okay, we have these large, you know, Claude has the, you know, big models and out there, you know, as an option, or, or a Llama model that's very large. Are you guys anticipating kind of pulling a lot more mind share towards some of the smaller ones? And do you anticipate

• 23:06Chris Benson?

  that you're going to continue to focus on, on these smaller, more efficient ones where people can actually get them deployed out there without, without breaking the bank of their organization? How, how, how does that fit in?

• 23:18Speaker 02

  Yeah. So, look, one thing to keep in mind is even without thinking about it, without trying, we're seeing small models are increasingly able to do what it took a big model to do yesterday. So you look at what a tiny, you know, two billion parameter, our Granite 2B model, for example, outperforms on numerous benchmarks, you know, a Llama 270B, which is a much larger, but older generation. I mean, it was state of the art when it was released. But it, the technology is just moving so quickly. So, you know, we do believe that by focusing on some of the smaller sizes, that ultimately we're going to get a lot of lift just natively, because that is where the technology is evolving. Like, we're continuing

• 24:03Speaker 02

  to find ways to pack more and more performance in fewer, fewer parameters and expand the scope of what you can accomplish with a small language model. I don't think that means we're going to. Ever get rid of big models? I just think if you look at where we're focusing, we're really looking at kind of where are the models? You know, if you think of the 80/20 rule, like 80% of the use cases can be handled by a model, you know, maybe eight billion parameters or less. That's, that's what we're targeting with Granite, and we're really trying to focus in. We think that there's definitely still always going to be innovation and opportunity and complex use cases that you need larger models to handle. And that's where we're really interested to see, okay, how do we expand the Granite family potentially,

• 24:48Speaker 02

  focusing on more efficient architectures, like mixture of experts, to target those larger models and more complex model sizes so that you still get a little bit more of a more practical implementation of a big model, recognizing that, again, it's not, you're always going to need, there's always going to be those outliers, those really big cases. We just don't think there's going to be as much business value, frankly, uh, behind those compared to really focusing and delivering value on the small to medium model space.

• 25:18Chris Benson?

  I, I think we've, that's one thing Daniel and I have talked quite a bit about is that we, we would agree with that. Uh, it's, I think the, the bulk of the use cases are, are for the smaller ones. While we're at it, you know, we've been talking about various aspects of Granite a bit, but could we take a moment and have you kind of go back through the Granite family and kind of talk about each component in the family? What it, you know, what it's called, what it does, and just kind of lay out the array of things that you have to offer?

• 25:47Speaker 02

  Absolutely. So the Granite model family has the language models that I, that went over. So between one billion to eight billion parameters in size. And again, we think those are like the, the workhorse models. You know, 80% of the tasks, we think you can probably get away with a model that's eight billion parameters or less. We also with three dot two recently released a vision model. So these models are for vision, understanding tasks. That's important. It's not vision or image generation, which is where a lot of the early like hype and excitement on generative AI came from is like Dolly and those. We're focused on models where you provide a image and a prompt. And then the output is text, the model response.

• 26:29Speaker 02

  So really useful for things like image and document understanding. Uh, we specifically uh prioritize a very large amount of document and chart Q and A type data in its training data set. Really focusing on performance on those types of tasks. So you can think of, you know, having a picture or an extract of a chart from a PDF and being able to answer questions about it. We think there's a lot of opportunity. Uh, so RAG is a very popular workflow in enterprise, right? Retrieval, augmented generation. Right now, all of the images and your PDFs and documents, they all get basically thrown away. But we really like are working on, can we use our vision model to actually include all of those charts, images, figures, diagrams?

• 27:13Speaker 02

  To help improve the model's ability to answer questions in a RAG workflow. So we think that's going to be huge. Uh, so lots of use cases on the, on the vision side. And then we also have a number of kind of companion models that are designed to work in parallel with a language model or a vision language, vision language model. So we've got our Granite Guardian family of models. And these are, are, we call them guardrails. They're meant to sit in right in parallel with the large language model that's running the main workflow. And they monitor all the inputs that are coming in to the model and all the outputs that are being, you know, uh, provided by the model. Looking for, you know, potential adversarial prompts, jailbreaking attacks, harmful inputs, harmful and biased outputs.

• 27:58Speaker 02

  They can detect hallucinations and model responses. So it's really meant to be a governance layer that can sit and work right alongside Granite. Could actually work alongside any model. So, you know, even if you've got a OpenAI model, for example, you've deployed. Uh, you can have Granite Guardian work right in parallel. And, you know, ultimately just be a tool for responsible AI. And you know, the last model I'll talk about is our embedding models, which again is meant to be, you know, assist a model in a broader generative AI workflow. So in a RAG workflow, you'll often need to take large amounts of documents or text and convert them into what are called embeddings, that you can search over in order to retrieve the most relevant info and give it to the model.

• 28:40Speaker 02

  So our Granite embedding models are used for that embedding step. So these are meant to do that conversion and can support in a number of different, similar kind of search and retrieval style workflows, working directly with the Granite large language model.

• 28:54Chris Benson?

  Gotcha. I know there was, there was some comment in the white paper also about time series.

• 29:00Speaker 02

  Yes.

• 29:01Chris Benson?

  Can you talk a little bit to that for a second?

• 29:02Speaker 02

  Absolutely. So I mentioned Granite is multimodal and that it supports vision. We also have time series as a modality, and I'm really glad you brought these up because these models are really exciting. So we talked about our focus on efficiency. These models are like one to two million parameters in size. That is teeny tiny in today's generative AI context, even compared to other forecasting models. These are really small generative AI-based time series forecasting models, but they are right now delivering top of the top marks when it comes to performance. So we just, as part of this release, submitted our time series models to Salesforce as a time series leaderboard called GIFT.

• 29:45Speaker 02

  They're the number one leaderboard on GIFT right now, number one model on GIFT's leaderboard right now. And we're really excited. They've got over 10 million downloads on Hugging Face. They're really taking off in the community. So it's a really excellent offering in the time series modality for, for the Granite family.

• 30:03Chris Benson?

  Okay. Well, thank you for going through kind of the, the layout of the family of, of models that you guys have. I actually want to go back and ask a quick question that you, you talked a bit about Guardian kind of providing guardrails and stuff. And, and that's something that if you take a moment to dive into, I think we often tend to focus kind of on, you know, the model and it's going to do X, you know, whatever. I love the notion of integrating these guardrails that Guardian represents into a larger architecture, you know, to address kind of the quality issues surrounding. The inputs and the outputs on that. How did you guys arrive at that? I'm just, you know, and how did you, you know, it's, it's pretty cool.

• 30:48Chris Benson?

  I love the idea that not only is it there for your own models, obviously, but that, you know, that you could have an end user go and apply it to something else that they're doing, maybe from a competitor or whatever. How did you decide to do that? And, you know, that's, I think that's a fairly unique thing that we don't tend to hear as much from other organizations.

• 31:08Speaker 02

  Yeah, you know, so Chris, the, one of the values again of being in the open source ecosystem is we get to like build on top of other people's great ideas. So we actually weren't the first ones to come up with it. Uh, there, there's a few other guardrail type models out there, but you know, IBM has quite a large, especially IBM Research, presence in security space. And we, there are challenges in security that are very similar to the large language models and generative AI that, you know, it's not totally new. And what I think we've learned as a company and as a field is that you always need layers of security when it comes to creating a robust system against, you know, potential adversarial

• 31:49Speaker 02

  attacks and dealing with even the model's own innate safety alignment itself. So, you know, when we saw some of the work going out in the open source ecosystem on guardrails, you know, I think it was kind of a no brainer from a perspective of this is another great way to add an additional layer on that generative AI stack of security and safety to better improve model robustness and figure out, you know, IBM's hyper focused on what is the practical way to implement generative AI. So what else is needed beyond efficiency? We need trust, we need safety. Let's create tools in that space. So it, it kind of, uh, you know, number of different reasons, all, all made it very clear, clear and easy when to go and pursue. And we are actually able to build on top of Granite.

• 32:33Speaker 02

  So Granite Guardian is a fine tuned version of Granite that's laser focused on these tasks of detecting and monitoring inputs going into the model and outputs going out. And the team has done a really excellent job, uh, first, starting at, you know, basic harm and bias detectors, which I think is pretty prevalent in other guardrail models that are out there. But now we've really started to kind of make it our own and innovate. So some of the new features that were released in the three dot two Granite Guardian models include hallucination detection, very models, few models do that today, specifically hallucination detection with function calling. So if you think of an agent, you know, whenever an LLM agent is trying to access or submit external information, it'll make what's called a tool call.

• 33:18Speaker 02

  And so when it's making that tool call, it's providing information based off of the conversation history, saying, you know, I need to look up, you know, Kate Soul's information in the HR database. This is her first name. She lives in Cambridge, Mass. XYZ. And we want to make sure the agent isn't hallucinating when it's filling in those pieces of information it needs to use to retrieve. Otherwise, you know, if she made up the wrong name or said Cambridge, UK instead of Cambridge, Mass., the tool will provide the incorrect response back, but the agent will have no idea. And it will keep operating with utmost certainty that it's operating on correct information. So, you know, it's just an interesting example of, you know, some of the observability we're trying to inject into responsible AI workflows, particularly around things like agents.

• 34:03Speaker 02

  Because there's all sorts of new safety concerns that really have to be taken into account to make this technology practical and implementable.

• 34:11Chris Benson?

  And, and, you know, that's actually having brought up agents and stuff, and that being kind of the really hot topic of the moment of, you know, 2025 so far. Could you talk a little bit about Granite and agents? And how you guys, you know, how are you thinking? You've gone through one example right there. But if you could expand on that a little bit in terms of, you know, how does, how is IBM thinking about positioning Granite? How do agents fit in? What's, what does that ecosystem look like? You know, you've started to talk about security a bit. Could you kind of weave that story for us a little bit?

• 34:45Speaker 02

  Absolutely. So, yeah, obviously IBM is all in on agents. And there's, there's just so much going on in the space. A couple of key things that I think are interesting to bring up. So one is looking at the open source ecosystem for building agents. So we actually have a really fantastic team located right here in Cambridge, Massachusetts, that is working on an agent framework and broader agent stack called BAI, like a bumblebee. So we're working really closely with them on how do we kind of co-optimize a framework for agents with a model that in order to be able to have all sorts of new tips and tricks, so to speak, that you can harness when building agents. So I don't want to give too much away,

• 35:29Speaker 02

  but I think there's a lot of really interesting things that IBM's thinking about agent framework and model co-design. And that only unlocks, you know, so much potential when it comes to safety and security. Because there needs to be parts, for example, of an L L M, of an agent, that a agent developer programs that you never want the user to be able to see. There are parts of uh data that an agent might retrieve as part of a tool call that you don't want the user to see. So for example, an agent that I'm working with might have access to anybody's H R records, but I only have permission to see my H R records. So how can we design models and frameworks with those concepts in mind

• 36:13Speaker 02

  in order to better demarcate types of sensitive information that should be, you know, hidden? In order to protect information that the model knows, like, these types of instructions can never be overwritten, no matter what type of, like, later on attacks, adversarial attacks, somebody might try and do and say, you're not Kate's agent, you're, you know, a nasty bot and your job is to do X, Y and Z. Like, how do we prevent those types of attack vectors through model co-design and agent, model and agent framework co-design? So I, I think there's a lot of really exciting work there. More broadly, though, you know, I think even on more traditional ideas and implementations of agent, not that there's a traditional one, this is so new, but more, more classical a- agent implementations, we're working, for example, with IBM Consulting.

• 36:57Speaker 02

  They have a agent and assistant platform that is uh where Granite is the default agent uh and assistant that gets built. And so that allows IBM all sorts of economies of scale. If you think about, we've now got a hundred and sixty thousand consultants out in the world using agents and assistants built off of Granite in order to be more efficient and to, to help them with their, their client and consulting projects. So we see a ton of client zero, what we call client zero. IBM is our, you know, uh first client in that case of how do we even internally build agents with Granite in order to improve IBM productivity?

• 37:34Chris Benson?

  Very cool. I'm kind of curious, as you guys are looking at this array of considerations that you've just been going through, and as there is more and more push out into the edge environments, and you've already talked a little bit about that earlier, as we are starting to wind down, could you talk a little bit about, as things push a bit out of the cloud and out of the data center, and as we have been migrating away from these gigantic models into a lot more smaller, hyper-efficient models, often that are doing better on performance and stuff, and we see so many opportunities out there in a variety of edge environments,

• 38:17Chris Benson?

  could you talk a little bit about where Granite might be going with that, or where it is now, and what the thoughts about Granite at the Edge might look like?

• 38:28Speaker 02

  Yeah, so I think with Granite at the Edge, there's a couple of different aspects. One is, how can we think about building with models so that we can optimize for smaller models in size? So when I say building, I mean building prompts, building applications, so that we're not designing prompts how they're written today, which I like to call it the YOLO method, where I'm going to give 10 pages of instructions all at once and say, "Go and do this," and hope to God the model follows all those instructions and does everything beautifully. Small models, no matter how much this technology advances, probably aren't going to get perfect scores on that type of approach. So how can we think

• 39:13Speaker 02

  about broader programming frameworks for dividing things up into much smaller pieces that a small model can operate on? And then how do we leverage model and hardware co-design to run those small pieces really fast? Uh, so you know, I think there's a lot of opportunity, you know, across the stack of how people are building with models, the models themselves and the hardware that the model's running on, that's going to allow us to push things much further to the edge than we've really experienced so far. It's going to require a bit of a mind shift again, like right now, I think we're all really happy that we could be a bit lazy when we write our prompts and just like, uh, you know, write kind of word vomit prompts down, but

• 39:55Chris Benson?

  Right.

• 39:55Speaker 02

  I think if we can get a little bit more like kind of software engineering mindset in terms of how you program and build, uh, it's going to allow us to break things into much smaller components and push those components even farther to the edge.

• 40:08Chris Benson?

  That makes sense. That makes a lot of sense. Um, I guess eh, eh, kind of final question for you as we talk about this, kind of eh, any other thought, you talked a little bit about kind of where you think things are going right there. Anything that you have to add to that in terms of kind of industry or specific to Granite, where you think things are going, what the future looks like when you are kind of winding up for the day and you're at that moment where you're kind of just, your mind wanders a little bit. Any, anything that appeals to you that kind of goes through your head?

• 40:39Speaker 02

  So I, I think the thing I've been most obsessed about lately is, you know, we need to get to the point as a field where models are measured by like how efficient their efficient frontier is, not by like, you know, did they get to point oh one higher on a, a metric or benchmark. So I think we're starting to see this with like the reasoning with Granite, you can turn it on and off with the reasoning with Claude, you can pay more, you know, have harder thoughts, you know, longer thoughts or shorter thoughts. But you know, I really want to see us get to the point, and I think we've got the, like, the, the table is set for this. Uh, we've got the pieces in place to really start to focus in on how can I make my model as efficient as possible, but as flexible as possible. So I can choose

• 41:22Speaker 02

  anywhere that I want to be on that performance cost curve. So if my mo- if my task isn't, you know, very difficult, I don't want to spend a lot of money on it. I'm going to route this in such a way with very little thinking to a small model. And I'm going to be able to achieve, you know, my acceptable performance. And if my task is really high value, you know, I'm going to pay more. And I don't need to like, think about this, it's just going to happen either from the model architecture, from being able to reason or not reason, from routing that might be happening behind an API endpoint to send my request to a more powerful model or to a less powerful but cheaper model. I think all of that needs to be, you know, we need to get to the point where no one's having to think about this or solve for it and design it.

• 42:07Speaker 02

  And I really want to see, I want to see these curves. And I want to be able to see us push those curves as far to the left as possible, making things more and more efficient. Versus like, here's a, here's a number on the leaderboard, like I spent another, you know, X godzillion dollars on compute in order to move that number up by point oh two. And you know, that's science, like I'm ready to move beyond that.

• 42:28Chris Benson?

  Fantastic. A great conversation. Thank you so much, Kate Soul, for joining us on the Pride Clay I podcast today. Really appreciate it. A lot of insight there. So thanks for coming on. Hope we can get you back on sometime.

• 42:41Speaker 02

  Thanks so much, Chris. Really appreciate you having me on the show.

• 42:51Speaker 00

  All right. That is our show for this week. If you haven't checked out our change log newsletter, head to changelog.com/news. There you'll find 29 reasons, yes, 29 reasons why you should subscribe. I'll tell you reason number 17, you might actually start looking forward to Mondays.

• 43:11Speaker 03

  Sounds like somebody's got a case of the Mondays.

• 43:14Speaker 00

  28 more reasons are waiting for you at changelog.com/news. Thanks again to our partners at fly.io, to Breakmaster Cylinder for the Beats, and to you for listening. That is all for now, but we'll talk to you again next time.