We've all done RAG, now what?

Longtime friend of the show Rajiv Shah returns to unpack lessons from a year of building retrieval-augmented generation (RAG) pipelines and reasoning models integrations. We dive into why so many AI pilots stumble, why evaluation and error analysis remain essential data science skills, and why not every enterprise challenge calls for a large language model. Featuring: Rajiv Shah – LinkedIn https://www.linkedin.com/in/rajistics/ Daniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaUpcoming Events: Join us at the Midwest AI Summit https://midwestaisummit.com/ on November 13 in Indianapolis to hear world-class speakers share how they’ve scaled AI solutions. Don’t miss the AI Engineering Lounge, where you can sit down with experts for hands-on guidance. Reserve your spot today! Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:03Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:48Speaker 01

  Welcome to another episode of the Practical AI Podcast. I'm Daniel Whitenack. I am CEO at Prediction Guard and not joined by Chris today, but very happy to be joined by a longtime friend of the podcast and friend of all things AI and data science online, Rajiv Shah from Contextual AI, where he's the chief evangelist and also, of course, TikToker extraordinaire and all things. So, welcome back to the show, Rajiv. It's great to have you.

• 1:23Speaker 02

  It's great to be back. Excited to talk.

• 1:25Speaker 01

  Yeah, and we're close neighbors around the Midwest as well and looking forward to seeing you at the Midwest AI Summit, which if listeners don't know, both Chris and I will be at the Midwest AI Summit, which is happening November 13 in Indianapolis. So, I think, as you put it on LinkedIn, Rajiv, if you live anywhere near corn and you like AI, then this is the place to be. And actually, Rajiv will be there giving a great talk about kind of this idea of, I think you're going to be talking about this stat, the 95% of pilots that fail, why that might be. Yeah, so great to have a fellow

• 2:08Speaker 01

  Midwest AI Silicon Prairie friend on the show as well.

• 2:13Speaker 02

  No, it's great to have these in-person events here in the Midwest. Usually you have to kind of fly out to some large city to get that. It's nice to be able to kind of get that right in Indy.

• 2:21Speaker 01

  Yeah, yeah. So if you're interested in that, go to Midwestaisummit.com and make sure and and register for that. Hopefully we'll see you there there in person. It's going to be it's going to be a fun time, but it has been a while since we've had you on the on the show, Rajiv, and I'm sure there's a lot to catch up on. It almost seems like last time we talked, there was kind of, we were all getting into rag and thinking about, you know, agents and reasoning models. We're trying things. People were kind of gradually developing maybe what they thought would be a direction for kind of best practices or ways to approach problems.

• 3:02Speaker 01

  It would be interesting to hear kind of from from then to now, how you've seen that world advance and kind of these archetypal use cases that that people are are addressing with retrieval and and related related frameworks.

• 3:18Speaker 02

  Yeah, no, it's been amazing, right? About two and a half years ago, we were really giddy about the possibility of AI. But there was a lot of different things that we can do. And kind of since then, we've seen AI develop in certain ways where, for example, in code completion, code development tools, it's accelerated dramatically, even over the last year like that. And we're all kind of using those tools on a regular basis. And then in other areas, right, like chatbots, for example, we've seen some progress, but we've also seen that there's some things that are difficult to do with chatbots as well. But I think, you know, it's been incredible to see just the continued development of AI in terms of the capabilities, the smartness, the ability to kind of use tools to find

• 4:00Speaker 02

  search information that there's still so much for all of us to do in our lives, the kind of those of us that work with AI.

• 4:07Speaker 01

  Yeah, yeah. And I guess, like, certainly, there's a lot of people still. I guess when I interact with customers or hear from other people in the industry, like, still, a lot of people are wanting, as their first things, to kind of, whatever is build a rag chatbot, which maybe for those listeners out there that are maybe not as familiar with this, they've maybe heard, you know, rag this and raggedy rag that and all of that. Maybe just remind, remind listeners, like, what rag is and like how retrieval, you know, may fit into some of this AI stuff.

• 4:46Speaker 02

  Yeah, and I'll start simply. Even a couple of years ago, I was kind of working with the earliest large language models and people would be like, I like ChatGPT, but it doesn't know anything about me or my company. Like, how can I train it to know all that knowledge? And what we figured out is that it's not about really training that model about the knowledge. It's instead finding that knowledge, searching that knowledge, passing that on to the model, and then be able to use that. And that's where kind of this idea of retrieval, finding stuff, and then augmenting the generation, augmenting the written response for that came about. And I think rag today is kind of one of the most important, or it's one of the most widely used use cases, I'd say, in the generative AI space.

• 5:31Speaker 01

  Yeah.

• 5:31Speaker 02

  Like, every company is probably running some type of rag at this point for searching its internal data, internal knowledge, right? Helping folks figure out through HR documentation, using it for customer support. There's a lot of use cases where we have huge amounts of information. We want to be able to find things from that, but we also want to use the smarts of AI for that. Like, we don't want to Google search results of 20 things, right? Like, we want a nice, like, AI summary of it. Or we want to extract all the information. I just want the names and dates out of all this. I don't need to read every one. And so that's where the AI really comes into it and why RAG is so powerful and we see it so widely used.

• 6:11Speaker 01

  Yeah, and I love what you were saying, just sort of contrasting this with training or even fine tuning. I find this, of course, to be a very widespread misconception about how these tools work. Even even ChatGPT, the application, like it does seem like it kind of quote trains on your information that you put in, your previous chat history, et cetera. And then like, it's almost like you have a model of your own, right? Which seems like, it almost seems like OpenAI has a separate model for every person on the planet, which is not feasible from the data science training perspective. So could you highlight some of those things?

• 6:54Speaker 01

  And just, just highlight, I just, I know you educate a lot as well, so I'm sure you've seen this as well. There's kind of this jargon of training thrown around a lot, which is confusing maybe in how it's, how it's used, so, yeah.

• 7:08Speaker 02

  Yeah, and I think partly is, is a lot of these companies want to make you think that they have like one thing that does everything in the world. But if you actually take a look at something like ChatGPT, it's a system, it actually composes of multiple parts. So if you were trying to build your own ChatGPT, these are the, some of the things that you would have to think about. So one is, I need to be able to retrieve from lots of different sources. I have all my knowledge inside my companies inside of Confluence. I need to be able to access and retrieve that. That could be one thing. So one thing is using RAG, being able to access tools. Another thing is memory. Like, I wanted to remember that last conversation I had, whether it was two things I said ago,

• 7:51Speaker 02

  or like, one of the nice things about ChatGPT is you can come back a conversation or a day later, and it remembers a lot about you. It could maybe a little bit scary sometimes how much it remembers about you and how well it can profile you. And so this is kind of what we see inside of AI engineering as context engineering, where there's a number of different parts of managing interactions with these models, whether it's RAG, whether it's remembering the memory, when it's knowing how to summarize conversations, to do things like multi-turns, we can have those repeated conversations, but keep track of what was said earlier in our conversations as well.

• 8:31Speaker 01

  Yeah, so I love that idea of context engineering. How is that idea of context engineering kind of different from like what we would maybe from the data science, traditional data science side, consider like model training, I guess? So there's almost like a pipelining mindset versus an actual like GPU model training piece?

• 8:56Speaker 02

  Yeah, you know, I think what's remarkable nowadays is the amount of information that's stored inside of these large language models. I do a thing when I kind of talk about large language models where imagine if you were sitting outside and kind of reading a book and you could read all day, all night. If you read for 10 years, I think that's on the order of something like about a billion tokens, and these models are trained on the order of like 15 trillion tokens. So it's just an inconceivable amount of information that these models hold inside there. And I think one of the biggest improvements we've seen in these models is not only kind of stuffing all that information in there, is effectively using all of that information

• 9:39Speaker 02

  in there, that these models really today are untapped in terms of everything they can do. And it's one of these things on the technical side is why you see that the capabilities of the models have continued to grow without them necessarily having to grow immensely in size, simply because we're better tapping into all the abilities that they have inside them.

• 10:00Speaker 01

  And I know, I know you're up for this because, uh, because we know each other, but I'll, I'll present a scenario to you that I, as sometimes comes up for us and, you know, whether I'm teaching a workshop or something, and you can help us understand how this retrieval type of thing fits in. So, so I often, maybe I'm talking to folks in healthcare and what's interesting is there's all of this, you know, huge amount of data that these models have been trained on. And there is good, let's say, medical information in that data, right? Or there's like care guidelines for nurses or medics or whatever it is. Let's take that

• 10:40Speaker 01

  as an example. The interesting thing is there's even like, if you think about factuality in a situation, it is kind of relative in the sense that if I'm a medic that works for this provider, let's say their care guidelines are different from another provider. Or if like I'm treating, you know, maybe children, those guidelines are probably different than if I'm treating adults or, or senior adults or, or something like that. So there can be these facts out there that are actually conflict with one another. And it's really, you know, you talked about the context. It's really about the context that you're working in, what it should pull. So these models, assuming that all that information is public, right? Like you said, and it's been scraped off

• 11:25Speaker 01

  the internet. All of that's kind of, you could imagine that maybe somehow that is embedded somewhere in the, in the model, the base model. So whether we're talking about, you know, Lama or GPT or whatever. How can, let, let's say I'm a healthcare company, but I'm interested now in my context. Now, how do I make that connection? So still use maybe, I'm not going to retrain that model, but how do I use the model then in this context of kind of conflicting facts?

• 11:56Speaker 02

  Yeah, absolutely. Like as much as the models know and been trained so widely, they don't know everything. And so sometimes that's, you have your own information, you want to pass that into the model. And so that's where that retrieval augmented generation comes in. We want to grab that information and then we want to pass it into the model. And this is where for folks who are in the space for a while, prompt engineering comes into play, where we think about the inputs that go into the model, where we take the facts that we know about our own given situation. Along with maybe some instructions for what we want along with it. And that all of that is manipulating the context, right? The larger world that this model is sitting within there. And by manipulating and giving it information, instructions and other facts,

• 12:41Speaker 02

  we can get outputs that more that better match kind of what we're looking for like that.

• 12:46Speaker 01

  Yeah, yeah, that's, that's great. And I guess, you know, there's another term that's been thrown around the last couple of years. One would be this kind of retrieval piece, which you've talked about. And this idea that in the context of a certain question or something, I'm able to connect to one of my internal data sources, get that right information. There's this other thing that would be related to reasoning. Now, some people might kind of just consider that all kind of generative AI models reason in one way or another, but it is kind of now almost like a term of art. It means kind of a certain thing when we're, when we're talking about these models. Could you, could you help highlight that also to put that in context?

• 13:29Speaker 01

  And then maybe we'll circle around and combine some of these things.

• 13:32Speaker 02

  Yeah, you know, a lot of times when we talk about models, we want to make it convenient for, to help people explain things. And so we kind of anthropomorphize them, right? We give them human-like qualities when they're not actually human-like qualities. And so, for example, reasoning is something we say for these models. Now, they're not reasoning like a human would through a problem. But what we typically mean kind of when we think about reasoning with these models is they're doing lots of extra steps. And they're doing these steps in a logical way to better solve a problem. So if you take kind of a math word problem, a great way to see this is if you take a word problem, like the train is moving east, you know, at 50 miles an hour, and another train is moving west another 40 miles an hour,

• 14:16Speaker 02

  and you have to figure out like what point they're going to cross, right? There's no kind of quick answer to that. You need to kind of calculate the first train, calculate the second train, and then you can figure out the solution. And what we've done, and that's what these reasoning models have done, is we've trained them that don't come up with the answer right away. Think through it, think about the first thing, think about the second thing, connect all the dots, and then put an answer in. And the way we've done this is we've literally given the model, well, we've trained it in multiple ways, but in some ways, we've literally give the model an example of, hey, this is how I solved this word problem. I did this, all these steps here. I want you to learn how to go through these problems step by step. And this is where we differ from two and a half years ago, when the first models didn't know how to do that.

• 15:02Speaker 02

  Those first models, all we had trained it up was like, hey, tell me if a movie is a good movie or a bad movie, or tell me if this sentiment is happy or sad. But since then, we've had time to develop more training data, we've given them more complex training, and the cool thing is, the models have picked up. They've been able to learn this capability for doing this. And so now we're able to do this much more complex, kind of, I'm doing the air quotes of reasoning, to solve these problems.

• 15:28Speaker 01

  Yeah, I guess one thing that people, and this is part of why I really enjoy watching your videos online, is you often break down a lot of this jargon and kind of help. It's almost like people feel the shock of all of this terminology and a new model coming out every, every, you know, 30 minutes or whatever it is, and, and just not really knowing how to deal with that. And there's kind of all of these things that have happened. So there's reasoning models, there's small language models, there's tool calling, there's retrieval, there's like all of these different kind of mix of things. Based on your experience, kind of, and, you know, both what you've implemented personally,

• 16:11Speaker 01

  also what you've seen and interacted with folks on, if people could kind of bring in some focus and maybe imagine a company's kind of getting into, you know, AI, however they view AI transformation within their organization, and they're thinking about some of the use cases that are kind of the initial ones on their roadmap. At the time being, like, what would you encourage them to kind of like pick the signal out of the noise? So what would you encourage them to maybe focus on to see, see some of that time to value upfront? Not that they don't want to explore. Some of the other topics that are happening or, you know, read about them or whatever that is.

• 16:56Speaker 01

  But how would you at least recommend kind of to get the best time to value? Or, or maybe just the things that are producing the most value? And you could maybe flip that as well and say, like, what are some of those things that are cool things? But maybe let's wait and see what happens and maybe, maybe just sort of don't get distracted at the moment.

• 17:19Speaker 02

  Yeah, I mean, you're taking me away from all the full, all the fun, cool technologies that are the latest, like automation, things that I can kind of fire up like that. I think, you know, when you're thinking about this from the perspective of kind of a company, if you're kind of a manager in these situations, you have to figure out what use cases. You want to put on a very different hat than just thinking about the technology itself. And I can tell you this, like, I was burned from this, from personal experience when I was just starting out in data science. I was entrapped by the latest technology. So I remember, right, we were talking about code development. Like 10 years ago, I was working at State Farm and I think Andridge Karpathy has written his paper

• 18:00Speaker 02

  about the, I think, like the unreasonable effectiveness of LSTMs. It was something like that. But part of that paper had the idea of you could translate code. You could complete code from one language to another. And I was like, hey, come on, guys, like, we've got a lot of this COBOL code sitting around. Like, give me some, give me a few GPUs and some data and I can, I'll solve this, right? Like, totally naive, like, you know, they didn't fund that project or anything. But I think, yes, it's, it can be very easy to be kind of seduced by the technologies, kind of what a shiny demo is versus when you're in an organization, you really have to think about kind of the problems that you have.

• 18:41Speaker 02

  Part of it will be, you know, how complicated is this from a technical point of view to get it up and running? That's one factor, but that's not the only factor. We also need to consider, like, what's the value to this organization? I talk to lots of enterprises on a regular basis and I see often what I call science experiments where teams like the latest technology to go out and kind of run this stuff, but there's no way for them to actually get that implemented inside the company in a useful way. And they're literally just kind of interesting experiments that people are running like that.

• 19:14Speaker 01

  Does that get partially to like the 95% of AI pilot failing type of report from MIT?

• 19:23Speaker 02

  Absolutely. Now, the 95% of course is like a little bit of a hype number that they like to put out in this. And I, right, for those of us that have been in the space for a while, we remember the fact of 80% of data science projects fail, I think was something that we had. And to some extent, that's okay. You can't expect every initiative, every experiment, everything that you start to succeed. You want things to fail because partly is if something works, that means you have to maintain it. You have to monitor, you have to put up a lot of guards around it. There's a cost for something that actually succeeds. But when we talk about AI, it's very easy to build a cool demo, but it's not only the value to the company, you have to figure out how to integrate this into

• 20:08Speaker 02

  people's everyday work life. And so you can build a very shiny widget that sits and can do something awesome. But if that's not inside somebody's regular workflow of how they work, the tools that they work in, if it's not, if they're not properly trained on how to use it, if their leadership isn't supporting you to use it, like there's lots of factors like that that go into why people might not actually adopt and use a technology. And it's really nothing about AI. It's really about organizational change and introducing technology into companies.

• 20:39Speaker 01

  Yeah, I know from just a founder perspective, it can be, you know, just from a different side of this, it can be frustrating when you see like, oh, there's this, you know, company over here that the technology side is fairly simple. Like, oh, it's just a simple model that does a simple thing, or a browser extension that does this. You're like, wow, you know, like I could have vibe coded that in a, in a weekend. Like, how are they, you know, scaling to the moon? And we have all this cool technology. And, and I think a part of it is that that side of it that you talked about, like,

• 21:19Speaker 01

  part of the hard problem is, is cracking what actually does provide value to your organization? What can be adopted? How you communicate that? How you tell that story? And how you deliver on your promises? How you, how you provide customer support? Like, a lot of that is not really related to the, to the technology and that component. So maybe, I don't know if this would be accurate to say, but maybe the first step people could take is just getting something off the ground that's fairly simple and interacts with an AI model. Maybe it's just to do a very simple task, but really pushing that through, like you say, to be embedded in a product or be embedded in a process.

• 22:05Speaker 01

  That may be the best way for, for people to kind of start that journey is to, yeah, really start from that simple side and deal with some of, in all honesty, the harder problems around the periphery of the technology.

• 22:19Speaker 02

  And, you know, I think a big part of that, like what you're saying, is just to get closer to those end users, the stakeholders. Because I think once you often cut through that, sometimes you figure out that really they don't necessarily need a fancy kind of GPT-5 model to solve their problems. Maybe you can solve it with almost a simple if then rule that you can just implement in someplace. And so this is where kind of looking at the data, spending time talking to those end users, like that often gives you a much better result, right? It's going to give you the biggest bang for your buck than going out and reading some archive paper.

• 22:56Speaker 01

  Yeah, yeah, that's, that's true. And I guess now you, you should just have the AI model read the paper for you and give, give you some, some summary points.

• 23:07Speaker 02

  I'm a big fan of that. On my walks, I often sit and I'll talk to chat GPT and we'll talk through papers and one of the main technical points and stuff. So.

• 23:15Speaker 01

  Yeah, I'm glad I'm not the only one. So thanks. Thanks for thanks for validating that. I guess getting back like more into the development retrieval kind of reasoning stuff. What are some of the now that we've been working with this technology for a while? We've got more cycles like we someone can spin up a rag pipeline and you know, whatever 10 minutes I can spin it up and I have something going. But it's another thing to kind of, of course, scale that maintain it over time, deal with some of the issues, you know, what? I guess my question is, what pitfalls are people falling into that? Maybe we didn't know about whatever it was a year ago when we were kind of just getting into these initial

• 23:56Speaker 01

  kind of naive rag sorts of things. What what challenges or consistent challenges and pitfalls? Do you see people kind of falling into?

• 24:05Speaker 02

  Yeah, I think the consistent thing I see with with something like rag is it's fairly easy to build a quick demo. You can grab an off the off the shelf embedding model to do that. You can combine that with a generation model like an open AI model and you can build yourself a quick kind of proof of concepts. I think the trouble that people get into with it is scaling it up is it's great on 100 documents. But now, all of a sudden, I have to go to 100,000 or a million documents. How am I going to do that? Or, you know, when I first did my demo, I did a couple of very simple queries, text extraction queries. But now, when I put it out in front of my users, I find out all of a sudden,

• 24:46Speaker 02

  they're not giving nice one sentence queries. They're just asking two words. And then I need to add, right, a query reformulation step or something to do that. Or, the accuracy is not kind of what I was looking for. And so I've added a bunch of pieces in there. I've added a re-ranker. I've added other steps. But now my latency is suffering, right? There's all these kind of trade offs as you kind of get to production. And then you're like, oh, you know, do I go back? You go and you look online and you see that, oh, wait a minute. There's like 15, no, I think that, I think there's like 25, 30 flavors of rag. And you're like, oh, did I set up my infrastructure wrong? Or, oh, do I go back and I changed my chunking strategies? I can see, you know, there's 10 different chunking strategies. People are doing that.

• 25:31Speaker 02

  And so I think this is the cycle of where it's very easy to get started. But getting to that final kind of production quality rag can kind of be a little wearisome.

• 25:40Speaker 01

  Yeah. And do you, do you think that that's where the real human engineering piece of this development still like will be with us for some time? Cause you know, to some degree, like you can describe, let's say I describe that sort of problem to my AI coding assistant. Is it reasonable for me to think that like that kind of debugging and process could help me or, or that kind of assistant type of thing could help me get to the bottom of this or update my, you know, retrieval pipeline and, and that sort of thing?

• 26:13Speaker 02

  So I'm kind of optimistic that the, the reasoning models that we have now are going to get us much farther towards helping you solve that problem. Now, of course, that reasoning model's got to understand like how you're thinking about how to solve that problem. But already today, if you take the traditional rag approach, for example, but you pair it with one of the reasoning models that can make tool calls, that can kind of look at the results that come back, think about it, decide, hey, I want to requery it in a different way. You can improve the quality of your results in that way by kind of using that reasoning to do that. So I'm pretty optimistic that we're going to keep finding new ways as long as there are workflows that we can train these models on that are kind of fairly, fairly, let's say logical.

• 26:57Speaker 01

  Archetypal or, yeah.

• 26:59Speaker 02

  Exactly.

• 26:59Speaker 01

  Gotcha.

• 27:00Speaker 02

  Like, like a way we can kind of connect the dots and teach the models to do that, that I think a lot of these things that if we have the, if we give it the budget for Spending time thinking, doing those tokens there.

• 27:12Speaker 01

  Right.

• 27:13Speaker 02

  It's going to cost us more latency, but we're going to see better results. And I think some of us, we already see that in using some of these tools like deep research, where we can see that by spending more time on the task, it's able to give us a better result.

• 27:25Speaker 01

  Yeah, yeah. And, you know, one area that we haven't talked about yet is this sort of world of agents, which I know is a loaded term. And it's probably related to what you were just talking about in terms of time of compute and steps in the process and the reasoning models and all of those sorts of things. Could you help us parse apart, you know, from your perspective? It almost seems to me like it's one of those, I don't know if you're, oh, I'm sure you remember, you know, when it seemed like every conversation I got on, like, the first part of the presentation was, what is the difference between data science and AI and machine learning?

• 28:08Speaker 01

  Or the difference between machine learning and AI or whatever? And at a certain point, I was like, well, these terms all mean nothing essentially because people use them so interchangeably. I feel like that's sort of where we're getting with agents and assistants and all of these sorts of terms. But from your perspective, I guess if that word agent kind of has a meaningful difference to you, what kind of stands out in your mind there?

• 28:33Speaker 02

  Yeah, and I think like we all like the idea of this agent, right? Like something I can give a problem to and it solves a problem. And now I think there's, where the definitions break down to is how much autonomy is this agent, how structured is what we do like that. But if we just think back about the bigger picture of like, I have a, I have a problem. It's not a straightforward problem that I want to give to an agent. Now, there's at least two different ways that we can kind of tackle this. One is I can give them a step-by-step list of instructions. And this is what we call a workflow often, like do this, do this, do this. And then I can check their work at every step and make sure that they're on the track to solve it. Or I could just be like, this is the difference maybe between my kind of five year old and a thirteen year old.

• 29:18Speaker 02

  My thirteen year old, like I'll give them the list and I'll cross my fingers and hope that they'll finish it. And, you know, usually they do, but not always. But I'm not involved in every step of the way. And I think one of the things is we're watching the agents evolve. And this is one of the big trade offs that developers have today is how much structure? How much babysitting am I doing for this agent? How do I do versus kind of the hands off? Now, I think the trend is, is we're going to be able to do much more hands off. Just like we've seen these models be able to gain the reasoning ability over the last two years or so. I have no doubt that we're going to be able to train them to do more complex tasks, to be able to follow those steps.

• 30:01Speaker 02

  It's just a matter of kind of giving them the training data, having the experience to do that. So my bet is in the long term for many of these tasks will be able to be much more hands off. And the models themselves will strive to be able to solve them themselves.

• 30:17Speaker 01

  I just thought it would be good to get your, your input on a theory I've been having, which is, is maybe related. I mean, maybe it's an offshoot from, from what we're talking about, which is really maybe the, the ability to use these vibe coding tools or others to, or assistants or agents to update, you know, retrieval processes or kind of architect our AI pipelines. If you, if you will, I've had this sort of thought and I, I want, I'm curious about your opinion because you also have a background kind of pre generative AI in the data science world that like previously I kind of had in my mind this mental model of

• 30:58Speaker 01

  on the one side you have kind of engineering, traditional software engineering, DevOps, infrastructure. On the other side, you have business and the product and, you know, marketing, all of those things. And in the middle is kind of data scientist because, you know, you translate the business problems and understand how to connect it to the data and the tech and produce your predictive model and like you're kind of living between. Between those two worlds and it's almost like I see that middle zone shrinking and shrinking and shrinking because those domain experts on the business side are actually able to use very sophisticated tools now to kind of self-serve themselves a lot of the kind of

• 31:43Speaker 01

  maybe stuff that would normally fit on the plate of a of a data scientist. So part of me is wondering like me as a me personally, like Daniel Whitenack as a data scientist, like what is the future of that data science world when this kind of middle is shrinking? I'm curious if you also see it that way or see it slightly differently and what your thoughts are in terms of that view?

• 32:06Speaker 02

  Yeah, so I don't think the data science world is changing at all. First of all, I'm excited that the bar is kind of dropping in terms of people being able to use code to build solutions. Like my nine-year-old can literally like vibe code of a game that he can play as well as my 23-year-old who has a who has a degree in computer science and you couldn't tell the difference between like the games that they built like that. Like, so there's a great ability in just allowing everybody to be able to kind of more participate and work kind of with code that we're being able to see now. Now, how does that change something like data science, right? Like data science, the original triangle was right, part of it was coding.

• 32:48Speaker 02

  I think data scientists were never thought of as really great coders, which is why, right?

• 32:53Speaker 01

  Which is why 80% of those projects didn't make it past pilot two.

• 32:58Speaker 02

  Exactly, right? Like they would not write production code and right, there was MLE engineers kind of became the offshoot to kind of do the production piece like that. Now, for me, it's a similar thing to like if you think of like journalists in media, right? Like, everybody says, oh, right, everything's going online, we're not going to have any journalists. Well, if you think at the end of the day, a journalist is a storyteller telling you about kind of the facts of what's going on. For me, the data science is a similar piece where it's still kind of a mission, a work that we're doing in terms of we're helping a business solve problems by looking at their data. There's the tools have changed, but the same problem still exists.

• 33:42Speaker 02

  And I think this is the most important thing is you still need a flexible mind as a data scientist to be able to look at data, to be able to talk to a stakeholder, to be able to go out and figure out what is the coding, what is the math, the algorithms to bring to solve that problem where you need a lot of this kind of left brain, right brain stuff. And so it's still a fairly unique role. And you can see this where you start talking to like AI engineers, where you have developers that are trying to kind of bridge this and solve the business problems. And we see one of the biggest problems they have is with evaluations. And for data scientists, they're trained on how to do evaluations coming up.

• 34:25Speaker 02

  Like you look at the data, you talk to people, like error analysis is something that's built into kind of data scientists. But I always look over and see like how we have to kind of teach software developers that skill if they really want to be able to kind of do the same kind of work like that.

• 34:41Speaker 01

  Yeah, that's super interesting. I kind of posed the question in maybe a in maybe a little bit of a controversial way. I think, I think I would, I would echo what you're saying. I mean, there's, there's elements of this in certain cases, like, you know, in certain industries where like, hey, if you're using computer vision to analyze parts coming off of, you know, medicine coming off of a pharma manufacturing line and needing to do that, you know, 10,000 times a minute, like this is not a problem that is like, hey, just prompt an LLM. And this like, that's a like, on one side, there's very hard kind of modeling problems

• 35:26Speaker 01

  that that need to be solved there. I think on the other side to what you're saying, I also see this kind of gap around AI engineering where it's, okay, we can architect the pipeline. And where I see a lot of people spinning their wheels is saying, well, it seems like this is working. That's kind of where they end. And like, well, we could also measure. If it's working and like construct a test set and you know, maybe automate that. And as we update our pipeline, we could test the retrieval and those sorts of things. So love, love that perspective around the evaluation especially. I guess, do you see that side being stressed more and more as like in maybe software engineers?

• 36:09Speaker 01

  See that the future is AI related and really want to push into that?

• 36:14Speaker 02

  So I, I, I see that there is kind of a growing emphasis for developers and engineers to understand evaluations and do that. My thing is that it's always going to be a little bit of a tension for those folks. Because often the folks that are really good at software development have a very black and white way of looking at the world. That they focus on optimization, that there is a best solution, which necessarily isn't the same type of mindset that you need. And of course, this is a graduated spectrum. Everybody's a little different.

• 36:48Speaker 01

  Yeah, yeah.

• 36:48Speaker 02

  Fills it. So this is where I think there's always going to be that gap between just having kind of software developers fully step into it. But I want to take up one other thing that you say is a lot of times like the hype we see around generative AI and Nvidia and stuff kind of draws out and makes kind of the problems that we can solve with generative AI kind of much bigger than I think the actual usefulness of them. So, and what I mean is that there's a lot of problems inside an enterprise that can be solved without large language models. And my worry is, is the folks inside them that have been doing data science for 10 years know that. They know that I could use operations or optimization to solve this problem, or this is a time series problem to do that.

• 37:33Speaker 02

  My biggest fear is the people coming into kind of AI and data science nowadays aren't seeing those types of problems and understanding that there's a whole set of tools to be able to solve those problems where often kind of everything comes, we're using generative AI as the hammer for solving every type of problem like that.

• 37:52Speaker 01

  Yeah, maybe this exists out there. And so, you know, I'm already building something. So someone can totally steal my idea if they want. But I wonder if there exists out there, you know, this kind of idea of some sort of assistant that would live at the orchestration level above these kind of traditional data science tools and help you like towards that analysis. So just by way of example, like, I'm assuming you could put sort of Facebook's profit for time series forecasting behind an MCP server and be able to have that discovered by this kind of orchestration level and maybe guide people to that. Now, that might not

• 38:34Speaker 01

  be the interface that you want to have for your time series modeling like in production. But it could potentially kind of guide folks to some of these, these kind of traditional data science tools and kind of help teach them maybe what they need to put in place that's not on the large language model side and actually have the large language model tell you that, hey, I'm not the best for this and, and you should use Facebook profit or whatever.

• 39:04Speaker 02

  Yeah, I'm, I'm hoping that there's, that we'll be able to get to that. Like, I think there's some elements of, right, we, the models are great for kind of brainstorming, thinking through things, solutions through too. But if the space is too large of possibilities of different ways to slice the problem, different ways to think about how you could set the predictions or what data to use, then even in LM, you're not going to be able to feed it all the relevant information to be able to actually kind of make that decision. And this is where as humans, we have to kind of often be the piece that takes in a lot of that different disparate information and figure out like, okay, this is what the business really focuses on. Let's zoom it down to this piece. And now I'm going to use my LM to help me think about, hey, there's three different tools here and strategies, like,

• 39:48Speaker 02

  tell me the trade offs. Let's figure out which, I was doing this earlier today, like which package should I spend my time learning how to use to solve this problem?

• 39:55Speaker 01

  Yeah, yeah, that makes, that makes sense. Well, Rajiv, I'm sure we'll have many other great conversations at the Midwest AI Summit and at upcoming or future podcast episodes. But as you, as we kind of get closer to the end here, and you look out towards the future, what is it that kind of excites you about the next steps of our journey in this, in this space?

• 40:21Speaker 02

  Yeah, no, I mean, it's just been a great time of innovation inside of data science, which is why I love it. I mean, everything from kind of going from XGBoost to CNNs to kind of where we are now. And so I'm looking forward to like more innovation, especially in kind of the area of large language models. But I also want to remind people like we were talking about, there's a great wake of tools that are out there that I still like to kind of point people to that there's, it might not get the most attention, but there's a lot of times a more efficient way of solving your problem as well.

• 40:52Speaker 01

  Yeah. And where would you like from, obviously, you, you produce a lot of content and that sort of thing. But as just a person that's more intimately familiar with that kind of ecosystem, if folks are like, hey, you know, I, I heard, for example, I'm at a rally innovation conference here in Indianapolis today, off in a corner, and I heard Kevin O'Leary, you know, from Shark Tank this morning, he was saying, you know, every day you should spend 30% of your mental capacity trying something new, like keep those juices flowing. So maybe it's our listeners today, they're, they're taking away, hey, I should

• 41:32Speaker 01

  try one of these non-gen AI things. And like, where would, where would I even go to, to start that? Any, any suggestions?

• 41:40Speaker 02

  Yeah, no, I, I, I love that idea of like, it's spending a 30 minutes or an hour a day, like continual learning is, is the, is the future like that. So I have my own content that I put out at logistics that tries to kind of inspire you to push you in different ways that kind of AI is doing, give, give people simple kind of nuggets like that. So I would, of course, kind of point to myself as well. I think the other area that I really like are newsletters. I think newsletters are a nice way to be able to take in all the information that's coming in, but in a little bit of a slower kind of meditative way, rather than just kind of reacting to the latest trending post.

• 42:18Speaker 01

  Yeah, that's awesome. And I'm sure we'll include a few links in our in our show notes to things that will be useful for people. But really appreciate you joining us again, Rajiv. Looking forward to seeing you in person. And yeah, keep up the keep up the great work. It's always it's always good to get to hear your perspective and looking forward to having you on the show again.

• 42:43Speaker 02

  Thanks so much. I think this is one of the longest running data science podcasts out there. So it's been great to be part of it. Thanks so much.

• 42:49Speaker 01

  Thanks.

• 42:57Speaker 00

  All right, that's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.