RAG continues to rise

Daniel & Chris delight in conversation with “the funniest guy in AI”, Demetrios Brinkmann. Together they explore the results of the MLOps Community’s latest survey. They also preview the upcoming AI Quality Conference https://www.aiqualityconference.com. Sponsors: The Hacker Mindset https://thehackermindset.com – “The Hacker Mindset” written by Garrett Gee, a seasoned white hat hacker with over 20 years of experience, is available for pre-order now. This book reveals the secrets of white hat hacking and how you can apply them to overcome obstacles and achieve your goals. In a world where hacking often gets a bad rap, this book shows you the white hat side – the side focused on innovation, problem-solving, and ethical principles. Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Demetrios Brinkmann – X https://x.com/DpbrinkmChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: MLOps Community https://linktr.ee/mlopscommunityAI Quality Conference https://www.aiqualityconference.comEvaluation Survey https://hq.yougot.us/primary/WebInterview/3AW6LW5D/StartRAG failover talk from Jerry Lui https://home.mlops.community/home/videos/a-survey-of-production-rag-pain-points-and-solutionsPrompt Templates the Song https://www.youtube.com/watch?v=g6WT85gIsE8Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am founder and CEO at Prediction Guard. I'm joined, as always, by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris?

• 1:00Speaker 02

  Doing good, Daniel. How's it going today, man?

• 1:02Daniel Whitenack?

  It's going great. I just landed in Boston and you were texting me and you're like, "Hey, Demetrius from the ML Ops community wants to hop on and record an episode." And I was like, "I've got to get out of this train station." So I just found the nearest stop and got out. And I don't have my normal setup, so I probably sound weird, but I was like, "These are the best times when we get to have our friend Demetrius on." How's it going, man?

• 1:28Speaker 02

  You guys can't get away from me. I've blackmailed your boss into letting me come back on here.

• 1:34Daniel Whitenack?

  Standing invitation.

• 1:36Speaker 02

  Absolutely.

• 1:37Daniel Whitenack?

  What's up, man? How you doing?

• 1:39Speaker 02

  I'm so excited because I try not to abuse this standing invitation. I had a great time the last time I was on here. We had a bunch of laughs and a lot of people reached out to me because we had, you know, this rag versus fine-tuning conversation. And I think things have kind of, like, the pieces have fallen, the cookies have crumbled. It feels like fine-tuning is not as popular as it was. I don't know what you all are seeing out there, but rags are the go-to in these days.

• 2:09Chris Benson?

  Yeah, that's what I'm saying. Everyone's ragging each other all over the place.

• 2:13Daniel Whitenack?

  And I think, yeah, along with that, there's sort of like, in my mind, has developed this category of like, So there was RAG, which of course you're augmenting the generative model with data, but not in the way that like people typically think of fine tuning. You're just doing retrieval. But I think also these other sort of workflows around calling external tools or like the neuro symbolic stuff that we're seeing of like combining a rules based algorithm or function or traditional quote unquote machine learning algorithm with generative models, maybe to get the inputs. And then like connections to databases, all of these different ways.

• 2:54Daniel Whitenack?

  Like it kind of seems like people are figuring out that generative models are great at being kind of assistants and automators, but not necessarily like predictors, right? Or, or other kind of functions to like analytics types of things and that, that sort of stuff.

• 3:12Chris Benson?

  I've noticed there's been a new term coined, at least for me, RAG is a service. Have you guys run across that?

• 3:18Daniel Whitenack?

  RAG.

• 3:19Chris Benson?

  RAG is a service is now like a thing.

• 3:22Daniel Whitenack?

  Is that acronym just RAG?

• 3:25Chris Benson?

  Sort of.

• 3:26Daniel Whitenack?

  RAG.

• 3:28Chris Benson?

  Don't even try. You're, you're just going to strain the vocal cords if you do that, man. Yeah, RAG is a service. We'll rag you. If you bring your thing, we're going to rag you. We're going to rag you all over the place. Yeah.

• 3:38Speaker 02

  Well, I think there, what you're saying, Daniel, it really speaks to something that I've been seeing too, which is the maturity in the last, whatever, six months. It's become very clear that there's traditional ML workloads and use cases that are kind of going to always be traditional ML workloads. You think about like your fraud detection models or like the term prediction or the recommender system even. And then you have your generative AI workloads or use cases. And that's something like these, like transcription, or you have the LLMs, which are doing all sorts of stuff, but rags are probably the biggest ones in that

• 4:22Speaker 02

  where you get that co-pilot or code generation, I think is a huge one. There's not like such a big overlap where you're saying, okay, the generative AI use cases.

• 4:33Daniel Whitenack?

  Or the generative AI models are going to dethrone the traditional models.

• 4:38Speaker 02

  I agree with that.

• 4:39Daniel Whitenack?

  Yeah.

• 4:40Speaker 02

  Definitely different use cases.

• 4:41Daniel Whitenack?

  I think one thing, we had our first Gen AI Mastery webinar with the podcast. When was that, Chris? A couple of weeks ago or something like that?

• 4:51Speaker 02

  Yeah.

• 4:51Daniel Whitenack?

  And we were talking about text to SQL specifically and that, you know, analytics like SQL is really good at doing analytics, right? Especially descriptive analytics and aggregations and all of that. And it just doesn't make any sense for you to take a big table and somehow figure out how to dump its contents in a prompt and have a model reason over it because it's probably going to get it wrong anyway. But also there's this existing tool which can be called, which is really awesome at doing those things.

• 5:23Speaker 02

  I love the idea and the exploration that's happening right now on how can we merge both of these worlds? And how can we see what different parts work well together? And which combination of the traditional ML plus the generative AI can go together?

• 5:43Daniel Whitenack?

  And I know that you do a series of surveys with the ML Ops community. I think the last time that we talked, we were talking about some of your survey work and some of the interesting findings. But I think you've gone through other iterations of this, right? So are you seeing interesting things pop up as the community around this technology matures?

• 6:06Speaker 02

  So 100% and I just come on here when I got survey insights to share, I guess. That's what I'll be known for. Whenever we have a nice survey.

• 6:15Daniel Whitenack?

  The pollster of AI.

• 6:18Speaker 02

  I'll come and share it with you all. But this one is cool because this time, so we did an evaluation survey and we launched it when we had our virtual conference, which had a huge turnout. It was over two days span.

• 6:33Daniel Whitenack?

  Which was awesome, by the way.

• 6:35Speaker 02

  It was great. You were part of one of the past ones, right?

• 6:37Daniel Whitenack?

  Yeah.

• 6:38Speaker 02

  And Daniel, you had an awesome spot, but the two day span, we tried something different and we said, What if we do two days, but since it's virtual, nobody has to fly anywhere. So instead of you trying to watch a live stream for eight hours a day on a Thursday and then a Friday, why don't we just do two Thursdays in a row? And then you don't have to feel like, wow, I just had 20%, 30% of my week eaten up by that eight-hour live stream or 16-hour live stream. Now you can tune in, tune out on a Thursday of your choosing. But we launched it there and the response has been amazing because normally maybe we'll get like 100, 150 people that will fill out the survey.

• 7:20Speaker 02

  This time we had 322 is the number of these, which is super cool to see. And so let me give you some of the clear insights. One, there's budget being allocated towards AI these days. I don't think that's going to surprise anybody. The fascinating part is that like 45% of the respondents said that they're using existing budget. And then a whole like whopping 43% here said, no, we're using a whole new budget. So you've got exploration happening in generative AI like never before.

• 8:03Speaker 02

  But when it comes to that, one other takeaway has been that like the ML ops, AI, ML engineers, they're really trying to figure out what the biggest leverage use cases are and how they can explain that. And I think what we're seeing is there's a lot of companies that are open to the exploration right now. And they're open to letting people say, all right, cool. What is the most valuable for our teams in our company? Is it a chatbot that is an internal chatbot? Is it an external chatbot? What does that actually look like? What is the use case?

• 8:43Chris Benson?

  It's kind of funny. We actually talked about that a little bit last week, Daniel and I did, in terms of trying to get non-technical people engaged in it. And I think that there are organizations all over the world right now that are doing exactly that, you know, to your, to your point on the result that you're seeing. There's a lot of effort and a lot of money being thrown at how do we start doing that?

• 9:05Speaker 02

  And it's all really like the shining star here was just rags. Obviously, it's very clear. Everybody's using rags and the participants self-identified as being intermediate in rags. That was the majority. So we had like 31% saying that we have some experience with LLMs and rags. And then you only had 6% saying, "We are at the frontier of LLMs and RAG model innovation."

• 9:51Speaker 04

  What's up, friends? There's a new book out there called The Hacker Mindset. This is a productivity cheat code to unlock new levels of success in your career, in your creative pursuits, and in your personal growth. This book is about leveraging the principles of white hat hacking and applying those skills to the broader world. It's available for preorder right now, and it's not your typical productivity guide. This is written by Garrett G., a seasoned white hat hacker with over 20 years of experience. This book reveals the secrets of hacking and how you can apply those skills to overcome obstacles and achieve your goals. So don't miss your chance to get ahead and get this book, The Hacker Mindset.

• 10:33Speaker 04

  You can preorder your copy today at thehackermindset.com. Be among the first of many to tap into this power of hacking for your success. Join the movement and embrace a new way of thinking. Again, that's thehackermindset.com.

• 11:01Chris Benson?

  So let me ask you guys a question, get some opinions going here on that. Do you think, you know, with all of these assistants and chatbots and all the other kind of, you know, focus things in Gen AI, with those using RAG, do you think those are for more kind of general use cases, having some domain knowledge in? Or do you, and do you think that maybe going back to the last time we, we had you on the show, when we were talking about fine tuning, do you think that kind of highly specialized fields will still stick with, with fine tuning instead of RAG? Do you think, in other words, the degree of expertise required, if you will, to get a job done productively, do you think that makes a difference on whether or not you go RAG or fine tuning?

• 11:45Chris Benson?

  Or do you think it has nothing to do with that?

• 11:47Speaker 02

  I would love to hear what Daniel has to say in a minute, but I've heard something said about like, fine tuning is for form. So if you're trying to get a different form on the output, or for example, if you're trying to get functions and the whole, basically like GPT functions are a perfect example of this. So if you're trying to get a homegrown model to do that type of thing, then fine tuning makes sense. Otherwise, it's not necessarily the good call because unless you're using a very small model, I think is the, the thing there that, okay, how, what's the trade off that you're doing?

• 12:31Speaker 02

  And the other thing that is probably worth talking about when it comes to like the difficulty levels that I've seen is people that want to go and do, use a small model, a small like domain specific model that is distilled, or it is very fine tuned and distilled, and it's on their own, in their own infrastructure and in like, they need a whole team to support that. That's like hard mode. You're playing on hard mode and you contrast that with just GPT four call. That's a whole different level of the game that you're playing. And so I kind of look at it that way, like, how much are you willing to trade off when

• 13:16Speaker 02

  you're trying to figure out, is this the way forward?

• 13:19Daniel Whitenack?

  I would agree with that. I think the only thing I would add is, There is still, at least as far as I've seen with our enterprise customers, like an inclination that they need to fine tune, like that's still a kind of general like, oh, we need to do that at some point. And I think once they kind of solve a few of their use cases without that, they kind of disillusion themselves of that notion in many cases. But the good thing, like you were saying, Demetrius, is like, you can probably prove to yourself without a huge amount of effort using an easy to use API, whether or not you'd need fine tuning and do

• 13:59Daniel Whitenack?

  that in like a day versus like just immediately going to jump to fine tuning and how do we get GPUs and how are we like, what model server are we going to use and all of that stuff. Like you say, that gets very much into hard mode very quickly and you don't sort of need that to validate your use case often. And even if you do fine tuning, sometimes it may be down the line when you've been running the pre-trained model for quite some time and you actually have a good prompt data set to fine tune with because most people don't start out with that either.

• 14:34Speaker 02

  That's a huge point, too. And this is probably like the biggest unlock that happened is that, okay, now that anybody can use the OpenAI API, you can quickly see if there's value in that crazy idea you have. And then you can go down the line like, all right, now I'm going to use an open source model, which is turning the knob to something harder. Or maybe it's not even using an open source model. Maybe it's just like, can we get the same results with a smaller model from OpenAI? So instead of GPT-4, we're going with 3.5 Turbo. Can we then go to an open source model and get the same results? Like, I almost look at it like a spectrum of how difficult you

• 15:16Speaker 02

  want to make your life and how much upkeep you're going to need and all of that. But as with everything, there are benefits if you go to that very small model, if you need it. And you just have to really like play out and see if you actually are going to need it.

• 15:30Daniel Whitenack?

  Speaking of another survey, just to make it an increasingly survey-driven show. I don't know if you saw the

• 15:41Chris Benson?

  Survey says.

• 15:42Daniel Whitenack?

  Yeah, survey says the Andreessen Horowitz post that they did another sort of survey, which was kind of interesting and part of what they drew out, I forget the range of participants that participated in the survey, we can link it in our show notes, I was just posted March 21st, we're recording this maybe week, a little more than a week later, and it was about enterprise, so 16 changes to the way enterprises are building and buying generative AI. And one of the things that they specifically highlight there was enterprises are moving towards a multimodal future and specifically a multimodal future driven at least partially

• 16:27Daniel Whitenack?

  by open models. And so I think the other kind of interesting trend that you're seeing is they have like graph with like how many model providers are people using per company or whatever. And you see like three, four, or five, like in many cases, and also a high adoption of open models. And I think what they're trying to draw out is like some of that is maybe driven by security, privacy things, but I think also it's driven by like control and flexibility. And once people start realizing, I also find it still a pretty big misunderstanding that people have that all of these models sort of behave the same.

• 17:10Daniel Whitenack?

  And in reality, pretty much every model has a character of its own and a specific behavior. And even just switching from OpenAI to an open model for like text to SQL, for example, a model that doesn't do other things well, but does that really well can prove to be really useful. Or maybe it's a specific language thing, like we're doing a Mandarin chat right now with one of our customers. And so Whether it's language or whether it's task, people are, I think, I think finding out that their future is multi-model or multi-model provider or whatever, mainly because of that behavior

• 17:50Daniel Whitenack?

  thing, but also because they can have some control over when they use this model or when they use that model and kind of create the mix that's right for them. And that's kind of a way, it's like a route around fine tuning in some ways because you can kind of assemble these reasoning chains, even with multiple models involved that do very specialized tasks. And that can kind of help you avoid spinning up and running up your own fine tune that does this very, you know, unique reasoning workflow. You can kind of bring all the experts in, bring all the expert models in to help you.

• 18:29Chris Benson?

  I can't help but wonder, you know, like, that's a built-in capability that you have at Prediction Guard, but I, I think that there's a maturity issue there with a lot of organizations on getting to that point. And so I'm, what I'm seeing is very mature organizations are going exactly to that, having multi-models capabilities and they have the ability to distinguish between which models they should use for which circumstances. I suspect, and maybe there's some survey data on this, but I suspect that that's a, still a fairly small group of even enterprise organizations that have gotten to that level of understanding of what they can do. And I think there's a spectrum falling off from there that the bulk of the world is in right now in terms of trying to figure out how to

• 19:14Chris Benson?

  make it work for them.

• 19:15Speaker 02

  Let me twist some statistics to play in your favor. Hold on, I'm gonna crunch some numbers.

• 19:20Daniel Whitenack?

  I love it. Live, you're probably not using an AI model to do it.

• 19:26Speaker 02

  Yeah, I'm gonna, I'm gonna weave that narrative that you've got, Chris, and I'm gonna go with it with the survey data as you're asking for. No, but I, actually, the biggest question, as you guys are talking about this, that goes on in my head is, you know what engineers really do not like, and I think it makes them very anxious, is having a single point of Failure. And so if you are relying on OpenAI's API and you have a lot riding on that, where does that all go if the CEO gets ousted? And so I imagine that a lot of people thought twice after there was that big drama that happened. And people started thinking, you know what?

• 20:09Speaker 02

  Maybe we should try and have a few redundancy options just in case. Now you do have to have a bit more maturity to say, okay, I can't use the same prompt as I use always. So I have to have this prompt suite or prompt tests or prompt templates. And I think that's another thing that's happened since the last time we talked, when we had the conference to try and get people to... Well, prompt ops is one, agent ops is another one. But I created a song called Prompt Templates. And I'll put it in the... Maybe we could play it real fast.

• 21:34Chris Benson?

  As someone who's in kind of the defense industry, I'm for agent ops because it sounds bad, doesn't it?

• 21:39Daniel Whitenack?

  Oh, I thought you were gonna say you were for prompt songs, which would bring smiles to people in the defense industry.

• 21:46Speaker 02

  That's what I was hoping to do. Exactly. Missed out on that one. But he is... I don't know if people can see this, but he's got a wonderful shirt on. That is for sure. And being in the defense industry, I don't know how you can get away with that.

• 22:01Daniel Whitenack?

  Work from home.

• 22:04Chris Benson?

  Work from home is how I get away with it.

• 22:06Speaker 02

  And for those that aren't watching, his shirt, it says, I hallucinate more than ChatGPT, which is a classic shirt.

• 22:14Chris Benson?

  Which Dimitri has sent to me, I have to say. So thank you very much. I love this shirt.

• 22:18Speaker 02

  I love that you wear it. That is what I'm very happy with. But the other thing that I wanted to mention about the survey, and then we can move on and keep talking about other stuff of the day, topical issues. But the data that people use and the data with which we evaluate the output, it seems like people just don't know what's going on there. We haven't figured that out yet. There's no consensus. It's not really clear. And like the classic data sets or the classic evaluation pieces that you use, they don't really hold up. So everyone's gotta be having their own data that they've created and that they're testing against the output.

• 23:02Speaker 02

  But it's really hard like to do that at scale, right? And it's really expensive also. So that's what we saw in the evaluation data or the evaluation survey data is that you've got to handpick these, you've got to match them up. And it's human curated, the testing data sets that you create. We had 42% are using data that they've created as their data sets to evaluate if the model is working or not.

• 23:30Daniel Whitenack?

  Yeah, that's crazy. And you mentioned expensive. It could be monetary, but it could also be sort of iteration time too. Typically, like when we were back when we were creating machine learning models, which lots of people still do because it's the thing driving basically all of the predictive stuff. But you know, you could run your model and evaluate it. Maybe it took a few seconds, but you know, a couple of minutes. But here, when you're thinking about running, especially against an API.

• 24:05Daniel Whitenack?

  That has like variable latency, or maybe the calls, each execution of your prompt chain is taking like 15 seconds. And even if you want to run that over like 100, 200, 300 example reference inputs, all of a sudden your iterations become really, really slow. That's something I've noticed that people kind of struggle with is really making their evaluation quick enough that they can iterate, feel like they can try a lot of things, even if they have like a big budget to try a lot of things. This kind of iteration time is really frustrating also because maybe there's other people that are involved that aren't technical, right?

• 24:48Daniel Whitenack?

  And they don't want to think about like concurrency in Python, right? They just want to go into an interface and like try some stuff. And yeah, so you've got all of these things mixed together, which make it a bit of chaos in many cases.

• 25:02Speaker 02

  That's so true. The iteration speed, the time, the, I mean, we see here that, this is crazy, 72% of ground truth labels were manually labeled by humans. And so to have to go and do that, and then also, how often are you doing that? How, like, there's so many questions and so many unknowns for what the best practice is. That's one thing that came up on the challenges is that it's just like, a lot of people called out something that we synthesized into lack of guidance. Like, nobody's saying, this is the best practice. This is what we've seen works really well for us. Because maybe some people say, well, this worked well for us sometimes.

• 25:47Speaker 02

  And you can try it and see if it works well for you, too. That's kind of the state of the industry right now.

• 25:53Chris Benson?

  There's something I want to tie back into something else that we've been talking about lately, and that is the fragmented nature of the community. And it's another thing Daniel and I've talked about recently, is that we do have communities, but we have multiple communities. And in many cases, not in your case, but in many cases, they're, they're very platform dependent, you know, vendor specific. And it makes it Compared to a lot of programming languages, it makes it harder for people to come in and find specific best practices. So I'm actually not at all surprised to hear the survey kind of playing that out. I think that that's kind of a a natural fallout of the challenges that we're having with community in general.

• 26:32Speaker 02

  So if my understanding is correctly, it's like because a lot of the communities are being built around certain tools, you have the best practices for those tools, but not necessarily for the industry. You can't generalize those best practices.

• 26:47Daniel Whitenack?

  Yeah, and I think also the different channels through which people are communicating kind of naturally develop their own bias. I don't mean bias in a in a bad way necessarily, but just the bias towards like emphasizing certain things, right? Like you get into the news research community. We had a great conversation about that and like, people are talking about, oh, like we're doing all this like activation hacking and representation engineering and like that's, but that's like not really talked about. Like if you're over here in the Lama Index Discord or Lance DB Discord or like whatever. And some of that's driven by the focus on what those tools do, but also like where people are coming from, right?

• 27:31Daniel Whitenack?

  And a more of the indie hacker building app sort of stuff or the rigorous like academic side or the like enterprise, like I really just want to get something into production side. There's like all these different slants people are coming from.

• 27:48Speaker 02

  That is so fascinating to think about like how each of these communities has their main focus. And since it is, there is so much surface area and there's so many areas that you can go, different areas to explore that each community is exploring their own area. And if you go into that, you can tap into what people are talking about in that area. Versus if you go into another community, you get, oh, whoa, what's going on here? What's the focus of this community?

• 28:20Chris Benson?

  This is a different outcome from what we've seen. If you, if you step out specifically of kind of the AI ML world and you look at more just computer science, computer programming communities out there, there's usually kind of a place to go and you kind of learn the same sets of, of skills and values, you know, around that. And that's a little bit different from this. It's been one of the challenges, I think, that, that the AI ML world has struggled with a little bit. So I'm not at all, like I said, I think your survey captured that essence in this.

• 28:54Speaker 02

  Thank you for sharing that with me because I'm going to steal it and I'm going to say it a bunch. Hopefully you don't mind.

• 29:00Chris Benson?

  Say it all you want.

• 29:02Speaker 02

  It's a great insight. I've seen it just in the ML Ops community, right? We have people that are really trying to productionize AI. And so what people in there are talking about is really like pragmatic and practical. How can I get this being used in my company so that I can either save money or make money? Like money is the ultimate metric there. If you go into, as you were mentioning, like these different communities, if you go into the Lama Index community, there's a lot of talk of rags. And actually, we had Jerry on in the conference and he showed this slide that I thought was so incredibly done. It wasn't him that did it. I can't remember the person who created it, but it was like

• 29:46Speaker 02

  the 11 ways that rags fail. And so it had all these different ways that you need to be aware of. And I think one that's coming to light that people are seeing is so important is how you need to get that retrieval evaluation correct. Because if you're not retrieving the right thing from the vector DB, then it doesn't matter what you give or what the output of the LLM is. If you give it some kind of crap, then it's not going to give you anything there. And the other piece that I think is fascinating is that, like, how do you make sure that all this data that you've got in the vector DB is up to date? And so we've talked about this a bunch.

• 30:30Speaker 02

  Again, this is in the ML Ops community, we're very industry focused and how can we make sure that we are productionizing this. So in a production scenario, you've got your HR chatbot that is using a RAG system and you say, alright, cool, we've updated the vacation policy. So we went from a European vacation policy to an American vacation policy. And you've got Daniel over here saying, alright, HR chatbot, like, how many days of vacation do I have? How do you make sure that everywhere in the vector database, it now is updated to the American vacation policy? And so, okay, cool. In the vector database, maybe you say, you know what, we were able to scrub

• 31:12Speaker 02

  everything, or we just pull from the most recent documents. But then you were a good engineer and you made sure to pull in a bunch of different data sources. So in Slack, turns out that you're grabbing some data from that and people talk about how it still is the European vacation policy. And now Daniel's been quoted of having 30 days of vacation when really only has two.

• 31:35Daniel Whitenack?

  That's unfortunate.

• 31:38Daniel Whitenack?

  Yeah, actually, this is a conversation we just had the other day with a customer because also, at least some of these databases, they have, depending what you go with, if you go, you know, with a plugin to an existing database, maybe there's kind of more traditional updating and upsetting sort of functionality. But some of these is just like, put a document in, get it out, delete it. There has to be a layer of logic on top of these that actually help you do some of that. So in their case, it was like, oh, we want to take in all the articles that we've had on our web, on this website, and that's going to be it. And then they're like, well, what if we update those?

• 32:21Daniel Whitenack?

  Do we just like blow everything away and redo it? And I think my ended up my answer like with the amount that they had, I was like, probably like if you can have something running in the background, honestly, that's probably the safest thing for you and it's going to take like a couple hours or something, but then at least you the going into the making sure everything synced up is in in that case, like they could just version of files of the embedded database. But yeah, it's an interesting set of problems.

• 32:53Speaker 02

  It is a fun one. And also, you know what I'd love to explore too, is the idea of like RBAC or role-based access control.

• 33:02Daniel Whitenack?

  Yeah.

• 33:02Speaker 02

  How are you seeing people go through that and do it well? Because that feels like another one that can be really misused.

• 33:11Daniel Whitenack?

  So for RAG is one thing, for text to SQL, some of that maybe can be kind of nice, because if you're embedding some function in an application, it's already has RBAC on the database, then you could use that credential and hopefully that carries through. But for the vector database side, we've interacted with people that have maybe like an internal chat and an external chat where the external chat is a subset or should use a subset of the documents from your internal chat. So in that case, you sort of have like two, it's bifurcated rather easily. And you know, that's like somewhat easy to deal with because you could just have like

• 33:56Daniel Whitenack?

  two tables or two collections, whatever that is in the vector database and kind of merge the retrieval or use them selectively in in certain ways. But as soon as then you have like many, many different roles or even user specific things, I don't know, like many vector databases, that would be however you manage that would be transparent to that vector database. So you'd have to somehow manage the metadata associated with it. There may be certain people, we'll have to follow up, Chris, we haven't had a Muta on for a while, but they're always thinking about these role-based access to really sensitive and private data. I'm sure there's people doing advanced things, but in terms of the main tooling that people

• 34:39Daniel Whitenack?

  are just grabbing off of the shelf, a lot of that logic is just absent.

• 34:44Speaker 02

  Exactly. Yeah, I want to hear if anybody is doing R-BAC and they've figured it out. That's one thing I'm fascinated with because it is a very, again, going to the community that I run with, that's something that productionizing, kind of, it comes hand in hand with that.

• 35:02Daniel Whitenack?

  Yeah, and it could also have to do with the guardrails that you put around the large language model calls because if it's like a public facing chat or something like that, that you may want to filter out PII or like prompt injections may be a very important thing versus like internally, ideally you trust people as long as you know how the data is flowing, like there might not be as many restrictions in terms of what can go in or who's accessing things and that sort of thing. But yeah, it's interesting.

• 35:54Speaker 00

  This is a changelog news break. Pierre-Carlo Longlaes announcing the release of Common Corpus on Hugging Face. Quote, "Contrary to what most large AI companies claim, the release of Common Corpus aims to show it is possible to train large language models on fully open and reproducible corpus without using copyright content. This is only an initial part of what we have collected so far, in part due to the lengthy process of copyright duration verification. In the following weeks and months, we'll continue to publish many additional data sets, also coming from other open sources such as open data or open science." End quote. Here is

• 36:35Speaker 00

  more info about this massive data set. Common Corpus is the largest public domain data set released for training LLMs. Common Corpus includes 500 billion words from a wide diversity of cultural heritage initiatives. Common Corpus is multilingual and the largest corpus to date in English, French, Dutch, Spanish, German, and Italian. Common Corpus shows it is possible to train fully open LLMs on sources without copyright concerns.

• 37:04Speaker 00

  You just heard one of our five top stories from Monday's change log news. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 37:31Daniel Whitenack?

  So in addition to this sort of evaluation stuff, we've spent a lot of time talking about data and evaluation and retrieval. What about on the on the model side? Do you think we'll ever escape the world of transformers, Mitros?

• 37:49Speaker 02

  So this is something I've been thinking about a ton, man. And I've got some thoughts on this that like, is everything that we're doing now in AI a Band-Aid because transformers just aren't the right tool for the job? Have you guys thought about?

• 38:06Daniel Whitenack?

  It's like one big workaround?

• 38:08Speaker 02

  Yeah, exactly. Am I crazy to think that?

• 38:11Daniel Whitenack?

  I don't think so. Actually, I was talking to one of our customers about this, like they have so much like logic around double checking the outputs of models or like formatting the outputs of models or and I'm talking like hundreds and hundreds and hundreds of lines of of code, thousands of lines of code, I don't know, written all around like this sort of workaround of like, and it's because they're using general purpose model, right? That you sort of have to massage into how you want it to behave, right?

• 38:42Speaker 02

  Is it a little bit ironic that, you know, use rag to clean up the problems with transformers? Is that what we're saying here?

• 38:50Daniel Whitenack?

  Oh, I get it. The rag. What we need is the Lysol wipes.

• 38:54Speaker 02

  There you go.

• 38:57Speaker 02

  But I I often wonder, like, are we having to over engineer this because the core of the problem, it's like we're trying to put a Band-Aid on something instead of going and fixing the root of the problem. And right now, it feels like there's nothing out there that can even stand a chance against the transformer architecture. So of course, we can't say, well, I would rather use XYZ. But I just get the feeling like when we think about AI in 2024,

• 39:35Speaker 02

  or like the chat GPT AI era, we're probably going to be laughing at the whole idea of transformers. If in 10 years, we're looking at that, it's going to be like, yeah, okay. Transformers were great, but they were a stepping stone.

• 39:52Chris Benson?

  I know that there's a quite a bit of research going on in general about doing different types of architectures. I know that there's a number of organizations that have been testing alternatives to transformers in the last couple of years. But I don't think anyone's gotten there, or if they have, then you should reach out to us and let us know so that we can be talking about it here on the on these podcasts. So I think there's a lot of folks out there that are really wondering what's next because we've kind of, you know, we're essentially taking one superset of architecture and we're doing everything we could possibly do with it. You know, and every big step forward in the last few years has been around what else can we do with this architecture?

• 40:33Chris Benson?

  So at some point, I agree with you, Demetrius, something's going to give, and we've got to try some new approaches in there.

• 40:40Speaker 02

  Yeah, that's what it feels like to me. It's just like, what's the next step? And I would love to also hear from whoever, if there's something that feels like it's promising. It's really exciting to me. I don't know enough about that. That's very much the research community that I don't get to spend a lot of time in. And so, and I'm sure there's a bunch of false flags and people get excited about something and then it turns out that after you throw a bunch of GPU at it, it doesn't work out like we thought it would, or like we, we saw a promise, but it didn't actually work out when it holds up to scale. So I understand that right now, we're in the era of transformers. I wonder how long we're going to stay in this era.

• 41:24Chris Benson?

  Not only around specific architectures in that capacity, but almost new approaches. For the first time in a while, you know, neuromorphic computing is really rising again as a topic of interest. And it's not there yet. You're talking about architectures, both on the hardware and the software side, that are not specific to, to either transformers or even GPUs underlying it and stuff. But it's been interesting to see the maturity that's developing. You know, you talked about the exposure to research. Even for me, that's the same case is that, you know, you have all the pure researchers out there, but now we're, we're starting to see them expand out in lots of ways and trying completely

• 42:05Chris Benson?

  different approaches. And I'm pretty excited that we're going to start seeing some interesting results over the next few years as people are looking for alternatives across both hardware and software architectures. I think we're pretty close to a turning point.

• 42:20Speaker 02

  Can you break down real fast, what was that big word you just used? Anthromorphic? What was that? I can't even say it. I got tongue tied.

• 42:27Chris Benson?

  Neuromorphic computing, I think is what you're, what you're talking about.

• 42:31Speaker 02

  Neuromorphic computing. That is a big word. What does that even mean? I don't know. I gotta Google that real fast.

• 42:36Chris Benson?

  So, so, and I am the last person on the face of the earth that should be trying to explain neuromorphic computing.

• 42:44Speaker 02

  I put you on the spot.

• 42:45Chris Benson?

  But have it, yeah, there, no worries. But having been exposed to that, the, the short version is almost like, um, you know, in the earlier days of AI and we, and people would say, uh, you know, in the marketing, people would talk about, oh, mimicking, you know, the neocortex. You know, the human brain and stuff like that. And we all kind of is this GPU and transformer based architectures. We're like, it's not really like the human brain. Well, the neuromorphic architecture is actually that. It's, it's the legitimately like, how does that the architecture of a brain? And I'm saying this, like, there's probably neuromorphic computing scientists out there listening to me now going, oh, my God, somebody take his mic away. That's

• 43:27Chris Benson?

  a terrible explanation. But in my, in my fairly primitive understanding, that's kind of where it is, is, you know, how do neurons really work in real life? And how do you do compute artificially in that capacity? So, but I know that there's definite interest, uh, doing that. Uh, I know Daniel has a relationship with Intel through Prediction Guard. And I know Intel, uh, has an interest in that field. I think they're one of the leaders in it.

• 43:52Speaker 02

  I Googled it. Intel's all over the first page, or I perplexed it. It was all cited from Intel. That is very true.

• 44:03Chris Benson?

  I would hesitate to say it out loud because I'm probably wrong, but they may very well be the global leader in that space right now. So.

• 44:09Speaker 02

  Yeah, makes sense. Well, that is awesome. I'm glad that you taught me about that. I appreciate you for teaching me neuromorphic. Now I can say it properly and everything.

• 44:19Chris Benson?

  Well, you know what? Now that you say that, we're gonna have to have a show on neuromorphic computing coming up pretty soon. So.

• 44:24Speaker 02

  Yeah, exactly. Let's get down into it. I want to listen to that for sure.

• 44:28Chris Benson?

  We'll dive into that. We'll, we'll get, Daniel can, can reach out to us contacts there.

• 44:32Speaker 02

  Oh, that's classic.

• 44:34Chris Benson?

  Well, dude, thank you very much for coming on as we, as we wind up here. Uh, it's always a pleasure. Anyone who has been listening to the show long knows that you're, uh, join us regularly on the show. It's always special for us. We have a great time with you. So thanks for coming on today. We will get the show notes for the survey, uh, and some of the other topics that you brought up today. So people can join and, uh, folks, if you haven't gotten to the ML Ops Community Podcast, uh, that Demetrius hosts, you definitely need to check that out. It is an awesome podcast, highly recommended by both myself and Daniel. So, uh, hope people join you over there.

• 45:13Speaker 02

  Oh, and can I also plug, we're gonna have an in-person conference. And I'm really excited about that, a little bit shaken in my boots. Because June 25th, it's gonna be our first in-person conference ever. And it's gonna be all about AI quality. And we've got some super cool speakers coming. We managed to get the CTO of Cruise to come and talk about what they've done since their little mishap. In regards to like, making sure that their AI is quality. We've also, I mean, there's so many great people, you can go to aiqualityconference.com. And we'll throw the link there in the show notes too. I'm very excited for it, but the speakers are gonna be awesome.

• 45:57Speaker 02

  The attendees are gonna be amazing. I think what I'm most excited for though, is that we're gonna have all kinds of fun, random stuff. You can imagine it's gonna be a conference, but it's probably gonna be more like a festival. I may have people riding around in tricycles, giving out coffee. Or we'll have a little DJ area, so, or a jam band's like breakout room. A bunch of Legos hanging around, I don't know yet. So if anybody has any ideas on how we can make it absolutely unforgettable, I would love to hear about that too.

• 46:30Chris Benson?

  And I'm gonna throw out one last plug for you. Is that when you say that, I believe you. Because I know that you've heard me say this when we were off the air. But just in case anyone doesn't know this, Demetrius is the funniest guy in the entire AI world. And does hilarious things. If you don't follow him on social media, you are missing some really great, great content. So, um, anyway, just wanted to say that I, that people, people should show up at the conference just to see what you're doing. If no other reason, even aside from the cool content you have, they'll enjoy it. So, thanks for coming back.

• 47:07Speaker 02

  I mean, there's gonna be great speakers, you're gonna learn a ton. But there's also gonna be some really random stuff that you're gonna be like, what is going on here? And hopefully you really enjoy it. Because that's kind of what I, that's what I'm going for.

• 47:20Chris Benson?

  Okay, well, thanks a lot, man. I'll talk to you next time.

• 47:23Speaker 02

  Likewise, see ya.

• 47:32Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. And join our free Slack team where you can hang out with Daniel, Chris, and the entire changelog community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.