Fine-tuning vs RAG

In this episode we welcome back our good friend Demetrios from the MLOps Community to discuss fine-tuning vs. retrieval augmented generation. Along the way, we also chat about OpenAI Enterprise, results from the MLOps Community LLM survey, and the orchestration and evaluation of generative AI workloads. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Demetrios Brinkmann – X https://x.com/DpbrinkmChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: MLOps Community https://mlops.community/LLM survey report https://mlops.community/surveys/llm/LLMs in Production Event - Part III https://home.mlops.community/public/events/llms-in-production-part-iii-2023-10-03Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm the founder at Prediction Guard and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. And today, Chris, I don't know if you've been listening to the Change Log, our sister podcast. They've been doing these like Change Log and Friends episodes where it's not necessarily like a guest interview, but it's like, hey, let's invite one of our friends on and just talk about cool stuff. And I feel like we've a little bit got like Practical AI and Friends today because we're joined by Demetrios from ML Ops Community,

• 1:24Daniel Whitenack?

  which, you know, you're involved in events and podcasts and reports and surveys. And basically, basically you run the whole AI world. So, you know, welcome and we're glad to have you back as our friend.

• 1:37Chris Benson?

  He's like the deep state, you know? He's the deep state. He's like running everything behind the scenes.

• 1:43Speaker 03

  I am honored to be considered a friend, first off. I just want to say that because I appreciate the amazing stuff that you all are doing here. And of course, whenever I get the opportunity to come and chat with you folks, I am going to jump at it.

• 1:58Chris Benson?

  Excellent.

• 1:59Daniel Whitenack?

  And I know that last time I saw you, I think, was at the one of the recent LLMs in production event, which was super fun. Give us, give us a little sense of like, what has the past few months look like in the ML Ops community and some of the events that you've been, been doing? It seems like, you know, there's so much. I think you had also like in-person things. So like, give us a sense, what's happening?

• 2:30Speaker 03

  Yeah, I appreciate you calling that out. And of course, I appreciate you presenting at the LLMs in production conference. That was a blast.

• 2:40Daniel Whitenack?

  Yeah, it's a good time.

• 2:41Speaker 03

  That event itself, we had two days and each day had three tracks. So there was two full tracks and then one workshop track. And there was over 82 speakers in the whole event. And man, that was a lot of work.

• 2:59Daniel Whitenack?

  Yeah, I bet so.

• 3:00Speaker 03

  Add on to that. So that was the virtual part of it. But add on to that, we had in-person parts of it. So in Berlin, there was a hackathon that we did. And in Amsterdam, we had a meetup. And then in London, we had a watch party. In San Francisco, we had a meetup that happened. And then a hackathon right after, and then a workshop. So there was all kinds of craziness that was going on. And that was just for the LLM in production event. Later on, I mean, we're in now 37 cities around the globe. So wherever you are, you probably have an ML Ops community meetup around you unless we're in like, North Africa and South Africa.

• 3:45Speaker 03

  But and then also even in Lagos, Nigeria, we're in Australia, which is always awesome because I keep threatening to go out there and just pop up at a meetup. Yeah, so that's the in-person stuff. There's people that have just gotten super excited about it and they decided to start a community chapter, which is the unbelievable power of community. I, I am blown away by it every time someone approaches me and says, hey, I want to do something in my city.

• 4:17Daniel Whitenack?

  Yeah, that's so cool. And I've been able to attend a couple in-person events this year related to AI stuff, and I will in the fall as well. I'm curious to know, like, from my perspective, it's cool to see things, I feel, transition a little bit from kind of hypothetical stuff, a lot of discussion, to people talking about like, hey, we did this, this is how we implemented our workflow. Hey, have you tried this? And so I, of course, love those types of conversations. Do you get a similar vibe? Or what, how

• 4:57Daniel Whitenack?

  does the community in terms of generative AI and LLMs, how, how does it seem different now than even like six months ago or something like that?

• 5:06Speaker 03

  Oh, yeah. That's so good, because it does feel like there are use cases that are becoming very clear on what LLM shine in and what they're not good at. And then there's also the stack that's forming. And we did a survey, probably, that's another thing that we did in the community on top of all the fun other stuff. We did a survey and we surveyed people that are actually using LLMs or even not using LLMs. And we asked them, why you're not using them? And we went through a bunch of stuff, or why you are using them, what are some big pain points? And it was becoming very clear that there are certain use cases that people are using LLMs for, and we can get into that

• 5:48Speaker 03

  in a minute. And then there's also this stack that is forming. And the stack was probably the most interesting. I know you guys mentioned it with the A16Z article, and Rajko is one of the authors of that, and he helped me with the report when I wrote it, too. So he's behind the scenes, like moving the puppets, the puppeteer. The cool thing with the stack is you, you kind of have, if I break it down for those who haven't seen the diagram that we put together, it's like you have the foundational model. And then you have some kind of vector database, which is like the hero, the champion in this whole LLM scene. You've got, if you need to do some fine tuning or model building,

• 6:31Speaker 03

  you have that component to it, but we can get into that. Uh, I'm very opinionated about the fine tuning part and I'll tell you why in a bit. And then you have stuff like developer SDKs. And this is, you know, like your Llama index or lang chains. And then on top of that, you have like the monitoring or experiment tracking, prompt tracking, those kinds of things, like a port key, a promptimize port, uh, prompt layer, there's all kinds that are coming out. And so what we didn't have in that moment that I think are starting to emerge more now,

• 7:09Speaker 03

  and I'm really excited about is like, how are people actually evaluating these models? And is it coming with your different tools? Or are you doing extra stuff on top of it? So that was the inspiration behind a whole nother survey that we're doing right now on evaluation.

• 7:33Daniel Whitenack?

  Yeah, I can say personally, I'm doing extra stuff on top. That's my like short and very short answer, which is of course much more involved. But um, yeah, I don't know. What, what is your sense of that? Am I just intuition wise, am I out of the norm or in the norm with that?

• 7:55Speaker 03

  No, you are completely in the norm. And I think the hardest part, and this is why we wanted to do a survey around it is because this is one thing that is super unclear and nobody really knows if they're doing it right. And they don't really know what the best practices are. And so you also don't really know what you're evaluating. Are you just evaluating the model? One thing's for sure, all these benchmarks are complete bull that we all know, right? That is very clear.

• 8:26Daniel Whitenack?

  How do you really feel about it?

• 8:29Daniel Whitenack?

  Yeah, and they are interesting in certain ways, but the types of evaluations that are going on there. So like, I think they serve a place, maybe, but they don't translate into like, okay, now I have this use case, right? If I take the model on the top of that leaderboard, I am very much not guaranteed to have like the, quote, "the best results for my use case," and I think that's what's confusing to a lot of people.

• 9:02Speaker 03

  A hundred percent. That's exactly it, is that these models and the use case that you have, who knows how it's going to match up against one another, right? And then it's not only that, but how are you monitoring or evaluating for toxicity? Or the ability for it to do the one thing that you care about? I mean, I don't care if it's, and also, it just kind of feels like a lot of marketing. At the end of the day, when you see the newest model comes out and everybody loves Sota, Sota, this is, you know, state of the art, Sota, beats ChatGPT on all these different metrics.

• 9:44Speaker 03

  And I just kind of laugh because it feels like I've desensitized to that these days.

• 9:49Daniel Whitenack?

  Yeah, I think also it's kind of often kind of funny to me that even like ChatGPT is being used as a static baseline for these things when it's not even, like ChatGPT isn't a model, right? It's a product that has layers on top of it for, you know, that handles all sorts of things. And so that's another misconception that I've seen is like, well, is it really fair to compare a model's output to the output of a product that has a lot of kind of functionality built around it and with it? Yeah, and that also kind of lets people know, when that's drawn out, it lets people know that, hey,

• 10:33Daniel Whitenack?

  the LLM here is not your application. There's this whole layer on. On top of it, which I know you were talking about retrieval based augmentation or retrieval augmented generation. As we are gearing up for this episode, there's of course, like whatever your opinion about prompt engineering is. There is an engineering element to how you call these models and chain things together. And like you say, evaluate things, validate things, filter things for whether it be toxicity or factuality or, or whatever it is.

• 11:10Daniel Whitenack?

  So there's just so much around that that's not the LLM that I think people confuse those concepts a lot of times.

• 11:20Chris Benson?

  You know, just as a little aside here, listening to you guys talking about this, and you guys are experts at this stuff. And I'm just thinking about all the poor people out there who are listening and maybe aren't at your level. It's a tough thing to try to figure out how to navigate this. When you think about it, you guys are debating this and are not completely in alignment yourself. I'm having a lot of empathy for, for people in the audience who are going, how the hell am I supposed to do this?

• 11:48Daniel Whitenack?

  Well, and then alignments, the other thing.

• 11:50Chris Benson?

  Yeah.

• 11:53Speaker 03

  Another buzzword that tick it off on the buzzword bingo. There you go.

• 11:56Chris Benson?

  There you go.

• 11:58Speaker 03

  Well, it's funny because when you are trying to figure out your use case and how to get the best performance out of an LLM. There are things that you go through, right? There's like almost stages and you figure out the debugging is quite difficult. As you were saying, is it the prompt that's giving me the problems? Or is it that something in my retrieval or the way that I'm creating these vector embeddings? Are those the problems? Like where exactly is my problem? And how do I isolate that so that I can make the whole system better? And that is, again, it goes back to evaluation and evaluating the whole system.

• 12:41Speaker 03

  How do you look at what you're doing as a whole as opposed to just like, oh, cool. There's this model and if I go to a hosted version of it and I ask it if the earth is flat, it tells me yes or no. And most of the time they say yes, which is crazy. It's like, oh, you've been trained on one too many flat earth subreddits.

• 13:02Daniel Whitenack?

  Or they've just seen a lot of answers that are positive. So it seems probable.

• 13:08Speaker 03

  Yes.

• 13:08Daniel Whitenack?

  Yeah, as you're talking, I think one of the things that I've realized is in the A16Z kind of stack, they call this layer orchestration, which I think a lot of these tools are amazing that fit into that layer and the things that they're doing, you know, Langchain, Llama Index, we've mentioned. Sometimes, though, it's just like how rapidly the field is advancing. Like if you import like whatever chain from Langchain and then whatever model, right? And then whatever vector database and then like you put it all together, you run the thing and then you get like an empty string output, right? Like you're saying like where is, where did it go wrong? Right? And how do you like divide and conquer debug that?

• 13:53Daniel Whitenack?

  Right? So I think a lot of what I've, just in my own applications that I'm working on for clients and other things, a lot of times, to be honest, for me, it's a lot simpler to write out my chain of my LLM reasoning in just regular Python logic, add like whatever exception handling I want, like make the call to the vector database like manually and like create some logic around that. So I feel that that sort of Python DIY side, it's like less convenient, but I usually end up getting there still. You know, again,

• 14:34Daniel Whitenack?

  it's part of the maturity of this field, I guess, and partly like how things are advancing quickly.

• 14:41Chris Benson?

  You know, they'll have a fix for that. You know, in the spirit of you may have heard, like, you know, when you have a problem with Facebook, what's the answer? More Facebook? Well, in that spirit, I'm sure Meta will put out debug Llama for you in no time and it will solve all of your problems right there.

• 14:56Speaker 03

  It's funny too, because it feels like that just forces you to stay simple.

• 15:02Chris Benson?

  Yes.

• 15:03Speaker 03

  Again, like going back to the KISS principle and realizing, you know what, maybe I'm trying to over engineer this and I can go far with, maybe I don't even need a vector database, which is kind of blasphemy.

• 15:16Daniel Whitenack?

  How dare you?

• 15:17Chris Benson?

  You did call it a hero and a champion a few minutes ago, I want to point out.

• 15:21Speaker 03

  Yeah, I mean, it is wild that you can go so far. I talk to startups almost every day in my day job, which is not the MO Ops community, and they are, they have real revenue. And a lot of them, I'm always asking them like, oh, so how are you doing this behind the scenes? And a lot of them aren't even using vector databases, but they're, they're like cashing in and they have a product that's working and it's working at scale. And so there is this misconception, I think, sometimes that we need all the bells and whistles. And this stack that I just was talking about and saying that the vector databases are the champions

• 16:03Speaker 03

  that potentially for your use case, like, do you really need it? Because it does add that complexity. And so going back to the tried and true principle, like, just keep it simple.

• 16:26Daniel Whitenack?

  You mentioned that you had some strong opinions related to retrieval and fine tuning. I think this is the time for the hot take officially declaring this practical AI and friends episode, so it's a safe space to declare your hot take.

• 16:45Speaker 03

  Safe space. I love it. Well, I know, I just think that I hear a lot about fine tuning. And I don't know if people who throw around the idea of fine tuning something really understand what you fine tune a model for, especially like LLMs. If it's a stable diffusion model, that's a whole different story. And I think sometimes these diffusion models, they give us the wrong idea of what fine tuning in LLM will do. So undoubtedly you saw the rise of like Lensa or Photo AI or all of these fine tuning.

• 17:27Speaker 03

  Basically what these companies were doing in the background is they were running some kind of diffusion model and you would upload. Your selfie, or your selfie, or a picture of your dog, and you would be able to bring it into the world of AI art. And if you take that concept over to LLMs, you think, oh, well, if I just fine tune an LLM on all of my emails, then the LLM will know how to write emails like me. But it's not like that. There's the misconception that it's not like equal in that regard. Fine tuning, you don't fine tune something so that it can understand you more,

• 18:09Speaker 03

  and you can call it out and say, now write like Demetrius. Because what you want to fine tune for, for that case, let's just be clear, that's where retrieval augmented generation shines. Because you just say, hey, here's a database, or a vector database of all of Demetrius's emails. And the most, you can do some few-shot prompting and say, write like this, here's like five styles of Demetrius writing a response to this. So make a sixth one. And you're golden. You don't need to go through like burning a lot of cash on GPUs, and GPUs are scarce these days, to fine tune some model that may or may not work after you've fine tuned it.

• 18:52Speaker 03

  So I think it's worth calling out like when you should be fine tuning things. Because I don't, I don't want to say like never fine tune it. I just want to say like, I've seen a lot of people talking about it, and also a lot of companies starting that will tote how easy it is to fine tune, and how you should be fine tuning and use your company's data to fine tune. It's like, fine tuning, I think the best way that I heard it talked about was in a recent ML Ops Community podcast that I had with Shaul S. He created RAGs, which is like an evaluation framework for RAGs. That's awesome, by the way, go check it out. But he's also very big in fine tuning, and he is also one of the main people that does the Open Instruct project, which is a whole

• 19:38Speaker 03

  Open source LLM project. And so he was mentioning how you want to fine tune when you have some new function or some way, some new output, something that you need to teach the LLM to do that it doesn't necessarily know how to do. So a perfect example of this is OpenAI's functions. So like ChatGPT or GPT-4 functions. This makes it much easier for you to get very, very clean data or data that's outputted in a certain way, right? And it gives you this structured data and that is a whole reason that you would want to fine tune something.

• 20:21Speaker 03

  Or the other example I think is Lama, the Code Lama. But I think where Code Lama falls down is that if the original base model doesn't see a lot of examples of code, then no matter how much you fine tune it, you're not going to get a good coding model out of it. And some people who are using Code Lama and loving it, like, come tell me because I haven't found anybody yet. So that's my rant about fine tuning versus rags. And I have the last thing that I will say is like, if you're looking at fine tuning and you've been brainwashed by the society out there or the community at large and you

• 21:04Speaker 03

  think you need to do it, like, let's just remember why ML was so hard before LLMs. Like, collecting data is not so easy. Labeling data, cleaning data, all of that stuff is quite difficult and you need to do that if you're going to fine tune, so.

• 21:24Chris Benson?

  I just want to say, I want to remind everyone that Daniel likes to clean data.

• 21:29Speaker 03

  That's something I miss. I feel like I, because of LLMs, I'm not doing it quite as much, which is a hole in my life.

• 21:36Chris Benson?

  I'm determined to remind our audience of that fact. You know, it's just kind of horrifying. So I kind of remind everybody once a year.

• 21:43Speaker 03

  It's therapeutic.

• 21:45Daniel Whitenack?

  I think along with some of what you've expressed, Demetrius, that there's a general misconception about the data that you would use to fine tune an LLM. So I've been in countless conversations where the idea is, oh, well, we're wanting to do like question answering on our documents or something like that, right? And we would love to fine tune a model on our internal company documents. And then it's going to be better at question answering.

• 22:25Daniel Whitenack?

  So what I generally tell those people is like, hey, like when you're fine tuning one of these models, just on that raw, unstructured text, in the best case scenario, what you're creating is a better autocomplete model to autocomplete your type of documents, right? What you're not doing is creating a better question answering model. Those are two way separate things, right? So if you wanted to fine tune a model like that, I'm not, like you're saying, there may be cases where that's useful, maybe it's a domain thing, maybe it's a very, like you want very structured type of output answers, like you're talking about.

• 23:05Daniel Whitenack?

  But in that case, what you want to do, you actually don't want to fine tune just on that raw text data. You want to create your own set of instruction prompts likely that are fed in with various questions and you have the answers and you have everything like set up and you have thousands and thousands of these examples, which that I think when you frame it that way, then people are like, oh, so it's like a lot of work to create that sort of data set. And yeah, like turns out it still is, right?

• 23:41Speaker 03

  Oh, that's so funny. That is so well put too. I love that because that's my rub on fine tuning is that people don't realize how difficult it is and how much effort and how much work goes into it. And that's kind of why Mosaic sold for a billion, you know? Because it's hard. And so companies that actually are doing it, that Mosaic was able to convince they needed to do it, they got a lot of money for it, but that's a, that's a whole different story.

• 24:13Daniel Whitenack?

  Yeah, and maybe for those that are less familiar in our audience, maybe they've interacted with LLMs, they like understand the concept of fine tuning, maybe they're not as familiar with retrieval augmented generation. I know that one of the things you mentioned is the ML Ops community, you're doing a whole course on this, it sounds like. So could you just give like the kind of high level pitch for like retrieval augmented generation? Like, what does that mean to you as a person who is obviously a promoter of this approach and something that people should try as they're getting into these

• 24:56Daniel Whitenack?

  technologies? Kind of maybe that general pitch? And then, and then we can talk about maybe a few more specifics that you'd like to highlight.

• 25:03Speaker 03

  But thank you for bringing that up. I mean, this is the first time that we're actually doing a course. So it is a little nerve wracking and at the same time, super exciting. And so I'm just going to clarify that it is not me leading the course. We got an expert, the expert of experts, who is Raul, who created the course for us. He is an engineer in San Francisco that's been doing this for a long time and he's been doing it at some serious scale. So he knows what's up and he goes and does everything from like the Kubernetes clusters to the N prompts and then monitoring the whole system.

• 25:43Speaker 03

  So again, I'm a big thinking about things in systems, but when it comes to like retrieval augmented generation, let's just go back to what is the hero of our story in this case. And it's not the vector database. I would say that like question answer systems are basically the hello world of working with LLMs these days. That's kind of what it feels like to me. I don't know about you guys, if you have that feeling too, where you want to get your feet wet with LLMs, you probably are going to do some kind of like talk to your data, or hey, if I ask this a question, what responses do I get?

• 26:28Speaker 03

  And in those use cases, the best way to set up and architect your system is through like retrieval augmented generation. Again, Going back to what I was saying earlier, not absolutely needed because I've seen viable businesses built without it, but if you are at that point where you're like, oh, well, this will add to my tool belt, you know, it's like another tool in my toolkit, then what we created was a course that goes over and we wanted it to be something that you don't need to spend like six weeks on. We wanted it to be that you could level up really quickly. And so you go through creating a data pipeline and pre-processing that data,

• 27:12Speaker 03

  then ingesting that data into a vector database, and then you can semantic search for the answers from the questions that you're getting from the end user, and then it compiles a response using an LLM. So that's like the basics of the course. And the reason we wanted to do this was because of a hackathon that we did, actually, funny enough, right before the LLMs in production conference, we did a hackathon in San Francisco, and it was all about how bulletproof your LLM stack is. And so we created a bunch of questions and we gave everyone that was part of the hackathon, all the data from the ML Ops Community

• 27:55Speaker 03

  Slack, which this Slack has been around since 2020, and there's like 17,000 people and it's very active. Like, all the channels are going off every day. It is very, very active. I can't remember how many megabytes there were, but for text data, it was a lot. It was, people were saying like, oh, my God, that's a serious amount of text. So everyone had access to that. And then we would rate everyone's stack on how accurate the answers were from the questions and answers that we gave them. So we basically asked people to build these different QA bots or chatbots, if you would call it that. And then

• 28:37Speaker 03

  At the end of the hackathon, we gave them 100 questions and we saw how accurate were these responses? And so the questions were some things that were from Slack or they were from, they were just random questions about ML and ML Ops. And the best ones were the ones that would give you a accurate answer. And then cite, "Oh, but you know, this is another way of looking at it and here's a thread on it." And so it would go back to the Slack thread. So, all that being said, yeah, we're excited about the course. We're excited to do that. I mean, there's all kinds of cool stuff that I want to do in the community. And the only way that I'm able to do it is that people in the community are participating in this.

• 29:21Speaker 03

  I said Raoul is the guy that created this course. We've got all kinds of other courses in the mix from other community members. So people that are experts on what they feel like they know best, they can propose topics and then we're just putting it on our learning platform.

• 29:38Daniel Whitenack?

  And where do people go to find said learning platform? Because I need to point a couple of my coworkers to it.

• 29:47Speaker 03

  Well, if you just want rants from me about how you shouldn't fine tune, then you can find us on the ML Ops Community Podcast. But the easy one is we have learn.mlops.community. And that should get you to the like, the learning section of our website. And yeah, as I said, it's exciting. It's a little bit like nerve wracking because we plan on doing two styles. This one that we just released is go at your own pace. You get it and then you can go through the lessons and hopefully you can get your company to pay for it. Because it's within the learning budget and it actually is for work, right?

• 30:32Speaker 03

  So that should be an easy sell. But if your boss is not interested in paying for it, just DM me and I will give you some amazing copy that will, you can send to your boss. A nice little email that will hopefully convince. Them to change their mind. But the other pieces that we're going to start doing like cohort based courses. And that is interesting on another level because we've got the whole ML Ops community and we've got everyone that is part of the courses. They can go into a special Slack channel. They can be with the teachers and the teachers assistants and all that fun stuff. I mean, it's not like we're breaking any new ground here.

• 31:14Speaker 03

  Courses are kind of a tried and true method of learning. So this is just us having fun with it.

• 31:21Daniel Whitenack?

  Yeah, that's awesome. And speaking of learning, you already mentioned the surveys that you've been doing. And there's a previous survey that it's already published. I know you're, you're working on the next survey that will come out. But people should look at the survey. We'll, we'll definitely link it in our show notes. And there's a lot of really interesting stuff in the survey. And some of the highlights, I'll just call out a few of the highlights here. And then I'd be curious to know like what stood out to you, Demetrius or, or Chris, whichever one. So some of the highlights, just at a, at a high level. Company use cases like text generation and summarization are useful, but participants are going deeper

• 32:05Daniel Whitenack?

  and exploring a lot of other ways to use LLMs like data enrichment and data labeling augmentation, generation for subject matter experts and other things. It's still unclear. The use of large language models in organization is still unclear due to some high costs and our unclear ROI. They talk about hallucinations, the speed of inference with LLMs being potentially a blocker for certain types of use cases. You talk about the infrastructure. We already talked about that stack a little bit. Some of the things around augmentation and consistency of models. So those are just some of the highlights from the survey, which people can go into a lot more detail on.

• 32:48Daniel Whitenack?

  But I guess one question I would have is as you were building this report. First off, is the whole report generated using an LLM? Because I didn't see that highlighted in the report.

• 33:03Speaker 03

  That would be the most meta thing ever, huh?

• 33:06Speaker 03

  Not the company meta, just the old term for meta. And so that is classic.

• 33:12Daniel Whitenack?

  Yeah.

• 33:14Speaker 03

  And honestly, I tried really hard, but one thing that I did, which I'm going to preface this with, I am not Gardner, and so I did not know how to create reports before I did this. Now I've learned since, I learned what was painful, and I spent months with this data and just tearing it apart, trying to figure out what are some clear signals here. And you can find the signals, but the reason that it was so difficult and that it was a blessing and a curse was every single question, instead of having answers given to you, it was just a free form text box. And so this is the report that we created, but the

• 33:59Speaker 03

  actual raw data, it's linked in the report and anybody can see it. And so I'm a big fan of that because I have my biases. I wrote this report, but the report is for the people that don't have the time or want to go and spend hours upon hours looking through all the raw data. Fine tuning a model on it. Yeah. They're like, give me the TLDR and I'm good. And so that's kind of what I set out to do. And I also wanted to see what are some big things that stood out to me, but not only me. Every time, I think the reason it took me so long to release this was that I would ask a cohort of friends to review it and they would give me feedback

• 34:43Speaker 03

  and I would incorporate that feedback and then be like, okay, I think I can send it off to the designer now and let's get it going. And next thing you know, I would ask another cohort of friends to review it or people in the community and boom, I would have all kinds of new feedback and oh, but you know, I noticed that there was this and people were talking about that. So I did that probably six or seven times and that gave me the confidence that it's still biased, but it's not like as crazy biased as I think it would have been if I just put it out myself. And I got a lot of input from other people. So if anything, it's, it's biased from a lot of different people. And so.

• 35:23Speaker 03

  There's that piece and what we're doing with again, like the evaluation survey, I want to do the same exact thing. I learned my lesson, so it's not only free text form answers that you have now. I kind of put a lot of multiple choice and check all boxes that apply and then also the other at the end. So hopefully that will help me be able to do more Excel fancy math on it and formulas. Cause even with L L Ms, I was even trying all these new ones that, you know, it was use chat G P T in your

• 36:02Speaker 03

  Google Sheets. It didn't work. I spent days trying to ask it questions and it just did not work, man. And I ended up getting kind of frustrated because I spent more time trying to get the L L M to give me some kind of insight than if I just spent the time with the data and got the insight. And I think everybody who has played around with L L Ms has probably had that experience once or twice where it's like, I've been prompting this for a really long time. I wonder if I just sat down and wrote the report or if I sat down and tried to think of things on my own and create something, I could have just done it in the amount of time that I've been prompt tuning this. But anyway.

• 36:46Chris Benson?

  I get a question as I'm looking at it, like obviously it's a population of people like us that are answering the questions because you know, right off the bat, it's like, how many of you are using L L Ms in your company? And it's 61%, you know, which is a pretty high number right there. But I'm curious, do you go through and like what constitutes using an L L M? Can it be as simple as pulling up a prompt for chat G P T and, you know, posing prompts to it? Or does it need to be like?

• 37:13Speaker 03

  Putting in your company data?

• 37:15Chris Benson?

  Right. There you go.

• 37:17Speaker 03

  Oh my God.

• 37:18Chris Benson?

  Everyone owns it now.

• 37:19Speaker 03

  Yeah.

• 37:21Speaker 03

  Oh, that's so funny. I would love to talk about that for a minute, too. But the, uh, yeah, it was, there was only a few people that were just like, oh yeah, I'm just using ChatGPT and I'm going directly to OpenAI's website. That was not the majority. I think it was like one person, if I remember correctly. And don't quote me on that because it's been probably two months since we put this out. And so I can't remember anything since we put it out. But it's more people that are trying to set up systems with LLMs. And so these systems may be the API calls to OpenAI, or they may be hosting their own open source LLMs.

• 38:06Chris Benson?

  Right.

• 38:07Daniel Whitenack?

  And, uh, what's unique or what questions are you intrigued to find the answers to in the upcoming survey that are maybe, maybe there's some commonalities that you'd like to see, you know, carry through the surveys, but what are you most curious about going forward into this next round?

• 38:27Speaker 03

  I'm really fascinated by how many people are using open source versus using OpenAI. And one of the most hated visuals of the whole report is this one on like page 10. And it's talking about who's using OpenAI and what size they are. And people were like, this does not explain anything you're, what you're trying to say. And the visuals do not match up at all. And so again, I am not Gardner. I am one random guy who has never written a report, never did a survey, but for some reason I felt compelled to do this three months ago, five months ago now.

• 39:08Speaker 03

  And I almost regret it, but seeing the final product, I'm very happy that I did it. And so.

• 39:14Daniel Whitenack?

  Yeah, you should be proud. There's always things to look back on, but.

• 39:17Chris Benson?

  Yeah, it's a nice report. It's a good one.

• 39:19Speaker 03

  Exactly. And we were able to move fast on it. And so I'm sure Gardner's going to put something out soon. But that's the beauty of the community that we can move a little bit faster. Anyway, back to this visual that people did not like. And I mean, like, a lot of people did not like it. So the whole idea was, are you using OpenAI?

• 39:42Speaker 03

  And we found that there's a bit of a correlation between people that are in super small startups, like zero to 50, zero, I mean, one to 50.

• 39:55Chris Benson?

  And so. It's an autonomous startup.

• 39:59Speaker 03

  Yeah, exactly. It's so starting, it's so startup-y. So anyway, the one to 50 range, they're not using OpenAI. And the thousand plus are not using OpenAI. But the 500 to 1,000, and the numbers that I'm throwing out here, I didn't preface this because I got into the startup thing in the zero to 50. The amount of employees that you have at your company. So if it's one to 50, then you're not using OpenAI. If it's a thousand plus, you're not using it. At least that's the preliminary data that we saw. And if you're in the middle, then you are.

• 40:42Speaker 03

  And so we had some theories about this and it was like, hmm, I wonder if it's because if you're a startup, you think that you can create a moat by not using it? And maybe your whole business is around using some kind of LLM and creating some kind of difference than OpenAI. And then if you're a larger company, A, this was before the enterprise scam that they've got going on. But that's, again, we can get into that in a minute. And so if you're a larger company, you probably have resources to figure it out yourself and you don't necessarily need to use OpenAI.

• 41:25Speaker 03

  And you're probably less comfortable with your data going outside of your walled garden.

• 41:30Chris Benson?

  I think it's the latter.

• 41:32Speaker 03

  Yeah. If you're in the middle, it's like, let's just go as fast as we can. And so I want to see if that theory, if that holds up in the next one.

• 41:41Chris Benson?

  Yeah. If you're over a thousand people, you have a legal department.

• 41:44Speaker 03

  Yeah.

• 41:45Chris Benson?

  And that's what's inhibiting it.

• 41:47Speaker 03

  Good point, Chris. Good point.

• 41:48Chris Benson?

  Yeah.

• 41:49Speaker 03

  But now, I mean, Chris, you tell me, man, like, do you think that this enterprise play is going to work out? I, do you think people are going to trust them?

• 41:59Chris Benson?

  In terms of Open.

• 42:00Speaker 03

  OpenAI's enterprise?

• 42:02Chris Benson?

  I do think it will work out from a business standpoint for them.

• 42:09Chris Benson?

  I get dangerously close to some conflict of interest here. So I'm going to pass, I'm going to pass on this one.

• 42:16Speaker 03

  The fifth.

• 42:17Chris Benson?

  I'm pleading the fifth. So I'm backing away from this question.

• 42:22Daniel Whitenack?

  It sounds like you're not on the enterprise hype train, Demetrius.

• 42:26Chris Benson?

  I work for a company that's a thousand plus is what I'm saying. So I'm going to back away from that question.

• 42:31Speaker 03

  He's got a legal department that yes.

• 42:33Chris Benson?

  We have a legal department and some of them might listen to this podcast.

• 42:38Speaker 03

  Exactly. I just wonder, I mean, I, I've been memeing with friends about it and I think it's kind of funny how they do say they specifically call out, we're not going to use your data to train any of our models. We're not going to know about any of your data. And I think that the biggest question is like, oh yeah, because they have the best track record of doing what they say.

• 43:05Chris Benson?

  There is a healthy skepticism, I would say, in among large companies on that. Definitely.

• 43:10Daniel Whitenack?

  Yeah. Well, also, I don't know, this is kind of avoiding the question a little bit, but I think that it is related in that you brought up the leaderboards earlier, Demetrius. And I think any company that like goes all in, it's almost like a new version of like vendor lock-in, like we used to talk about where like now you have like model family lock-in where, hey, you know, these models, they're good. Like I'm no doubt, you know, GPT models really good. Are they going to be the models that are going to be best for

• 43:50Daniel Whitenack?

  your use case, either in terms of like output or in terms of the other things that were highlighted in your survey, right? Like latency and resources and like a lot of these practicalities and how you can control them and, and all this stuff.

• 44:07Chris Benson?

  You know what they need, don't you?

• 44:08Daniel Whitenack?

  Yeah, exactly. And so I think there is an element here of like, hey, do I want to go all in on a single model family? Or is my strategy play more to have a bit more of model agnostic approach? Where I can pivot between different models for different uses? Maybe fine tune when I need to, but even if I don't fine tune, I have a lot of potential options to use. And in a privacy concerning way. So yeah, I think that that's another element. However, that works out, it will work out. But I think there's this kind of side element here, which is how the model landscape is evolving versus a single model family is evolving, which is good to highlight.

• 44:54Speaker 03

  So true. And actually, you know, it's funny, like, I don't want to say that there is not an immense amount of value in ChatGPT and GPT-4, because one thing has become very clear after interviewing a ton of people who are using large language models in production. They are able to get up and running and proving value with their LLM so quickly. And I was just talking to Thibault, Thibault, I'm gonna have to check how I pronounce his name. He's French and it's spelled very different than how it's pronounced. And he's running the LLMs at Angel List.

• 45:39Speaker 03

  And I asked him, hey, so are you worried about that vendor lock-in type thing? Because you're only on OpenAI. Have you messed around with even just Anthropic or Coher? And I thought his answer was fascinating because he told me, look, you know what? There are so many other pieces of surface area that I would like to cover. So many other features that I would like to implement with these LLMs and be able to use in our product. That if I'm stuck on one feature and trying to figure out what the best model is, then that's going to slow me down. I just want to go and get as many features plugged in as possible.

• 46:23Speaker 03

  And I know that ChatGPT works really well. And so I'm just going to go as hard as I can and incorporate these features. Because I have a laundry list of them that I want to do. And then, once I get all of that out of the way, then I can start going back and saying, "Okay, let's figure out, should we bring a model in-house? Should we use Claude or something else?"

• 46:48Chris Benson?

  You know, it's really interesting to hear you say that because that's such a startup mentality. You know, we just have to run really fast and get as much done as we possibly can in the shortest possible time. And then you get to that large organization thing and they're worried about the lock-in, they're worried about where their data is going, and they go much slower, you know, as a result of that. It's almost like an inverted, you know, approach based on size of the company and maturity.

• 47:11Speaker 03

  And that's what I was trying to show in this horribly positioned visual graph that you see on page 10 of the report.

• 47:19Daniel Whitenack?

  So we've finally got to the conclusion that the graph, we all agree on the point that the graph is showing. And to leave the graph out of it, I think that that's good. Yeah. Don't look at a graph.

• 47:35Chris Benson?

  I'm sorry, he's laughing because we can all see each other even though this is audio only, and he's laughing at himself.

• 47:40Daniel Whitenack?

  Not to belabor the point, but I think like, you all are exactly right. And I think I see this because almost every lead that's coming into Prediction Guard, just in terms of where people are at, regardless of whether they're a good fit for what we're doing or not, but almost every lead that's coming in, it's almost laughably predictable that they say, "Hey, we've prototyped out something very quick with OpenAI that shows like there's huge value here. Now what do we do?" It's almost every conversation is starting like that. So yeah, I think that there's even like a you know, we can make your graph, I think we should make your graph like sideways and add like

• 48:25Daniel Whitenack?

  a temporal element, make it 3D. Over time, people like that more. Where like, you know, at the beginning of a project, I think a lot of people are doing that, whether they're authorized to do it or not in their organization. And then they get to that point where like, how do we scale that up, especially if we're in this larger organization environment?

• 48:45Speaker 03

  Yeah, it's like, oh, I gotta go present this to the C-suite. We gotta erase any use of sending data outside of our company. We can't tell anybody about that.

• 48:55Chris Benson?

  Yeah, exactly.

• 48:58Daniel Whitenack?

  As we're coming close to the end here of our and friends episode, which I hope is only the second of many times we'll get to hear from you on the show, Demetrius. As you're looking to the next, I don't even feel like we could go to the next year. Like as you're looking to the next couple of months of AI life, what are you hyped about? Like just generally across the industry, like what positive kind of trends are you seeing that give you hope for where things are headed?

• 49:30Speaker 03

  Let's see. That is a great question. Leading question, but great question. It's gotta be the positive side, huh? We gotta stay.

• 49:39Daniel Whitenack?

  End on a positive note.

• 49:40Chris Benson?

  You can dip though, before you get there if you want to though, just for fun. Cause I want to hear what he has to say.

• 49:46Speaker 03

  Well, for those that are just listening, I am wearing tie dye. So it is all peace and love here. And I am excited because right now, anyone who wants to mess around with machine learning and AI, they can. I have seen so many scenarios where a product person has said, you know what? Let's try and throw some AI with this. And they've been able to create an enormous amount of value for their company by adding some features that just

• 50:24Speaker 03

  call chat GPT or Claude or whatever. And that for me is really enticing because the barrier to entry has just been destroyed. You know, it almost was like the last couple of years, the first couple of years of the ML ops community, we were, I mean, we still are, there's still a lot of traditional machine learning where I put it in quotation marks because it's not that old. But there's a lot of really hard stuff happening with quote unquote, traditional machine learning. But with the advent of L L Ms, a lot of that has become really easy.

• 51:05Speaker 03

  And so all these N L P tasks that were really hard up until 2022, they're not as hard. And so you're seeing the creativity of people being able to put L L Ms into their products. I love that. That is something that now I didn't realize how much I enjoyed product and the idea of speaking with product people until Chat G P T came out. Because now I'm like, oh, I want to talk to more product people. The product owners are really great people to talk to because they have these wild ideas and they know how to figure out if this is actually a success or not.

• 51:49Speaker 03

  So there's that piece and dovetailing on that. So we're having another L L Ms in production conference on October 3rd. I don't know if I could told you guys this, but definitely come and I will explain why. I'll try and sell it to you as much as possible right now. But I wanted to create as many talks from product people as possible. And some of the stuff that we are talking about are like how to build an economical L L M solution and how to prioritize L L M use cases and how to put L L Ms into your product. So those are very much for the product owners and the product engineers.

• 52:33Speaker 03

  And it's because of that, it's really like, I got really excited now that this whole space has been opened up to the product owners. And so can I tell you what you can expect in the? Conference? Of course. Sure. I'm just gonna tell you why it is the greatest conference on the internet right now. Because where else, and Daniel can attest to this, alright? Live music interludes? Yes. Where else can you prompt me? Not in LLM, you can put in the chat what you want me to sing about, and I will sing it in real time, just improvising on my guitar. And last time, we had a whole song about catastrophic forgetting.

• 53:19Speaker 03

  And for those who do not know, that is where, actually, I'm not even sure I fully understand what happens. But basically, like, when you fine tune, it's all that damn fine tuning. When you fine tune, sometimes a model will forget something, because its new data is replaced with, it replaces the old data. But the catastrophic forgetting song was a hit. You've also got some semi-illegal betting going on during the breaks, and you win swag. And then- Semi-illegal betting? There is a gray area, Chris. Depends on what country you're in, alright?

• 54:00Speaker 03

  And so, that you also can expect. I mean, I'm just not sure that I've seen any conference that goes into the amount of technical details that we go into. And I wanna highlight this piece, and we can end with this. It is really hard, but it is very important for me to have a fully diverse field of speakers. And so, I cannot tell you how much work it is. And it frustrates me now that I've looked at other conferences, and I see, it's almost like, oh, these organizers were lazy. You know, because there are amazing-

• 54:45Speaker 03

  People out there from underrepresented groups that are doing some incredible stuff, but you almost have to look a little more because they're not necessarily on the conference circuit. They're busy shipping. Like, let's be honest, they're not out there talking about it. They're actually just doing it. And so I've had to look really hard, but I am very excited about the speakers that we have and the diversity of our speakers. And I think that that's probably like, out of all the things I'm proudest about, that's probably what I'm the most proud about.

• 55:22Daniel Whitenack?

  That's awesome. Yeah, I can't wait. And you said it was October 3rd? Is that right? October 3rd. Awesome.

• 55:31Speaker 03

  I'm gonna be, so we have one sponsor for this event and they rented a whole studio in Amsterdam and Amsterdam's like four hours from where I live. And so hopefully, you know, everything goes all right. I don't like eat any mushrooms or anything or smoke too much weed and not show up for the actual event. But in case that does happen, we have shirts. Did you see the shirts, Daniel?

• 55:58Daniel Whitenack?

  I think I've only seen the ones I hallucinate more than ChatGPT.

• 56:02Speaker 03

  That is it. I may live that in Amsterdam. You never know.

• 56:07Daniel Whitenack?

  Oh, how can I get one of those?

• 56:09Speaker 03

  Yeah, I'll, I'll share the link with you. We've got special, it's, they only pop up for sale during the conferences. So you can't get them now, but hopefully about a week before the conference starts, we'll, we'll start selling them again. And yeah, that's it. That's what I've been up to. Not, not too much, you know, just trying to

• 56:28Daniel Whitenack?

  Just keeping, keeping things chill. Yeah.

• 56:33Daniel Whitenack?

  That's awesome. Well, thanks so much, Dimitrios, for joining us. I hope that we see you again here in some number of months. Don't, uh, don't stay away too long and we'll look forward to, to hearing about the results of the survey, the, the events, and I'm sure all the like 15 new things that you're doing next time around that you're not doing this time around.

• 56:56Speaker 03

  Totally. Dude, I got so many ideas. I have so many. I mean, Yeah, I really appreciate you guys letting me come on here and rant about fine tuning and talk about the cool stuff you're doing in the ML Ops community. And I love what you all are doing. So, thank you.

• 57:12Chris Benson?

  It was fun.

• 57:12Speaker 03

  Thanks. We'll see you soon.

• 57:22Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a long time listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.