The state of open source AI

The new open source AI book from PremAI starts with “As a data scientist/ML engineer/developer with a 9 to 5 job, it’s difficult to keep track of all the innovations.” We couldn’t agree more, and we are so happy that this week’s guest Casper (among other contributors) have created this resource for practitioners. During the episode, we cover the key categories to think about as you try to navigate the open source AI ecosystem, and Casper gives his thoughts on fine-tuning, vector DBs & more. Sponsors: Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Casper da Costa-Luis – GitHub https://github.com/casperdclChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: State of Open Source AI Book - 2023 Edition https://book.premai.io/state-of-open-source-ai/index.html Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO of Prediction Guard, and I'm joined as always by my co-host, Chris Benson. How you doing, Chris?

• 0:55Chris Benson?

  Doing great. How's it going today?

• 0:57Daniel Whitenack?

  It is going awesome. It is, I don't know if you heard, but it is the advent of Gen AI. That is, I'm participating in this advent of Gen AI hackathon with Intel. So people are getting hands-on with a bunch of open source models and different hardware. So I've been in Slack all day answering questions and seeing cool prompts and seeing cool output. So it's just been a ton of fun.

• 1:22Chris Benson?

  What's the most interesting thing that's been in terms of what you've seen so far? I'm just curious before we go in.

• 1:27Daniel Whitenack?

  The first challenge, so we're only a day in. The first challenge was to generate a series of images that kind of go in a sequence, kind of like a comic strip that tell a narrative. But there were some really amazing ones. One's kind of a child growing up and then him having a son, and the images were really compelling and the narrative was really interesting. Yeah, just very, very creative output is something that I've noticed. And today is all about chat. So we're going to see some chatbots popping up in the hackathon and really looking forward to that. Sounds like fun. Yeah, and you know, the Hackathon is all centered around these openly accessible

• 2:10Daniel Whitenack?

  or open source or permissively licensed generative AI models. I think it's really fitting because we have with us Casper, who is a longtime open source enthusiast, but also one of the contributors to the recently published State of Open Source AI book from Pram. So, welcome, Casper. It's great to have you with us.

• 2:31Speaker 03

  Hello. Yes, yeah, great to be here.

• 2:33Daniel Whitenack?

  Yeah, well, I mentioned you're a longtime open source enthusiast. How did you kind of get enthused about open source AI specifically? So what was your own kind of journey into open source AI, maybe kind of leading up to this book and what it's become?

• 2:51Speaker 03

  Hey, that's a good question. I've been around for long enough that AI didn't really exist as a thing back when I got into open source. And it was honestly just purely a hobby. I never even considered it as a career. This was, I must have been, what, 15 years ago or something? And I, in fact, I felt ashamed and embarrassed every time I was working in open source because it felt like I should have been spending that time working on an actual career, right? It felt like it was just a toy. I had a very long commute to get between my home and the workplace on a train, and I was just coding away on my phone. I've actually installed Debian site loaded on my Android. And yeah, that got me hooked on open source purely as a hobby. And I mean, if you contribute enough and you're happy making mistakes in public,

• 3:35Speaker 03

  you know, eventually you build something that loads of people start using. It spirals out of control. Before you know it, it suddenly turns into a career. So I probably entered into this whole space in unconventional way. I didn't intend to, you know, make things that would become famous, but they just wound up becoming famous, which is, you know, quite pleasant. I mean, there's pros and cons because also things that become successful aren't necessarily things that you expect to become successful, right? You can put a lot of effort into something and the world determines it's not really of much value. And so they don't use it. And something you barely put much effort into could explode, right? So that was my sort of background. I'm kind of an academic slant as well. So I did a lot of

• 4:16Speaker 03

  machine vision type things in university. Didn't really want to shoe on myself into any particular one area, though. And also, I didn't want to do pure academia, right? I much prefer industry and having stakeholders and actual products that you build at the end of the day. And I mean, there's pros and cons definitely to both. But yeah, so that's obviously how I wound up, like the rest of the industrial world, seemingly moving towards AI, because that's a buzzword and that's what everyone wants you to work on effectively. So yeah, what started off as initially being machine vision, pre-machine learning, became machine learning type, machine vision type stuff. And now, of course, LLMs are all the rage. So that's why we thought of doing a bit of extra research and try and consolidate all

• 5:00Speaker 03

  of the noise out there and various different blog posts, people effectively shouting into the ether. And we thought we might as well write a book and release some of our research in the wild, get some feedback on that before we actually, you know, start building more things.

• 5:14Daniel Whitenack?

  Yeah, that's awesome. And you even allude to this in in the sort of intro to the book, this sort of fast-paced nature of the field and a lot of people feeling sort of FOMO, like, how do I even categorize all of the things that are happening in open source AI? So maybe one, one kind of general question about the structure of this. Chris and I have worked through some of these categories in various episodes on the podcast, but sometimes it is hard to sort of think about, like, how do you categorize all the things that are happening in open source AI? Because they do go beyond just models, but they include

• 5:56Daniel Whitenack?

  models and a lot of things are sort of interconnected. So how did you kind of, was it organic in how the structure of this book came together? Or how did you come up with the major categories in your mind for what's going on in open source AI?

• 6:09Chris Benson?

  And that's what I was really wondering as well. You literally said, Daniel, exactly what was in my head just now. So I just.

• 6:17Daniel Whitenack?

  Yeah, we're in tune.

• 6:19Speaker 03

  Yeah, no, I mean, it is a big ask because, I mean, my philosophy in general is that the universe exists as a cohesive whole. And, you know, we split it up into different subjects like physics and chemistry and maths, just as a, as a way for humans to actually parse everything that exists in this small little bite-sized chunks. But they're not really independent subjects, right? And the same goes with AI. I mean, there's so many different categories of AI. So, I mean, the nice thing about working in the open source space is that there's lots of different people you can have conversations with, get some feedback. Everyone kind of chipped in their own ideas about how to, let's say, break down a book into different chapters. Ultimately, I think what made the most sense is that it doesn't matter too much what those chapter titles are. It's more about the content within them

• 7:03Speaker 03

  being, let's say, not too repetitive and actually, you know, distilling the ideas that people are talking about. And if you can do that really well, it maybe almost doesn't matter quite how you subcategorize things. But I would say, Filippo Eddersini is probably the one who came up with the actual final, let's say, 10 chapters. But then past that, in terms of, you know, the actually writing those chapters, probably about a dozen people have actually worked on them. Which is, again, really nice that you can do this in the open source space, right? No, no single person is really the author of this book. It seemed fairly obvious to me, based on my own particular passion and research, that licensing should definitely be a chapter. And that's something that developers often neglect, because it's just sort of outside their field of interest and expertise.

• 7:47Speaker 03

  And it's just a bit of red tape that maybe they have to be aware of in the back of their mind. But, um, yeah, so that, I mean, I basically wrote a chapter on licenses, which I think everyone else was happy about. Nobody else wanted to do it. But sure, I mean, it was just effectively topics that we felt are big, major things that there's a lot of confusion over. Maybe we ourselves were confused about it as well. So like, evaluation and data sets, what's the best way to evaluate a model anyway? Right? So that seemed like a big topic. Let's make that a chapter. So it seemed fairly organic coming up with these titles. And of course, as we were writing this, again, it was all fully open source in the whole writing process. We thought maybe we should split up a chapter. So we split off models into two chapters, let's say, one for specifically unaligned models

• 8:30Speaker 03

  versus aligned models. So, yeah, it was an iterative process.

• 8:34Daniel Whitenack?

  On that front, I definitely hear the passion coming through for that sort of licensing element of that. And I see that upfront in the book. And maybe, so I'm also very, very much, like we've mentioned on the podcast multiple times, that people need to be reviewing these things, especially as they see, you know, whatever, 400,000 models on, on Hugging Face and kind of parse through these things. But could you kind of give us maybe the pitch for engineering teams or tech teams that are considering open models, but might not be aware of the kind of various flavors of openness that are occurring within kind

• 9:16Daniel Whitenack?

  of quote, open source AI? Could you just give us a little bit of a sense of maybe why people should care about that? And maybe just at a high level, what are some of these kind of major flavors that you see going on in terms of openness and access?

• 9:33Speaker 03

  Right. Yeah. I mean, I suppose first I should have a disclaimer, which is the quiet part that nobody usually says, which is almost a counterargument. It might not matter because in practice, nobody is going to sue you if you do something illegal unless you're fairly big and famous, right? That's just a harsh truth. And it's very frustrating that, you know, laws and enforcement are, tend to be two separate things. And there is a precedent in law that you're not meant to create a law unless you know definitely you can enforce it. So to a large extent, a lot of these licenses out there are questionable in that regard. The other thing is a lot of these licenses are not actually, let's say, tested in court. They're not actually formally approved by, you know, any government

• 10:16Speaker 03

  or legal process. So it's not necessarily legal just to write something in a license. You should probably be aware of recent developments in the EU, for example, they've proposed the two new laws, the CRA and PLA, two new acts, I should say, that are effectively saying the no warranty clause in all of these open source licenses might be illegal if you are in any way benefiting, let's say monetarily, even if it's indirectly. So you're completely releasing open source things purely for advertising purposes, but you're not directly gaining any money from it. We're still going to ignore the no warranty clause. So yeah, there's interesting stuff in that space. But I would say as a developer, the things that you should be aware of when it comes to model openness is that there's a difference between weights, training data and output. Like those are the three main categories, really.

• 11:01Speaker 03

  So licenses usually make a distinction with, or it's not licenses, this is more about the source. So are the model weights available? That's often the only thing that developers care about in the first instance, because that means they can download things and just play with that, right? But if you actually care about explainability or in any way alignment in order to figure out how you might be able to make a model aligned or unaligned or whatever you want to do with it, you probably do need to know a bit about the training data. So is the training data at least described if not available? And when I say described, I said more than just a couple of sentences saying how the data was obtained, but you know, actual full references and things. So a lot of models are not actually open when it comes to the training data.

• 11:43Speaker 03

  And then of course, the final thing is the licensing around the outputs of the model. Do you really own it? Are you allowed to use it for commercial purposes? And even if you are, it's highly dependent on the training data itself, right? Because if the training data is not permissively licensed, then technically you shouldn't really have much permission to use the output either, right? So I think even developers are kind of confused about the ethics around the permissions. So certainly legally, we're super confused as well.

• 12:12Chris Benson?

  I have two questions for you as follow up, but they're unrelated, but I'm going to go ahead and throw both of them out. Number one, the quick one, I think, is could you define what an aligned model versus an unaligned model is? Just to compare those two for those who haven't heard those phrases. And then I'll go ahead, just as you finish that and say, and what's the reason that I noticed, you know, licenses is addressed at the very top of the book? And is that framing the way you would look at the rest of the book? Or is that more just happen chance that it came there? I was just wondering how that fits into the larger story you're telling.

• 12:45Speaker 03

  Yeah. So for those who don't know, unaligned models, it's effectively. If you train a model in a bunch of data, it is by default considered unaligned. But in the interest of safety, what most of the famous models that you've heard of do, like ChatGPT, for example, is add safeguards to ensure that the model doesn't really output sensitive topics, issues, anything illegal. It's still probably capable of outputting something quite bad, but there are safeguards. And the process of adding safeguards to a model is called aligning a model, as in aligning with good ethics, I suppose that's the implicit.

• 13:23Chris Benson?

  Gotcha. Thank you very much. And then I was just wondering, like I said, the positioning of licensing at the front, is that relevant or is that just happen, Chance?

• 13:32Speaker 03

  We did sort of think of an order of chapters, let's say, and licensing just seemed like a good introduction, let's say, because it's before you get into the meat and the details of actual implementations and where you can download things and where the research is going, let's say.

• 13:47Daniel Whitenack?

  Well, Casper, as you were kind of, you were just describing the kind of framing of the book and also some of these concerns around licensing. I'm wondering if we could kind of take a little bit of a step back as well and think about like, what are some of the main kind of components of the open source AI ecosystem? The book kind of details all of these, but what are some of like the big major components of the AI ecosystem, maybe beyond models? Because people obviously have maybe thought about or heard of generative AI models or LLMs or text-to-image models. But there's a lot sort of around the periphery of those models that make AI applications work

• 14:32Daniel Whitenack?

  or be able to run in a company or in your application or whatever you're building. So could you describe maybe a few of these things that are either orbiting around the models, if you view it that way, or part of this ecosystem of open source AI?

• 14:48Speaker 03

  Sure, I mean, there's huge issues, I would say, regarding, I'd say, performance per watt, effectively electrical watt. There is a lot of development in the hardware space. And, you know, we have new Mac M1 and M2s, which might actually mean you can fairly easily do some fine tuning and, or at least inference on a humble laptop without ever needing CUDA. It seems like there's a lot of shifts and paradigm changes when it comes to the actual engineering implementations. Web GPU is a big upcoming thing, which, I mean, it has technically been going on for a decade or more, but it might actually have reached a point where possibly we can just write code once and it just works in all operating systems on your phone. You can get an LLM just working wherever.

• 15:33Speaker 03

  But yes, I mean, there's effectively a lot of MLop style problems. It's one thing to have a theory of how to actually create an LLM, but quite another thing to actually train a thing, fine tune it or deploy it in a real world application. So there are a lot of competing, let's say, software development toolkits, desktop applications. And I don't think anyone's really settled on one that's, you know, conclusively better than anything else. And really, based on your individual use cases, you have to do an awful lot of market research just to find something that's suited to your use case.

• 16:05Daniel Whitenack?

  I ask this because we've had a number of discussions on the show about sort of training, fine tuning, and then this sort of prompt or retrieval based methodologies. So from your perspective, as someone that's kind of taken survey of the open source AI ecosystem and is operating within it and building things, what is your kind of vision for where things are kind of headed in terms of more sort of fine tunes getting easier and fine tunes being everywhere? Or kind of pre-trained models getting better? And people just sort of implementing fancy prompting or retrieval based methods on top of those? Do you have any opinion on that sort of development?

• 16:50Daniel Whitenack?

  I know it's, it's something that's on people's mind because they're maybe thinking about, oh, this is harder to fine tune, but is it worth it? Because I'm getting maybe not ideal results with my prompting.

• 17:01Speaker 03

  Yeah, no, it makes sense. I would say basically, if you're not doing some form of fine tuning, you're not producing anything of commercial value. Effectively, it's very much like hiring an intelligent human being to work for you without them having any particular expertise. And not even knowing what your company does, right? That's what a pre-trained model is effectively, right? So you do need to fine-tune these things or add some amount of equivalent, anything else that's equivalent to fine-tuning, let's say. In terms of things that actually predate LLMs, I think there's a lot of stuff that is very useful and even maybe far more explainable that people seem to be discounting just because it's easy to get some result out of an LLM just by prompting it. So people view it as good enough and they start using it,

• 17:45Speaker 03

  even though it's maybe not safe, right? So one thing I would really recommend people look at is embeddings. Just by doing a simple vector comparison in your embeddings, you can find, you know, related documents. You don't really need an LLM to drive that because LLM is effectively, instead of explicitly making embedding of your query, you know, conversing your query into a vector and then comparing it to other vectors in your database that correspond to, let's say, documents or paragraphs that you're trying to search through. Your LLM is automatically doing that entire process and it might make mistakes while it does that, right? It's going to paraphrase things which it might get wrong because it can't even do simple, basic mathematics. It doesn't understand logic, right? So, yeah, and whenever it comes to things like, let's say,

• 18:29Speaker 03

  medical imaging where there's a lot of interest in how can we use AI to improve this? People tend to get frustrated with how slow the uptake of AI is, but there's a reason for that, which is explainability is important, right? So the way I see things going is, yes, far more fine-tuning, more retrieval augmented generation type stuff, so RAG stuff, and then also probably push into explainability. I don't really think there's much explainability in LLMs right now in general.

• 18:57Chris Benson?

  Everyone's been so focused on LLMs with large vision models are kind of one of the newer things on the rise. What is your take on on large vision models in the future and how they how they start integrating in? I was just, Andrew and Guy is talking about some of them now and I would love your take on it.

• 19:14Speaker 03

  Sure, I mean, we didn't quite get to covering this in the book. I mean, that's how fast-paced things are. So multimodal things are super interesting. To me, my feeling is that it's effectively gluing together existing models into pipelines. And it hasn't been historically something that I was that interested in because that's more an application and it's not so much something you need to research per se. It's very similar to how the OpenAI people were very surprised that ChatGPT exploded in popularity, even though technically the technology is quite old. It's just, you know, you lower the entry barrier a little bit and then everyone actually starts using it because they can, right? So to me, the multimodal type stuff is similar. It could result in really innovative new companies popping up and new solutions

• 19:58Speaker 03

  that are actually usable by the general public. But in terms of the underlying technology, it doesn't seem that particularly novel to me.

• 20:05Daniel Whitenack?

  As you kind of looked at the landscape of models itself and the licensing of those models, the support for those models and underlying ML ops sort of infrastructure, the support for in underlying kind of like model optimization, you know, toolkits and that sort of thing. Some people out there might hear all of these words like, oh, there's these Llama 2 models and there's now Mistral and then there's, you know, now Yi and like all of these. As you were going through and researching the book and also kind of doing that as an open source community, can you orient people at all in terms of the kind of major model families?

• 20:48Daniel Whitenack?

  So you already distinguished between sort of models and unaligned models. Is there any kind of categories within the models that you looked at that you think it would be good for people to have in their mind in terms of, hey, I have this application or I have this idea for working on this. I maybe want, I'm listening to Casper, I wanna maybe fine tune a model. I've got some cool data that I can work with. Where might be a sort of well-supported or reasonable place for people to start? In terms of, you know, open LLMs or open text-to-image models, if you also want to mention those?

• 21:23Speaker 03

  Sure, yeah. I mean, because there's just a new model basically being proposed every day. I mean, often it's a small incremental improvement over a previous model. So in terms of actually trying to compare them from a theoretical level without looking at their results, there isn't really much to talk about in terms of, you know, large model families. They might be an extra type of layer that has been added to a model in order to give it a new name, let's say. Nothing particularly stands out there. I mean, we do have a chapter on models where we try and address some of the more popular models over time, the proprietary ones and then the open source ones. But I would say nothing particularly stood out to me of there. I suppose the more interesting thing in terms of actually implementing something for your own particular use case

• 22:08Speaker 03

  is starting with a base model that has pretty good performance on presumably other people's data that looks as close as possible to the data that you actually personally care about. So you don't have to wait too long when then fine tuning it on your own data. So for that, I think the most important thing is to take a look at the most up to date leaderboards, right? And there are quite a few different leaderboards out there. We do also have a chapter on that and that was interestingly also a nightmare to keep up to date because the leaderboards themselves are also changing regularly. New leaderboards are being proposed for different things. And take a look at a leaderboard, pick the best model performing there and then start doing some fine tuning. That would be my MO.

• 22:50Daniel Whitenack?

  This kind of gets to one of the natural questions that might come up with a book on this topic, which is things are evolving so quickly. And you mentioned kind of the strategy with this book being to have the book be open source, have multiple contributors. And I'm assuming part of that is also with a goal for it to be updated over time and kind of be an active resource. How have you seen that start to work out in practice? And what is your hope for that sort of community around the book or contributors around the book to look like going into the future?

• 23:29Speaker 03

  Sure, yeah. I mean, like for the evaluation and data set thing, we already have more than a dozen leaderboards, just the names of the leaderboards and links to them and then what benchmarks they actually implicitly include. And yeah, we have comments at the bottom of each chapter, which are driven by GitHub, effectively powered by utterances, which is this integration tool helper. So you don't need to maintain a separate comments platform, let's say. It also encourages people to, you know, open issues, open pull requests. If we've, you know, made any mistake or something is out of date in the book, I mean, we definitely encourage people to fix things or complain about things, which I suppose it's also good from the perspective that nobody can sue you for writing something wrong, because in the first instance, what they really should do is just correct it,

• 24:14Speaker 03

  right? You can't really open a cold case. And for that reason, I think it's also lowering the entry barrier for people to contribute in the first place. They don't have to worry about what they write and whether or not people will disagree, because if they disagree, they can fix it, right? They can start a discussion. Nobody's going to immediately file a lawsuit. And yeah, so we've had quite a lot of interesting discussions already on the individual chapters. The other thing that we, you know, highlight is that as soon as you make a contribution to anything, your name is automatically displayed at the bottom of the individual chapter, as well as, you know, the list of contributors in the front. So yeah, it's a good way to get your name as a co-author in a way of a book. I mean, it's a 21st century book as well, so it lives fully online. Everything that is committed to the repository is automatically built and published immediately.

• 24:59Daniel Whitenack?

  And before we get too much further, some people in the audience might be wondering, like, I mentioned the name of the book, and of course, you can find it by Googling it, I'm sure, but what is the best place to find the book? And then also, as a contributor, you mentioned the links at the bottom of the pages, but I'm assuming there's a GitHub associated with the book. Do you just want to mention a couple ways for people to find it?

• 25:22Speaker 03

  Sure. I mean, the easiest is probably to go to book.premai.io. Yeah, apologies that there's an AI and an IO. It seems to be a thing. But yeah, so book.premai.io, or, I mean, you can also just probably Google Prem AI and you can find our GitHub, which is also github slash prem ai dash io.

• 25:47Speaker 03

  That's a thing. All the AIs and all the IOs. Exactly. We have quite a few repositories that, I mean, some of them were just archived right now because we're constantly running different experiments, changing the entire architecture of the things that we're building. So effectively, our strategy was to first do a lot of research. We didn't mind publishing this for the general public to have a look at, so we released it in a book. And now we're working on actually reading our own book and maybe taking some of its advice and building things. And we have this very much fast-paced startup style. Let's build lots of different things, try lots of different experiments. It's fine if we throw things away.

• 26:42Speaker 00

  This is a change log news break. One year after ChatGPT brought a seismic shift in the entire landscape of AI, a group of researchers set out to test claims that its open source rivals had achieved parity, or even better, on certain tasks. In the linked paper, they provide an exhaustive overview of this success, surveying all tasks where an open source LLM has claimed to be on par or better than ChatGPT. Their conclusion, quote, "In this survey, we deliver a systematical review on high-performing open source LLMs that surpass or catch up with ChatGPT in various task domains. In addition, we provide insights, analysis, and potential issues of open source LLMs.

• 27:27Speaker 00

  We believe that this survey sheds light on promising directions of open source LLMs and will serve to inspire further research and development, helping to close the gap with their paying counterparts," end quote. "It's becoming increasingly clear to me that the data models powering future AI rollouts will be commoditized and democratized, thanks to the competitive nature and hard work of both academia and industry." What a relief. You just heard one of our five top stories from Monday's Change Log News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention.

• 28:11Speaker 00

  Once again, that's changelog.com/news.

• 28:19Chris Benson?

  So Casper, I want to actually do a quick follow up of something you were just saying as we were going into the break. And that was talking about, you know, now we're going to start going through the book ourselves and taking the advice. And that brings up kind of a business oriented question I wanted to ask about it. And so you go out today, you've listened to the podcast, downloaded the book, and there's so much great information in all of these chapters and the comparisons and the what, you know, the different options that each chapter addresses are good or bad and things like that. If someone is just getting going, or maybe they're starting a new project and they're using your book as a primary source to kind of help them make their initial evaluations, how best to use that book?

• 29:02Chris Benson?

  Because there's a lot of material in here in terms, you know, all these different categories. They need to come up with their pipelines and, you know, go back to the leaderboards and select the models that they, the architectures they're just been doing and, and all that. If you were looking at this initially with a new set of eyes, but also having the insight of been one of the authors and editors of this, how would you recommend to somebody that they best be productive as quickly as possible and getting all their questions sorted? How would they go about that process?

• 29:34Speaker 03

  Right. I mean, that's not really a question I was thinking of addressing with, you know, writing a book. So I suppose what you're referring to is a case where someone has a particular problem that they want to solve.

• 29:46Chris Benson?

  Sure.

• 29:47Speaker 03

  And an actual, let's say, business model or target audience. So, I mean, if there's actually something that you're trying to solve, the book hasn't been really written from that perspective. It's more. For a student who kind of wants to learn about everything, right? Or a practitioner who just hasn't kept up to date with the latest advancements in the last year. So the intention is that you can skim through the entire book, really. You're not meant to necessarily know it in advance, which specific chapters might have or spur an innovation or an idea that you can actually implement to help you. In terms of that, I mean, what probably might be more useful is looking through a couple of blog posts that actually take you from, you know, zero to, here is an example application

• 30:31Speaker 03

  that, for example, will download a YouTube video, automatically detect the speech, do some speech to text recognition type things, and then give you a prompt and you can type in a question and it will answer it based on that video, right? We do, in fact, have a few blogs giving you these kind of examples, right? And I think that would probably be more useful if you're actually trying to build a product, to find existing write-ups of people who've built similar things and just follow that as a tutorial, right? The book is more just to get an overview of what's happened in the last year in terms of the recent cutting-edge state-of-the-art, right?

• 31:04Daniel Whitenack?

  Yeah, and I think that's a good call-out. And I think one of the ways I'm viewing this is like, I am having a lot of those conversations as a practitioner with our clients about, you know, how are we going to solve this problem? And something might come up like, oh, now we're talking about a vector database. How does that fit into like, the whole ecosystem of what we're talking about here? And why did we start talking about this? I think that the way that you formatted things here and laid them out actually really helps put some of these things in context for people within the whole of what is open source AI, which is really helpful. So I just mentioned vector databases, which we have talked about quite a bit on the show

• 31:46Daniel Whitenack?

  and is something that, of course, is an important piece of a lot of workflows. But there's one thing on the list of chapters here that maybe we haven't talked about as much on this show, and that's desktop apps, which we've talked a lot about, whether it be like that orchestration or software development toolkit layer, like you're talking about LangChain. And Llama Index and other things, or the models, or the ML Ops, or the vector database. But I don't think we have talked that much about sort of desktop apps, quote unquote, associated with this ecosystem of open source AI. Could you give us a little bit of framing of that topic? Like, what is meant by desktop app here?

• 32:27Daniel Whitenack?

  And maybe highlighting a couple of those things that people could have in their mind as part of the ecosystem?

• 32:34Speaker 03

  Sure, I mean, I should probably quickly say about vector databases. I don't quite understand why there's so much of hype over it. To me, embeddings are actually the important thing. The database that you happen to store your embeddings in is almost like a minor implementation detail. Unless you're really dealing with huge amounts of data, it shouldn't really matter which database you pick, right?

• 32:53Daniel Whitenack?

  Sure, valid point.

• 32:54Speaker 03

  I don't know if you have a different opinion there, though.

• 32:56Daniel Whitenack?

  No, no, I think it's not necessarily a one or the other, but there's use. In my opinion, there's use cases for both, but not everyone should assume that they they fit in one of those use cases. And so they figure out what's relevant for their own problem, so.

• 33:14Speaker 03

  But yeah, in the desktop space, I think maybe there aren't that many developers who talk about it because it's it's almost front end type applications as opposed to getting stuck into the details of implementing, fine tuning, and all that stuff tends to be more backend, let's say, in inverse commerce. So I think that might be one of the reasons why there aren't that many desktop applications being produced, because you kind of need both, both front end and back end. And that maybe naturally lends itself to more the sort of resources that only a closed source company might be willing to dedicate. So maybe that just might be why there's not so much in the open source space. Just takes a lot of development effort. But yeah, there are a few that we do mention in the book.

• 33:58Speaker 03

  There's LM Studio, GPT-4, all, Cobalt. All of them are still very new because, I mean, the thing that they're effectively giving you a user interface for itself is very new. Yeah, I mean, there are some common design principles that are maybe being settled on. You know, you do expect a prompt if you're dealing with language models. You do expect a certain amount of configuration for images. If you're dealing with images, like how many, what's the dimensions? And some basic preprocessing that has nothing to do with artificial intelligence, but you might still expect to see this sort of thing in one place, rather than having to switch between a separate image editor and your pipeline.

• 34:39Speaker 03

  Things that I'm kind of interested in is improving the usability or the end user pleasure, let's say, of using these desktop apps far more so. Can you sort of graphically connect these pipelines together? Like some sort of a node editor, so you can drag and drop models around and like drop their inputs, connect their inputs and outputs to each other so that you can have a nice visual representation of your entire pipeline. But yeah, excited to see what happens in that space. To some extent, I think Prime itself is probably interested in developing a desktop app itself.

• 35:11Chris Benson?

  As you've gone through the process of putting the book together, and I think one of the things that in any project that folks do is kind of like, when to go ahead and put it out there. You know, there's a point where you have to kind of put a pin in it and say, that's this one right now. But our brains never stop working, obviously, on these problems. As to that effect, you get the book out there. Is there anything? And you have conversations like this one that we're having right now, where we're talking about it and you're like, well, it wasn't meant for that, but it was meant for this. Is there anything in your head that you're starting to think, well, maybe that should have been a topic or, you know, something we should have put in the book, maybe next time? With this landscape evolving so fast, where has your post-publishing brain been at on these collection of topics?

• 35:55Speaker 03

  We definitely have yet another 10 more chapters planned. So there's definitely going to be a second edition of this book. Or maybe I should say second volume. It's not even a second edition. It's not correctionist of the kind of thing. It's 10 whole new chapters, yes. Literally V2. That's going to include a lot of interesting stuff about. Things that happened in the latter half of 2023 and hopefully will be developed in 24 as well.

• 36:21Daniel Whitenack?

  Among the things that people are talking about, I mean, we already talked about vector databases a little bit and maybe you're like, you don't see the hype there. What are some things in the ecosystem that you're really, really excited about? And then some things that maybe like, are there any, is there anything else that you're like, ah, like, people are talking about this a lot, but I don't, I don't really see it going anywhere. Any, any hot takes?

• 36:46Speaker 03

  Um, I mean, I probably already covered some of these things, right? What I'm super interested in is fine tuning and lowering entry barriers for other. Things that I'm not all that convinced by are pretending that AI is AGI. They're not the same. I'm sorry. And I don't see it. And I don't trust these models to be more intelligent right now than at best a well-trained secretary. They're considerably faster. So, you know, there are applications where being able to churn through a lot of text really quickly is actually a value in which ACS great apply one of these things. But apart from that, I don't, I don't really buy the hype.

• 37:22Daniel Whitenack?

  Yeah, that's fair, I think. And as we kind of get closer to an end here, I'm wondering, maybe there's some in our listener base that don't have the kind of history in open source that you do. And of course, there's contributions to this book that would be relevant, but there's also contributions within this whole ecosystem of OpenAI, whether it's in the toolkits or it's in the desktop apps or it's in the, in the actual models or data sets or evaluation techniques themselves. For those out there that maybe are newer to open source,

• 38:02Daniel Whitenack?

  do you have any recommendations or suggestions in terms of more people getting involved in open source AI? Obviously, the book is a piece of that because it's open source and people could contribute to that. But maybe more broadly, do you have any encouragement for people out there in terms of ways to get started in contributing to open source AI rather than just consuming?

• 38:27Speaker 03

  Sure, yeah, no, I would say that basically every time you consume, you are ninety percent of the way there to contributing back as well. So you have probably cloned a repository somewhere in order to run some code, right? You probably encountered some issues. And a lot of those issues probably are genuine bugs because these are fast-moving things. People just write some code without necessarily doing full, proper, robust testing. We don't have time to do robust testing, right? A lot of the time, they just throw away experiment-type things. So we're in make and break mode. Yeah, so if you find an issue, rather than quietly fixing it yourself, feel free to open a pull request. And maybe, you know, you're not new, but you're kind of new to this and you're scared of opening a pull request. You're scared that it's not perfect code that you have written as well.

• 39:10Speaker 03

  Well, I mean, bear in mind that the code you fixed was even less perfect, right? And I can say, as an open-source maintainer, I'm always super happy when people contribute anything, whether it's an issue, a pull request. And I think generally, people are far more happy and helpful and kind than you might expect. I would say that when it comes to actually writing code, people aren't necessarily the same trolls that you might find on Twitter, right? Or social media in general, right? These are people who have a mindset that they're thinking about what's being written and they care about the actual project and they don't care about, you know, fighting you on a political front, let's say. So if you are trying to be helpful, that counts a lot more than are you actually helpful in your own opinion or anyone else's opinion, right?

• 39:54Speaker 03

  And even if your pull request doesn't get accepted or merged in, you will definitely have some useful feedback. It might, you know, help you in your own expertise, your own growth as a student or contributor. And I would say, you know, there are definitely times where you might rub somebody up the wrong way and you're not happy with an interaction. But it's such a small percentage of the time that it's definitely worth it.

• 40:17Daniel Whitenack?

  Yeah, well, I think that's a really great encouragement in this conversation with. And of course, Chris and I as well would encourage you to get involved. And even if it's something small initially, get plugged into a community, start interacting and contribute to the ecosystem. Because I would agree with you, Casper, it can be both useful for the projects, but also very rewarding and beneficial for the contributors in terms of the community and the things you learn and the connections that you make and all of that. So yes, very much encourage people to get involved. Also encourage people to check out the Open Source AI book, which we'll link in our show notes.

• 41:01Daniel Whitenack?

  So make sure you go down and click and take a look. It's very easy to navigate to, and you'll see all the categories that we've been talking about through the episode. So dig in, and if you see things to add, definitely contribute them. Appreciate you joining, Casper. Yes, and thanks for sharing the link. You just shared it with me. So book.premai.io/statedofopensourceai with dashes. We'll link it in the show notes as well, so people can click easily. But yeah, thank you so much for joining, Casper, and also thank you for your contributions to the book. We're really thankful that you've done this.

• 41:38Speaker 00

  Sure, yeah. Thanks for having me on.

• 41:50Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now, if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.