Cambrian explosion of generative models

In this Fully Connected episode, Daniel and Chris explore recent highlights from the current model proliferation wave sweeping the world - including Stable Diffusion XL, OpenChat, Zeroscope XL, and Salesforce XGen. They note the rapid rise of open models, and speculate that just as in open source software, open models will dominate the future. Such rapid advancement creates its own problems though, so they finish by itemizing concerns such as cybersecurity, workflow productivity, and impact on human culture. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Stable Diffusion XL 0.9 https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9OpenChat https://huggingface.co/openchat/openchatZeroscope XL https://huggingface.co/cerspense/zeroscope_v2_XLSalesforce XGen https://huggingface.co/Salesforce/xgen-7b-8k-baseAI is Eating The World https://txt.cohere.com/ai-is-eating-the-worldLLM university https://docs.cohere.com/docs/llmuUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another fully connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest news and also dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I'm a founder and data scientist at Prediction Guard, and I'm joined as always by Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 1:11Chris Benson?

  I'm doing very well, Daniel. It's more interesting times ahead of us. You know, I'm thinking about changing jobs. I'm thinking about like a job title called something like, I don't know, generative juggler. What do you think?

• 1:24Daniel Whitenack?

  Generative.

• 1:25Chris Benson?

  Yeah, because it sounds fun, you know? I mean, I can totally see.

• 1:29Daniel Whitenack?

  Llama wrangler?

• 1:30Chris Benson?

  Oh, I love that. That's perfect for me, too. I'm all over that. Okay.

• 1:36Daniel Whitenack?

  Of course, our listeners know that you're a big animal advocate. What is an animal advocate's perspective on the use of all of this llama, camel, all this sort of different usage of animals? Do you find it fun and interesting?

• 1:50Chris Benson?

  Of course. We should all have animals on the mind all the time. I mean, it makes us better people.

• 1:55Daniel Whitenack?

  Yes.

• 1:57Daniel Whitenack?

  Yeah, I'm traveling. My wife just sent me a picture of our dog laying on the floor in a funny position, looking out of the corner of his eye. So it made me happy going into this recording. So that's always good.

• 2:08Chris Benson?

  That sounds good. You know, the pet pictures are really important when you're traveling. My wife does that with me.

• 2:13Daniel Whitenack?

  Yeah.

• 2:14Chris Benson?

  She'll send a good moment. So you know, in the face of all this technology change constantly coming at us, it keeps our humanity intact.

• 2:22Daniel Whitenack?

  Yeah, yeah. And it is a crazy time in the AI community with. So we use these fully connected episodes to update people on different news and that sort of thing. And one of the things I was realizing this week as we were prepping for this episode is I've even seen there's people, and I think there's a website, talking about the Cambrian explosion of models or the proliferation of models. So, you know, there's just in the past couple of weeks, there's so many different ones that have come out. It is really a proliferation. So I thought it'd be good to highlight a few of those. We can't

• 3:02Daniel Whitenack?

  get to all of them because there's just so many. But one thing as a tip to people, sometimes how I look at this is I'll go to Hugging Face and just go to the models tab. And if you make sure that it's sorted by trending, that's kind of a cool way to see, hey, what's at the top? And it, you know, you can filter by different types of models, but I found it kind of interesting to just look at what's trending overall because as of now on the Hugging Face hub, it's a mix between kind of video generation, image generation, language generation models. And over time, you can see kind of which of those categories is trending up or down. I don't know, there's probably an app that needs to be made to track that sort of thing, but I'll let someone else do that.

• 3:47Daniel Whitenack?

  One of the ones that I wanted to highlight was the new Stable Diffusion XL 0.9. Also, these model names are getting a little bit more complicated over time, I've found. But Stable Diffusion XL 0.9, or SDXL, this is, of course, people probably remember Stable Diffusion. This is an image generation model, so you put in a text prompt and then out comes an image. So, something like astronaut riding a horse on the moon, photorealistic, or something like that, and you get an image out. This one is kind of interesting.

• 4:30Daniel Whitenack?

  I think it was back in April, they announced some kind of private access to this, or beta access. Now, the model is up on Hugging Face. It is available, but under only a research-only license. But the images, I don't know if you've seen some of these, Chris.

• 4:47Chris Benson?

  I'm looking at them now while we're talking.

• 4:49Daniel Whitenack?

  Yeah, you played with Stable Diffusion back, you know, when the previous kind of iteration. What is your thought in terms of the progression of this?

• 4:57Chris Benson?

  Oh, I mean, it was like, I remember when we were playing, we were actually doing it on one of our episodes. And we were coming up raccoons all over the place, I remember, at the time. There were raccoons everywhere, not just us. There seemed to be lots of raccoons coming out of Stable Diffusion. Regardless, I was rather wondering about that. But, no, I'm looking through some of the things, and just like, the imagery has come so far, and the capability and what you can do, and that's just a few months since we were doing that. So, I'm in awe right now as I look at these shots as we're talking.

• 5:28Daniel Whitenack?

  At least the last time I checked, and this might be different now that, if you're listening to this episode, it might be different. But at the time, today, there's a blog post about the release from Stability, and they mentioned that there's going to be a follow-up, more technical, deep dive. I don't know if it's a full paper or just a deep dive post, but there are some general descriptions of how this is working, and... You can dig into it a little bit. So instead of there being sort of one step or a one model kind of situation in this image generation. Apparently this model consists of a two step pipeline. It's still diffusion based, but there's one model that generates,

• 6:11Daniel Whitenack?

  they say latence of the desired output size in the second step. It's specialized to generate this sort of high resolution image. So it's like an image to image model. They combine these in the second stage of the model then kind of adds finer details to the generated output. So that's one interesting thing, which also is kind of interesting. I don't know if you've been following all of the, everyone talking about what's going on, quote unquote, in GPT four. But I think there's a lot of speculation and evidence that that also is a sort of mixture of experts, multiple models together, not just a single model call. So I find this trend kind of interesting.

• 6:54Chris Benson?

  Do you have any thoughts around like, what is the virtue of having the kind of the multi-step, multi-model approach? And is that, do you think that that's likely to be kind of a, a general architecture that we see continually overboard? Instead of just having the model, I mean, even going back to the stable diffusion, I noticed the two models you mentioned. And interestingly, the second model is basically twice the size of the first one in terms of parameters. Any thoughts around the science or math around that or why, why you would take that approach?

• 7:22Daniel Whitenack?

  Yeah, as you scale up your data set and you scale up your compute for a given model size, you're going to get diminishing returns on the performance of that model. So in some ways, given a certain amount of data and a model architecture, you know, what are you going to improve more? You could train for longer, you could train on more data, but at the levels that some of these models are at now, thinking particularly about OpenAI, you know, what more can they do right now with respect to training longer, with the same model architecture, more data? So what's a natural way to improve output, but combining multiple models, a pipeline together? Now, I think that you'll see probably advances in architectures.

• 8:07Daniel Whitenack?

  So different model architectures will continue to come out and maybe break some of that, that trend. Another way that you see this kind of multiple models being applied is in things like the RLHF process, which we talked about on the show, the reinforcement learning from human feedback, which things like this have been around for quite some time. So GANs, for example, include two different models, right? A generative model and the discriminator. These sort of like multi-model workflows that produce a instruction tuned or tuned model out the other end, I think we'll continue to see a lot of that as well, even if the model that's produced or used for inference at the end is a single inference.

• 8:51Chris Benson?

  I got one other question before we dive into the rest of the models to release. You know, one of the things that was notable was, you know, OpenAI kind of commented, you know, after GPT-4 that there was, there was only so much vertical growth you could have there, you know, given the, the data set, you know, basically, I mean, the whole internet, you know, in a model. So there's only, you know, you can't just keep growing them like that. Here we find ourselves in this, you know, what we've kind of described as the proliferation kind of episode, talking about all these models coming out. Do you think part of what we're looking at today is generated by the fact that when you lose the potential for further vertical growth, because you basically used all the data that's out there, does that give all of these other model creators a chance to catch up to some degree?

• 9:34Chris Benson?

  So that you kind of, you had the surging of the leader, but once they hit kind of a barrier there, now you're seeing many, many catching up and comparing themselves to that. Is, is that a fair assessment in terms of kind of what we're looking at now?

• 9:46Daniel Whitenack?

  Yeah, people probably have seen this post that went sort of viral, which is supposedly a leaked document from Google saying, you know, we have no mode and neither does OpenAI. And they, they talk about how basically, I think the phrase they use is open sources eating our lunch. Like, we're not positioned as major players to compete necessarily. So I think that that's where that sentiment is probably coming from, wherever that document originated. That would be the sentiment that's being expressed there. So the ability to have a foundation model is no longer the sort of moat that separates you, because

• 10:29Daniel Whitenack?

  now there's open source models, there's really good open source models, that maybe the base model, let's say the base model doesn't perform as good in a general purpose way as GPT-4 or something like that.

• 10:44Daniel Whitenack?

  Well, the reality is that like in your business environment, you don't need a general purpose model. That's usually not what you need, right? Right. What you need is a model that performs really well for your task. And so in that sense, having a really good open access, whether it's a language model or an image generation model, and then having the ability, which we have now, to adapt or fine tune that model with your own private data, actually, it is kind of part of what we're seeing with this proliferation, I would say. An example of this is the next model I was gonna highlight, which I think is a really good example of this. So I saw this in a tweet. I don't know

• 11:28Daniel Whitenack?

  the actual day it was released, but the open chat models. So if you just go to hug and face slash open chat. So there was a model that kind of outpaced ChatGPT in some benchmarks. So there's a Vakuna benchmark. That model wasn't as open, but these open chat models are the first open models to outpace ChatGPT with GPT 3.5 in this benchmark. And what's interesting is, this is another very much a trend that we're seeing more and more and more of, is actually using the closed proprietary, but really impressively performing models

• 12:07Daniel Whitenack?

  like GPT-4, to actually create data for you to fine tune an open model, which then performs, or maybe performs better than the closed models, at least in certain scenarios. So that's what they did. They used 6,000 conversations generated out of GPT-4 to fine tune this model, which actually outperforms and is available publicly. And this we're seeing over and over. So there's other models, like people are generating this data for less than $1,000, right? They're using the OpenAI API, less than $1,000, to create these models that are really impressive in how they perform. Now, I think there's all sorts of interesting

• 12:49Daniel Whitenack?

  implications of that. And part of me wonders, well, how is OpenAI going to shift its business model to make that sort of thing less? Or other providers of foundation models. One result of this might be that we see providers like OpenAI try to prevent usage like this, where you're just using their API to generate data to create a model that works better for you than using their API. I don't know, we'll see.

• 13:17Chris Benson?

  If you kind of back away for a second and look at the history of this, it's starting to look a lot like the way software development went open source. You know, if you look back around, you know, 2000 or even down into the 90s before that, you saw all of these proprietary programming languages. You know, you'd pay for them, you had to pay for environments and stuff like that. And gradually, open source overtook it. And from my perspective, it's feeling a lot of the same right now as we're making a shift. I will leave it by saying I'm wondering if to that point about your unknown source document earlier, whether or not that's kind of an inevitable destination we're going to.

• 14:00Speaker 00

  This is a change log news break. Can you trust ChatGPT's package recommendations? Hmm, not so much. The team at Vulcan have published a new security threat vector. They're calling AI package hallucination. That sounds good. I'll have that. It relies on the fact that ChatGPT sometimes answers questions with hallucinated sources, links, blogs, and statistics. It'll even generate questionable fixes to CVEs and offer links to libraries that don't actually exist.

• 14:32Speaker 03

  What about the ROUSs? Rodents of unusual size? I don't think they exist.

• 14:38Speaker 00

  Quote, "When the attacker finds a recommendation for an unpublished package, they can publish their own malicious package in its place. The next time a user asks a similar question, they may receive a recommendation from ChatGPT to use the now existing malicious package." End quote. These AI tools like ChatGPT are a real boost to developer productivity, but be careful out there. You just heard one of our five top stories from Monday's Change Log News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 15:27Daniel Whitenack?

  We've been talking about open source models, some things that we talked about even like two months ago on this show, like someday these things will happen. I remember us talking about the graph that I think Clem from Hugging Face posted on Twitter where, you know, you kind of got this linear progression of these closed source models. And then eventually there's this kind of exponential increase of open models that surpasses the performance of the closed models. And I don't know if we're totally in that place yet, but it kind of seems like it's happening to some degree.

• 16:03Chris Benson?

  Yeah.

• 16:03Daniel Whitenack?

  And maybe not in, in certain ways. So I think still for like general purpose, like this model can do whatever you ask it to do. Those sorts of use cases, still the closed models are winning, I think. But like I said, how many business use cases do you need a model that does that sort of thing? The majority, you don't need that. So maybe for the actual proliferation of these models in business use cases, all that really matters is that you can have open models that perform really well for all of those business use cases. And that brings up, of course, a lot of other concerns and practical implications. So

• 16:46Daniel Whitenack?

  open models are great. If they perform better, that's great. But there is a lot of, I mean, it's not only that OpenAI or Coher or Anthropic or whoever who are running these models, that the model is good. They also have a really nice and easy to use API that generally is up, although I think ChatGPT was down the other night. But, uh,

• 17:08Chris Benson?

  Yeah.

• 17:08Daniel Whitenack?

  But yeah, generally is good and like well maintained and all that. You don't get that with these open models. You have to figure that bit out yourself, which has other sort of engineering implications and infrastructure implications.

• 17:21Chris Benson?

  And obviously, uh, you know, going back to someone I know here, uh, there are business opportunities available to help people onboard on those things. So there's a lot, it's so much happening right now. As you said, two months ago, and now it's already changing. And I think people need to get used to the new speed of, of how fast this is happening at this point. Um,

• 17:42Daniel Whitenack?

  Yeah.

• 17:42Chris Benson?

  Later on, I'll come back to that, but uh,

• 17:44Daniel Whitenack?

  One other one that is trending, at least this week on Hugging Face is ZeroScope XL. Version two is the one that I'm looking at, but if you just search for ZeroScope. You'll find it. This is a video generation model, which is pretty cool. So it's video generation. It produces watermark-free videos. And one of the things I find interesting about, so like this model, the ZeroScope model, and also the Stable Diffusion model that we mentioned a second ago, is you can run these on some sort of commodity hardware. Maybe not the cheapest of commodity hardware, but this model supposedly uses 15, a little over 15 gigabytes of GPU memory,

• 18:28Daniel Whitenack?

  rendering 30 frames at 1024 by 576. So that sort of hardware is definitely within reach for a lot of people, even in platforms where you can access some of that for free for at least some time. So yeah, that's one of the things that I find interesting about some of these models as well.

• 18:50Chris Benson?

  That's cool. We're seeing more and more video generation recently. It wasn't long ago, it was earlier this year that we were talking about, you know, kind of moving there as we were coming into 2023 and the fact that we were expecting it, but it hadn't really arrived yet. And now it's already, to your Cambrian point, it has blown up and we're seeing multiple opportunities in terms of these models already in open source versions as well. So how do you, I'm kind of curious, Daniel, how do you as a practitioner, as you're looking at this explosion of these different options coming at you, how do you make an evaluation? I've had people ask me that recently, like so much is happening now. I don't even know how to evaluate one option versus another. Do you have any thoughts on framing that?

• 19:35Daniel Whitenack?

  I think there's a bunch of different axes that you could kind of narrow down your choices along. So let's say that you have a commercial use case, right? That alone is a filter by which you can knock out a huge amount of models. Because just looking at the ones we've listed so far. Zero scope, released under Creative Commons, non-commercial, can't use it. Open chat, released under the Llama license, can't use it for commercial. Stable diffusion XL 0.9, available only for research, can't use it. So, not that you couldn't prototype with it, or that it, versions of this wouldn't be eventually released.

• 20:16Daniel Whitenack?

  Or you could access them in other commercial products, but that kind of does narrow down your cases quite a bit, whereas you look at certain models like the MPT family from Mosaic, released under licenses that allow you to use them for commercial purposes, etc. So, that's an easy one. What is your use case? Are you commercial? Well, that knocks out a whole bunch. Then you have a smaller set, and then I think you need to do a second layer of filtering, which is, think about your practical use of this model. So, for example, let's say that I want to use an LLM to extract a bunch of information from a huge number of unstructured documents. I've got maybe millions of documents, and I want to extract information from them.

• 21:02Daniel Whitenack?

  Okay, well, if each inference is gonna take 20 or 30 seconds for me, and I would need to extract a bunch of information, then that's gonna become a major problem. So then, I need to think about, like, how am I gonna use this? And what are the constraints around, like, the inference speed and the interaction with the model? Or the context length that I'm putting in, in the case of large language models? Do I need to put in a bunch of information or a small amount? And that narrows down to models that are maybe smaller, that can be run faster for inference, or models that support larger inputs, right? So there's those concerns. And then, finally, once you get down to that, let's say you've found one that fits your use case and the constraints that you're working under.

• 21:46Daniel Whitenack?

  Then I think it gets down to this sort of, I guess we call it old-fashioned, oh, it's not that old-fashioned, create yourself a test set. That's still the best way to do this, right? If you have... You know, even 100, 200 examples that you've manually labeled as this is what I would like to go in and this is what I would like to come out. Then you should just check the output and see, you know, what is the accuracy or how does the output compare? How would I rate these as failure or what? That's still the way to do it, right?

• 22:18Chris Benson?

  So the last two minutes is my favorite part of this episode so far. You just put the practical in practical AI in terms of how to go about actually doing this stuff in real life. So much appreciated on that.

• 22:30Daniel Whitenack?

  Yeah, of course. Yeah. Well, one of those things that was mentioned, well, two things. The licensing and the context link that we just talked about. So for those that aren't aware, most of these generative models accept a prompt that is some amount of text. That is kind of auto completed. The result is an auto completion. Most of the large language models that we're dealing with are auto completion models. So they predict next words. The image generation, one of the video generation, when you kind of think of the image or the video as the completion of a prompt as well. Cause you're putting in text. But these models generally have a constraint around the amount of text that you can put in as your prompt.

• 23:13Daniel Whitenack?

  Many of the open models are kind of around 2000 ish tokens of input. So for example, you couldn't put in maybe a whole chapter of a book or something. That's not what you could put in there. There are some like trickeries that have been introduced that take like a model that was trained on a smaller context link. Then kind of extend the context link. But something we've seen in the past couple of weeks is some really seemingly very powerful models that are open and are available for commercial usage.

• 23:48Daniel Whitenack?

  Under their licensing that support a longer context length, one of these being the Salesforce XGen model. So if you go on Hugging Face, just search for XGen. It's a 7 billion parameter model with an 8,000 input sequence length, which is obviously quite a bit more than that 2,000. And one of the things I find interesting about this model as well, kind of fitting with similar trends that we saw in the other model, the 7 billion parameter is kind of an important piece of it because 7 billion parameter, once you kind of go beyond that, you lose some of your ability to deploy models on more commodity hardware.

• 24:31Daniel Whitenack?

  And so that 7 billion is a very strategic number. And that's why you see a lot of 7 billion, 6.9 billion parameter models, is it allows you to kind of run these models on more reasonable hardware, single GPU cards, that sort of thing.

• 24:47Chris Benson?

  What is the technical distinction there when you exceed the 7 billion parameter? Is this something as simple as, you know, kind of like the bus width of data bits going in? Or I mean,

• 24:56Daniel Whitenack?

  It's really the model on fitting into the GPU memory.

• 25:01Chris Benson?

  Got it.

• 25:01Daniel Whitenack?

  And not exceeding it. So unless you want to quantize your model, which we had a whole episode with Neural Magic. So I'd recommend people listen to that. That was really cool.

• 25:11Chris Benson?

  But It was.

• 25:12Daniel Whitenack?

  Unless you're very careful. So quantization means like each of these 7 billion parameters of this model are some sort of floating point numbers. Right? And most of them, so if you load them in, are not used that much, or you don't need sort of full float 32 precision to get good output. So one thing people do is they quantize those down to float 16 or even an eight or four bit or whatever. If you're not really careful about how you do that, or if you don't kind of retrain with that precision, oftentimes you lose a lot of performance. So the thing here is like the 7 billion parameter model with these larger cards now that you can get,

• 25:55Daniel Whitenack?

  single cards, even if it's an A100 or something like that, that's fairly expensive, but it's a single card and it will fit and run one of these models fine. But if you go to like 40 billion parameters, 60 billion parameters, these larger models, now you're kind of getting into multi GPU zone, which makes things much more difficult. So there is a balance here, like you can quantize or optimize the larger models and run them on commodity hardware, but it's not always straightforward how to do that.

• 26:27Chris Benson?

  Gotcha. So in general, you want to get, if you're just a practitioner out there and you're in a small or medium sized business, you're kind of doing it on your own or with your company's stuff, kind of focusing in that five, six, seven billion parameter so that you can be productive as opposed to and not escalate costs out of your control. Is that a fair way of looking at it?

• 26:47Daniel Whitenack?

  Yeah, yeah, I would say basically if you try to work in that seven billion or fewer zone, your life is much easier infrastructure wise, I would say. And that's probably will also change over time, but I think it's the reality now. And one other Salesforce thing, I love it when people post this, that they posted that the training cost was around $150,000 USD on Google Cloud using TPUs. And this model is released under Apache 2, which is cool for me. The other one that I mentioned that was the 8K context length was the MPT 30 billion model, which was released recently. But also note there, the difference in parameter

• 27:30Daniel Whitenack?

  size, right? The XGen model from Salesforce supports that context like that seven billion parameters. And for MPT, you kind of have to go up to that 30 billion, which the MPT models are really great. I love them. I've been using them. But that's just the differentiation. Like you can see why maybe Salesforce XGen is trending because of their focus on this sort of thing.

• 27:52Chris Benson?

  It's more accessible.

• 27:53Daniel Whitenack?

  Yeah.

• 28:12Daniel Whitenack?

  Well, Chris, I think that some of what we've talked about here with the open models is quite interesting because, as we already mentioned, we were talking about this a couple of months ago and thinking, oh, at some point, these open models are going to proliferate and kind of take market share, whatever you want to say, from the closed proprietary models. And I think we are seeing this trend. One of the evidences that I saw in the news this, yeah, I forget if it was this week as we're recording this, but was the acquisition of Mosaic ML. So Mosaic is the one that created the MPT family of models, which, again, I've already said are really great choices if you're looking

• 28:55Daniel Whitenack?

  for some LLMs to play with. But Mosaic was acquired by Databricks or, quote, agreed to join, which, I don't know.

• 29:05Chris Benson?

  The prices on these things are just astronomical, you know, in terms of...

• 29:09Daniel Whitenack?

  It's crazy. Yeah. So, I mean, it's public information, at least this one is public information. So, a total of 1.3 billion for Mosaic ML, which has 62 employees. So, that's $21 million per employee.

• 29:26Chris Benson?

  That's a valuable employee right there.

• 29:28Daniel Whitenack?

  And I was talking to someone about this, and I wasn't in the strategy meetings with Databricks when they're talking about, like, why are we doing this? And how does this position us? But, I mean, think about, I remember Databricks and Spark and Hadoop back in the sort of big data days, leading into data science days, and really focusing on this Spark sort of thing. And think about that use case I gave earlier of the data extraction, right? How are people going to do large-scale data processing in the future? Or large-scale analytics in the future? Well, there will likely always be data warehouses and SQL queries and analytic

• 30:11Daniel Whitenack?

  systems, but there's going to be a large portion of what people are doing analytics wise or kind of big data analysis quote unquote wise by extracting information or doing reasoning with LLMs. The problem with that is for an enterprise, you can't do that with a proprietary closed API because you can't leak your private data to that API. And it's not cost effective to do it anyway, because those charge per token. So how are you going to do that? You're going to proliferate open models that are trained on your own private data and make that easier and easier. And that's what Mosaic's doing, right? So I think, once you kind of

• 30:54Daniel Whitenack?

  think about that positioning, I'm not one to comment on business strategy necessarily, but that's, that's how I've kind of thought about this is yeah, like that's the valuable trajectory of where we're headed.

• 31:06Chris Benson?

  I think it's inevitable, because you know, you run in, I know in business, I have seen many, many cases where these closed models and the licenses surrounding them and the concern about proprietary data, it's a big challenge for people that are trying to get into them as quickly as possible to navigate that through. It throws a whole bunch of legal concern around it. And then you need guardrails, which slows it down. So it makes perfect sense to go and consume and participate in the open community. And I think, just like software, it's inevitable. Business will force us into that direction. So it's not, it's not people doing it out of the goodness of their hearts. It's people doing it for the betterment of their businesses, because it's the only sustainable viable

• 31:48Chris Benson?

  option they have right now. The rug can't get yanked out from under you very quickly.

• 31:52Daniel Whitenack?

  Yeah.

• 31:53Chris Benson?

  So yeah, I think we had no idea a couple of months ago that we were going to have this conversation now, though. I think you and I probably expect things to happen faster than most people out there because we're neck deep in the stuff all the time. But I don't even think we realized just how fast that would happen. I think it's gonna, I'm trying to adjust my own brain for the fact that that will keep happening and will probably accelerate. So we're gonna have a lot to talk about in the days ahead.

• 32:19Daniel Whitenack?

  Yeah, the trend that we're seeing is happening very quick that I thought would take much longer with these open models. So if you look back at these decisions that have been made around like funding new GPU clusters for different startups trying to produce new foundation models. So think about like we saw funding $1.3 billion for Inflection. And the latent space guys in their posts on AI engineers have highlighted some of these for Mistral. And other ones that there's this sort of hoarding of GPUs which is taking place. But those strategy decisions were made, you know, how long ago? Like two months.

• 33:03Daniel Whitenack?

  And now is having that sort of compute infrastructure? Is that the differentiator that is going to make the difference in the business world? I think, you know, I'm not saying it's not going to be good for those businesses. I think probably they'll do great things and be wonderful. But you don't have to now have this sort of large GPU cluster with thousands of GPUs to be a player and create value in the marketplace. So yeah, it's interesting to also see how the dynamics of funding and business strategy are kind of getting intermixed with this rapid proliferation and kind of individual developers,

• 33:45Daniel Whitenack?

  small groups of developers, creating these models like Open Chat and other ones.

• 33:49Chris Benson?

  We're also seeing, you know, going back to our conversation just a moment ago about, you know, seven billion parameters being kind of an over/under decision point because it changes how you're going to go implement with that under seven billion. You're going to have whole industries focusing on things like that because they may be working out on the edge and now they're looking at in the not so distant future. You know, whereas we might've said eventually it will, blah, blah, blah. Now we're like, let's go do LLMs on the edge. Let's go, you know, I can have a GPU that's a single board in whatever edge device we're talking about. And you're going to see whole industries pop up around the ability to do that because you're within that, as you pointed out, the RAM available on the GPU.

• 34:33Chris Benson?

  So that's going to create a whole bunch of new business cases. I think one of the things in my mind right now is it's such an explosive kind of Wild West moment for us here is is always the case, all of the concerns that touch on to these issues such as cybersecurity, such as how it affects your workforce, your productivity, how do you integrate the tooling in, what does it mean for changing business strategy and opportunities? This is all trailing distantly behind, even things like AI ethics, which we've covered quite a bit, you know? How do you, you know, legal frameworks to different countries and different municipalities and such? How do you catch up?

• 35:14Chris Benson?

  All of those things with the fact that we're having this amazing Cambrian explosion in terms of model availability and accessibility and fragmentation into many different use cases that were not thought of two months ago.

• 35:27Daniel Whitenack?

  Yeah. You had highlighted the security side of this. I think it's a really good note because one thing I've seen is you go to, for example, the hugging face LLM leaderboard, right? Like, let's say I'm a person, I want to use the greatest open LLM that I can find. Let's say that for one, maybe a lot of the licensing causes problems for me, but let's say all the licensing problems are equal. Then I go to the leaderboard, I click on some of those that are high up on the leaderboard, and the lack of information around the data processing, the training set, the fine tuning

• 36:06Daniel Whitenack?

  set, the testing and security vulnerabilities, potentially like prompt injection vulnerabilities, all of these things similar to like, you go to GitHub, it's the same with open source code, right? You can search for some tool and it might have a little bit of information in the readme and might say, okay, great, import, solves my problem, and move on. But that's a recipe for introducing vulnerabilities into your code. It's why, it's why products like Snake, which I think is a cool way that I've found for developers to deal with that sort of issue on the code side is, you know, analyzing your dependencies to look for known vulnerabilities in open source projects.

• 36:49Daniel Whitenack?

  But there's nothing like that for LLMs, right? Which of these LLMs has more? Hallucinations than another one, which of them has more toxicity than other ones? Which of them are more prone to prompt injection type of things than other ones? All of that's not on the leaderboard, right?

• 37:08Chris Benson?

  And one of the things to also note here is we're kind of addressing all of the kind of the technical, and I don't necessarily mean like code technical, but you know, things like the legalities and documentation and how do you put in compliance around it? All these things, but I've also noticed, and I'm just kind of mentioning it passing right now because we can't delve into it. There's a huge cultural thing that we're also trying to digest right now. You know, we've talked this year about how this, you know, 2023 is really the year that it's been huge in the public's consciousness. People are using the stuff and they're aware they're using it in many parts of their lives, things like the chat GPT app, you know, everyone's using it on their phones and such these days.

• 37:52Chris Benson?

  I think I've had more conversations in the last three months around people trying to figure out, not just like the business aspect of how do I adopt, but also a lot of fear and a lot of concern about that. And so I think that is becoming part of what we need to be able to think about from a business strategy standpoint, isn't just the cybersecurity and the compliance and all these issues, but also how do you bring the humans along for the ride?

• 38:19Daniel Whitenack?

  Yeah.

• 38:19Chris Benson?

  And get them integrated in as we're making these massive leaps forward. So don't forget your humans in the equation as you, as you try to take advantage of all this amazing LLM goodness.

• 38:31Daniel Whitenack?

  Really good point, and I think some of the writing that I wanted to share as our learning resources at the end of this highlights some aspects of those points that you just mentioned, which is, and I've been trying to tell people this recently, that the LLM or the generative model, the image generation model, in some ways, people are thinking about those things like applications, but really they're tools that are embedded in applications. So you're building an application for real people, users, that might make use of a tool like an LLM or an image generation model, but application development is still part of it and coding and engineering is part of it and security is part of it and.

• 39:15Daniel Whitenack?

  Your UI UX around how you interact with your customers is part of it. So that sort of thinking about these things as embedded tools within an application, I think, is important. It's one thing that Jay Almar, who was a previous guest on our show, he has a really great article, which I would recommend as a learning resource if you're thinking about this sort of how to create competitive advantage or moats with your AI applications. He has an article called AI is Eating the World, and he gives some really good analysis of thinking about, okay, where are their competitive advantages and where aren't there? And he has this really nice diagram of like, models are down here, your application is here,

• 39:57Daniel Whitenack?

  that's where you live, right? The application level above that, maybe in the custom model, like fine tuning level, and then above that, there's all of these things that are unrelated to the model, or not unrelated, but are more so like business concerns, right? How is it distributed? You know, what sort of proprietary or sensitive data are you dealing with? What sort of domain expertise do you have that can be infused in your application? Et cetera, et cetera. Those are the sorts of things that can differentiate you. And I found his writing on this very helpful in framing my mind. So I would recommend people look at that.

• 40:33Chris Benson?

  I like it in addition because it reminds us to stay grounded and be practical. And while the world is changing out from under us in so many ways, the kind of the workflow of how you think about applications and getting productivity out to people is still largely the same. New tools and stuff like that, but the same concerns exist. And so sometimes maybe you take a deep breath and you go, I know how to do this. We've been doing this even before this moment.

• 41:00Daniel Whitenack?

  Good point. I think that's a good, good statement to end with. So thanks for journeying through the Cambrian explosion or proliferation with me, Chris. This has been fun.

• 41:10Chris Benson?

  That's right. It's a space warp here of models flying by us. Good times, Daniel. Thanks a lot.

• 41:16Daniel Whitenack?

  All right. Talk to you soon.

• 41:25Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Brake Master Cylinder, for continuously cranking out the best beats in the biz. That's all for now, we'll talk to you again next time.