Stable Diffusion

The new stable diffusion model is everywhere! Of course you can use this model to quickly and easily create amazing, dream-like images to post on twitter, reddit, discord, etc., but this technology is also poised to be used in very pragmatic ways across industry. In this episode, Chris and Daniel take a deep dive into all things stable diffusion. They discuss the motivations for the work, the model architecture, and the differences between this model and other related releases (e.g., DALL·E 2). (Image from stability.ai) Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Announcement blog post https://stability.ai/blog/stable-diffusion-announcementStable diffusion paper https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdfBlog post about the model from Marc Päpper https://www.paepper.com/blog/posts/how-and-why-stable-diffusion-works-for-text-to-image-generation/Stable Diffusion on Hugging Face https://huggingface.co/CompVis/stable-diffusion-v1-4Hugging Face Diffusers library https://github.com/huggingface/diffusersUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  With these technologies, one of the first things that I thought about was how artists, you know, were getting frustrated with the fact that you would have machine learning practitioners come in and creating art with these things and all that. And that's in a very immediate, you can do it today kind of situation. As we've watched these multimodality evolutions coming through these models over the months, it's not hard to envision that at some point down the road, this will move into video and we'll see other modalities being added to it. And as we do that, you're now moving into that creative space that previously it took a great deal of effort. You know, if we're talking about the entertainment industry and

• 0:40Chris Benson?

  movie making and special effects, this could really revolutionize how special effects are cheap and make some, some amazingly phenomenal special effects as we see iterations going forward, become very accessible.

• 1:05Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:41Chris Benson?

  Welcome to another fully connected episode of the Practical AI Podcast. In these episodes, we keep you up to date with everything that's happening in the AI community and take some time to dig into the latest things in the AI news. And we'll share some learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I'm a data scientist with S I L International and I'm joined as always by my co-host, Chris Benson.

• 2:10Daniel Whitenack?

  Who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:14Chris Benson?

  I am doing very well, Daniel. Having a good day. Gosh, we got cool stuff to talk about today.

• 2:18Daniel Whitenack?

  Yeah, but the biggest question, though, did you watch Rings of Power?

• 2:23Chris Benson?

  So this is the conflict in my family because I mentioned in the last episode, you know, that I was, but I'm waiting. I'm being a good husband and a good dad till they're ready. Okay, okay.

• 2:37Daniel Whitenack?

  I won't give any spoilers and I, I probably shouldn't on the podcast anyway, but Chris and I, for our listeners are both big Lord of the Rings fans, so.

• 2:47Chris Benson?

  Thanks for torturing me here at the beginning of the episode.

• 2:49Daniel Whitenack?

  Yeah, no, no worries. Any, anything I can do. Yeah, I won't, I won't indicate one way or the other. So, yeah, I mean, this isn't revealing anything, but I was really interested in and kind of analyzing a lot of the visuals of Rings of Power as I was looking through it. And of course, Rings of Power, Lord of the Rings in general, it's, it's set in a fantasy world of, of Middle Earth. And so there's all sorts of interesting, interesting visuals and creative elements. A lot of them with a lot of effort put in from designers and artists and graphics people. And it got me thinking a lot more about stable diffusion, which is what we're going to talk about today.

• 3:34Daniel Whitenack?

  Because really this model, and it's the latest in a series of models, but this kind of stream of models, these diffusion models are really kind of taking over and dominating a lot of the discussion in the AI community. And Chris and I thought, thought it would be good to, uh, good to spend some time chatting about them in a lot more detail than we have in previous episodes. So if you're wondering more about stable diffusion, what it means, what it is, what it can do, that's what we're going to dig into. Yeah. How have you been, uh, been thinking about stable diffusion? Where it's, where has it been entering into your life, Chris?

• 4:15Chris Benson?

  So it is one of those, you know, we've been talking about. The different, kind of these different disciplines within machine learning, crossing modalities and joining up. And that's been some, we've had a pretty exciting year in terms of what's happened already. And I think for me, as I know I've expressed to you offline, this is the most exciting thing. And not just for what it is, but for what may be to follow. So I think this is, I hope that listeners are as excited as we are, because this is one of those moments that, that I think is going to really turn into something quite wonderful. And it already is looking super cool.

• 4:56Daniel Whitenack?

  Yeah, for sure. So maybe it would just be good to set the stage for what stable diffusion is in terms of like, what it can do and the motivation behind it. Cause it wasn't created in a vacuum, right? This is kind of the latest, the latest model in a series of these so-called diffusion models, which I think primarily are associated with right now, or how they've got the most sort of attention is for text to image tasks. So you put in a text prompt and it will generate an image corresponding to that text prompt. What are some of the interesting ones that you've, you've seen, Chris, or the, these

• 5:41Daniel Whitenack?

  sort of images generated from text prompts that have been interesting for you?

• 5:45Chris Benson?

  I think actually, I think some of the things that we've, we've shared a little bit back and forth and that are in some of these articles are pretty cool. You know, being the geeks that we are and seeing things like Lord of the Rings showing up with blended with Star Wars characters in one of those, you know, there's one that has Gandalf and Yoda mixed together. They're just fun. And so I'm enjoying the creativity out of it. But it's really, it's really like, I can think of so many uses that aren't necessarily just like cool imagery from a creative standpoint that are really functional. And we can get to that later on in the conversation. But this is one of those that has popped up from time to time that has, it kind of has a sense of magic about it. And of course,

• 6:26Chris Benson?

  it's not, I'm sad to say, but it definitely, it definitely has that surprise awe factor and what, and what you're able to do as you look at how the different parts of the system work together. And I know we're going to talk about that kind of workflow in terms of how the model works. But the backend, what arises out of that is, is definitely surprising.

• 6:48Daniel Whitenack?

  Yeah, and I think, like you said, the sort of text to image stuff is maybe the most accessible thing for people to try. And so that's what you've seen most. But I've seen really interesting integrations and demos of the model already, because you can do not only sort of just a raw text to image, but you could do sort of like in painting. So you could freeze a part of the image and fill in the rest, or like recover parts of an image after kind of, or, you know, if you have an image of a street and you want to take this person out, you could kind of remove them and then fill in the gap. All sorts of interesting things like that that you could do as part of the workflow. And then there's also this sort of image to image tasks,

• 7:33Daniel Whitenack?

  kind of doing some sort of translation of image style or something like that. But yeah, a lot, a lot of things that are integrating the stable diffusion model. One of the reasons because it's, it's open and people can access it.

• 7:46Chris Benson?

  Yeah, it's fully open source. And I think going back to what you were just talking about for a second there, I think one of the coolest things about it is you can change the representation that's fed into the diffusion model. So, you know, like, as you said, from an accessibility standpoint, you kind of start with this, you know, writing the text out. And the train model, which has been trained on so many things in human culture and civilization, you know, has these, these great components to draw, to pull from within the train model. But, you know, you mentioned the image to image, and we've seen some interesting things where, you know, they, you know, you can take things out of an image. And I know there are other techniques out there, obviously, for, for doing this. But the representation can be text, it can be

• 8:30Chris Benson?

  images, it can be lots of different things, which really opens up the possibilities. And I think we'll, we'll kind of span all the disciplines that we commonly talk about in the space.

• 8:40Daniel Whitenack?

  Yeah, so to give people an idea of the accessibility, even just this morning, I had a Google Colab notebook open. It did have a GPU on it, but it was just a Google Colab notebook. I used the Hugging Face Diffusers library where you can import the stable diffusion model. There's a pipeline built for, you know, using the pre-trained stable diffusion model. So I'm just counting after my imports. I have one, two, three, four, five, six, seven, eight lines of code to go from text to image. So this sort of, there's two factors here.

• 9:23Daniel Whitenack?

  One is like there's great tooling from Hugging Face, which is something we talk about all the time. So continual great work there. But the other side of it is this is just running in a Google Colab notebook, and I'm able to access it via my browser. I don't have to like spin up an instance in the cloud with a big beefy GPU or set of them. This side of the accessibility, both the open source release of the model and the ability to use the model in a computationally efficient way. Those were two of the sort of big motive, in my understanding, and I should be explicit. I'm not, wasn't, I didn't have anything to do with training this model.

• 10:05Daniel Whitenack?

  But in my understanding from the, the teams that, that train this, which included a sponsor called Stability, that's where it gets its name, stable diffusion. Runway ML was involved, which we've, I think, mentioned on the show here before, that has tools for kind of creative uses of machine learning. And then academic researchers from Ludwig Maximilian University in, in Germany. So this group kind of explicitly set out with motivations around accessibility and specifically with accessibility.

• 10:41Daniel Whitenack?

  More computationally efficient, a more computationally efficient diffusion model, and one that would be explicitly open source. And I think that's why this has exploded is because if people can access something easily, and they don't need really fancy compute to run it, then it's going to kind of spread very quickly, right?

• 11:01Chris Benson?

  Yeah, I mean, it's been noted in multiple places that, you know, if you have a computer with a graphics card that, you know, that's a GPU, that you're probably good to go. It doesn't have to be the latest, greatest thing, and you can... So it really opens up to people, you know, everywhere that can use this. And probably most people that might be interested in it already have the equipment, you know, even without going to a cloud solution like Colab or something, you have it in your house probably already, and you can do it.

• 11:28Daniel Whitenack?

  Yeah, on a laptop with a card or a desktop, or just a cloud instance that's less expensive, right, than trying to do something. I was reading that for other diffusion models, so we should be explicit, too. This isn't the first of these types of models. We already talked about DALL-E 2, which is, has a lot of similarities with stable diffusion, and we'll, we'll kind of point out the differences as we continue the conversation. But also a model that's capable of doing this amazing text-to-image generation, right? And, and these other applications like in-painting and that sort of thing. But it's fairly computationally expensive, and it's not as, as open, right?

• 12:12Daniel Whitenack?

  You have to kind of sign up on a waitlist, get access, use it via API, that sort of thing. And I think I was reading, so for some other diffusion models, I read a one statistic that was like 50,000 samples takes about five days to do inferencing on, on a single A100. So most people don't have access to an A100, and maybe don't want to spend five days waiting around for, for the processing of a bunch of samples. Now, 50K is, is a lot as well. But yeah, so that, that's one just kind of baseline or foundational number that, hey, these things did exist before, but they were extremely computationally expensive.

• 12:54Chris Benson?

  You know, it's just as a, as a kind of a single point that you mentioned about it being open source. We've had, and we've talked about this with previous model releases on the show, different approaches to releasing of different types of models. And, you know, there have been things where there's been concern about how it would be used or security and things like that. And incremental, some things stay proprietary with just kind of a front end interface to it. Other things have been released incrementally where the big model is withheld, but a smaller, reduced functional version is, is offered. And here we are, and we just went through, you know, Dolly, which as you pointed out, you know, it has, has constraints there.

• 13:36Chris Benson?

  And here we are with this open source release that's quite powerful and quite amazing, and yet quite accessible to pretty much anybody who would like to, to start working with it. What, what are your thoughts around, around the fact that, I mean, this is a feeling a little bit more like the open source software world that you and I have both come from, you know, in the past. And how do you think this may change the space going forward? If others as well with, with both this and other releases going forward, we, it tends to be more straight out open source with the level of accessibility. What, how do you, how does that change the space we're in?

• 14:12Daniel Whitenack?

  Yeah, I think that there's a few elements of this. I think it has been interesting last, um, last episode that we, that we had, we talked about these open rail licenses. And, um, one is utilized by stable diffusion. And so there is some explicit things you have to agree to when downloading the model. On hugging face, for example, you have to, you know, click a button that says, I agree to this stuff. And then you can download it and you have to use your hugging face token to download it. But it is open in, in that sense, under, in a sort of unique way. But I think that if we look at models like this and ones that are released open source, I think you saw, you kind of saw in software, I think over time,

• 14:57Daniel Whitenack?

  as it was open sourced, a lot of software applications or kind of specialized software things going from kind of specialized expert groups using them to a general purpose technology that was used and integrated into a whole variety of things that the original creators didn't even have in mind, right? So I think we're in a similar place here where we're going from maybe models that were being experimented with in sort of siloed places. But now, as you were mentioning, there's all sorts of ways you could imagine using this model. And because I can access it and because I can run it without expensive, you know, hardware and because there's good tooling like

• 15:39Daniel Whitenack?

  the diffusers library, which I can pull in and do this in eight lines of code, then who knows how people will use this and sort of hack it in a good way, right? So hacking it for useful, kind of pragmatic purposes.

• 15:55Chris Benson?

  I agree. I'm actually looking forward to seeing, as it really gets out beyond its core community and reaches all those people and people become aware of it, because we're still very early days, it'll be interesting to see some of the ideas that come out of it, both the creed of art that we've seen already, but also some of the kind of innovative, maybe kind of business oriented, you know, novel ways of using it that, that we are not likely to think of today.

• 17:01Daniel Whitenack?

  Okay, Chris, you know, I like to get into the weeds sometimes. I say we just dive into this model and see kind of how it works a bit. We'll kind of take the listeners along with us and go through and figure out how this happens. How do we go from text to image? And also, how is this thing trained?

• 17:23Chris Benson?

  Let's diffuse the weeds.

• 17:25Daniel Whitenack?

  Yes, diffuse the knowledge or whatever. Yeah. And Chris, I think there's certain things to listen for as we go through this process. You and I have talked about some of these building blocks that continually show up, one of them being transformers and the attention mechanism that has been applied. Of course, diffusion models have been applied in a variety of ways, encoder, decoder models, word embeddings or text embeddings, all of these things show up as we go through this. So again, this is not kind of popping up out of nowhere. It's an assembly of things that we've talked about before.

• 18:03Chris Benson?

  Yes. This has been a little bit of a magical past year as we've seen things come about largely from that cross-pollination of different technologies that have arisen on different paths. But now they're getting blended and some pretty cool things are coming out of it.

• 18:17Daniel Whitenack?

  Yeah, so the stable diffusion model, if you were kind of having your mind, and we can't show you a picture because this is an audio podcast, but if you have in your mind going from a text input to an image output, the sort of general process is that that text is embedded into some representation, that embedding plus some noise is then de-noised to an image, and that image is then upscaled or decoded into a larger image that's not compressed. So those are the like general stages of the pipeline. You've got text embedded plus

• 19:00Daniel Whitenack?

  random noise, de-noised, and then decoded or upscaled to an image.

• 19:05Chris Benson?

  Do you want to take a moment and let's just kind of talk about for those who are kind of coming into it? The idea of introducing noise and then de-noising, what do you get out of that productively?

• 19:15Daniel Whitenack?

  Yeah.

• 19:16Chris Benson?

  What's the reason for that in the workflow?

• 19:17Daniel Whitenack?

  Yeah. So I mean, you can think about, it doesn't have to be a text to image model, but this sort of de-noising or diffusion type model is useful because it can take a sort of noisy input and de-noise it. So the sort of bigger idea here is that I could take a set of images in my training set, right? And then introduce noise into those images in a, via a certain steps of noising. And then I could train my model to, in a series of steps, de-noise those images. And so this could be used both for like fixing corrupted images or upscaling images and that sort of thing.

• 20:02Daniel Whitenack?

  So it doesn't have to be for like text to image. But this is the general idea is that you have an, an original output or original, um, original set of images that you can kind of corrupt intentionally and then train your model to de-corrupt those or de-noise them. And then that model can be used to perform that sort of de-noising or, um, or upscaling type of, type of action afterwards.

• 20:33Chris Benson?

  As we talk about, you know, the fact that attention is used in here, and I know in some of the discussions around it, it's referred to as cross-attention. What is cross-attention as a form of attention? Does that just mean different modalities coming in or how, how would you define that?

• 20:49Daniel Whitenack?

  Yeah. So I think it would be good with that to kind of describe the, maybe the overall components or modules of, of this system. So there's, there's three main components of stable diffusion to like make it what it is. The first is text encoder or a language model that takes your text and converts it into an embedded representation and, or encode. That text. The next major component is an auto encoder. We'll come back to that because this is a key piece of what makes stable diffusion different is what they did with the auto encoder. But the the auto encoder, basically, you can think about it as a way to train something to upscale your image.

• 21:34Daniel Whitenack?

  So to go from a compressed image to a non compressed image. And then the third is this diffusion model, which is is a U net model. This is the the type of architecture. It is a U net model. And this is that model that takes a noisy input and then de noises it. So again, the text encoder encodes your text to an embedded representation, just a series of numbers, series of floating point numbers. Your auto encoder is a way to really a way to get to a decoder, which can decompress images or upscale them. And then your diffusion model, which is based on this unit architecture,

• 22:15Daniel Whitenack?

  which takes Gaussian noise or some noise and de noises it to get closer to the text representation that you input. So those are the three main components. And what happens is that we mentioned this diffusion model that takes noise and de noises it to something that's close to your text representation, while somehow you have to combine that noise and your text representation. So if you imagine text comes in to your text encoder, a language model, that's converted to a series of numbers and embedding, a learned embedding for that text. And then that learned embedding is combined with this random noise.

• 23:00Daniel Whitenack?

  And that's where the cross attention happens. So cross attention is this way of mapping, mapping your text representation, your encoded text onto this random noise, which the word that they use for this is condition. It conditions the random noise with your text representation. And that's how the diffusion model, which de noises. That's that's how it knows what it's kind of after. That's how that's how it gets to a semantically relevant image that's relevant to your text input is because it's been combined with your text embedding in this cross attention mechanism and the random noise.

• 23:45Chris Benson?

  And the diffusion model is a form of convolutional model. Is that accurate?

• 23:49Daniel Whitenack?

  Yeah, the diffusion model, the at least the one that was used in this stable diffusion piece is called UNet. It's not, it's used in for other purposes as well, but it it sort of has a series of convolutional layers, one that kind of takes your image and shrinks shrinks the image down in the convolutions and one that does the inverse of that. So this is like a down path, up path thing. And then there's combinations between those two things. But yeah, it's it's it's a series of convolutions that are combined in a certain way, which makes it UNet.

• 24:27Chris Benson?

  You know, it's it's interesting as as you have kind of catalogued these different components in their workflow and all. We have talked about all of these things in previous episodes. These are all existing technologies, but they had they found a way to put them together to a remarkable effect. And I it's it's very interesting that we we keep returning to that cross modality being, you know, kind of the source of of the current wave of creativity in the AI space. And I think this is a great example. Individually, I know what all those things are, would never have imagined putting them together to to achieve this. So it was a pretty, pretty cool way of doing this.

• 25:08Daniel Whitenack?

  Yeah, and I think that the the key piece to emphasize about what was done here is is really with the piece that we kind of glossed over quickly, which is this auto encoder and particularly how they trained both the diffusion model and the auto encoder. So it's not new to use this sort of auto encoder to compress and decompress images. That that's been done before. If you imagine you have Have a model that can encode an image and then decode it. The encoding is sort of like the compressing of that image. The decoding is the decompressing of that image. And so you can train a model, you can train an encoder and a decoder jointly

• 25:52Daniel Whitenack?

  to do that compression and then do a corresponding decompression or decoding. And then the diffusion model sort of operates in that on those compressed images. So this, this is not new, this sort of combination of autoencoder and diffusion model, in my understanding. What is new is that the stable diffusion team, this team from, from Stability and a group in, in Germany, I'll mention some of their names because Rob and Rambock at all are on the paper, we'll link it in the show notes. But the thing that they wanted to do, remember the motivation that they were after was to make a more computationally efficient diffusion model. That was at least one of the accessibility things they were after.

• 26:36Daniel Whitenack?

  And so what they did was instead of jointly training the autoencoder and the diffusion model, they separately trained the autoencoder and the diffusion model. And this does two things, sort of, it separates out the autoencoder and lets you train the autoencoder for what it needs to be good at, which is compressing and decompressing images. But it also means that the diffusion model only operates on these compressed images in the training. And those compressed images require like 64 times less memory for your diffusion model,

• 27:16Daniel Whitenack?

  which is why you can run the stable diffusion model on a consumer GPU card, because they've strategically separated out the training of this autoencoder and the diffusion model, which allows the diffusion model to operate on compressed images, but still allows you to get high quality, upscaled images out because you're using the decoder still.

• 27:41Chris Benson?

  And we've seen the decoder and encoder being used. I mean, I think you see that in typical graphic software.

• 27:47Daniel Whitenack?

  Right. Or machine translation models, all sorts of things. Yeah.

• 27:52Chris Benson?

  It's used often to clean that up. So, so the diffusion model is kind of where, if I'm understanding you correctly, is kind of going through that, that noising and then de-noising. It kind of blends what, what is available from the trained model together. And then in that compressed format, and then when the decoder takes the result of that and kind of up scales it back to the uncompressed model, it kind of, in a very non-technical phrase, it kind of cleans it up and makes it, you know, what it, what it is at that point. Is that close to being how, is that, is that approximately fair?

• 28:25Daniel Whitenack?

  Yeah. So if you can imagine really small images which are generated out of random noise based on the diffusion model, de-noising that noise.

• 28:35Chris Benson?

  Yeah.

• 28:35Daniel Whitenack?

  Then those really small images are then decoded to a larger image, image which is inferred, which uses this separately trained decoder, which was trained in this sort of autoencoder methodology.

• 28:50Chris Benson?

  I have a question, a random question for you. Given that they're training them kind of as these separate components, does that potentially, if I'm thinking in terms of outside of this space and software, we often mix different components together to achieve new things. Do you think that'll help accelerate some of the exploration and experimentation in this by, by keeping those bits separate so that you combine them as you want?

• 29:14Daniel Whitenack?

  Yeah. Well, I think that there's the clear computational advantage, but I think as an additional advantage, basically separating out this encoder or the autoencoder from the diffusion model makes it to where you can use the same autoencoder model for all sorts of different downstream diffusion models. So this is another kind of shift that we've seen in other areas, right? Where a portion of what you're doing is general purpose and then you're kind of bolting on what you need for the downstream tasks that you care about, whether that be image to image sort of tasks or. Text to image task, or maybe even another thing that would be like a text to audio task,

• 29:58Daniel Whitenack?

  or there's all sorts of different things that you could imagine doing downstream. So yeah, I think that this decouples the two. There's a computational advantage. There's also a sort of functional advantage.

• 30:34Daniel Whitenack?

  Well, Chris, I think one last thing to mention in terms of in the weeds stuff is I think it's, it is really interesting to look at how a model was was trained. So it's probably worth mentioning a couple of those things where this model again was trained in two distinct phases. There's this universal auto encoding stage, which is trained once and can be utilized for multiple diffusion models, model trainings downstream. And then there's the second phase, which is actually training the diffusion model. And this model was was trained on approximately 120 million image text pairs.

• 31:17Daniel Whitenack?

  So there were, well, there were 120 million image text pairs from approximately 6 billion image text pair data set. Those, that data set is freely accessible that you can look at that as well. And we'll link it in our, in our show notes. But I think we also talked last, in our last conversation about how it wasn't, I mean, it's expensive, but it wasn't a crazy number to actually train this model. So it took 256 A100s, about 150 K hours, which would kind of equal at least that at market price around 600 K. And I'm getting that from one of the, the team members on, on Twitter. So yeah, pretty, pretty interesting.

• 32:01Daniel Whitenack?

  I mean, I don't know if you have 600 K lying around, Chris, but it's certainly a more accessible number than like, you know, training a model for 500 million, or you know, something.

• 32:13Chris Benson?

  And no, I don't have the pocket change of 600K laying around, but you know, as we're looking at separating these trainings out and the fact that, you know, if you kind of think, you know, we talked a little bit about the idea of the magic arising out of this earlier. And the fact that, you know, you have so much human semantics captured in the diffusion model, you know, in terms of how it was trained. So there are many concepts, you know, we talked earlier about the Gandalf Yoda imagery that we had seen and clearly the training had included, you know, the concept of Yoda and the concept of Gandalf that were combined. As we go forward,

• 32:53Chris Benson?

  is, do you think there is the idea of a, of kind of a diffusion marketplace that arises, both open source and maybe some not, not open source, where depending on the costs that you want and things like that, you can kind of get into the level of sophistication that you can support for your application. Is, do you think that becomes a reality as we talk about making these accessible across a wide range of users and use cases?

• 33:21Daniel Whitenack?

  Yeah, I mean, I think if you draw a parallel with what's happened with other models that have caught on in similar ways, like if you imagine back to BERT and these large language models, part of the magic of those was that the weights were open source. You could pull down a pre-trained version and then fine tune it for a particular task, right? So I have no doubt that, and I think people are looking into this and there's explicit notes on the stable diffusion page about limitations and bias and all that. So you can read that there. But certainly there's bias in the data set on which it was trained. But I think the power

• 34:03Daniel Whitenack?

  comes is if you're able to open source the model in some sort of way with tooling that, that will allow for the fine tuning of it, I'm sure that people will sort of fine tune or create different versions based off of the parent using maybe it's imagery for

• 34:23Daniel Whitenack?

  Particular styles of books or publications or imagery for or in painting for, you know, creative arts or for video processing or for all of these different things. I think people will create their own versions of these and probably some of them will be, those fine tuned kind of purpose built models will be commercially available for purchase. As we've seen with language, certain language models in the marketplace and some will be open source for people's usage, just like we've seen kind of a general purpose BERT. And then we've got like a science document BERT and we've got a legal document BERT and these sorts of things. And those are open, but also there's companies that are,

• 35:08Daniel Whitenack?

  you know, making money because they're processing legal documents with BERT and they're using, they have their own proprietary version, or maybe they're using the open source version and just have good tooling around it.

• 35:18Chris Benson?

  So to extend kind of your answer there just a little bit, you know, one of the things that we often ask guests when we have guests on the show is kind of that, you know, wax poetic a little bit and tell us, you know, kind of where you see some of these things going. And I know that as we were diving into this topic for, for today's show and kind of exploring what we wanted to share with the audience, I could see so many possibilities as could you. And so let's wax poetic for a few minutes on, on where this might go and what might come down the road. You talked a little bit about the marketplace of, you know, that where people can find resources to move forward. With these technologies, one of the first things that I thought about was we were just talking in the last episode or two about

• 36:03Chris Benson?

  how artists, you know, were getting frustrated with the fact that you would have machine learning practitioners come in and, and creating art with these things and all that. And that's in a very immediate, you can do it today kind of, kind of situation. But it's, as we've watched these multimodality evolutions coming through these models over the months, it's not hard to envision that at some point down the road, this will move into video and we'll see other modalities being added to it. I think that would be consistent with the recent history that we've seen. And as we do that, it really, you're now moving into that creative space that previously it took a great deal of effort.

• 36:46Chris Benson?

  You know, if we're talking about the entertainment industry and movie making and special effects, since we started with the Lord of the Rings, this could really revolutionize how special effects are cheap and make some amazingly phenomenal special effects as we see iterations going forward, become very accessible to people at home. You know, you're no longer, you're no longer the big special effects company, but you're, and those companies would have access too. But I could see so many industries. There's obviously, there's security concerns, there's art things, there's business things. What are some of the, what are some of the what ifs that you could see, maybe not just with this particular model, but with what we might expect to see not too far down the road?

• 37:31Daniel Whitenack?

  Yeah, I think the two areas that I'm thinking about are, one, the expansion of modalities, like you talked about. So diffusion models applied to audio, for example, and what, what that means for both things like speech synthesis, or even creative things like music generation, or that sort of thing. So I think that that, that area is quite interesting to me, and I think it will happen. But the other what if in my mind is how this set of technologies will be combined with others that we've seen to be very powerful already, that already exists.

• 38:14Daniel Whitenack?

  So for example, I could have a dialogue system, or I could have prompts that were not created by me and fed into stable diffusion. But what if I create a prompt using GPT-3, or automate the, the sort of dialogue I'm having in a chatbot, right? With language model generated prompts, along with imagery or video that's created using something like stable diffusion. Or, you know, you could even imagine- Imagine creating a storybook with both, you know, language models and sort of visual

• 38:54Daniel Whitenack?

  elements from something like stable diffusion. So I think that the sort of creativity or the uses are also interesting in how they are integrated with existing technologies, both that are AI related and maybe not AI related. So things like chatbots could be driven by an AI model, like a dialogue system model that's state of the art. They could also just be like decision tree based, you know, bots that are rule based. But maybe you integrate visual elements from something like this that in a more controlled way. So I think that this combination of the technology with both existing technologies and other, other language models,

• 39:39Daniel Whitenack?

  other models that are out there is an area that I, I think will kind of expand quite a bit and we'll see some interesting things happen.

• 39:47Chris Benson?

  I think we're looking at the birth of a kind of creative entrepreneurship, being able to, to really take some of this model and other recent models and some of the, the new things that we expect to come in the not so distant future and really have some amazing creative outputs on that. You know, we started with Lord of the Rings and so I'll, I'll, I'll make a suggestion to the Tolkien business, if you will. You know, it would be interesting to see maybe in a few years when they've decided they need to refresh those stories again. Maybe it's done with some of these technologies and it's done, you know, kind of entirely with this set of

• 40:26Chris Benson?

  creative technologies. And to your point, maybe it is released in many, many languages simultaneously, kind of native instead of being translated, you know, in that capacity. And so we can all share in that experience and maybe even variations to adapt to different cultures and different, all sorts of different races, cultures and everything and stories can be, you can take a storyline and make it pretty special in terms of being multimodal itself. So I can, I can imagine a lot of pretty cool things.

• 40:59Daniel Whitenack?

  Yeah, I always think back to that conversation we had with Jeff Adams from the Cobalt Speech Company, talking about how, you know, his vision for the future also was this sort of more holistic, holistic treatment of both language and other things, because language touches everything. So I think that that's some of what you're meaning. While you were talking, just to kind of show you how accessible things are, I typed into "Stable Diffusion: Map of the USA in Lord of the Rings style." And there's definitely,

• 41:40Daniel Whitenack?

  I'm sure you'll recognize, certain elements of that map that I just posted in our Slack channel, Chris, that are Lord of the Rings-esque. So, pretty interesting.

• 41:51Chris Benson?

  I actually, there's a book, I don't remember what it's called right now, but there's a book of Lord of the Rings, of Middle Earth maps. And it does remind, so it's the US, that we're, you know, the eastern and central US that we're looking at here. But it definitely has that Lord of the Rings style going to it. So, yeah, that's, I'm enjoying that.

• 42:12Daniel Whitenack?

  Yeah. In terms of learning resources for people, I think what Chris and I would recommend that you do is just get hands-on with this model. There's ways to do it, even if you don't code, there's ways to do it through the dreamstudio.ai app. Or on Hugging Face, you can actually download the model and use their diffusers library to run the model. If you search for "Stable Diffusion" on Hugging Face, you can find it. Also, another post, so we'll put this in our show notes, but we leveraged a blog post, which was quite useful, written by Mark Popper, that described a lot of the things that we talked about

• 42:55Daniel Whitenack?

  here. So if you want some visuals and that sort of thing to aid your understanding of the model, we'll link that in our show notes. So definitely take a look. It's been fun to diffuse some of these ideas with you, Chris. Enjoyed it. Very much.

• 43:10Chris Benson?

  I did too. I hope our audience enjoyed this as much with these, these shows where we get to explore bold new places. I, I really get excited about. So until next, I'm sure there'll be something super cool coming up that we'll be talking about again. But until then, thanks for joining today, Daniel.

• 43:25Daniel Whitenack?

  Bye.

• 43:34Speaker 01

  All right. That is our show for this week. If you dig it, don't forget to subscribe. Head to practical AI.fm for all the ways. And if practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.