The Future of AI Artistry with Suhail Doshi from Playground AI

Multimodal models are making it possible to create AI art and augment creativity across artistic mediums. This week on No Priors, Sarah and Elad talk with Suhail Doshi, the founder of Playground AI, an image generator and editor. Playground AI has been open-sourcing foundation diffusion models, most recently releasing Playground V2.5. In this episode, Suhail talks with Sarah and Elad about how the integration of language and vision models enhances the multimodal capabilities, how the Playground team thought about creating a user-friendly interface to make AI-generated content more accessible, and the future of AI-powered image generation and editing. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Suhail Show Notes: (0:00) Introduction (0:52) Focusing on image generation (3:01) Differentiating from other AI creative tools (5:58) Training a Stable Diffusion model (8:31) Long term vision for Playground AI (15:00) Evolution of AI architecture (17:21) Capabilities of multimodal models (22:30) Parallels between audio AI tools and image-generation

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Hi listeners and welcome to another episode of No Priors. Today, we're talking to Suhail Doshi, the founder of Playground AI, an image generator and editor. They've been open sourcing foundation diffusion models, most recently Playground 2.5. We're so excited to have Suhail on to talk about building this model in conjunction with the Playground community and the future of AI pixel generation. Welcome, Suhail.

• 0:26Speaker 01

  Thanks for having me.

• 0:28Sarah Guo?

  So this is your third company. You started Mixpanel, Mighty, now you're working on Playground. How did you decide this was the next thing?

• 0:36Speaker 01

  I think like back in April of 2022, I think that was just a place that time. It was like GPT-3, 5 kind of came out and then Dolly 2 came out. And I was actually working on the second company, Mighty. And at that time, I was trying to figure out how to like do something with AI inside of a browser address bar. But when I saw Dolly 2 came out, it was just this like very big, strange, eye-opening moment where I think a lot of people didn't think that we'd be able to do like weird, interesting art things so soon. And so, and I think then soon after that, I think Stable Diffusion came out around June or July of that same year and I got early access, maybe a couple of weeks access to early to SD1.4.

• 1:21Speaker 01

  And I just kind of blew my mind what people could do with that. And I just thought that it seemed odd that all of this was being done in a Google Colab notebook. Shouldn't there be like a UI that makes it really easy, that sort of thing?

• 1:34Sarah Guo?

  From the start, were you just thinking we will open source? We will train our own models from scratch? Did you think about other modalities?

• 1:44Speaker 01

  Yeah, I was, I mean, there have been a lot of people that thought I should do like something in music, but I just, I, because music has been like a huge hobby of mine for like six years or so. I like produce music, but I just couldn't wrap my, my brain around like what useful thing I would end up making for people. Although now there's like a lot of very interesting, cool, useful things for music. And then it seemed like a lot of people were very focused on language and I had really enjoyed. I already work with lots of creative tools. Like when I was in high school, I used to make logos or I would make music or whatever. So I was excited that finally I could find something where it was a combination of creativity, tooling, images have like really amazing like built in distribution. People want to share those kinds of things.

• 2:27Speaker 01

  So it ended up just like being this perfect thing that I was excited to work on.

• 2:32Sarah Guo?

  How do you think the like overall landscape for competition is different in language versus images versus music, right? Like, how did you think about in what ways you guys would want to like build advantage and stand out?

• 2:48Speaker 01

  I think with language, there's like, I don't know. I don't know how many language companies there are. You guys would probably know better than me, but it seems like there's like over 20. And then maybe like five or like eight of them have a billion dollars worth of funding. I also didn't want to work on something if there were already extremely passionate people, really working hard at a thing, people that I like really respected that were working on that thing. And so at the time with with images, there was just sort of, I think there was mid journey. There was OpenAI doing some Dolly stuff and then you saw sort of stable diffusion. But for some of these companies, it didn't seem like there was going to be a longstanding concerted effort to keep making them better. It was sort of unclear like who was doing this as like a fun demo versus who was doing this as something they would like spend and invest tons of their time in.

• 3:32Speaker 01

  And so once I had kind of figured out, you know, to what extent OpenAI was going to invest in it or to what extent it seemed like the folks at Stability AI were like sort of focused on like seven different kinds of things. And I just thought like, hmm, there's just not enough people that want to do this one thing and do it really, really great. So I think for me, it was just about what were there enough capable people that wanted to do this?

• 3:53Speaker 02

  Can you talk a little bit about the specific direction you decided to take with Playground as well? And I know you thought really deeply about some of the applications or use cases for it. So I was just curious if you could share a bit more about that.

• 4:03Speaker 01

  Yeah, I think one thing that has sort of been surprising, and it hasn't changed too much, actually, from maybe around June or July of August 2022, was that like a lot of, a lot of people think about like text to image is just, it's right now, it's kind of not, it's not even text to image, it's like text to art.

• 4:21Sarah Guo?

  Sorry, what's the difference?

• 4:23Speaker 01

  The difference is that these models have, they don't, they don't, they haven't quite reached the potential of like what maybe what its utility could be. Right now, for the most part, we take, we formulate a prompt, which is really just a caption of what the image is, and then it diffuses into an image, a set of pixels, but a lot of those pixels are primarily used for art. But what we haven't done is, we haven't done anything beyond that. We haven't really done something like editing, for example. Like, why can't we take an image that you already have, and why can't we like sort of insert something into that with like the correct lighting and stuff? Why can't we like stylize an existing thing? Why is there not a blend of like real and synthetic imagery into a single image that could then be used for a lot more things than just pure art? And

• 5:06Speaker 01

  so right now, it's a lot of just people making art, but not a lot of people, sometimes that reduces its like practicality or its utility.

• 5:13Speaker 02

  Yeah, that makes sense. I think one of the things that you folks did as well that I thought was really interesting is you built your own models, right? Or you trained your own models, and a lot of people in this space just take stable diffusion and fine tune it, or do other approaches like that. And you've gotten, you just launched 2.5, the model is performing incredibly well, like it creates really beautiful imagery, and it's super high quality. And I'm all big curious if you could share a bit more about how you went about training your model and hiring a team specifically for that purpose, and how you thought about it and approached it.

• 5:43Speaker 01

  Yeah, it turns, turns out that, you know, I think when, if, you know, a set of strong engineers, like their first thought is that like you just take a model architecture, you find a lot of data, you get, you fund yourself with enough compute, and you just sort of like throw these things into like a mixture of sorts, and like out comes out like something like Dolly 2 or Dolly 3. It turns out that it's just way more complex than that, and way more complex than I even imagined. I had a sense that it was more complicated than that, but then it's still. Further, it's more complicated than even that. You know, so I think there are a couple of things that we did. One of the things that we were really focused on with that model was that we wanted to see how far we could push the architecture of something that already existed. This was mostly like a test.

• 6:26Speaker 01

  It was a test to see whether, how far we could get as a research team before like the next model change. And so we wanted to take something that we knew was a recipe that worked already, which was Stable Diffusion XL's architecture, which is like a U net, right? And CLIP and the same VAE that Robin Rambock trained. All this stuff. And then we sort of said, okay, what if we try to get something that's just at least better than SDXL, better than the open source model? And we weren't really sure by how much. And so our only goal was to like, just be better and try to deliver on the number one state of the art, open source model that we could release. And, and so we kind of learned two things. One is that when we looked at some of the images from something like SDXL,

• 7:09Speaker 01

  we noticed that there was sort of this like average brightness. It was really confusing. It didn't quite have like the right kind of color and contrast. And in fact, I became so used to this, I became so surprised about the average brightness when comparing it to the images of our model that I thought it was a bug during eval. Like I literally was like looking at the images and I was like, these cannot be the right images. And my team was sort of like, hey, I think you're actually just getting used to the images of the new model. And so we employed this thing called like this EDM formulation, which like samples a noise slightly differently. And it's like a really clever kind of math trick. And there's probably, there's a paper that you could probably read on it. But it's surprising how this like one little like very clever trick what can produce images that have like incredibly

• 7:54Speaker 01

  great color and contrast, but like the blacks are really vibrant with like a bunch of different colors. And this average brightness kind of goes away. So that's like one thing.

• 8:02Speaker 02

  You know, that's a really interesting example of really optimizing for one aspect of creating, you know, aesthetically pleasing imagery. And there's a few other aspects like that. So I'm just curious how much you have to sort of hand tune different parameters versus it's just something that you get as you,

• 8:20Speaker 02

  you know, train a model or post-train a model.

• 8:23Speaker 01

  Yeah, I mean, there's just so many, there's different dimensions of these models. I mean, one is just like, there is understanding of knowledge, but then for like aesthetics, it's really tricky. Honestly, I think like the field itself is just so nascent that like every month, there's like a new trick. There's like a new thing that we all sort of develop or find, find out. I think there are, there's an element of some of that being like a lot of different tricks. Like there's like this new trick that hasn't been like well-employed or well-exploited yet by this guy named Teo Karas. And he basically does this weird thing called Power EMA. Anyway, like basically helps converge training really fast. And, and so that's like one trick. And then there's EDM trick and there's this thing called offset noise. And so there is a lot of tricks for things like color and contrast.

• 9:06Speaker 01

  There's even a trick called DPO for like, that I think works in the language model world and also the image world. Right? So I think there are all these, there are like lots of tricks that sometimes get you like 10, 20, sometimes 2X improvements. But I think the number one trick is like, really just like that last phase of, you know, a supervised fine tune. We're finding like really great curated data. And it's hard to say how much of that is a trick because it's actually just a lot of meticulous work. So I think there's a kind of a combination of some of these things being tricks and techniques. And then there's just like this other thing that's just like really hard meticulous, like there has to be deep care. And with images, maybe more so than language, there has to be like taste and judgment.

• 9:45Speaker 02

  Yeah. How do you think about that from the perspective of the eval you do? Because not that many people have amazing taste aesthetically, right? And some are a little bit curious how you end up determining what is good taste or is it just user feedback? You know, thumbs up, thumbs down. Like, how do you think about that?

• 10:00Speaker 01

  One thing that I've noticed is that every time we do an eval, we try to make our evals better. And we try to try to make them better than the predecessor eval. And so one thing I always notice though, with each successive run is that I find out much later after eval that the model has like all these gaps. So an example of a gap that we recently had was like, we did well on our eval, but one area that I thought we did poorly, that I wish we had done better on was photorealism. Sometimes it would make faces look like they hadn't gone to sleep for three days or something. And so I think that most evals evals in the industry are relatively flawed. Like, a lot of them are doing benchmarks on things that maybe are valuable from the purposes of marketing,

• 10:42Speaker 01

  but are not necessarily well correlated with what maybe users care about. And so, like, a simple example would be with large language models. Like, there's probably a good reason. There's a reason why they're probably good at homework. It's because a lot of the evals are related to things that are related to things that could be homework, like solving an LSAT or a bio test or math test. And so, some of these evals just don't have the necessary coverage. So I think with things like judgment and taste, my feeling is that overall, the evals need to get way stronger. And so, one thing that we tend to do is we just tend to really look at a lot of images across a lot of grids. And we're really being exacting about what thing could be off. But you have to look at thousands of images across lots of different grids,

• 11:27Speaker 01

  across different checkpoints, to basically find and pick sort of release candidates. But I still think that our own evals are not sufficiently strong enough. And they could be better at world knowledge, whether that's its ability to reproduce a celebrity, if that's what you want, or paintings. Sometimes paintings are difficult, or 3D, or illustrations, or logos. Those kinds of things are all like, overall, I think, coverage is pretty tricky.

• 11:54Sarah Guo?

  So, one of the things that you guys do is have voting schemes or user studies within the product itself. So, I don't know if it's grids, but you're asking users to express preferences more so than I think perhaps other research efforts are. Can you talk about, just generally, your data curation strategy? If there's some sort of overall framework, or if community is a big piece of it?

• 12:20Speaker 01

  Generally, we try to keep something very simple, because we know that users are, they have like, they're there to make images. They're not there to necessarily help us label images, or annotate things, or tell us everything about their preferences. And so, we kind of have a very sophisticated process of how we sort of curate images and how we're collecting data from these users to help us kind of rank and make sure we're choosing the right sort of things that we want to curate. And so, I think these things are very, they might seem very simple when you encounter it, but beneath that is something very, very complex. But yeah, it's a little tough to go into it too deeply, because yeah, it does feel like a little bit of a secret sauce, I suppose.

• 13:01Sarah Guo?

  Yeah, well, at least I'll feel good that my guess as to what's interesting is right. Can you just characterize like where a playground does really well today? Like where you stand out and what sort of use cases you're focused on like winning?

• 13:13Speaker 01

  Maybe like probably like number two, I suspect, I guess, like text to art at the moment. Just because we're training these models from scratch, we're getting, we're closing the gap really rapidly as rapidly as we can around all the various like kind of use cases. But I think that we'll probably diverge from some of the other companies in part because we care, I think we start, we're going to care a lot more about editing.

• 13:36Speaker 01

  People just have like a lot of images on their phone or they want to take some image that they love, whether that's made as art or something that they found. And they want to like tweak it a little bit. It's a little annoying that like you make this image and then you can't really change too much about it. You can't change the likeness of it. Maybe there's like a dog or like your face or like something, character consistency issues. It feels a little like a loot box right now. And I think that because it's so much of a loot box, it feels, hmm, it feels like it's too much effort, I guess, to get something that you really, really want. So I think more where we're navigating is like, how can we help you take an image that you love?

• 14:19Speaker 01

  Maybe it's your logo, maybe or incorporate something like your logo or put it in some sort of situation that that you would prefer. Text synthesis is like something that we want to do, for example. So those are some areas that we want to head towards where there's like higher utility and less like you make an image and you just post it to Instagram or something like that.

• 14:40Speaker 02

  Where do you want to take the company and the product over the next few years? Like, what is a long-term vision of what you're doing?

• 14:46Speaker 01

  If people are out there kind of like working on scaling text, we're basically trying to focus on like scaling pixels. And the first area that we're basically started on is just images. And the reason why we're working on images instead of say something like video or 3D or something like that is one part, one issue with 3D is that it tends to be better to work on 3D. If you're like making the content, like you're making Pixar movies, the tools in 3D tend to not like make as much money. And then the other thing with video is like videos is just extraordinarily computationally expensive to.

• 15:17Speaker 01

  To do inference or even training on. And a lot of the video models first train, like pre-trained with like a billion images first anyway, to like have a rich semantic understanding of like pixels. We just think that video, that images is like, like maybe the most obvious place to start because A, the utility is quite low and B, and B, it's like actually somewhat efficient computationally like to do. So, long-term, I think that we're trying to make a large vision model. There's not really like a word, I guess, like we have L L Ms, but I'm not really sure what the word is for vision or pixels if you're trying to make like a, you know, a multitask like vision model. And so the goal would be to try to like do three, three areas of a large vision model would be to be able to create things,

• 16:00Speaker 01

  edit things and then understand things. And so like understanding would be like G P T four V, or if you're using something open source like cog V L M, or there's all these amazing vision language models that are happening. And then editing and creating are things that we've kind of talked about, but it would be really amazing at some point that if you made this like really amazing large vision model that it could do things like, not just like create things like art, but like maybe like help some kind of robot like traverse like some sort of path or like maze. And then there's like things in the middle that are sort of like, you know, maybe you have like a video camera or surveillance system or something and it's like able to understand what's going on in that. But I think right now we're just really focused on graphics.

• 16:40Speaker 02

  And then how do you think about the underlying architecture for what you're doing? Cause you know, traditionally a lot of the models have been diffusion model based and then increasingly, you know, you see people now starting to use transformer based architectures for some aspects of image gen and things like that. How do you think about where all this is heading from an architectural perspective? And what sort of models will exist in the next year or two?

• 17:07Speaker 01

  My kind of like controversial take perhaps is that, you know, there's this thing called D I T, which people allegedly believe like Sora is based on and, and then there are variants of D I T. There's this thing called like, I think M M D I T, which I think stable division three is supposed to be based on by that research team, that's stability A I. And my overall feeling is that transformers are definitely, I think transformers are definitely like the right direction, but I don't think that we're going to get a lot of you enough utility if we're not like somewhat trying to figure out a way to combine.

• 17:39Speaker 01

  The great amazing knowledge of like a language model and or and then just like using, you know, something like DIT, which is completely trained from some kind of video caption or image caption to an image because there's not enough like interpretable knowledge, I suppose. Like you're not able to interpret anything about the input, which a language model is really great at. But then there's like these models that are just trained on these captions that emit images. And it's kind of unclear like how we might marry these two things. And so it'd be sure would be nice if like somehow we could combine these two things. So I think the architecture is like mostly going is most likely going to change. I don't think that DIT is like the right architecture, but transformers certainly.

• 18:18Speaker 02

  And just for people on the who are listening, DIT just stands for diffusion transformers. So in case people are wondering.

• 18:26Sarah Guo?

  One belief held by some of the large labs focused mostly on language today is in the end, we end up with like one truly multimodal general model, right? That it is like not like we don't end up with a language model and a video model and an audio model and an image model. It is any modality in giga brain knowledge, reasoning, long context and any modality out. Like, do you do you believe in that worldview? Or how do you see it differently?

• 18:59Speaker 01

  I definitely think the model is going to be multimodal. And in fact, like, that's kind of what I mean about like some of these models that are just like strictly trained through like, you know, a diffusion transformer, like a diffusion transformer that's only taking like caption image inputs is just like completely lacks sort of somewhat some knowledge. And then conversely, if you look at just like the language models, you know, we know that language is at a much lower dimensionality than say like an image, which has like all these pixels that like sort of tell us about lighting or physics or spatial relationships or size and shapes. So, you know, for example, if you were to like, take a glass and like shattered on the floor, and then I asked you to describe it and I asked, and then I described

• 19:41Speaker 01

  it, we would both come up with like completely different descriptions if Elad had to go and like draw it. Right? So we know that like pixels have an enormous amount of high information density compared to language and language is just really between me and you. It's like a compressed way that you and I can converse with each other at a higher somewhat of a higher bandwidth, right? Like we have an abstract view of like those what those words mean. So I think that the models. There has to be something like, language is really great because it's compressed information and then like, and then like, vision is really great because it's so information-rich, but it's been hard to annotate until recently. It's only because vision language models exist that it's now suddenly a lot easier

• 20:24Speaker 01

  to like, sort of label or annotate or understand what's going on in an image. So I think that these two things are very, are going to be very likely married. The only question is, is like, does, does language, to me, it's a kind of a question of, does language, language has this wonderful trait where it's like, you can use language to control things, which is pretty cool because of its low dimensionality. But my question would be like, I wonder if language will hit a ceiling, has like, a lower ceiling than, say, vision? Because it's very easy to get lots of pixel data, and that pixel data is like, very, very high-density.

• 20:57Sarah Guo?

  Very easy to get more pixel, like, additional pixel data to the already-collected data from the internet that's gone into these models.

• 21:05Speaker 01

  Yeah, I mean, there's an assumption, maybe, one assumption I tend to question is like, whether the internet data is sufficient. Like, the internet's very big, but maybe there's like, some kind of boat collapse even with internet data, whereas like, with vision, at least you can like, you can like, make a robot that just like, travels down the street and like, just keeps taking pictures of everything. You can get like, infinite training data with vision, but it might be trickier to like, sort of, filter and clean internet data, especially as like, more synthetic data ends up on the internet.

• 21:32Speaker 02

  Well, another area that I know you spend a lot of time is music. You know, you make your own music and produce it, and, you know, there've been a number of different applications, refusions, you know, etc., that have sort of come up on the music side. I was just curious how you've been paying attention to that, what you think of it, and, you know, where you think that whole space evolves to.

• 21:52Speaker 01

  Yeah, I love, I love audio. That'd be the other kind of thing that I would go work on if it weren't for Playground. Partly, I didn't work on music because the music industry is like, only, the whole industry is $26 billion, so it was a little hard for me to figure out like, how big like, a music thing could be, but I definitely think audio is going to be enormous, like, things like Eleven Labs are very interesting. But anyway, I, yeah, I mean, the way that I've been, I've been trying to find ways to figure out how to use it as a user, because that gives me like, a really, that gives me a stronger sense of maybe where things are going. And so, one thing that I've been waiting for, for many years, is that

• 22:32Speaker 01

  Instrumentals in music are actually very easy to get or to make. You know, there's a wide variety of like quality, of course, but generally instrumentals in a song, like if you hear a song from Taylor Swift or whoever, a rap song, those beats or those instrumentals are fairly easy to make and easy. What's hard is to get lyrics and vocals. And that's always been like a difficulty of mine. Like, how do I find a singer? And then how do I get them to like, write lyrics and then sing it? That's much, that's a much more scarce resource in the music world. And, and so for the first time with like something like Suno AI, it was really cool because it's the first time that I heard,

• 23:14Speaker 01

  you know, them be able to make like a rap song where the rapper has like good flow. Flow is just like the swing of lyrics to a beat. And, or like, you know, you hear like actually really good lyrics that feel like very emotional, have the right breathiness. Doesn't sound like, like it's all made on like auto-tune, I guess. And so I have this like little flow where I like make a song in Suno and then I use a different AI tool, AI tools all the way down, I guess, to like split the stems and just grab the lyrics, but then throw away the instrumental. And then I get to like make a song with the instrumental and the vocal. Anyway, so I mean, I put some songs on my Twitter where, where like I basically tried to do this and it sounds, you know, so I can get to like a higher quality song,

• 23:59Speaker 01

  I guess, because I make the instrumental, doesn't, there are still some like weird errors in the songs, but, but that's been like a really cool way to use AI, in my opinion.

• 24:08Speaker 02

  So Hal, thanks so much for sharing everything that you're working on at Playground with us.

• 24:13Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at nodashbuyers.com.