AI Engineering for Art — with comfyanonymous, of ComfyUI

Applications for the NYC AI Engineer Summit, focused on Agents at Work, are open http://apply.ai.engineer! When we first started Latent Space, in the lightning round we’d always ask guests: “What’s your favorite AI product?”. The majority would say Midjourney. The simple UI of prompt → very aesthetic image turned it into a $300M+ ARR bootstrapped business as it rode the first wave of AI image generation. In open source land, StableDiffusion was congregating around AUTOMATIC1111 https://github.com/AUTOMATIC1111/stable-diffusion-webui as the de-facto web UI. Unlike Midjourney, which offered some flags but was mostly prompt-driven, A1111 let users play with a lot more parameters, supported additional modalities like img2img, and allowed users to load in custom models. If you’re interested in some of the SD history, you can look at our episodes with Lexica https://www.latent.space/p/sharif-shameem, Replicate https://www.latent.space/p/replicate, and Playground https://www.latent.space/p/suhail-doshi. One of the people involved with that community was comfyanonymous https://github.com/comfyanonymous/, who was also part of the Stability team in 2023, decided to build an alternative called ComfyUI https://github.com/comfyanonymous/ComfyUI, now one of the fastest growing open source projects in generative images, and is now the preferred partner for folks like Black Forest Labs https://blog.comfy.org/day-1-support-for-flux-tools-in-comfyui/’s Flux Tools on Day 1 https://blog.comfy.org/day-1-support-for-flux-tools-in-comfyui/. The idea behind it was simple: “Everyone is trying to make easy to use interfaces. Let me try to make a powerful interface that's not easy to use.” Unlike its predecessors, ComfyUI does not have an input text box. Everything is based around the idea of a node: there’s a text input node, a CLIP node, a checkpoint loader node, a KSampler node, a VAE node, etc. While daunting for simple image generation, the tool is amazing for more complex workflows since you can break down every step of the process, and then chain many of them together rather than manually switching between tools. You can also re-start execution halfway instead of from the beginning, which can save a lot of time when using larger models. To give you an idea of some of the new use cases that this type of UI enables: * Sketch something → Generate an image with SD from sketch → feed it into SD Video to animate * Generate an image of an object → Turn into a 3D asset → Feed into interactive experiences * Input audio → Generate audio-reactive videos Their Examples https://comfyanonymous.github.io/ComfyUI_examples/ page also includes some of the more common use cases like AnimateDiff, etc. They recently launched the Comfy Registry https://registry.comfy.org/, an online library of different nodes that users can pull from rather than having to build everything from scratch. The project has >60,000 Github stars, and as the community grows, some of the projects that people build have gotten quite complex: The most interesting thing about Comfy is that it’s not a UI, it’s a runtime. You can build full applications on top of image models simply by using Comfy. You can expose Comfy workflows as an endpoint and chain them together just like you chain a single node. We’re seeing the rise of AI Engineering applied to art. Major Tom’s ComfyUI Resources from the Latent Space Discord Major shoutouts to Major Tom on the LS Discord who is a image generation expert, who offered these pointers: * “best thing about comfy is the fact it supports almost immediately every new thing that comes out - unlike A1111 or forge, which still don't support flux cnet for instance. It will be perfect tool when conflicting nodes will be resolved” * AP Workflows from Alessandro Perili https://perilli.com/ai/comfyui/ are a nice example of an all-in-one train-evaluate-generate system built atop Comfy * ComfyUI YouTubers to learn from: * @sebastiankamph https://www.youtube.com/@sebastiankamph * @NerdyRodent https://www.youtube.com/@NerdyRodent * @OlivioSarikas https://www.youtube.com/@OlivioSarikas * @sedetweiler https://www.youtube.com/@sedetweiler * @pixaroma https://www.youtube.com/@pixaroma * ComfyUI Nodes to check out: * https://github.com/kijai/ComfyUI-IC-Light * https://github.com/MrForExample/ComfyUI-3D-Pack * https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait * https://github.com/pydn/ComfyUI-to-Python-Extension * https://github.com/THtianhao/ComfyUI-Portrait-Maker * https://github.com/ssitu/ComfyUI_NestedNodeBuilder * https://github.com/longgui0318/comfyui-magic-clothing * https://github.com/atmaranto/ComfyUI-SaveAsScript * https://github.com/ZHO-ZHO-ZHO/ComfyUI-InstantID * https://github.com/AIFSH/ComfyUI-FishSpeech * https://github.com/coolzilj/ComfyUI-Photopea * https://github.com/lks-ai/anynode * Sarav: https://www.youtube.com/@mickmumpitz/videos ( applied stuff ) * Sarav: https://www.youtube.com/@latentvision (technical, but infrequent) * look for comfyui node for https://github.com/magic-quill/MagicQuill * “Comfy for Video” resources * Kijai (https://github.com/kijai) pushing out support for Mochi, CogVideoX, AnimateDif, LivePortrait etc * Comfyui node support like LTX https://github.com/Lightricks/ComfyUI-LTXVideo , and HunyuanVideo https://github.com/Tencent/HunyuanVideo#-community-contributions * FloraFauna AI https://x.com/florafaunaai/status/1856047561536086293 and Krea.ai https://krea.ai * Communities: https://www.reddit.com/r/StableDiffusion/, https://www.reddit.com/r/comfyui/ Full YouTube Episode As usual, you can find the full video episode https://youtu.be/Hc31HotThA0 on our YouTube (and don’t forget to like and subscribe!) Timestamps * 00:00:04 Introduction of hosts and anonymous guest * 00:00:35 Origins of Comfy UI and early Stable Diffusion landscape * 00:02:58 Comfy's background and development of high-res fix * 00:05:37 Area conditioning and compositing in image generation * 00:07:20 Discussion on different AI image models (SD, Flux, etc.) * 00:11:10 Closed source model APIs and community discussions on SD versions * 00:14:41 LoRAs and textual inversion in image generation * 00:18:43 Evaluation methods in the Comfy community * 00:20:05 CLIP models and text encoders in image generation * 00:23:05 Prompt weighting and negative prompting * 00:26:22 Comfy UI's unique features and design choices * 00:31:00 Memory management in Comfy UI * 00:33:50 GPU market share and compatibility issues * 00:35:40 Node design and parameter settings in Comfy UI * 00:38:44 Custom nodes and community contributions * 00:41:40 Video generation models and capabilities * 00:44:47 Comfy UI's development timeline and rise to popularity * 00:48:13 Current state of Comfy UI team and future plans * 00:50:11 Discussion on other Comfy startups and potential text generation support Transcript Alessio [00:00:04]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI. swyx [00:00:12]: Hey everyone, we are in the Chroma Studio again, but with our first ever anonymous guest, Comfy Anonymous, welcome. Comfy [00:00:19]: Hello. swyx [00:00:21]: I feel like that's your full name, you just go by Comfy, right? Comfy [00:00:24]: Yeah, well, a lot of people just call me Comfy, even when they know my real name. Hey, Comfy. Alessio [00:00:32]: Swyx is the same. You know, not a lot of people call you Shawn. swyx [00:00:35]: Yeah, you have a professional name, right, that people know you by, and then you have a legal name. Yeah, it's fine. How do I phrase this? I think people who are in the know, know that Comfy is like the tool for image generation and now other multimodality stuff. I would say that when I first got started with Stable Diffusion, the star of the show was Automatic 111, right? And I actually looked back at my notes from 2022-ish, like Comfy was already getting started back then, but it was kind of like the up and comer, and your main feature was the flowchart. Can you just kind of rewind to that moment, that year and like, you know, how you looked at the landscape there and decided to start Comfy? Comfy [00:01:10]: Yeah, I discovered Stable Diffusion in 2022, in October 2022. And, well, I kind of started playing around with it. Yes, I, and back then I was using Automatic, which was what everyone was using back then. And so I started with that because I had, it was when I started, I had no idea like how Diffusion works. I didn't know how Diffusion models work, how any of this works, so. swyx [00:01:36]: Oh, yeah. What was your prior background as an engineer? Comfy [00:01:39]: Just a software engineer. Yeah. Boring software engineer. swyx [00:01:44]: But like any, any image stuff, any orchestration, distributed systems, GPUs? Comfy [00:01:49]: No, I was doing basically nothing interesting. Crud, web development? Yeah, a lot of web development, just, yeah, some basic, maybe some basic like automation stuff. Okay. Just. Yeah, no, like, no big companies or anything. swyx [00:02:08]: Yeah, but like already some interest in automations, probably a lot of Python. Comfy [00:02:12]: Yeah, yeah, of course, Python. But I wasn't actually used to like the Node graph interface before I started Comfy UI. It was just, I just thought it was like, oh, like, what's the best way to represent the Diffusion process in the user interface? And then like, oh, well. Well, like, naturally, oh, this is the best way I've found. And this was like with the Node interface. So how I got started was, yeah, so basic October 2022, just like I hadn't written a line of PyTorch before that. So it's completely new. What happened was I kind of got addicted to generating images. Alessio [00:02:58]: As we all did. Yeah. Comfy [00:03:00]: And then I started. I started experimenting with like the high-res fixed in auto, which was for those that don't know, the high-res fix is just since the Diffusion models back then could only generate that low-resolution. So what you would do, you would generate low-resolution image, then upscale, then refine it again. And that was kind of the hack to generate high-resolution images. I really liked generating. Like higher resolution images. So I was experimenting with that. And so I modified the code a bit. Okay. What happens if I, if I use different samplers on the second pass, I was edited the code of auto. So what happens if I use a different sampler? What happens if I use a different, like a different settings, different number of steps? And because back then the. The high-res fix was very basic, just, so. Yeah. swyx [00:04:05]: Now there's a whole library of just, uh, the upsamplers. Comfy [00:04:08]: I think, I think they added a bunch of, uh, of options to the high-res fix since, uh, since, since then. But before that was just so basic. So I wanted to go further. I wanted to try it. What happens if I use a different model for the second, the second pass? And then, well, then the auto code base was, wasn't good enough for. Like, it would have been, uh, harder to implement that in the auto interface than to create my own interface. So that's when I decided to create my own. And you were doing that mostly on your own when you started, or did you already have kind of like a subgroup of people? No, I was, uh, on my own because, because it was just me experimenting with stuff. So yeah, that was it. Then, so I started writing the code January one. 2023, and then I released the first version on GitHub, January 16th, 2023. That's how things got started. Alessio [00:05:11]: And what's, what's the name? Comfy UI right away or? Yeah. Comfy [00:05:14]: Comfy UI. The reason the name, my name is Comfy is people thought my pictures were comfy, so I just, uh, just named it, uh, uh, it's my Comfy UI. So yeah, that's, uh, swyx [00:05:27]: Is there a particular segment of the community that you targeted as users? Like more intensive workflow artists, you know, compared to the automatic crowd or, you know, Comfy [00:05:37]: This was my way of like experimenting with, uh, with new things, like the high risk fixed thing I mentioned, which was like in Comfy, the first thing you could easily do was just chain different models together. And then one of the first things, I think the first times it got a bit of popularity was when I started experimenting with the different, like applying. Prompts to different areas of the image. Yeah. I called it area conditioning, posted it on Reddit and it got a bunch of upvotes. So I think that's when, like, when people first learned of Comfy UI. swyx [00:06:17]: Is that mostly like fixing hands? Comfy [00:06:19]: Uh, no, no, no. That was just, uh, like, let's say, well, it was very, well, it still is kind of difficult to like, let's say you want a mountain, you have an image and then, okay. I'm like, okay. I want the mountain here and I want the, like a, a Fox here. swyx [00:06:37]: Yeah. So compositing the image. Yeah. Comfy [00:06:40]: My way was very easy. It was just like, oh, when you run the diffusion process, you kind of generate, okay. You do pass one pass through the diffusion, every step you do one pass. Okay. This place of the image with this brand, this space, place of the image with the other prop. And then. The entire image with another prop and then just average everything together, every step, and that was, uh, area composition, which I call it. And then, then a month later, there was a paper that came out called multi diffusion, which was the same thing, but yeah, that's, uh, Alessio [00:07:20]: could you do area composition with different models or because you're averaging out, you kind of need the same model. Comfy [00:07:26]: Could do it with, but yeah, I hadn't implemented it. For different models, but, uh, you, you can do it with, uh, with different models if you want, as long as the models share the same latent space, like we, we're supposed to ring a bell every time someone says, yeah, like, for example, you couldn't use like Excel and SD 1.5, because those have a different latent space, but like, uh, yeah, like SD 1.5 models, different ones. You could, you could do that. swyx [00:07:59]: There's some models that try to work in pixel space, right? Comfy [00:08:03]: Yeah. They're very slow. Of course. That's the problem. That that's the, the reason why stable diffusion actually became like popular, like, cause was because of the latent space. swyx [00:08:14]: Small and yeah. Because it used to be latent diffusion models and then they trained it up. Comfy [00:08:19]: Yeah. Cause a pixel pixel diffusion models are just too slow. So. Yeah. swyx [00:08:25]: Have you ever tried to talk to like, like stability, the latent diffusion guys, like, you know, Robin Rombach, that, that crew. Yeah. Comfy [00:08:32]: Well, I used to work at stability. swyx [00:08:34]: Oh, I actually didn't know. Yeah. Comfy [00:08:35]: I used to work at stability. I got, uh, I got hired, uh, in June, 2023. swyx [00:08:42]: Ah, that's the part of the story I didn't know about. Okay. Yeah. Comfy [00:08:46]: So the, the reason I was hired is because they were doing, uh, SDXL at the time and they were basically SDXL. I don't know if you remember it was a base model and then a refiner model. Basically they wanted to experiment, like chaining them together. And then, uh, they saw, oh, right. Oh, this, we can use this to do that. Well, let's hire that guy. swyx [00:09:10]: But they didn't, they didn't pursue it for like SD3. What do you mean? Like the SDXL approach. Yeah. Comfy [00:09:16]: The reason for that approach was because basically they had two models and then they wanted to publish both of them. So they, they trained one on. Lower time steps, which was the refiner model. And then they, the first one was trained normally. And then they went during their test, they realized, oh, like if we string these models together are like quality increases. So let's publish that. It worked. Yeah. But like right now, I don't think many people actually use the refiner anymore, even though it is actually a full diffusion model. Like you can use it on its own. And it's going to generate images. I don't think anyone, people have mostly forgotten about it. But, uh. Alessio [00:10:05]: Can we talk about models a little bit? So stable diffusion, obviously is the most known. I know flux has gotten a lot of traction. Are there any underrated models that people should use more or what's the state of the union? Comfy [00:10:17]: Well, the, the latest, uh, state of the art, at least, yeah, for images there's, uh, yeah, there's flux. There's also SD3.5. SD3.5 is two models. There's a, there's a small one, 2.5B and there's the bigger one, 8B. So it's, it's smaller than flux. So, and it's more, uh, creative in a way, but flux, yeah, flux is the best. People should give SD3.5 a try cause it's, uh, it's different. I won't say it's better. Well, it's better for some like specific use cases. Right. If you want some to make something more like creative, maybe SD3.5. If you want to make something more consistent and flux is probably better. swyx [00:11:06]: Do you ever consider supporting the closed source model APIs? Comfy [00:11:10]: Uh, well, they, we do support them as custom nodes. We actually have some, uh, official custom nodes from, uh, different. Ideogram. swyx [00:11:20]: Yeah. I guess DALI would have one. Yeah. Comfy [00:11:23]: That's, uh, it's just not, I'm not the person that handles that. Sure. swyx [00:11:28]: Sure. Quick question on, on SD. There's a lot of community discussion about the transition from SD1.5 to SD2 and then SD2 to SD3. People still like, you know, very loyal to the previous generations of SDs? Comfy [00:11:41]: Uh, yeah. SD1.5 then still has a lot of, a lot of users. swyx [00:11:46]: The last based model. Comfy [00:11:49]: Yeah. Then SD2 was mostly ignored. It wasn't, uh, it wasn't a big enough improvement over the previous one. Okay. swyx [00:11:58]: So SD1.5, SD3, flux and whatever else. SDXL. SDXL. Comfy [00:12:03]: That's the main one. Stable cascade. Stable cascade. That was a good model. But, uh, that's, uh, the problem with that one is, uh, it got, uh, like SD3 was announced one week after. Yeah. swyx [00:12:16]: It was like a weird release. Uh, what was it like inside of stability actually? I mean, statute of limitations. Yeah. The statute of limitations expired. You know, management has moved. So it's easier to talk about now. Yeah. Comfy [00:12:27]: And inside stability, actually that model was ready, uh, like three months before, but it got, uh, stuck in, uh, red teaming. So basically the product, if that model had released or was supposed to be released by the authors, then it would probably have gotten very popular since it's a, it's a step up from SDXL. But it got all of its momentum stolen. It got stolen by the SD3 announcement. So people kind of didn't develop anything on top of it, even though it's, uh, yeah. It was a good model, at least, uh, completely mostly ignored for some reason. Like swyx [00:13:07]: I think the naming as well matters. It seemed like a branch off of the main, main tree of development. Yeah. Comfy [00:13:15]: Well, it was different researchers that did it. Yeah. Yeah. Very like, uh, good model. Like it's the Worcestershire authors. I don't know if I'm pronouncing it correctly. Yeah. Yeah. Yeah. swyx [00:13:28]: I actually met them in Vienna. Yeah. Comfy [00:13:30]: They worked at stability for a bit and they left right after the Cascade release. swyx [00:13:35]: This is Dustin, right? No. Uh, Dustin's SD3. Yeah. Comfy [00:13:38]: Dustin is a SD3 SDXL. That's, uh, Pablo and Dome. I think I'm pronouncing his name correctly. Yeah. Yeah. Yeah. Yeah. That's very good. swyx [00:13:51]: It seems like the community is very, they move very quickly. Yeah. Like when there's a new model out, they just drop whatever the current one is. And they just all move wholesale over. Like they don't really stay to explore the full capabilities. Like if, if the stable cascade was that good, they would have AB tested a bit more. Instead they're like, okay, SD3 is out. Let's go. You know? Comfy [00:14:11]: Well, I find the opposite actually. The community doesn't like, they only jump on a new model when there's a significant improvement. Like if there's a, only like a incremental improvement, which is what, uh, most of these models are going to have, especially if you, cause, uh, stay the same parameter count. Yeah. Like you're not going to get a massive improvement, uh, into like, unless there's something big that, that changes. So, uh. Yeah. swyx [00:14:41]: And how are they evaluating these improvements? Like, um, because there's, it's a whole chain of, you know, comfy workflows. Yeah. How does, how does one part of the chain actually affect the whole process? Comfy [00:14:52]: Are you talking on the model side specific? swyx [00:14:54]: Model specific, right? But like once you have your whole workflow based on a model, it's very hard to move. Comfy [00:15:01]: Uh, not, well, not really. Well, it depends on your, uh, depends on their specific kind of the workflow. Yeah. swyx [00:15:09]: So I do a lot of like text and image. Yeah. Comfy [00:15:12]: When you do change, like most workflows are kind of going to be complete. Yeah. It's just like, you might have to completely change your prompt completely change. Okay. swyx [00:15:24]: Well, I mean, then maybe the question is really about evals. Like what does the comfy community do for evals? Just, you know, Comfy [00:15:31]: Well, that they don't really do that. It's more like, oh, I think this image is nice. So that's, uh, swyx [00:15:38]: They just subscribe to Fofr AI and just see like, you know, what Fofr is doing. Yeah. Comfy [00:15:43]: Well, they just, they just generate like it. Like, I don't see anyone really doing it. Like, uh, at least on the comfy side, comfy users, they, it's more like, oh, generate images and see, oh, this one's nice. It's like, yeah, it's not, uh, like the, the more, uh, like, uh, scientific, uh, like, uh, like checking that's more on specifically on like model side. If, uh, yeah, but there is a lot of, uh, vibes also, cause it is a like, uh, artistic, uh, you can create a very good model that doesn't generate nice images. Cause most images on the internet are ugly. So if you, if that's like, if you just, oh, I have the best model at 10th giant, it's super smart. I created on all the, like I've trained on just all the images on the internet. The images are not going to look good. So yeah. Alessio [00:16:42]: Yeah. Comfy [00:16:43]: They're going to be very consistent. But yeah. People like, it's not going to be like the, the look that people are going to be expecting from, uh, from a model. So. Yeah. swyx [00:16:54]: Can we talk about LoRa's? Cause we thought we talked about models then like the next step is probably LoRa's. Before, I actually, I'm kind of curious how LoRa's entered the tool set of the image community because the LoRa paper was 2021. And then like, there was like other methods like textual inversion that was popular at the early SD stage. Yeah. Comfy [00:17:13]: I can't even explain the difference between that. Yeah. Textual inversions. That's basically what you're doing is you're, you're training a, cause well, yeah. Stable diffusion. You have the diffusion model, you have text encoder. So basically what you're doing is training a vector that you're going to pass to the text encoder. It's basically you're training a new word. Yeah. swyx [00:17:37]: It's a little bit like representation engineering now. Yeah. Comfy [00:17:40]: Yeah. Basically. Yeah. You're just, so yeah, if you know how like the text encoder works, basically you have, you take your, your words of your product, you convert those into tokens with the tokenizer and those are converted into vectors. Basically. Yeah. Each token represents a different vector. So each word presents a vector. And those, depending on your words, that's the list of vectors that get passed to the text encoder, which is just. Yeah. Yeah. I'm just a stack of, of attention. Like basically it's a very close to LLM architecture. Yeah. Yeah. So basically what you're doing is just training a new vector. We're saying, well, I have all these images and I want to know which word does that represent? And it's going to get like, you train this vector and then, and then when you use this vector, it hopefully generates. Like something similar to your images. Yeah. swyx [00:18:43]: I would say it's like surprisingly sample efficient in picking up the concept that you're trying to train it on. Yeah. Comfy [00:18:48]: Well, people have kind of stopped doing that even though back as like when I was at Stability, we, we actually did train internally some like textual versions on like T5 XXL actually worked pretty well. But for some reason, yeah, people don't use them. And also they might also work like, like, yeah, this is something and probably have to test, but maybe if you train a textual version, like on T5 XXL, it might also work with all the other models that use T5 XXL because same thing with like, like the textual inversions that, that were trained for SD 1.5, they also kind of work on SDXL because SDXL has the, has two text encoders. And one of them is the same as the, as the SD 1.5 CLIP-L. So those, they actually would, they don't work as strongly because they're only applied to one of the text encoders. But, and the same thing for SD3. SD3 has three text encoders. So it works. It's still, you can still use your textual version SD 1.5 on SD3, but it's just a lot weaker because now there's three text encoders. So it gets even more diluted. Yeah. swyx [00:20:05]: Do people experiment a lot on, just on the CLIP side, there's like Siglip, there's Blip, like do people experiment a lot on those? Comfy [00:20:12]: You can't really replace. Yeah. swyx [00:20:14]: Because they're trained together, right? Yeah. Comfy [00:20:15]: They're trained together. So you can't like, well, what I've seen people experimenting with is a long CLIP. So basically someone fine tuned the CLIP model to accept longer prompts. swyx [00:20:27]: Oh, it's kind of like long context fine tuning. Yeah. Comfy [00:20:31]: So, so like it's, it's actually supported in Core Comfy. swyx [00:20:35]: How long is long? Comfy [00:20:36]: Regular CLIP is 77 tokens. Yeah. Long CLIP is 256. Okay. So, but the hack that like you've, if you use stable diffusion 1.5, you've probably noticed, oh, it still works if I, if I use long prompts, prompts longer than 77 words. Well, that's because the hack is to just, well, you split, you split it up in chugs of 77, your whole big prompt. Let's say you, you give it like the massive text, like the Bible or something, and it would split it up in chugs of 77 and then just pass each one through the CLIP and then just cut anything together at the end. It's not ideal, but it actually works. swyx [00:21:26]: Like the positioning of the words really, really matters then, right? Like this is why order matters in prompts. Yeah. Comfy [00:21:33]: Yeah. Like it, it works, but it's, it's not ideal, but it's what people expect. Like if, if someone gives a huge prompt, they expect at least some of the concepts at the end to be like present in the image. But usually when they give long prompts, they, they don't, they like, they don't expect like detail, I think. So that's why it works very well. swyx [00:21:58]: And while we're on this topic, prompts waiting, negative comments. Negative prompting all, all sort of similar part of this layer of the stack. Yeah. Comfy [00:22:05]: The, the hack for that, which works on CLIP, like it, basically it's just for SD 1.5, well, for SD 1.5, the prompt waiting works well because CLIP L is a, is not a very deep model. So you have a very high correlation between, you have the input token, the index of the input token vector. And the output token, they're very, the concepts are very close, closely linked. So that means if you interpolate the vector from what, well, the, the way Comfy UI does it is it has, okay, you have the vector, you have an empty prompt. So you have a, a chunk, like a CLIP output for the empty prompt, and then you have the one for your prompt. And then it interpolates from that, depending on your prompt. Yeah. Comfy [00:23:07]: So that's how it, how it does prompt waiting. But this stops working the deeper your text encoder is. So on T5X itself, it doesn't work at all. So. Wow. swyx [00:23:20]: Is that a problem for people? I mean, cause I'm used to just move, moving up numbers. Probably not. Yeah. Comfy [00:23:25]: Well. swyx [00:23:26]: So you just use words to describe, right? Cause it's a bigger language model. Yeah. Comfy [00:23:30]: Yeah. So. Yeah. So honestly it might be good, but I haven't seen many complaints on Flux that it's not working. So, cause I guess people can sort of get around it with, with language. So. Yeah. swyx [00:23:46]: Yeah. And then coming back to LoRa's, now the, the popular way to, to customize models is LoRa's. And I saw you also support Locon and LoHa, which I've never heard of before. Comfy [00:23:56]: There's a bunch of, cause what, what the LoRa is essentially is. Instead of like, okay, you have your, your model and then you want to fine tune it. So instead of like, what you could do is you could fine tune the entire thing, but that's a bit heavy. So to speed things up and make things less heavy, what you can do is just fine tune some smaller weights, like basically two, two matrices that when you multiply like two low rank matrices and when you multiply them together, gives a, represents a difference between trained weights and your base weights. So by training those two smaller matrices, that's a lot less heavy. Yeah. Alessio [00:24:45]: And they're portable. So you're going to share them. Yeah. It's like easier. And also smaller. Comfy [00:24:49]: Yeah. That's the, how LoRa's work. So basically, so when, when inferencing you, you get an inference with them pretty efficiently, like how ComputeWrite does it. It just, when you use a LoRa, it just applies it straight on the weights so that there's only a small delay at the base, like before the sampling to when it applies the weights and then it just same speed as, as before. So for, for inference, it's, it's not that bad, but, and then you have, so basically all the LoRa types like LoHa, LoCon, everything, that's just different ways of representing that like. Basically, you can call it kind of like compression, even though it's not really compression, it's just different ways of represented, like just, okay, I want to train a different on the difference on the weights. What's the best way to represent that difference? There's the basic LoRa, which is just, oh, let's multiply these two matrices together. And then there's all the other ones, which are all different algorithms. So. Yeah. Alessio [00:25:57]: So let's talk about LoRa. Let's talk about what comfy UI actually is. I think most people have heard of it. Some people might've seen screenshots. I think fewer people have built very complex workflows. So when you started, automatic was like the super simple way. What were some of the choices that you made? So the node workflow, is there anything else that stands out as like, this was like a unique take on how to do image generation workflows? Comfy [00:26:22]: Well, I feel like, yeah, back then everyone was trying to make like easy to use interface. Yeah. So I'm like, well, everyone's trying to make an easy to use interface. swyx [00:26:32]: Let's make a hard to use interface. Comfy [00:26:37]: Like, so like, I like, I don't need to do that, everyone else doing it. So let me try something like, let me try to make a powerful interface that's not easy to use. So. swyx [00:26:52]: So like, yeah, there's a sort of node execution engine. Yeah. Yeah. And it actually lists, it has this really good list of features of things you prioritize, right? Like let me see, like sort of re-executing from, from any parts of the workflow that was changed, asynchronous queue system, smart memory management, like all this seems like a lot of engineering that. Yeah. Comfy [00:27:12]: There's a lot of engineering in the back end to make things, cause I was always focused on making things work locally very well. Cause that's cause I was using it locally. So everything. So there's a lot of, a lot of thought and working by getting everything to run as well as possible. So yeah. ConfUI is actually more of a back end, at least, well, not all the front ends getting a lot more development, but, but before, before it was, I was pretty much only focused on the backend. Yeah. swyx [00:27:50]: So v0.1 was only August this year. Yeah. Comfy [00:27:54]: With the new front end. Before there was no versioning. So yeah. Yeah. Yeah. swyx [00:27:57]: And so what was the big rewrite for the 0.1 and then the 1.0? Comfy [00:28:02]: Well, that's more on the front end side. That's cause before that it was just like the UI, what, cause when I first wrote it, I just, I said, okay, how can I make, like, I can do web development, but I don't like doing it. Like what's the easiest way I can slap a node interface on this. And then I found this library. Yeah. Like JavaScript library. swyx [00:28:26]: Live graph? Comfy [00:28:27]: Live graph. swyx [00:28:28]: Usually people will go for like react flow for like a flow builder. Yeah. Comfy [00:28:31]: But that seems like too complicated. So I didn't really want to spend time like developing the front end. So I'm like, well, oh, light graph. This has the whole node interface. So, okay. Let me just plug that into, to my backend. swyx [00:28:49]: I feel like if Streamlit or Gradio offered something that you would have used Streamlit or Gradio cause it's Python. Yeah. Comfy [00:28:54]: Yeah. Yeah. Yeah. Comfy [00:29:00]: Yeah. Comfy [00:29:14]: Yeah. logic and your backend logic and just sticks them together. swyx [00:29:20]: It's supposed to be easy for you guys. If you're a Python main, you know, I'm a JS main, right? Okay. If you're a Python main, it's supposed to be easy. Comfy [00:29:26]: Yeah, it's easy, but it makes your whole software a huge mess. swyx [00:29:30]: I see, I see. So you're mixing concerns instead of separating concerns? Comfy [00:29:34]: Well, it's because... Like frontend and backend. Frontend and backend should be well separated with a defined API. Like that's how you're supposed to do it. Smart people disagree. It just sticks everything together. It makes it easy to like a huge mess. And also it's, there's a lot of issues with Gradio. Like it's very good if all you want to do is just get like slap a quick interface on your, like to show off your ML project. Like that's what it's made for. Yeah. Like there's no problem using it. Like, oh, I have my, I have my code. I just wanted a quick interface on it. That's perfect. Like use Gradio. But if you want to make something that's like a real, like real software that will last a long time and will be easy to maintain, then I would avoid it. Yeah. swyx [00:30:32]: So your criticism is Streamlit and Gradio are the same. I mean, those are the same criticisms. Comfy [00:30:37]: Yeah, Streamlit I haven't used as much. Yeah, I just looked a bit. swyx [00:30:43]: Similar philosophy. Comfy [00:30:44]: Yeah, it's similar. It's just, it just seems to me like, okay, for quick, like AI demos, it's perfect. swyx [00:30:51]: Yeah. Going back to like the core tech, like asynchronous queues, slow re-execution, smart memory management, you know, anything that you were very proud of or was very hard to figure out? Comfy [00:31:00]: Yeah. The thing that's the biggest pain in the ass is probably the memory management. Yeah. swyx [00:31:05]: Were you just paging models in and out or? Yeah. Comfy [00:31:08]: Before it was just, okay, load the model, completely unload it. Then, okay, that, that works well when you, your model are small, but if your models are big and it takes sort of like, let's say someone has a, like a, a 4090, and the model size is 10 gigabytes, that can take a few seconds to like load and load, load and load, so you want to try to keep things like in memory, in the GPU memory as much as possible. What Comfy UI does right now is it. It tries to like estimate, okay, like, okay, you're going to sample this model, it's going to take probably this amount of memory, let's remove the models, like this amount of memory that's been loaded on the GPU and then just execute it. But so there's a fine line between just because try to remove the least amount of models that are already loaded. Because as fans, like Windows drivers, and one other problem is the NVIDIA driver on Windows by default, because there's a way to, there's an option to disable that feature, but by default it, like, if you start loading, you can overflow your GPU memory and then it's, the driver's going to automatically start paging to RAM. But the problem with that is it's, it makes everything extremely slow. So when you see people complaining, oh, this model, it works, but oh, s**t, it starts slowing down a lot, that's probably what's happening. So it's basically you have to just try to get, use as much memory as possible, but not too much, or else things start slowing down, or people get out of memory, and then just find, try to find that line where, oh, like the driver on Windows starts paging and stuff. Yeah. And the problem with PyTorch is it's, it's high levels, don't have that much fine-grained control over, like, specific memory stuff, so kind of have to leave, like, the memory freeing to, to Python and PyTorch, which is, can be annoying sometimes. swyx [00:33:32]: So, you know, I think one thing is, as a maintainer of this project, like, you're designing for a very wide surface area of compute, like, you even support CPUs. Comfy [00:33:42]: Yeah, well, that's... That's just, for PyTorch, PyTorch supports CPUs, so, yeah, it's just, that's not, that's not hard to support. swyx [00:33:50]: First of all, is there a market share estimate, like, is it, like, 70% NVIDIA, like, 30% AMD, and then, like, miscellaneous on Apple, Silicon, or whatever? Comfy [00:33:59]: For Comfy? Yeah. Yeah, and, yeah, I don't know the market share. swyx [00:34:03]: Can you guess? Comfy [00:34:04]: I think it's mostly NVIDIA. Right. Because, because AMD, the problem, like, AMD works horribly on Windows. Like, on Linux, it works fine. It's, it's lower than the price equivalent NVIDIA GPU, but it works, like, you can use it, you generate images, everything works. On Linux, on Windows, you might have a hard time, so, that's the problem, and most people, I think most people who bought AMD probably use Windows. They probably aren't going to switch to Linux, so... Yeah. So, until AMD actually, like, ports their, like, raw cam to, to Windows properly, and then there's actually PyTorch, I think they're, they're doing that, they're in the process of doing that, but, until they get it, they get a good, like, PyTorch raw cam build that works on Windows, it's, like, they're going to have a hard time. Yeah. Alessio [00:35:06]: We got to get George on it. Yeah. Well, he's trying to get Lisa Su to do it, but... Let's talk a bit about, like, the node design. So, unlike all the other text-to-image, you have a very, like, deep, so you have, like, a separate node for, like, clip and code, you have a separate node for, like, the case sampler, you have, like, all these nodes. Going back to, like, the making it easy versus making it hard, but, like, how much do people actually play with all the settings, you know? Kind of, like, how do you guide people to, like, hey, this is actually going to be very impactful versus this is maybe, like, less impactful, but we still want to expose it to you? Comfy [00:35:40]: Well, I try to... I try to expose, like, I try to expose everything or, but, yeah, at least for the, but for things, like, for example, for the samplers, like, there's, like, yeah, four different sampler nodes, which go in easiest to most advanced. So, yeah, if you go, like, the easy node, the regular sampler node, that's, you have just the basic settings. But if you use, like, the sampler advanced... If you use, like, the custom advanced node, that, that one you can actually, you'll see you have, like, different nodes. Alessio [00:36:19]: I'm looking it up now. Yeah. What are, like, the most impactful parameters that you use? So, it's, like, you know, you can have more, but, like, which ones, like, really make a difference? Comfy [00:36:30]: Yeah, they all do. They all have their own, like, they all, like, for example, yeah, steps. Usually you want steps, you want them to be as low as possible. But you want, if you're optimizing your workflow, you want to, you lower the steps until, like, the images start deteriorating too much. Because that, yeah, that's the number of steps you're running the diffusion process. So, if you want things to be faster, lower is better. But, yeah, CFG, that's more, you can kind of see that as the contrast of the image. Like, if your image looks too bursty. Then you can lower the CFG. So, yeah, CFG, that's how, yeah, that's how strongly the, like, the negative versus positive prompt. Because when you sample a diffusion model, it's basically a negative prompt. It's just, yeah, positive prediction minus negative prediction. swyx [00:37:32]: Contrastive loss. Yeah. Comfy [00:37:34]: It's positive minus negative, and the CFG does the multiplier. Yeah. Yeah. Yeah, so. Alessio [00:37:41]: What are, like, good resources to understand what the parameters do? I think most people start with automatic, and then they move over, and it's, like, snap, CFG, sampler, name, scheduler, denoise. Read it. Comfy [00:37:53]: But, honestly, well, it's more, it's something you should, like, try out yourself. I don't know, you don't necessarily need to know how it works to, like, what it does. Because even if you know, like, CFGO, it's, like, positive minus negative prompt. Yeah. So the only thing you know at CFG is if it's 1.0, then that means the negative prompt isn't applied. It also means sampling is two times faster. But, yeah. But other than that, it's more, like, you should really just see what it does to the images yourself, and you'll probably get a more intuitive understanding of what these things do. Alessio [00:38:34]: Any other nodes or things you want to shout out? Like, I know the animate diff IP adapter. Those are, like, some of the most popular ones. Yeah. What else comes to mind? Comfy [00:38:44]: Not nodes, but there's, like, what I like is when some people, sometimes they make things that use ComfyUI as their backend. Like, there's a plugin for Krita that uses ComfyUI as its backend. So you can use, like, all the models that work in Comfy in Krita. And I think I've tried it once. But I know a lot of people use it, and it's probably really nice, so. Alessio [00:39:15]: What's the craziest node that people have built, like, the most complicated? Comfy [00:39:21]: Craziest node? Like, yeah. I know some people have made, like, video games in Comfy with, like, stuff like that. So, like, someone, like, I remember, like, yeah, last, I think it was last year, someone made, like, a, like, Wolfenstein 3D in Comfy. Of course. And then one of the inputs was, oh, you can generate a texture, and then it changes the texture in the game. So you can plug it to, like, the workflow. And there's a lot of, if you look there, there's a lot of crazy things people do, so. Yeah. Alessio [00:39:59]: And now there's, like, a node register that people can use to, like, download nodes. Yeah. Comfy [00:40:04]: Like, well, there's always been the, like, the ComfyUI manager. Yeah. But we're trying to make this more, like, I don't know, official, like, with, yeah, with the node registry. Because before the node registry, the, like, okay, how did your custom node get into ComfyUI manager? That's the guy running it who, like, every day he searched GitHub for new custom nodes and added dev annually to his custom node manager. So we're trying to make it less effortless. So we're trying to make it less effortless for him, basically. Yeah. Alessio [00:40:40]: Yeah. But I was looking, I mean, there's, like, a YouTube download node. There's, like, this is almost like, you know, a data pipeline more than, like, an image generation thing at this point. It's, like, you can get data in, you can, like, apply filters to it, you can generate data out. Comfy [00:40:54]: Yeah. You can do a lot of different things. Yeah. So I'm thinking, I think what I did is I made it easy to make custom nodes. So I think that helped a lot. I think that helped a lot for, like, the ecosystem because it is very easy to just make a node. So, yeah, a bit too easy sometimes. Then we have the issue where there's a lot of custom node packs which share similar nodes. But, well, that's, yeah, something we're trying to solve by maybe bringing some of the functionality into the core. Yeah. Yeah. Yeah. Alessio [00:41:36]: And then there's, like, video. People can do video generation. Yeah. Comfy [00:41:40]: Video, that's, well, the first video model was, like, stable video diffusion, which was last, yeah, exactly last year, I think. Like, one year ago. But that wasn't a true video model. So it was... swyx [00:41:55]: It was, like, moving images? Yeah. Comfy [00:41:57]: I generated video. What I mean by that is it's, like, it's still 2D Latents. It's basically what I'm trying to do. So what they did is they took SD2, and then they added some temporal attention to it, and then trained it on videos and all. So it's kind of, like, animated, like, same idea, basically. Why I say it's not a true video model is that you still have, like, the 2D Latents. Like, a true video model, like Mochi, for example, would have 3D Latents. Mm-hmm. Alessio [00:42:32]: Which means you can, like, move through the space, basically. It's the difference. You're not just kind of, like, reorienting. Yeah. Comfy [00:42:39]: And it's also, well, it's also because you have a temporal VAE. Mm-hmm. Also, like, Mochi has a temporal VAE that compresses on, like, the temporal direction, also. So that's something you don't have with, like, yeah, animated diff and stable video diffusion. They only, like, compress spatially, not temporally. Mm-hmm. Right. So, yeah. That's why I call that, like, true video models. There's, yeah, there's actually a few of them, but the one I've implemented in comfy is Mochi, because that seems to be the best one so far. Yeah. swyx [00:43:15]: We had AJ come and speak at the stable diffusion meetup. The other open one I think I've seen is COG video. Yeah. Comfy [00:43:21]: COG video. Yeah. That one's, yeah, it also seems decent, but, yeah. Chinese, so we don't use it. No, it's fine. It's just, yeah, I could. Yeah. It's just that there's a, it's not the only one. There's also a few others, which I. swyx [00:43:36]: The rest are, like, closed source, right? Like, Cling. Yeah. Comfy [00:43:39]: Closed source, there's a bunch of them. But I mean, open. I've seen a few of them. Like, I can't remember their names, but there's COG videos, the big, the big one. Then there's also a few of them that released at the same time. There's one that released at the same time as SSD 3.5, same day, which is why I don't remember the name. swyx [00:44:02]: We should have a release schedule so we don't conflict on each of these things. Yeah. Comfy [00:44:06]: I think SD 3.5 and Mochi released on the same day. So everything else was kind of drowned, completely drowned out. So for some reason, lots of people picked that day to release their stuff. Comfy [00:44:21]: Yeah. Which is, well, shame for those. And I think Omnijet also released the same day, which also seems interesting. Yeah. Yeah. Alessio [00:44:30]: What's Comfy? So you are Comfy. And then there's like, comfy.org. I know we do a lot of things for, like, news research and those guys also have kind of like a more open source thing going on. How do you work? Like you mentioned, you mostly work on like, the core piece of it. And then what... Comfy [00:44:47]: Maybe I should fade it in because I, yeah, I feel like maybe, yeah, I only explain part of the story. Right. Yeah. Maybe I should explain the rest. So yeah. So yeah. Basically, January, that's when the first January 2023, January 16, 2023, that's when Amphi was first released to the public. Then, yeah, did a Reddit post about the area composition thing somewhere in, I don't remember exactly, maybe end of January, beginning of February. And then someone, a YouTuber, made a video about it, like Olivio, he made a video about Amphi in March 2023. I think that's when it was a real burst of attention. And by that time, I was continuing to develop it and it was getting, people were starting to use it more, which unfortunately meant that I had first written it to do like experiments, but then my time to do experiments went down. It started going down, because people were actually starting to use it then. Like, I had to, and I said, well, yeah, time to add all these features and stuff. Yeah, and then I got hired by Stability June, 2023. Then I made, basically, yeah, they hired me because they wanted the SD-XL. So I got the SD-XL working very well withітhe UI, because they were experimenting withámphi.house.com. Actually, the SDX, how the SDXL released worked is they released, for some reason, like they released the code first, but they didn't release the model checkpoint. So they released the code. And then, well, since the research was related to code, I released the code in Compute 2. And then the checkpoints were basically early access. People had to sign up and they only allowed a lot of people from edu emails. Like if you had an edu email, like they gave you access basically to the SDXL 0.9. And, well, that leaked. Right. Of course, because of course it's going to leak if you do that. Well, the only way people could easily use it was with Comfy. So, yeah, people started using. And then I fixed a few of the issues people had. So then the big 1.0 release happened. And, well, Comfy UI was the only way a lot of people could actually run it on their computers. Because it just like automatic was so like inefficient and bad that most people couldn't actually, like it just wouldn't work. Like because he did a quick implementation. So people were forced. To use Comfy UI, and that's how it became popular because people had no choice. swyx [00:47:55]: The growth hack. Comfy [00:47:56]: Yeah. swyx [00:47:56]: Yeah. Comfy [00:47:57]: Like everywhere, like people who didn't have the 4090, they had like, who had just regular GPUs, they didn't have a choice. Alessio [00:48:05]: So yeah, I got a 4070. So think of me. And so today, what's, is there like a core Comfy team or? Comfy [00:48:13]: Uh, yeah, well, right now, um, yeah, we are hiring. Okay. Actually, so right now core, like, um, the core core itself, it's, it's me. Uh, but because, uh, the reason where folks like all the focus has been mostly on the front end right now, because that's the thing that's been neglected for a long time. So, uh, so most of the focus right now is, uh, all on the front end, but we are, uh, yeah, we will soon get, uh, more people to like help me with the actual backend stuff. Yeah. So, no, I'm not going to say a hundred percent because that's why once the, once we have our V one release, which is because it'd be the package, come fee-wise with the nice interface and easy to install on windows and hopefully Mac. Uh, yeah. Yeah. Once we have that, uh, we're going to have to, lots of stuff to do on the backend side and also the front end side, but, uh. Alessio [00:49:14]: What's the release that I'm on the wait list. What's the timing? Comfy [00:49:18]: Uh, soon. Uh, soon. Yeah, I don't want to promise a release date. We do have a release date we're targeting, but I'm not sure if it's public. Yeah, and we're still going to continue doing the open source, making MPUI the best way to run stable infusion models. At least the open source side, it's going to be the best way to run models locally. But we will have a few things to make money from it, like cloud inference or that type of thing. And maybe some things for some enterprises. swyx [00:50:08]: I mean, a few questions on that. How do you feel about the other comfy startups? Comfy [00:50:11]: I mean, I think it's great. They're using your name. Yeah, well, it's better they use comfy than they use something else. Yeah, that's true. It's fine. We're going to try not to... We don't want to... We want people to use comfy. Like I said, it's better that people use comfy than something else. So as long as they use comfy, I think it helps the ecosystem. Because more people, even if they don't contribute directly, the fact that they are using comfy means that people are more likely to join the ecosystem. So, yeah. swyx [00:50:57]: And then would you ever do text? Comfy [00:50:59]: Yeah, well, you can already do text with some custom nodes. So, yeah, it's something we like. Yeah, it's something I've wanted to eventually add to core, but it's more like not a very... It's a very high priority. But because a lot of people use text for prompt enhancement and other things like that. So, yeah, it's just that my focus has always been on diffusion models. Yeah, unless some text diffusion model comes out. swyx [00:51:30]: Yeah, David Holtz is investing a lot in text diffusion. Comfy [00:51:34]: Yeah, well, if a good one comes out, then we'll probably implement it since it fits with the whole... swyx [00:51:39]: Yeah, I mean, I imagine it's going to be a close source to Midjourney. Yeah. Comfy [00:51:43]: Well, if an open one comes out, then I'll probably implement it. Alessio [00:51:54]: Cool, comfy. Thanks so much for coming on. This was fun. Bye. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Happy New Year, friends. Thanks for all the love on the Latent Space Live and 100th episode end of year recap. Your support has boosted us 30 places in the podcast charts, and that always helps us book great guests and organise more industry events for you. We don't say this enough, but thank you to everyone who has left a review on Apple Podcasts or subscribed to our new YouTube channel. Last year, we broke new ground when we interviewed our first public company CEO with Drew Houston, and first technology cabinet member with Minister Josephine Tao, and first year with full coverage of leading labs across Meta, OpenAI,

• 0:39Speaker 00

  Anthropic, Raker, Liquid, and Google DeepMind. For our 101st episode, we are proud to introduce another first with our first anonymous guest. As Swyx mentions in the episode, latent space was started in the immediate aftermath of stable diffusion, and the uncredentialed software engineers it enabled set the stage for the LLM wave that was to come with ChatGPT. The earliest winner of the stable diffusion tooling wars was SD Web UI, a Gradio app by the anonymous young creator Automatic1111 that quickly amassed over 100,000 GitHub stars for how it rapidly shipped plugins and usable interfaces for the

• 1:23Speaker 00

  rapidly growing stable diffusion ecosystem. However, these days, the power tool of choice is now Comfy UI by today's guest, Comfy Anonymous, who is gracing us with his first ever podcast appearance today. The shift from Automatic1111 to Comfy UI reflects a shift away in the image diffusion space from prompting and tweaking settings in 2022 to more complex and parallel workflows chaining together different models and orchestrating long-running operations that can also include video processing visualised on an intuitive canvas instead of long YAML or code

• 2:03Speaker 00

  Because Comfy UI is open source, there are now multiple Y Combinator startups built off of a comfy workflow or offering comfy UI as a service directly. Interestingly enough, this same workflow tooling has not seemed to take off for other modalities yet, but perhaps 2025 is the year diffusion tooling diffuses to non-image domains. In other news, we have just announced the second AI engineer summit in New York City. We are bringing back the surprisingly successful AI leadership track from World's Fair. And also, the single-track AI engineering track is now wholly focused on agents at work. If you are building agents in 2025, this is the single best conference to attend.

• 2:46Speaker 00

  Head to apply.ai-engineer and see you there. Watch out and take care.

• 2:55Speaker 01

  Hey, everyone. Welcome to the Late in Space Podcast. This is Alessio, partner and CTO of Dasebo Partners, and I'm joined by my co-host Swyx, founder of Small AI. Hey, everyone. We are in the Chrome Studio again, but with our first-ever anonymous guest, Comfy Anonymous. Welcome.

• 3:12Speaker 01

  I feel like that's your full name. You just go by Comfy, right?

• 3:15Speaker 02

  Yeah, well, a lot of people just call me Comfy, even though, even when they know my real name. They say, "Hey, hey, Comfy."

• 3:22Alessio Fanelli?

  Yeah, Swyx is the same. Not a lot of people call you Sean.

• 3:27Speaker 01

  Yeah, you have a professional name, right? People know you by, and then you have a legal name. Yeah, that's fine. How do I phrase this? I think people who are in the know know that Comfy is the tool for image generation and now other multimodality stuff. I would say that when I first got started with Stable Diffusion, the star of the show was Automatic 111, right? And I actually looked back at my notes from 2022-ish. Like, Comfy was already getting started back then, but it was kind of like the up-and-comer, and your main feature was the flowchart. Can you just kind of rewind to that moment, that year, and how you looked at the landscape there and decided to start Comfy?

• 4:02Speaker 02

  Yeah, I discovered Stable Diffusion in October 2022.

• 4:08Speaker 02

  And well, I kind of started playing around with it. Yes, I am. And back then I was using Automattic, which was what everyone was using back then. And I, so I started with that. Cause I had the, cause when I started, I had no idea like how diffusion models work, how any of this works.

• 4:27Speaker 01

  Oh yeah. What was your prior background as an engineer?

• 4:30Speaker 02

  Uh, just a software engineer. Yeah. Boring software engineer.

• 4:35Speaker 01

  But like any, any image stuff, any orchestration, distributed systems, GPUs?

• 4:40Speaker 02

  No, I was doing basically nothing interesting.

• 4:45Speaker 01

  Crud web development?

• 4:47Speaker 02

  Yeah, well, not web development, just yeah, some basic, maybe some basic like automation stuff.

• 4:53Speaker 01

  Okay.

• 4:54Speaker 02

  Just, yeah, no, like no big companies or anything.

• 4:59Speaker 01

  Yeah, but like already some interest in automations, probably a lot of Python.

• 5:03Speaker 02

  Yeah, yeah, of course, Python, but I wasn't actually used to like the node graph interface. Before I started config UI, it was just, I just thought it was like, oh, like what's the best way to represent the diffusion process in the user interface. And then my, oh, wow. Like natural, this is the best way I found. And this was like with the node interface. So how I got started was, yeah, so basically October 2022, just like I hadn't written a line of PyTorch before that.

• 5:42Speaker 02

  So it's completely new. What happened was I kind of got addicted to generating images.

• 5:50Speaker 01

  As we all did.

• 5:50Speaker 02

  Yeah. And then I started experimenting with like the high risk fixed in Auto, which was for those that don't know, the high risk fixes just to generate, since the diffusion models back then could only generate at low resolution. So what you would do, you would generate low resolution image, then upscale, then kind of refine it again. And that was kind of the hack to generate high resolution images. I really liked generating like higher resolution images, so I was experimenting with that.

• 6:28Speaker 02

  And so I modified the code a bit. Okay, what happens if I if I use different samplers on the second pass? So I was edited the code of auto. So what happens if I use a different sampler? What happens if I use a different like a different settings, different number of steps? And because back then, the high risk fix was very basic, just so.

• 6:56Speaker 01

  Yeah, now there's a whole library of just up samplers.

• 7:00Speaker 02

  I think I think they added a bunch of of options in the high risk fix since since since then. But before that was just so basic. So I wanted to go further. I wanted to try, okay, what happens if I use a different model for the second second pass? And then, well, then the auto code base was wasn't good enough for like it would have been harder to implement that in the auto interface than to create my own interface. So that's when I decided to create my own.

• 7:35Speaker 01

  And you were doing that mostly on your own when you started? Or did you already have kind of like a subgroup?

• 7:39Speaker 02

  No, I was on my own because it was just me experimenting with stuff. So, yeah, that was it. So I started writing the code January one, twenty twenty three. And then I released the first version on GitHub, January sixteen, twenty twenty three. That's how things got started.

• 8:02Speaker 01

  And what's what's the name Comfy UI right away?

• 8:05Speaker 02

  Yeah, Comfy UI. The reason the name my name is Comfy is people thought my pictures were comfy. So I just just named it. It's my comfy UI. So, yeah, that's.

• 8:18Speaker 01

  Is there a particular segment of the community that you targeted as users, like more intensive workflow artists, you know, compared to the automatic crowd or, you know?

• 8:28Speaker 02

  This was my way of like experimenting with with new things. I like the high risk fix thing I mentioned, which was like in comfy, the first thing you could easily do was just chain different models together. And then one of the first things, I think the first times it got a bit of popularity was when I started experimenting with the different like applying prompts to different areas of the image.

• 8:55Speaker 02

  Yeah, I called it area conditioning, posted it on Reddit and it got a bunch of upvotes. So I think that's when like, when people first learned of config UI.

• 9:08Speaker 01

  Hmm. Is it mostly like fixing hands?

• 9:10Speaker 02

  I don't know. That was just like, let's say, well, it was very, well, it still is kind of difficult to like, let's say you want a a mountain, you have an image and then, okay, I want a mountain here and I want the like a a fox here.

• 9:28Speaker 01

  Hmm. Yeah. So compositing the image.

• 9:30Speaker 02

  Yeah. My way was very easy. It was just like when you run the diffusion process, you kind of generate, okay, you do pass one pass through the diffusion model. Every step you do one pass, okay. This place of the image with this prompt, this place, place of the image with the other prompt and then the entire image with another prompt and then just average everything together every step. And that was area composition, which I call it. And then then a month later, there was a paper that came out called multi-diffusion, which was the same thing. But yeah. That's uh.

• 10:11Speaker 01

  Could you do area composition with different models? Or because you're averaging out, you kind of need the same model?

• 10:17Speaker 02

  Could do it with, but yeah, I hadn't implemented it for different models. But you you can do it with with different models if you want, as long as the models share the same latent space. Like.

• 10:32Speaker 01

  We we're supposed to ring a bell every time someone says latent space.

• 10:36Speaker 02

  Yeah. Like, for example, you couldn't use like Excel and SD one point five, because those have a different latent space, but like, yeah, like SD one point five models, different ones, you could, you could do that.

• 10:50Speaker 01

  There's some models that try to work in pixel space, right?

• 10:54Speaker 02

  Yeah, they're very slow.

• 10:56Speaker 01

  Of course.

• 10:57Speaker 02

  That's the problem. That's the, the reason why stable diffusion actually became like popular like cause was because of the latent space.

• 11:05Speaker 01

  Yeah, small and yeah, because it used to be latent diffusion models and then they trained it up.

• 11:10Speaker 02

  Yeah, because the pixel, pixel diffusion models are just too slow, so.

• 11:15Speaker 01

  Yeah. Have you ever tried to talk to like, like Stability, the latent diffusion guys, like, you know, Robin Ronback, that, that crew?

• 11:22Speaker 02

  Yeah, well, I used to work at Stability.

• 11:25Speaker 01

  Oh, I actually knew.

• 11:26Speaker 02

  Yeah, I used to work at Stability. I got, I got hired in June twenty twenty three.

• 11:33Speaker 01

  Ah, that's the part of the story I didn't know about. Okay.

• 11:37Speaker 02

  So the, the reason I was hired is because they were doing SDXL at the time. And they were basically, SDXL, I don't know if you remember, it was a base model and then a refiner model. Basically, they wanted to experiment, like chaining them together. And then they saw, oh.

• 11:55Speaker 01

  Right.

• 11:56Speaker 02

  Come, oh, this, we can use this to do that. Well, let's hire that guy.

• 12:01Speaker 01

  But they didn't, they didn't pursue it for like SD three?

• 12:04Speaker 02

  What do you mean?

• 12:04Speaker 01

  Like the SDXL approach?

• 12:06Speaker 02

  Yeah, the reason for, for that approach was because basically they had two models. And then they wanted to publish both of them. So they, they trained one on lower time steps, which was the refiner model. And then they, the first one was trained normally. And then they, during their test, they realized, oh, like if we string these models together, our like quality increases. So let's publish that.

• 12:37Speaker 01

  It worked.

• 12:38Speaker 02

  Yeah. But like right now, I don't think many people actually use the refiner anymore. Even though it is actually a full diffusion model, like you can use it on its own and it's going to generate images. I don't think anyone, people have mostly forgotten about it. But.

• 12:56Speaker 01

  Can we talk about models a little bit? So stable diffusion, obviously, is the most known. I know Flux has gotten a lot of traction. Are there any underrated models that people should use more? Or what's the state of the union?

• 13:08Speaker 02

  Well, the latest state of the art, at least, yeah, for images, there's, yeah, there's Flux. There's also SD 3.5. SD 3.5 is two models. There's a, there's a small one, 2.5B, and there's the bigger one, 8B. So it's, it's smaller than Flux, so. And it's more creative in a way. But Flux, yeah, Flux is the best. People should give SD 3.5 a try because it's, it's different. I won't say it's better. Well, it's better for some like specific use cases. If you want some, to make something more like creative, maybe SD 3.5.

• 13:51Speaker 02

  If you want to make something more consistent than Flux, it's probably better.

• 13:57Speaker 01

  Do you ever consider supporting the closed source model APIs?

• 14:01Speaker 02

  Uh, well, we do support them with our custom nodes. We actually have some official custom nodes from different.

• 14:10Speaker 01

  Ideogram.

• 14:11Speaker 02

  Yeah.

• 14:12Speaker 01

  I guess Dolly would have one.

• 14:14Speaker 02

  Yeah, that's, it's just not, I'm not the person that handles that.

• 14:19Speaker 01

  Sure, sure. Quick question on, on SD. There's a lot of community discussion about the transition from SD 1.5 to SD 2 and then SD 2 to SD 3. People still like, you know, very loyal to the previous generations of SDs?

• 14:33Speaker 02

  Uh, yeah, SD 1.5 and still has a lot of, a lot of users.

• 14:38Speaker 01

  The last based model.

• 14:40Speaker 02

  Yeah, then SD 2 was mostly ignored because it wasn't, it wasn't a big enough improvement over the previous one.

• 14:49Speaker 01

  Okay, so SD 1.5, SD3, Flux.

• 14:52Speaker 02

  Yeah, SDXL.

• 14:53Speaker 01

  SDXL.

• 14:54Speaker 02

  SDXL, that's the main one.

• 14:55Speaker 01

  Stable Cascade?

• 14:56Speaker 02

  Stable Cascade, that was a good model, but the problem with that one is it got, like SD3 was announced one week after.

• 15:07Speaker 01

  Yeah, it was like a weird release. What was it like inside of Stability, actually? I mean, statute of limitations expired, you know, management has moved, so it's easier to talk about now.

• 15:18Speaker 02

  Yeah, and inside Stability, actually that model was ready like three months before, but it got stuck in red teeming. So basically, the problem, if that model had released or was supposed to be released by the authors, then it would probably have gotten very popular since it's a step up from SDXL. But it got all of its momentum stolen by the SD3 announcement, so people kind of didn't develop anything on top of it, even though it's, yeah. It was a good model, at least completely, mostly ignored for some reason, like.

• 15:58Speaker 01

  It seemed, I think the naming as well matters. It seemed like a branch off of the main tree of developments.

• 16:06Speaker 02

  Yeah, well, it was different researches that did it.

• 16:09Speaker 01

  Different arch, yeah.

• 16:10Speaker 02

  Yeah, very, like, good model, like it's the Worsterson authors, I'm not sure if I'm pronouncing it correctly.

• 16:17Speaker 01

  Worshend, yeah.

• 16:19Speaker 02

  Worshend, yeah.

• 16:19Speaker 01

  I actually met them in Vienna.

• 16:21Speaker 02

  Yeah, they worked at Stability for a bit and they left right after the Cascade release.

• 16:26Speaker 01

  This is Dustin, right?

• 16:27Speaker 02

  No.

• 16:28Speaker 01

  Dustin's SD3.

• 16:29Speaker 02

  No, Dustin is SD3, SDXL. So that's Pablo and Dome. I think I'm pronouncing his name correctly. That's very good.

• 16:42Speaker 01

  It seems like the community is very, they move very quickly.

• 16:46Speaker 02

  Yeah.

• 16:46Speaker 01

  Like when there's a new model out, they just drop whatever the current one is and they just all move wholesale over. Like they don't really stay to explore the full capabilities. Like if the Stable Cascade was that good, they would have A/B tested a bit more. Instead, they're like, okay, SD3 is out, let's go, you know?

• 17:02Speaker 02

  Well, I find the opposite, actually. The community doesn't, like, they only jump on a new model when there's a significant improvement. Like, if there's only, like, incremental improvement, which is what most of these models are going to have, especially if you, because stay the same parameter counter.

• 17:23Speaker 01

  Yeah.

• 17:23Speaker 02

  Like, you're not gonna get a massive improvement, like, unless there's something big that changes, so.

• 17:31Speaker 01

  Yeah. And how are they evaluating these improvements? Like, because it's a whole chain of, you know, Comfy workflows.

• 17:38Speaker 02

  Yeah.

• 17:39Speaker 01

  How does, how does one part of the chain actually affect the whole process?

• 17:43Speaker 02

  Are you talking on the model side specific?

• 17:45Speaker 01

  Model specific, right? But like, once you have your whole workflow based on a model, it's very hard to move.

• 17:52Speaker 02

  Uh, not, well, not really, well, it depends on your, depends on the specific kind of workflow.

• 17:59Speaker 01

  Yeah. So I do a lot of, like, texted image.

• 18:02Speaker 02

  Yeah. When you do change, like, most workflows are kind of gonna be compatible between different models. It's just like, you might have to completely change your prompt, completely change.

• 18:15Speaker 01

  Okay, well, I mean, maybe the question is really about evals. Like, what does the Comfy community do for evals? Just, you know.

• 18:22Speaker 02

  Well, that they don't really do, it's more like, oh, I think this image is nice. So that's.

• 18:29Speaker 01

  They just subscribe to Forfer AI to see, like, you know, what Forfer is doing.

• 18:34Speaker 02

  Yeah, well, they just, they just generate, like, like, I don't see anyone really doing, like, at least on the Comfy side. Comfy users, they, it's more like, oh, generate images and see, oh, this one's nice, this is like.

• 18:47Speaker 01

  Yeah.

• 18:48Speaker 02

  Yeah, it's not, like, the, the more, like, scientific, like, like, checking, that's more specifically on, like, model side.

• 19:01Speaker 02

  Yeah, but there is a lot of vibes also because it is a like artistic. You can't create a very good model that doesn't generate nice images. Because mo-most of the images on the internet are ugly, so if you if that's like, if you just, oh, I have the best model that can, it's super smart, I created on all the like, I trained on just all the images on the internet. The images are not gonna look good, so.

• 19:33Speaker 01

  Yeah, yeah.

• 19:34Speaker 02

  They're gonna be very consistent, but yeah, people like, it's not gonna be like the the look that people are gonna be expecting from a, from a model, so.

• 19:46Speaker 01

  Can we talk about Laura's? Because we talk, we talk about models, then like, the next step is probably Laura's. Before, I, I actually, I'm kind of curious how Laura's entered the toolset of the image community. Because the Laura paper was twenty twenty one. And then like, there was like, other methods like textual inversion that was popular at, at the early SD stage.

• 20:04Speaker 02

  Yeah, I can't even explain the difference between that. Like textual inversions, that's basically what you're doing is you're, you're training a, because well, yeah, stable diffusion, you have the diffusion model, you have text encoder. So basically, what you're doing is training a vector that you're gonna pass to the text encoder. It's basically you're training a new word.

• 20:28Speaker 01

  Yeah, it's a little bit like representation engineering now.

• 20:30Speaker 02

  Yeah, basically, yeah, you're just, so yeah, if you know how like, the text encoder works, basically you have a you, you take your, your words of your prompt, you convert those into tokens with the tokenizer. And those are converted into vectors. Basically, yeah, each token represents a different vectors of each word. Presents a vector and those, depending on your words, that's the list of vectors that get passed to the text encoder, which is just yeah, just a stack of, of attention. Like, basically, it's a

• 21:07Speaker 02

  Very close to L L M architecture. Yeah, yeah, so basically what you're doing is just training a new vector. We're saying, wow, I have all these images and I want to know which word does that represent? And it's gonna get like you train this vector and then and then when you use this vector, it hopefully generates like something similar to your images.

• 21:33Speaker 01

  Yeah, I would say it's like surprisingly sample efficient in picking up the concept that you're trying to train it on.

• 21:39Speaker 02

  Yeah, well, people have kind of stopped doing that, even though back at like when I was at stability, we we actually did train internally some like textual versions on like T five X X L. It actually worked pretty well, but for some reason, yeah, people don't use them. And also they might also work like like, yeah, that's just something you'd probably have to test. But maybe if you train a textual inversion like on T five X X L, it might also work with all the other models that use T five X X L. Because same thing with like, like the textual versions that that were trained for S D one point five,

• 22:21Speaker 02

  they also kind of work on S D X L because S D X L has the has two text encoders. One of them is the same as the as the S D one point five clip out. So those they actually were they don't work as strongly because they're only applied to one of the text encoders. But and the same thing for S D three three, S D three has three text encoders. So it works, it's still you can still use your textual version S D one point five on S D three, but it's just a lot weaker because now there's three text encoders. So it gets even more diluted.

• 22:56Speaker 01

  Yeah, do people experiment a lot on just on the clip side? Uh, there's like SigLip, there's Blip, like, do people experiment a lot on on?

• 23:03Speaker 02

  You can't really replace.

• 23:05Speaker 01

  Yeah, because they're trained together, right?

• 23:06Speaker 02

  Yeah, they're trained together, so you you can't like, well, what I've seen people experimenting with is a long clip. So basically someone fine tuned the clip model to accept longer prompts.

• 23:19Speaker 01

  Oh, it's kind of like long context fine tuning.

• 23:22Speaker 02

  Yeah, so so like it's it's actually supported in Core comfy.

• 23:26Speaker 01

  How long is long?

• 23:27Speaker 02

  Regular clip is seventy seven tokens.

• 23:30Speaker 01

  Yeah.

• 23:30Speaker 02

  Long clip is two hundred fifty six.

• 23:34Speaker 01

  Okay.

• 23:34Speaker 02

  So but the hack that like you if you use stable diffusion one point five, you've probably noticed, oh, it still works if I if I use long prompts, prompts longer than seventy seven words, well, that's because the hack is to just, well, you split, you split it up in chunks of seven, your whole, your big prompt, say you you give it like the massive text like the Bible or something. And it would split it up in chunks of seven and then just pass each one through the clip. And then just cut anything together at the end. It's not ideal, but it actually works.

• 24:17Speaker 01

  Like the positioning of the words really, really matters then, right? Like, this is why order matters in prompts.

• 24:23Speaker 02

  Yeah, yeah, like it it works, but it's it's not ideal, but it's what people expect. Like, if if someone gives a huge prompt, they expect at least some of the concepts at the end to be like present in the image. But usually when they give long prompts, they they don't like, they don't expect like detail, I think. So that's why it works very well.

• 24:50Speaker 01

  And while we're on this topic, prompt weighting, negative prompting, all all similar part of this layer of the stack.

• 24:56Speaker 02

  Yeah, the the hack for that which works on clip, like it basically, it's just for SD one point, well, for SD one point five, the prompt weighting works well because clip L is a is not a very deep model. So you have a very high correlation between you have the input token, the index of the input token vector and the output token. They're very, the concepts are very close closely. So that means if you interpolate the vector from what, well, the, the way Kung Fu UI does it is it has, okay, you have the vector, you have a empty prompt.

• 25:42Speaker 02

  So you have a, a child, like a clip output for the empty prompt. And then you have the one for your prompt. And then it interpolates from that depending on your prompt weight, the weight of your, of your tokens. So, so if you, yeah, so that's how, how it does prompt weighting. But this stops working the deeper your text encoder is. So on T5X, it doesn't work at all. So.

• 26:10Speaker 01

  Wow. Is that a problem for people? I mean, because I'm, I'm used to just move, moving up numbers.

• 26:15Speaker 02

  Probably not. Yeah.

• 26:16Speaker 01

  Well. So you just use words to describe, right? Because it's a bigger language model.

• 26:20Speaker 02

  Yeah. Yeah. So honestly, it might be good, but I haven't seen many complaints on Flux that it's not working. So, because I guess people can sort of get around it with, with language. So.

• 26:37Speaker 01

  Yeah. And then coming back to LoRAS. Now the, the popular way to, to customize models is LoRAS. And I, I saw you also support LoCon and LoHa, which I've never heard of before.

• 26:47Speaker 02

  There's a bunch of, because what, what the LoRAS is essentially is instead of like, okay, you have your, your model and then you want to fine tune it. So instead of like, what you could do is you could fine tune the entire thing.

• 27:01Speaker 01

  Yeah, full fine tune. Yeah.

• 27:02Speaker 02

  But that's a bit heavy. So to speed things up and make things less heavy, what you can do is just fine tune some smaller weights, like basically two.

• 27:16Speaker 02

  Two matrices that when you multiply like two low rank matrices, then when you multiply them together gives a represents a difference between trained weights and your base weights. So by training those two smaller matrices, that's a lot less pain.

• 27:36Alessio Fanelli?

  And they're portable. So you're gonna share them.

• 27:38Speaker 02

  Yeah, and also smaller. Yeah, that's the how LoRa's work. So basically, so when when inferencing, you get an inference with them pretty efficiently, like how Comfy UI does it, it just, when you use a LoRa, it just applies it straight on the weights, so that there's only a small delay at the beginning, like before the sampling to when it applies the weights, and then it just same speed as as before. So for for inference, it's it's not that bad. But and then you have, so basically, all the LoRa types, like LoHa, LoCon, everything, that's just different ways of representing that, like, basically, you can call it kind

• 28:22Speaker 02

  of like compression, even though it's not really compression. It's just different ways of represented, like, just, okay, I want to train a different on the difference on the weights, what's the best way to represent that difference. There's the basic LoRa, which is just, oh, let's multiply these two matrices together. And then there's all the other ones, which are all different algorithms. So.

• 28:48Alessio Fanelli?

  Yep. So let's talk about what Comfy UI actually is. I think most people have heard of it. Some people might have seen screenshots. I I think fewer people have built very complex workflows. So when you started, automatic was like the super simple way. What were some of the choices that you made? So the node workflow, is there anything else that stands out? It's like this was like a unique take on how to do image generation workflows?

• 29:13Speaker 02

  Well, I feel like, yeah, back then, everyone was trying to make like easy to use interface. I'm like, well, everyone's trying to make an easy to use interface.

• 29:23Alessio Fanelli?

  Let's make a hard to use interface.

• 29:28Speaker 02

  Like, so like, I like, I don't need to do that.

• 29:33Speaker 02

  And everyone else doing it. So let me try something, like, let me try to make a powerful interface.

• 29:39Alessio Fanelli?

  Yeah, yeah.

• 29:40Speaker 02

  That's not easy to use, so.

• 29:44Speaker 01

  So like, yeah, there's a sort of node execution engine. Your reading actually lists, has this really good list of features of things you prioritize, right? Like, um, let me see, like, uh, sort of re-executing from, from any parts of the workflow that was changed. A-se只要有Q system, smart memory management, like, all this seems like a lot of engineering that.

• 30:03Speaker 02

  Yeah, there's a lot of engineering in the, in the backend to make things. Cause I was always focused on making things work locally very well, cause that's, cause I was using it locally. So everything, uh, so there's a lot of, uh, a lot of thought and work in like getting everything to run as well as possible. So, yeah, ConfyUI is actually more of a backend, at least, well, now the front end's getting a lot more development. But, but before, before it was, I was pretty much only focused on the backend.

• 30:40Speaker 01

  Yeah, so V zero point one was only August this year.

• 30:45Speaker 02

  Yeah, before there was no versioning, so.

• 30:47Speaker 01

  Yeah, yeah. And so what was the big rewrite for the point one and then the one point O?

• 30:52Speaker 02

  Uh, well, that's more on the frontend side. That's, cause before that, it was just like the UI, what? Cause when I first wrote it, I just, I said, okay, how can I make like, I can do web development, but I don't like doing it. Like, what's the easiest way I can slap a node interface on this? And then I found this library, LightGraph, like JavaScript library.

• 31:17Speaker 01

  LightGraph?

• 31:18Speaker 02

  LightGraph.

• 31:19Speaker 01

  Usually people will go for like React Flow for like a flow builder.

• 31:22Speaker 02

  Yeah, but that seems like too complicated.

• 31:26Speaker 02

  So I didn't really want to spend time like developing the, the front end. So I'm like, well, oh, LightGraph, this has the whole node interface. So, okay, let me just plug that into, to my backend and.

• 31:40Speaker 01

  I feel like if Streamlit or Gradio offered something, you would have used Streamlit or Gradio, cause it's Python.

• 31:44Speaker 02

  Yeah, Streamlit and Gradio, yeah, Gradio, I don't like Gradio.

• 31:48Speaker 01

  Why?

• 31:48Speaker 02

  It's bad, like, the, that's, that's one of the reasons why, like, automatic was very bad. It's great, cause, uh. The problem with Gradle, it forces you to, well, not forces you, but it kind of makes your your interface logic and your backend logic and like just sticks them together.

• 32:11Speaker 01

  It's supposed to be easy for you guys for if you're a Python main, you know, I'm a JS main, right?

• 32:15Speaker 02

  Yeah.

• 32:16Speaker 01

  If you're a Python main, it's supposed to be easy.

• 32:17Speaker 02

  Yeah, it's it. Well, it's easy, but it makes your whole software a huge mess.

• 32:21Speaker 01

  I see, I see. So you're mixing concerns instead of separating concerns?

• 32:25Speaker 02

  Well, it's because. Like front end and backend. You front end and backend should be well separated with a defined API. Like, that's that's how it was supposed to do.

• 32:35Speaker 01

  People, smart people disagree, but yeah.

• 32:37Speaker 02

  It just stick, it sticks everything together. It makes it easy to like a huge mess. And also, it's, there's a lot of issues with Gradle. Like, it's very good if all you want to do is just get like slap a quick interface on your like to to show off your like your ML project, like that's what it's made for.

• 33:01Speaker 01

  Yeah, yeah.

• 33:02Speaker 02

  Like, there's no problem using it like, oh, I have my, I have my code. I just want a quick interface on it. That's perfect. Like, use Gradio. But if you want to make something that's like a real, like real software that will last a long time and will be easy to maintain, then I would avoid it.

• 33:23Speaker 01

  Yeah, so. So your criticism is Streamlit and Gradle are the same, I mean, those are the same criticisms?

• 33:29Speaker 02

  Yeah, Streamlit, I haven't. Haven't used it. Yeah, it's just looked a bit.

• 33:34Speaker 01

  Similar philosophy.

• 33:35Speaker 02

  Yeah, it's similar. It's just, it just seems to me like, okay, for quick like AI demos, it's perfect.

• 33:42Speaker 01

  Yeah. Going back to like the the core tech, like asynchronous queues, slow re-execution, smart memory management, you know, anything that you you're very proud of or was very hard to figure out?

• 33:51Speaker 02

  Yeah, the thing that's the biggest pain in the ass is probably the memory management.

• 33:56Speaker 01

  Yeah. Or you're just paging models in and out, or?

• 33:59Speaker 02

  Yeah, before it was just, okay, load the model, completely unload it, load the new model, completely unload it. Then, okay, that that works well when your your model are small. But if your models are big and it takes sort of like, let's say, someone has a like a A490. And the model size is ten gigabytes. That can take a few seconds to like load and load, load and load. So you want to try to keep things like in memory, in the GPU memory as much as possible. What comfy UI does right now is that it tries to like estimate, okay, like, okay, you're gonna sample this model. It's gonna take probably this amount of memory.

• 34:42Speaker 02

  Let's remove the models like this amount of memory. And you load that's been loaded on the GPU and then just execute it. But so there's a fine line between just, because try to remove the least amount of models that are already loaded. Because as like Windows driver. And another problem is the Nvidia driver on Windows by default. Because there's a way to, there's an option to disable that feature. But by default, it like, if you start loading, you can overflow your GPU memory

• 35:25Speaker 02

  and then it's, the driver's gonna automatically start paging to RAM. But the problem with that is it's, it makes everything extremely slow. So when you see people complaining, oh, this model, it works, but oh shit, it starts slowing down a lot, that's probably what's happening. So it's basically have to just try to get, use as much memory as possible, but not too much, or else things start slowing down, or people get out of memory. And then just find, try to find that line where, oh, like the drive around window starts paging and stuff.

• 36:03Speaker 01

  Yeah.

• 36:03Speaker 02

  And yeah, the problem with PyTorch is it's, it's high levels. Don't have that much fine grain control over like specific memory stuff. So I have to leave like the memory freeing to Python and PyTorch, which is, can be annoying sometimes.

• 36:23Speaker 01

  So, you know, I think one thing is, as a maintainer of this project, like you're designing for a very wide

• 36:29Speaker 02

  Yeah.

• 36:29Speaker 01

  surface area of compute, like you even support CPUs.

• 36:33Speaker 02

  Yeah, well, that's, that's just PyTorch. PyTorch supports CPUs. Yeah, it's just, that's not, that's not hard to support.

• 36:41Speaker 01

  First of all, is there a market share estimate, like, is it like 70% Nvidia, like 30% AMD, and then like miscellaneous on Apple, Silicon, or whatever?

• 36:49Speaker 02

  Uh, for comfy?

• 36:51Speaker 01

  Yeah.

• 36:51Speaker 02

  Yeah, I'm like, yeah, I don't know the market share.

• 36:55Speaker 01

  Can you guess?

• 36:55Speaker 02

  Uh, I think it's mostly Nvidia.

• 36:58Speaker 01

  Right.

• 36:59Speaker 02

  Yeah. Because AMD, the problem, like AMD works horribly on Windows. Like on Linux, it works fine. It's, it's slower than the price equivalent Nvidia GPU. But it works, like, you can use it, generate images, everything works. On Linux, on Windows, you might have a hard time. So that's the problem. And most people, I think most people who bought AMD probably use Windows.

• 37:29Speaker 02

  Probably aren't gonna switch to Linux. So, so until AMD actually like ports their like Rockem to, to Windows properly. And then there's actually PyTorch. I think they're, they're doing that. They're in the process of doing that. But until they get, they get a good like PyTorch, Rockem build that works on Windows, it's like, they're gonna have a hard time.

• 37:57Speaker 01

  Yeah. We gotta get George on it.

• 37:59Alessio Fanelli?

  Yeah, well, he's trying to get Lisa Su to do one, but let's talk a bit about like the node design. So unlike all the other text to image, you have a very like deep, so you have like a separate node for like clip and code. You have a separate node for like the case sampler. You have like all these nodes. Going back to like the making it easy versus making it hard. But like, how much do people actually play with all the settings? You know, kind of like, how do you guide people to like, hey, this is actually gonna be very impactful versus this is maybe like less impactful, but we still want to expose it to you?

• 38:31Speaker 02

  Uh, well, I tried to expose, like, I tried to expose everything or, but, yeah. At least for the, but for things like, for example, for the samplers, like, there's, uh, like, yeah, four different sampler nodes which go in easiest to most advanced. So, yeah, if you go like the easy node, the regular sampler node, that's, you have just the basic settings. But if you use like the sampler advanced, custom advanced node, that, that's when you can actually, you'll see you have like different nodes.

• 39:10Alessio Fanelli?

  I'm looking it up now.

• 39:11Speaker 02

  Yeah.

• 39:12Alessio Fanelli?

  What are like the most impactful parameters that you use? So, it's like, you know, you're gonna have more, but like, which ones like really make a difference? C F G.

• 39:21Speaker 02

  Yeah, they all do. They all have their own, like, they all, like, for example, yeah, steps, usually you want the steps, you want them to be as low as possible, but you want, if you're optimizing your workflow, you want to, you lower the steps until like the image will start deteriorating too much, because that, yeah, that's the number of steps you're, you're running the diffusion process. So if you want things to be fast, that's, lower is better. But, yeah, C F G, that's more, you can kind of see that as the contrast of the image. Like, if your image looks too burnt out, then you lower the C F G.

• 40:04Speaker 02

  So, yeah, C F G, that's how, yeah, that's how strongly the, like, the negative versus positive problem. Because when you sample a diffusion model, it's, it's basically a negative prompt, it's just, yeah, positive prediction minus negative prediction.

• 40:23Alessio Fanelli?

  Contrastive loss.

• 40:24Speaker 02

  Yeah. Positive minus negative and the CFG does the multiplier.

• 40:29Alessio Fanelli?

  Yeah.

• 40:30Speaker 02

  That's yeah, so.

• 40:32Alessio Fanelli?

  What are like good resources to understand what the parameters do? I think most people start with automatic and then they move over and it's like step CFG sampler name scheduler denoise. Reddit.

• 40:44Speaker 02

  But honestly, well, it's more, it's something you should like try out yourself. I don't know, you, you don't necessarily need to know how it works to like what it does. Cause even if you know like CFGO, it's like positive minus negative problem.

• 41:01Alessio Fanelli?

  Yeah.

• 41:01Speaker 02

  So the only thing you know with CFG is if it's 1.0, then that means the negative problem isn't applied. You know, so maybe sampling is two times faster, but yeah. But other than that, it's more like you should really just see what it does to the images yourself. And you'll, you'll probably get the more intuitive understanding of what these things do.

• 41:25Alessio Fanelli?

  Any other nodes or things you want to shout out? Like I know the animated if IP adapter, those are like some of the most popular ones. Um, yeah, what else comes to mind?

• 41:35Speaker 02

  Uh, not nodes, but there's a like, what I like is when, when some people sometimes they make, they make things that use comfy UI as their backend. Like there's a, a plugin for Creta that uses comfy UI as its backend. So you can use like all the models that work in comfy in Creta. And I think I've tried it once, but I know a lot of people use it and find it really nice, so.

• 42:06Alessio Fanelli?

  What's the craziest node that people have built, like the most complicated?

• 42:12Speaker 02

  Craziest node like, I like, yeah, I know some people have made like video games and.

• 42:20Speaker 02

  In comfy, we like stuff like that. So like someone, I gotta remember, like, yeah, last, I think it was la- last year, someone made like a, like Wolfenstein through the comfy. And then one of the inputs was, oh, you can generate the texture and then it changes the texture in the game. So you could plug it to like workflow. And there's a lot of, if you like that, there's a lot of crazy things people do. So, you know.

• 42:51Alessio Fanelli?

  And now there's like a node register that people can use to like download nodes and.

• 42:55Speaker 02

  Yeah, like, well, there's always been the like the compute one manager, but we're trying to make this more like, I don't know, official, like with, yeah, with the the node registry. Cause be- before, before the node registry, the it like, okay, how did your custom node get into the compute one manager? That's the guy running it who like every day he searched GitHub for new custom nodes and added them manually to his, to his custom node manager. So we're trying to make it a less effort for him, basically.

• 43:31Alessio Fanelli?

  Yeah, but I was looking, I mean, there's like a YouTube download node. There's like, thi- this is almost like, you know, a data pipeline, more than like an image generation thing at this point. It's like you can get data in, you can like apply filters to it, you can generate data out.

• 43:45Speaker 02

  Yeah, you can do a lot of different things. Yeah, so I think, I think what I did is I made it easy to make custom nodes. So I think that that helped a lot for the ecosystem. Cause it is very easy just make a node. So, yeah, a bit too easy sometimes. Then, then we have the issue where there's a lot of custom node packs which share similar nodes. So, but well, that's.

• 44:19Speaker 02

  Yeah, something we're trying to solve by maybe bringing some of the functionality into core.

• 44:25Alessio Fanelli?

  Yeah, yeah, yeah.

• 44:26Speaker 02

  Yeah.

• 44:28Alessio Fanelli?

  And then there's like video, people can do video generation.

• 44:31Speaker 02

  Yeah, video, that's, well, the first video model was like stable video diffusion, which was last, yeah, exactly last year, I think, like one year ago. But that wasn't a true video model, so it was.

• 44:46Alessio Fanelli?

  It's like moving images?

• 44:48Speaker 02

  Yeah, generated video, what I mean by that is it's like, it's still 2D latents. It's basically what they did is they took SD2 and then they added some temporal attention to it and then trained it on videos and so it's, it's kind of like animated, like same, same idea, basically. Why I say it's not a true video model is you still have like the 2D latents. Like a true video model like Mochi, for example, would have 3D latents.

• 45:23Alessio Fanelli?

  Which means you can like move through the space, basically, is the difference. You're not just kind of like reorienting.

• 45:29Speaker 02

  Yeah, and it's also, well, it's also because you have a temporal VAE. But also, like Mochi has a temporal VAE that compresses on like the temporal direction also. So that's something you know have with like, yeah, animated diff and stable video diffusion. They only like compress spatially, not temporally. So, yeah, so these models, that's why I called it like true video models. There's, yeah, there's actually a few of them, but the one I've implemented in Kaffee is Mochi because that, that seems to be the best one so far.

• 46:06Alessio Fanelli?

  We had AJ come and speak at the stable diffusion meetup. Other open one I think I've seen is CogVideo.

• 46:12Speaker 02

  Yeah, CogVideo, yeah, that one's, yeah, it also seems decent, but, yeah.

• 46:18Alessio Fanelli?

  Chinese, so we don't.

• 46:20Speaker 02

  No, it's fine, it's just, yeah, I could, yeah, it's just, it, there's a, it's not the only one. There's also a few others which I.

• 46:28Alessio Fanelli?

  The rest are like closed stores, right, like cling.

• 46:30Speaker 02

  Yeah, closed stores, there's a bunch of them, but I mean, open, I've seen a few of them. Like, I can't remember their names, but there's Cog, Cog videos, the big, the big one. Then there's also a few of them that released at the same time. There's one that released that at the same time as SD 3.5, same day, which is why I don't remember the name.

• 46:54Alessio Fanelli?

  We should have a release schedule so we don't conflict on each of these things.

• 46:56Speaker 02

  Yeah, I think SD 3.5 and Mochi released on the same day. So everything else was kind of drowned, completely drowned out. So for some reason, lots of people picked that day to release their stuff.

• 47:12Speaker 02

  Yeah, which is, well, shame for those in gas. And think Omni Jet also released the same day, which also seems interesting.

• 47:21Alessio Fanelli?

  Yeah, what's Comfy? So you are Comfy and then there's like comfy.org. I know we do a lot of things for like news research and those guys also have kind of like a more open source and on thing going on. How do you work? Like you mentioned, you must have worked on like the core piece of it and then what?

• 47:38Speaker 02

  Maybe I should fill it because I, yeah, I, yeah, I feel like maybe, yeah, I only explained part of the story.

• 47:45Alessio Fanelli?

  Right.

• 47:46Speaker 02

  Yeah, maybe I should explain the rest. So, yeah. So yeah, basically, January, that's when the first January 2023, January 16th, 2023. That's when MFI was first released to the public. Then, yeah, did a Reddit post about the area composition thing somewhere in, uh, if you don't remember exactly, maybe the end of January, beginning of February. And then someone, a YouTuber made a video about it, like Olivia. He made a video about Comfy in March 2023. I think that's when it first real burst of attention.

• 48:27Speaker 02

  And by, yeah, that time I was continuing to developing it and it was getting, uh, people were starting to use it more, which unfortunately meant that I, my, I had first written it to do like experiments, but then my time to do experiments went, started going down.

• 48:49Speaker 02

  Yeah, because people were actually starting to use it and like I had to and I say, well, yeah, I need to add all these features and stuff. Yeah, and then I got hired by Stability in June twenty twenty three. Then I made the basically, yeah, they hired me because they wanted the SDXL. So I got SDXL working very well in comfy UI because they were experimenting with it. Actually, the SD, how the SDXL released worked is they released, for some reason, like they released the code first. But they didn't release the model checkpoint. Oh, yeah, so they released a couple.

• 49:30Speaker 02

  And then, well, if since the research was with the code, I released the code in comfy two. And then the checkpoints were basically early access. People had to sign up and they only allowed a lot of people from edu emails. Like if you had an edu email, like that they gave you access basically to the zero SDXL zero point nine. And well, that leaked. Right. Of course, because of course it's kind of leaky if you do that. Uh, well, the only way people could easily use it was with comfy. So, yeah, people started using it and then I fixed a few of all the issues people had.

• 50:13Speaker 02

  So then the big one point zero release happened and well, comfy UI was the only way a lot of people could actually run it on their computers. Because it just like automatic was so like inefficient and bad that most people couldn't act like it just wouldn't, wouldn't work. Like, because he, he did a quick implementation, so people were forced to use comfy UI. And that's how it became popular because people had no choice.

• 50:46Alessio Fanelli?

  The growth hack.

• 50:47Speaker 02

  Yeah. Yeah, like everywhere, like people who didn't have the forty ninety, they had like, who had just regular GPUs. Yeah, yeah. They just, they didn't have a choice, so.

• 50:57Alessio Fanelli?

  Yeah, I got a forty seventy, so think of me. And so today, what's, is there like a core comfy team or?

• 51:04Speaker 02

  Uh, yeah, well, right now, uh, yeah, we are hiring, actually. So right that core could like, um, the core core itself, it's, it's me. And by because the reason we're folk like all the focus has been mostly on the front end right now. Cause that's the thing that's been neglected for a long time. So, uh, so most of the focus right now is, uh, all on the front end, but we are, uh, yeah, we will soon get, uh, more people to like help me with the actual backend stuff.

• 51:40Speaker 02

  Because that's, uh, once the, once we have our V one release, which is gonna be a package comfy one with the nice interface and easy to install on Windows. And hopefully Mac, yeah. Yeah. Once we have that, uh, we're gonna have to, lots of stuff to do on the backend side and also the frontend side, but, uh.

• 52:05Alessio Fanelli?

  What's the release date? I'm on the waitlist. What's the timing?

• 52:09Speaker 02

  Uh, soon, soon. Yeah, like I don't want to promise a release date because, yeah, we, we do have a, a release date we're targeting, but yeah, I'm not sure if, if it's public.

• 52:23Alessio Fanelli?

  Yeah.

• 52:24Speaker 02

  Yeah. And how we're gonna, like, we're, we're still gonna continue, like, from doing the open source, like making comfy Y the best way to run, uh, like stable diffusion models, like at least the, the open source side and like, it's gonna be best way to run all models locally. But we will have a few, like a few things to, to make money from it, like, uh. Cloud entrants or that type of that type of thing. So and maybe some like some things for some enterprises.

• 52:59Speaker 01

  I mean, a few questions on that. How do you feel about the other comfy startups?

• 53:03Speaker 02

  I mean, I think it's great. They're using your name. Yeah, well, it's better to use comfy than to use something else. That's true. Yeah, like, yeah, it's fine. I don't like, we're like, yeah, I'm, we're gonna try not to, we don't want to like, we want them to people to use comfy. So like I said, it's better that people use comfy than something else. So as long as they use comfy, it's, I think it helps, it helps the ecosystem. Because because more people, even if they don't, they like, even if they don't contribute directly, the fact that they are using comfy means that like people are more likely to like join the ecosystem. So, yeah.

• 53:49Speaker 01

  And then would you ever do text?

• 53:50Speaker 02

  Yeah, well, you can already do text with some custom notes. So, yeah, it's something we like, yeah. It's something I've wanted to eventually add to core, but it's more like not a, not a very high priority. But because a lot of people use text for like prompt enhancement and like other things like that. So it's, yeah, it's just that my focus has always been like diffusion models. Yeah, unless some text diffusion model comes out.

• 54:21Speaker 01

  Yeah, David Holtz is investing a lot in.

• 54:24Speaker 02

  Yeah, well, if a good one comes out, then well, I'll probably implant it since it fits with the whole.

• 54:30Speaker 01

  Yeah, I mean, I imagine it's going to be close source to Midjourney, so.

• 54:33Speaker 02

  Yeah, well, if, yeah, if an open one comes out.

• 54:38Speaker 02

  Yeah, then, yeah, I'll probably, yeah, yeah, I'll probably implement it.

• 54:45Alessio Fanelli?

  Cool coffee. Thanks so much for for coming on. This is fun.