Pod Insights Unsupervised Learning with Jacob Effron Ep 16: VP of Generative AI at Adobe Alexandru Costin on The Future of Content Creation About this episode Patrick and Jacob sit down with Alexandru Costin, the VP of Generative AI and Sensei at Adobe, and discuss how Adobe’s early projects with generative AI in 2019 helped them move quickly upon the release of LLMs and diffusion models. Before leading Adobe’s generative AI efforts he founded InterAKT, a web development company, and led Adobe Romania for 10 years.
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 Welcome to Unsupervised Learning Red Points AI Podcast. I'm Patrick Chase. I am joined by Jacob Effron. Today, we spoke with Alexandru Costin, the VP of Generative AI and Sensei at Adobe. Alexandru started his career as an entrepreneur. He founded Interact, which was acquired by Adobe 18 years ago. We talk about how Adobe is building AI into their suite of products.
0:20 Speaker 01 You actually don't even have to type in a prompt. When you click generate, we're going to use the rest of the image as an implied prompt and generate something that matches.
0:28 Speaker 00 He explains how Adobe is leveraging their millions of users to refine their models.
0:32 Speaker 01 Our terms of use allow us to store the prompts and the generated images and use them in training, which is basically our way of doing R and A.
0:38 Speaker 00 And finally, Alexandru shares his vision for Adobe's Creative Cloud in 10 years.
0:42 Speaker 01 We see Creative Cloud evolving into enabling our customers to come from visual designers, so more like creative directors of this meta experiences that will be then hyper-personalizable.
0:53 Speaker 00 Thank you so much for joining us. We hope you enjoyed the episode.
0:58 Speaker 00 Alexandru, I looked this up. Fun fact, the day you started at Adobe, Adobe's market cap was 11.9 billion. Today, their market cap is 237 billion, so a pretty incredible run over the 18 years. So thank you so much for joining us. Really excited about this.
1:16 Speaker 01 Happy to be here. And yes, I did help a little bit, but it's 30,000 people doing hard work with Adobe, serving our customers that got our stock where it is today.
1:25 Speaker 00 I had kind of figured that it was you solely that had created the 200. I'm struggling with the math now. 200 and 10 billion in enterprise value. Maybe to start it off, can you tell us a little bit about your incredible journey at Adobe and some of the projects you've worked on there?
1:42 Speaker 01 Yeah, happy to do so. So originally from Romania, I had Interact, as I said, a small startup doing plugins for Macromedia Dreamweaver. And when Adobe, and we were a Macromedia partner, doing things for them, doing a lot of PHP development. When Adobe acquired Macromedia, the Macromedia folks sent Adobe to us. Adobe was interested in opening up a center in Eastern Europe. They found us. We had amazing web expertise. So they acquired our company and I ended up leading the Romania site. Adobe Romania for ten beautiful years and started doing AI even in Romania but by trajectory kind of accelerated towards the eyes throughout the years.
2:20 Speaker 00 Amazing, incredible background. And maybe just for our listeners, could you walk through kind of some of the AI products now? There's obviously Sensei and a bunch of incredible things that you guys are doing with AI at Adobe like Firefly and some of those other things. But maybe can you walk through the kind of suite of AI products and projects at Adobe?
2:43 Speaker 01 Of course, of course. So Adobe has three big business units. There is the Creative Cloud business unit, the Document Cloud business unit, then the Experience Cloud business unit, and each of them is Creative Cloud. We have Photoshop, we have Illustrator, Adobe Express, many products for content creation and authoring. The Document Cloud is the Acrobat franchise. And the Experience Cloud, we have various content management, this commerce solution, analytics and CRM data hosting for various enterprise customers. And AI is part of all of those, part of the products in each of those, in each of those business units. In Creative Cloud, we train models on what we call the Sensei platform. This is our internally homegrown,
3:27 Speaker 01 and using various public clouds, but homegrown infrastructure that enable hundreds of researchers to train various models that then are tech transferred into Photoshop, into Adobe Express, etc. And things like the masking, auto masking in Photoshop, things like content aware fill, things like object detection in Photoshop or neural filters, these are some of the capabilities that are already deployed in Photoshop with AI. In Document Cloud, in Acrobat, we have this capability called Liquid Mode that enable mobile, we use it AI to reflow documents on mobile phones. So you can read archive papers easier. That's actually a pretty nice hint for everybody that's following AK on Twitter and reads archive paper every day.
4:11 Speaker 01 This Liquid Mode is a really nice way to read them on a phone every morning. If you're, and in the Experience Cloud, we have various
4:20 Speaker 01 AI capabilities to do anomaly detection in analytics, to do content generation, marketing, content brand generation in Adobe Experience Manager, which is our CMS, etc. So throughout the years, actually since they started in 2016 and throughout the years, we've been doing a lot of those smaller models integrated in various workflows. Then last year, we've decided to embrace these larger models as diffusion models and LLMs kind of showed that the quality of the content generated, the quality of the content summarization, it all increased. We've decided to fully embrace large models, both large language models and large diffusion models and create the Firefly suite of models that now we've launched in March.
5:05 Speaker 01 We've been working for a year or so and is doing really great. We've launched as beta. It's not yet out of beta, but even during the beta, we have more than a billion images generated across multiple surfaces. We have the firefly dot adobe dot com website. We have this deep integration in Photoshop called Gerity Fill. We have integration in Adobe Express, which is our marketer slash communicator suite, design suite. So throughout all and in Illustrator, it's gonna, it's gonna be integrated everywhere. But even through this initial wave of integrations, we've seen tremendous success and customers being super excited. But what the Firefly models can do for them, enabling them to be more productive and creative.
5:47 Speaker 00 That's incredible. I mean, it's, it's an amazing overview and just the, the scope of what you guys have done with AI, I think is, is really impressive. And I feel, I don't know, you see a lot of the demos with Photoshop and things like that, which are very, you know, they're very sexy and people get them and they make for great demos. But it was, it's really fascinating to hear about, you know, not just the AI stuff you guys are doing in Photoshop, but also across all of the different pillars of the business. Would love to talk a little bit about kind of specifically on Photoshop, because I think that is something that's, that's really top of mind for, for people. You mentioned a billion photos generated. I think the It was maybe the CEO of Microsoft or someone that was saying that GitHub,
6:28 Speaker 00 forty percent of the new code checked in is now being generated by AI. Do you have a sense of what proportion of design is now being done by AI? Or how you see those workflows changing now that you've launched a bunch of these features?
6:43 Speaker 01 I'll share our strategic perspective. As we haven't gotten out of beta, the Firefly content is not yet usable for commercial purposes. It will be soon when we launch. But even with that, so we could say this one billion images were people playing with it and trying to get and give us feedback. Throughout this last four decades of Adobe history, we've helped our customers navigate various waves of technological disruption in content production, even starting with the first wave, which was digital publishing. Actually, Adobe was partly maybe the key leader in inventing the PostScript format and laser printing and at the time moving publishing to computer or digital publishing.
7:28 Speaker 01 This was a big transition that we have led. And at the time, there were a lot of questions about what will happen with the jobs of the people doing normal publishing. But throughout this transition, digital publishing, and then we had digital photography, we had the internet, we had the mobile wave. Throughout each of those waves of the innovation, we have seen the same pattern applied, the same pattern where content creation gets cheaper and easier to do and more accessible to a larger audience. But which triggers the concerns for the creators about the role of their job security, the role they will play in this new world, et cetera.
8:12 Speaker 01 But with every transition, what happens is the demand for digital content or for content in general increases exponentially, increases more than the decreases in cost of creating content. So because of that, we've actually seen for every transition, we've seen an opportunity for our customers that we serve through various product lines. Become relevant, stay relevant in this new world and actually embrace the new technology. Don't be disrupted by it and and become ten X more productive or creative. So I assume that behind your question, a lot of content will be touched by AI. We do believe content will be touched by AI one way or another.
8:54 Speaker 01 Fully generated, we don't think it's going to be fully automated. We actually do believe deeply that creative people will stay to be the ones in control of both the creation process and the curation of the content being generated. But in the same time, demand will grow exponentially. So we think that customers embracing AI will continue to stay relevant. And this is, this is what we're trying to do because we do think most content will be touched by some AI technology. It already is in Photoshop. If you select something, you're already using some GANs most of the time or selection models, maybe not generative models, but you're already using AI.
9:34 Speaker 01 And with capabilities like generative fill in Photoshop, you generate pixels and they, they, but it's through the process of editing a photo. So even if you're not generating a full new kind of art, you're editing a photo, you're using GNI. So mostly through editing and through some generation too. We think that the majority of the content will be touched by AI and we need to embrace it. Our customers need to embrace it to stay relevant in this new world.
10:02 Jacob Effron? Totally. And I think, you know, obviously you, you've had these features, even though they're already in beta, they're, they're used really widely. I imagine as you were developing them, there were all sorts of like interesting design decisions you had to make about how to like incorporate them into the product. And I think this is something that so many people are trying to figure out right now in, in, in kind of the broader AI world. I, I'd love to hear from you about maybe if there's some specific ones that come to mind where you guys were toying with different ways these could be incorporated into the existing product suite and, and how it kind of came to take the form that it, that it does today.
10:29 Speaker 01 Great question. So we, we do, as you know, Photoshop, we have maybe too many controls and, and knobs and capabilities that were, were added throughout the three plus decades history of the product. So it's like. Some people say that the cockpit of a Boeing seven four seven, so it can take you to fly anywhere, but it's like really powerful, but you need to know how to master it. And there's a lot of mastery that goes into building up knowledge about how to use Photoshop. And when we've seen some of those GANs, actually, part of my history with Firefly is I was leading this project called Neural Filter.
11:10 Speaker 01 So even before, before generative and diffusion models were, were, became so, so good, GANs were, were the best of breed three, four years ago. And this is when we've decided, we've, we've, we saw the power in GANs, the power of, of using AI to basically reinvent the content processing process. Because even GANs, they're very controllable with user intent. You can make somebody smile. I'm not sure if you guys remember some of the demos we had for Neural Filters. We have a feature called Smart Portrait that can change the yo of somebody or make somebody smile or age them or de-age them. And instead of, normally you would do this in Photoshop using adjustment layers and pixel manipulation. It's a very complicated manual process.
11:53 Speaker 01 But what we've started to introduce with Neural Filters was a intent-driven editing, where the GAN would understand the scene and then you would have some sliders, like a smile slider, an age slider, a yo slider. And those sliders already kind of introduced a new, very user-friendly editing paradigm that we were, we're still very proud of because we think it's kind of, it's interactive and it's very delightful. So we've been toying with innovation in user experience for a while. And when we saw prompt-based user interfaces, we, of course, we, they, they do lower the entry barrier massively even for folks that don't know how to use Photoshop.
12:34 Speaker 01 And just by expressing in an English sentence or, or any, any language sentence, what you want to create, something gets created for you. And we've embraced it too, and we started playing with it. The reality is our customer base, they do expect, they do expect more control than just a prompt. So when we think about where those User experiences will land. We think it's got to be a mix of language with pointing and sliders. I don't think we've reached the end state of where the human computer interaction for creativity will reach, because we do think when you just type something, normally, most of the time, you don't get what's in your mind, you get something else. So there's some frustration.
13:18 Speaker 01 I have a daughter, she's 11, Rebecca. First time I showed her Firefly, she started typing in things and she said, yeah, this is not what I imagined. It's not working there. It's like, but I couldn't know what's in her head when she was typing pink rabbit or whatever the particular scene was at the time. But I realized immediately, and of course, we get that from customers a lot, especially from the pixel-perfect audience that is the Photoshop customer, that they do want control because they do know in their head what beautiful, what a good design is, how to describe beautiful, so they want to control it. So we're investing in both enabling the ease of user dialogue, user-computer dialogue,
13:57 Speaker 01 and in the same time, more control mechanisms that would enable them to really create the art they want to create.
14:06 Jacob Effron? For the people that you've seen that like use it best or seem to really get it, like what kind of workflows do you see them use that are kind of maybe different than like traditional workflows?
14:14 Speaker 01 I think actually the most interesting thing we've seen, which, and I'm going to make self-driving car parallel here, but when we came up with the idea of using LLMs in Photoshop, et cetera, the first thing that comes to mind is to use an LLM to index help system about Photoshop and then through a dialogue, guide the user to what do they need, what tools, existing tools they need to use to get to the output they need. But through our engagement with the Photoshop team, they actually stepped back and said, look, if we think about what a co-pilot is, like a co-pilot, because we always had this vision of building what we called it a creative assistant at times or a creative co-pilot.
14:57 Speaker 01 What do people really want is to get to the end result, not to be instructed on how to use a complex tool to get to the end result. So I call it level four creativity. And with generative film and Photoshop, for example, you select something, which is like selection is the most used feature in Photoshop. You select something aided by AI, and then you have a prompt where you can describe what you want to happen. You want to remove something that's like remove a distractor, you want to insert an object, or you want to retouch a particular selection on the scene. And we've seen that interaction being the most natural. And through that interaction, actually, the customers along the whole spectrum of novices to professionals, they basically reach their destination sooner.
15:41 Speaker 01 It's like Uber, you don't know necessarily, you don't have to know how to drive. You need to get to a particular destination. So what we've seen that impressed us more was the openness of the whole range of customers from novices to professionals to this co-pilot experiences that basically do the work for you. You're still in control, you're still guiding the computer. It's not like automated, but it's a change in the user, in the human computer interaction by this co-pilot, and we want to embrace it more. We think that intercepting some of our most frequent workflows in our products and bringing generative AI in context is a strategy that will make our customers more productive.
16:24 Speaker 01 And we're going to continue to do that along multiple modalities, from images to video to 3D vectors, et cetera.
16:33 Jacob Effron? Super interesting. And I guess like, how have you thought about like, obviously, there's some folks that probably just get this intuitively right away, and they know exactly how to use these tools. And then I imagine there's obviously a big user education element to it too, right? And sometimes people come across like a blank prompt screen, and they have no idea what to prompt or how to use it. And so obviously, you have this like built-in user base that knows how to use tools in a certain way. How have you thought about like educating the user base? And also solving that kind of like cold start problem of, you know, what do I write for the first prompt?
17:00 Speaker 01 Good point. So first, as I said, Firefly is integrated in across the segment of customers. So for consumers, we have firefly.adobe.com that targets basically either next generation creatives or people that would are just not willing to spend the time to learn one of our tools. We have free tools too, but they just want to get the result fast. Then we have what we call the communicator segment and we serve that with Adobe Express, which is a both design and video editing tool, of course integrates Firefly, but that enables through the use of templates, the creation of a lot of fast social content and to help these communicators or marketers stand out.
17:44 Speaker 01 Then we have the creative professional community. They use Photoshop, Illustrator, Premiere, depending on their medium of choice, and they're basically doing this as a job. And then we have enterprises that use a mix of those tools with some custom tools. We have tools like Workfront to help guide and manage themes of creatives. So across those four segments, they will have different different levels of creative mastery. Like consumers, they basically they're coming in all fresh. And for them, on firefly.adobe.com, we have an inspiration stream allowing them to remix the existing prompts.
18:27 Speaker 01 We have curated actually from the community itself. When we launched firefly.adobe.com, we have a really powerful and very useful feedback mechanism. So customers, the beta customers can tell us what they like, what they don't like. They can report bias issues, harm issues, et cetera, but they can also submit their own creations to the inspiration stream. So we have a process to handpick the best creations and then put them in the inspiration work stream. So new users are coming in and they basically immediately find something they like and they start riffing from it. This is a very important process. You click on something, you see the prompt and then you can riff. Another thing we wanted to change in terms of promptology, I'm going to call it like that,
19:11 Speaker 01 is the fact that we see with other open source models or other models from that already exist, that you need to write this very complicated prompts and use a lot of very complicated words or esoteric words to try to guide your creation in a particular style. So Adobe, we're well-known for our property inspectors and Lightroom presets, etc., which were very popular with photographer and with artists in general. So we thought about how can we actually use AI, some specific custom models we've created to bring this capacity of styling to Firefly without requiring you to write the style as a very convoluted prompt, like 8K and the names of various 3D rendering engines, etc.
19:56 Speaker 01 So we've created the Adobe Style Engine. So if you go to fireflyadobe.com, on the right-hand side, you will see styles that you can pick, you will see light conditions, you'll see composition rules. So you can basically describe your scene by pre-selecting from a series of options that are curated by our design team that enable you to create easily and then the prompt is just a very simple sentence and then you can play with those styles and create amazing outputs while preserving the style structure. So that's another advantage. Normally, if you change the prompt, you might lose the identity of the subject. So you lose your scene and you need to start from scratch. With the Style Engine, you keep your structure. If you found the perfect emotion on somebody's face, you keep that and you change the style.
20:41 Speaker 01 You can make it, you can change it to a golden hour and then the image is preserved, the person is preserved, but then the lighting changed to make the image more appealing. So that's what we're doing for the consumer segment. And of course, we're investing into prompt auto-completion models. We do want to bring in all of this knowledge we have from Adobe Stock, for example, where when you start typing, we do have this search auto-completion. So we're going to bring some of those models too for the consumer audience to give them guidance along the way. For the Express customers, we think templates are their entry gateway. So they search for templates and we're going to bring in generative capabilities in the templates themselves.
21:26 Speaker 01 So sometimes you'll have, you search for a template, you find a template, the template will have a... Firefly image generating in it, carrying in the prompt. So it's like an inspiration workflow, but it's more complex because it has compositing. So it has layers, it has effects, it has remove background. So it's going to be a little bit more complex. But that's what we're planning to do for communicators or marketers. Give them templates that are baked in with generative AI capabilities that they immediately start to use intuitively. For the professionals, what we've seen with Photoshop, generate fill, having this contextual bar. So we've introduced this contextual bar when you select. You don't even have to orient yourself in the UI. The contextual bar is there. It has a, it appears for every selection.
22:11 Speaker 01 It's right near the selection and it has some contextual options depending on what you have selected. One of them is generate the fill. And when you click generate the fill, you actually don't even have to type in a prompt. When you click generate, we're going to use the rest of the image as a, as an implied prompt and generate something that matches. So you don't even have to think, but, but we, we haven't found, in particular for CPROs, creative professionals, we haven't found this dialogue to be complicated. For them, it's like a 10x easier. So they, they actually, they enjoy it and, and they, now, now it's more about prompt understanding and trying to collect feedback about how do they describe various sections? So we react correctly to them because most of the time, instead of putting in an object,
22:55 Speaker 01 they will put in the full sentence, replace his face or her face with, with a blonde hair or something. So we need to do more in terms of LLM understanding of their actual sentence. And finally, for enterprises, we're also thinking of introducing those capabilities in context. For example, if you're in AEM, which is the Adobe Experience Manager or CMS, and you, you, you're writing a page or a blog prompt, we're going to introduce contextually recommendations on how to auto generate some portions of that text, respecting your brand voice. So it's all, it's all about in-context introduction with, with various, either databases of contents to help you orient or technologies to help you configure the prompt
23:38 Speaker 01 to generate what you want and/or other AI models to auto-complete your prompt. Those are the types of things we're doing across those four segments to, to help our customers create, you know, onboard easier into this world.
23:53 Speaker 00 It's something that we've heard across a bunch of domains is kind of like putting guardrails or templates or, you know, giving people Some guidance as they're working with the AI to create what they want to create, but it's really cool to hear about everything that you guys have done.
24:07 Speaker 01 It is. And just to add to this, it's a very important point that for our enterprise customers in particular, one thing they obsess about and for the right reason is brand guidelines compliance. So they want to make sure that content created either respects the company's brand and/or if you're in a particular campaign and you generate additional images, they have to respect the campaign brand because they might have sub-brands. And this is where actually generative AI can play a key role and we're investing in innovating in this space. How can we enable enterprises or the marketers or the creative department inside an enterprise create content that is automatically compliant with the brand?
24:52 Speaker 01 So you don't have to QA it a lot and it's beautiful out of the gate. So we have some really nice innovation we're working on to really enable this content velocity problem, solve this content velocity problem while respecting the brand guidelines.
25:11 Speaker 00 Yeah, that makes a lot of sense. I can imagine especially for someone that's at the scale that you're at, that trust and safety and making sure that these generations, if you are generating a billion photos or different types of generations, that trust and safety is top of mind. Can you say a little bit more about how you guys go about solving that problem? Like whether it's on the modeling side, if you're training custom models or maybe putting in different kind of checks and balances before you're sending a prompt? Like how have you guys thought about that?
25:41 Speaker 01 Yeah, happy to do so. So trust and safety or removing bias or minimizing the chance harmful content can be generated is top of mind for us. We do have a very high bar there. For us, it all started with actually the data we train on. One of the things I didn't mention, actually, that is very important and is part of our differentiated Firefly strategy is we are training on data we have in our stock database of content. So this is, we don't train Firefly models on data we scrape from the internet. We think this is raising concerns with our customers, the artists that feel that they don't have consent, control, or compensation for that.
26:24 Speaker 01 So training on data from Adobe Stock not only gives us high-quality content, gives us the right to train on data that is stored in our marketplace, but also gives us reduced risk of IP infringement, reduced risk of actually harmful and biased content because, not bias, actually, there is some bias, I'll talk about bias for a second, but harmful content, we just don't approve it in Adobe Stock. We do have a both automated and manual curation process that every asset submitted to Adobe Stock has to pass our content moderation process when it has some high bar in terms of what we accept in the marketplace. So because we train only on Adobe Stock, we can say that we're already in a good place
27:08 Speaker 01 in terms of reducing harmful content because we don't have like illegal content in there and also reducing the potential of our customers generating infringing content that might infringe somebody else's IP. There is bias though, because it's a marketplace, we have various contributors from different countries all over the world, but there will be an inherent bias in the training data set. Like every data set is biased one way or another and the actual process of training models is to teach the model the actual bias. So when we launched the first internal version of Firefly,
27:49 Speaker 01 Text to image, we knew we will have some bias issues and we had an internal Adobe test with tens of thousands of Adobe employees that came in and gave us a really good and healthy stress test about what were the areas we were not doing well on. And we had a huge, not a huge, we had, we discovered maybe 10 big problems. Fortunately, our research team was working on solutions for those. So we got this feedback actually in February, late February, and we launched in late March. That month from February, that month, or I would say that quarter from February 20th to March 20th was pretty intense because we enabled all of those research mechanisms
28:34 Speaker 01 that we've been working on to both reduce bias and reduce harmful content. So one of the important things we did actually that we're very proud of, we built a person detector, we have a model that helps us understand if there's a person referenced in a prompt. And most of the time when you refer a person, you might refer it through a job, you could say a lawyer. And in a biased model, when you put in lawyer, you will get Caucasian in their 60s males. That's basically kind of what the dataset bias would teach the model to do. But we know that lawyers are actually across all skin tones, across multiple genders and across all age ranges.
29:19 Speaker 01 So what we did is we detect, we detect people, we detect jobs, and we de-bias the actual content to introduce the right distribution of skin tones and the right distribution of genders and age groups from the country of origin of the request. So we actually have a database with what is the skin tone distribution in a various countries. So you always get a good representation, a fair representation of your country's distribution of skin tones. And of course, we do age and gender de-biasing. So if you go to firefight dot wo dot com and you type in one of those prompts, you will see that we actually generate very de-bias results because we've invested significantly
30:02 Speaker 01 in making sure that we don't let the training dataset bias permeate through the model.
30:09 Speaker 01 Similarly, for harm and bias, we have various toxicity detector models that we've trained to make sure we can differentiate various terms. And because we do wanna, we do wanna enable all ages, all age ranges to access Firefly, so we don't wanna generate not safe for work content. And in order to do that, we have toxicity detectors, and we also have various denial lists and block lists. But more importantly, we do, we do have also NSFW filters at the end of the generation. And another area that became evident through this internal testing was that their children in particular, they need to be catered for, specifically, do wanna show children in relationship
30:54 Speaker 01 with tobacco, for example. So we've created various systems to help us detect a prompt that references children and then make sure we reduce the chance or basically avoid or do negative prompting to minimize the chances of generating by mistake, something that's not okay in relationship with children. So a lot, a lot of both ML models and algorithms and rules that we've built during, again, that months to put us in a good place. And of course, we know this is not perfect, and this is why we have a very good feedback mechanism inside firefly.com and inside Photoshop to enable customers to tell us when
31:35 Speaker 01 we basically give us new training data points for the rules and for the, for these models to help continue to bake it in and then refine the algorithm to, to reduce both biased content and harmful content. So I'm actually very proud of the work we did and we're continuing to do on this front. And it is, it is, I would say, At the edge, at the frontier of technology, but we want to make sure that content generated by Firefly is usable commercially. And this is why we want to minimize NSFW content.
32:11 Speaker 00 Amazing. Yeah, I'd love to ask you a little bit about the customer feedback mechanism. You were talking about it and it seemed like there's a bunch of different ways that kind of customer data influences the models and the AI that you're building. Maybe one of them is, you know, as you were talking about the explicit feedback. It seemed like you're also gathering data from them, which then after this curation process goes into your data set that you're training on. And then also at the beginning, you were talking about the prompting and, you know, maybe curating good prompts from the customers. But could you just kind of double click on some of the ways that customer data or customer feedback influences the models and AI?
32:52 Speaker 01 Yeah, and just to start on this front, there was some Twitter controversy about us training on data, customers storing Creative Cloud, which we don't. So we never shipped models trained on customer data. This is handled, customer data stored in Creative Cloud. It could be a brand's next generation campaign. We, this is separated. We don't train on it. We train on Adobe Stock and we're adding museum and some other art or content with copyright expired. But on fireflyadobe.com, which is our community engagement website, we have our terms of use allow us to store the prompts and the generated images and use them in training, which is basically our way of doing RLHF.
33:33 Speaker 01 We think RLHF is essential to be able to create an aligned generative model. And RLHF comes in multiple shapes and forms. So we have this explicit signals like dislike report that we take into account and we log. And also we have implicit signals like download or save or share. And those are also pretty strong signals that the customer liked a particular generation. So we're in the process right now for the next generation of Firefly models to.
34:04 Speaker 01 Integrate this feedback loop and basically teach the generation through a reinforcement learning to generate content we know users like most and also avoid generating content that we know users disliked. So we're doing that for a future version of Firefly. But that's basically that we're using Firefly.w.com as our RLHF system that gives us a lot of insights into what users like and don't like.
34:37 Jacob Effron? I feel like you guys obviously did so many things to make these models usable, whether it was training on your own data to avoid copyright issues, some of the guardrails on the brand voice, bias reduction, not safe for work filters, a lot of stuff we've talked about. You are like one of the only folks that have really deployed these models at like large scale. So I am curious, is there anything else you guys needed to build to make this work at the scale that you've... I feel like we see all these demos all the time and then the question is, well, what's required to actually put those into production? And so I'd love to hear from you about some of the other problems you guys had to solve to do that.
35:12 Speaker 01 That's a great question. And we had a recent VP Summit at Adobe and I was asked, together with Eric Snowden, our VP of Design, we were asked to present Firefly Innovation at the Speed of Light. And because everybody was like, I moved under Eli Greenfield, the Adobe DME CTO and my current boss, I moved under him on January 12th and then we shipped on March 20th. And everybody was like, this is just insanely fast for a large company. How do you make it happen? And my presentation was actually quite funny because I started my presentation showing the light moving at a trillion frames per second. And light actually moves quite slow. It filmed at a trillion frames per second. And the reality
35:53 Speaker 01 with building Firefly, we've started a long time ago and it wasn't called Firefly and it wasn't the diffusion models, it was GANs. But we've started in the research organization in 2019. I was showing some screenshots. Gavin Miller, he's our Head of Research and he invited me in 2019 to what he called an Adobe Research Retreat where researchers, and I was in the business unit at the time, they present their latest discoveries and ideas for the business unit to connect to it. And I'll tell you more about how this tech transfer works if you're interested later on in this podcast. I remember entering this room and seeing this researcher presenting something at the projector, something called GenShop, Generative Photoshop.
36:37 Speaker 01 And again, this was 2019 and the generation was through GANs. But I was looking at it, I was like, wow, this is really something. And it looked unbelievably good. I mean, you can tell Photoshop what to do and it understands and it changes the pixels. It actually looked a little bit science fiction to me, but in terms of quality. But the vision was there. And then I reached out to the researcher and I said, okay, let's work something. Let's make it happen. So together, we've set up what we call the Neural Filters team. And Neural Filters was this idea that inside Photoshop, we're going to bring, we're going to accelerate the tech transfer, the productization of this research-created AI models into the product and start engaging with customers. And this is how, I think, we've shipped almost or more than 100 models.
37:21 Speaker 01 They were small models, small GANs doing various things from colorization to JPEG artifact production, to smart portrait, to season changing. I mean, we did a lot of very creative things. But through the process, what we actually did, we were building up both the expertise, the customer engagement approach, and an amazing team that was capable of both training distributed models and doing optimized inference. Most of the Neural Filters actually run on-device inside Photoshop using Core ML and Wheel ML. But we've built this team that was ready. When we made the bet on Generative AI in 2022, the team was there so we could start training larger models.
38:05 Speaker 01 And the team was there to build an optimized inference platform. Because again, right now, these models are too big to be run on-device and at the right level of quality. Of course, we're looking into it, but we're not there yet. So we run everything in the cloud. But we've capitalized on our deep understanding on how to run models fast. We were building, again, for Neural Filters, various I would say inference graphs that we basically deployed in the cloud. So we had a lot of technology. We had a team, we have a technology stack, we have knowledge. And again, in Adobe Research, Adobe Research has 200 plus researchers. And every summer we welcomes 300 PhDs and they work with us to create the next generation of of models and we publish a lot of papers.
38:50 Speaker 01 But Adobe Research was always having a lot of ideas and and models in play. And what we needed was just a productization arm. So when I moved under Eli, I moved with my team and we've created this applied science and machine learning group, which is basically applied research. And we are now helping all these models from research be productizable. So both we refine them, we train the last 20% of the model, which takes 80% of the time and of the GPU cycles. And then we deploy them as services. So teams like Photoshop or Adobe Express or Illustrator, they can call our API. So we have been building the pieces of that.
39:31 Speaker 01 And the first quarter of this year, we just assembled them and deployed them in the cloud. And we do have other internal Adobe systems for container management, like we have a layer on top of the containerization system like EKS or other reuse multiple public clouds. But we had various systems in-house that we were building up that they all partnered to ship Firefly. And again, it was, it was starting from Shantanu, our CEO, and David Wadwani, the president of Digital Media, which is, which is in charge of Creative Cloud. They believed in this vision. So we had their full support and many other functional groups like from communications,
40:15 Speaker 01 go to market, customer support, all of those groups besides engineering and product and design helped ship Firefly in a really, really short period of time. But again, because we were building it for a long time.
40:30 Jacob Effron? And one thing I'm struck by in the way you talked about it is obviously like it seems like you guys had kind of interesting org structure around this too, right? Where you had kind of a centralized AI team that built some of these services that then you know, other products could call. And I think it's a question a lot of people are thinking about now of like, what is the right way to structure AI teams? Do you kind of embed AI engineers in each product line? Do you kind of build a more horizontal team? I'm curious like what you've learned or how you think about that, given, you know, given the work you guys have done.
40:56 Speaker 01 Yeah, and I would say we're doing both. So we have a, we have a horizontal layer. And then we have what we call tech transfer teams in each of the big product lines. And let me, let me detail a little bit what that means. So on the horizontal side, again, Adobe Research is, is reporting into Eli, into the CTO. And they do have different labs for modality. They have labs for images, for video, for vector, for design, for 3D. And those labs, they, they do research. They try to publish papers, they try to stay ahead. And also they try to bring these models closer to production, but they don't, they're not in charge of productizing them. So this is the research organization and they do use one of my team, the Sensei team.
41:41 Speaker 01 So I have this trade distributed training platform that is in charge of both enabling researchers and applied researchers to train large models and small models in a multi-tenant environment. And also providing curated, high-quality data sets for training. So the Sensei team does both EA training data, management and, and fast training and the distributed training infrastructure. So this is another horizontal that supports their research org. So once these models are ready, they move into the applied, they're ready for pre-production. They move into the applied research org where they're basically taken in and refined. And in some cases, this means more training.
42:22 Speaker 01 In some cases, it means we change the architecture completely because again, the world is moving so fast. But there's a lot of work happening in the applied research to prepare those models for productization. And then they move in the, what we call the GNI services team, where they create services that run those models in an efficient way. And we do a lot of things to, for efficient inference. And again, some of the open source technologies that exist these days to enable you to, to both train efficiently. We're a big PyTorch shop, but PyTorch 2 did help us a lot with, with various things we did with BF16 data type. So to, to make use of the many thousands of GPUs we have available. But also on the inference side, that we're using many open source technologies and some custom.
43:05 Speaker 01 Again, custom orchestration layers and custom computing layers, in particular on the imaging pre and post-processing stack, to make those APIs be efficient and fast. We do have, I think our average response time is below 15 seconds for a 1K by 1K image, and we're working on both increasing the quality and increasing the performance. So it's a race, it's a race against faster compute against quality. So this is the horizontal side of the house. Now, in each product line, like the Adobe Photoshop team, or Adobe Express team, or Adobe Illustrator team, they have what's called, again, a tech transfer subgroup in that team. So those folks, they know some applied, they're applied researchers.
43:50 Speaker 01 They can train their own models using the same infrastructure, but most of the time, they will take one of the APIs my team has created and build an amazing user experience around it, because they are the product experts, they're the segment experts, and they know best what is the core workflow that needs AI most. And this is more of the vertical portion. And this is what we've seen works for us best, because there are economies of scale in this horizontal, where we do things in a unified way. And how I like to call it, it's an AI superhighway that enables this fast tech transfer to become services. So once that's built, it's very easy for the product teams
44:33 Speaker 01 to basically build workflows on top of those. And of course, it's an internal mono repo. They can submit pull requests and improve this managed model zoo with new models. So we're trying to operate the AI superhighway as a managed model zoo. And we're early in the process, but we're very happy with the early results. And we think it's gonna accelerate, again, this is why we're calling it the AI superhighway, accelerate innovation and the transfer towards customers.
45:03 Speaker 00 That's super cool, and I even like the term tech transfer. I think that kind of captures what these folks in each vertical are doing. Would love to ask you a little bit about pricing, which is something that's kind of top of mind for a lot of companies as they're introducing AI features. I think Notion came out with Notion AI, and then I think we're charging $20 a month for it, or kind of this additional upcharge for AI features. How have you thought about pricing when it comes to AI? Whether that's upcharging or potentially including it as part of the platform. Obviously, there's a huge suite of products that you guys offer. But how have you thought about pricing?
45:42 Speaker 01 I'm not the right person to talk about pricing. What I can tell you is how we're looking at the levers that influence our costs with some of those models. That's maybe not the answer you're looking for, but it's an answer that at least helps understand how we think with these models. So these models are, as you know, big, and they have to run in the cloud, so there will be a cost associated with them. And most of the time, we want to be the best models for content creation across modalities and editing. Images, video, vectors, which is something unique to Adobe. As I said, we invented PostScript, which is a vector format, and 3D. So we're gonna invest in all of those modalities.
46:27 Speaker 01 And being the best means the quality of the generated output or the edit needs to be the best. So these are kind of the levers that increase the cost. Bigger models, more billion parameters, bigger encoders to understand the prompts better. So there are a lot of levers to increase the cost by just making the models bigger to generate the best output. On the other side, of course, we're looking into an amazing user experience. So models, they take 60 seconds. Our customers, they don't want to wait 60 seconds. So we need to find the efficient frontier between quality and interactivity. And with higher interactivity also comes lower costs. So all of this equation is complicated.
47:13 Speaker 01 But of course, we're looking at things like... Distillation, pruning, quantization, there are many techniques that again, applying various, the latest and greatest open source technologies, or even exploring new silicon types that are optimized, we're looking at all of those to keep costs in check, but there is a cost that comes with running some of those models, and again, especially as we're trying to have the best quality. But this is, this is the, the type of challenge I'm facing within my team, and we hope that, and there will be, once we go GA, I'm sure there will be more clarity about, about how we plan to commercialize it.
47:56 Jacob Effron? Well, I mean, you know, as we, you know, sadly come near the end here, I'd be remiss not to ask you, I know it's crazy to predict anything in this space, given how fast things are moving, but since we have you on, I thought it would be fun, like, what, what do you think the Adobe Creative Suite and this broader space looks like in like five, 10 years, just given the space, the pace at which these models are progressing?
48:16 Speaker 01 Yeah. I just think, and I was listening recently to Clem's, Clem's podcast too, I, and I realized that we think alike, actually, I'm a big fan, fan of Pugging Face, and I do think that software companies will become AI companies in the years to come, and they might not call, as Clem said, they might not call themselves AI companies, because this is like just a new way of software, but we call ourselves a software company for creativity today. So I think my, my hope is that we will continue to embrace these models, and again, I would say, model architectures are going to continue to change from LLMs and diffusion models
48:58 Speaker 01 to many other, the innovation in this space, it's exponential, actually. So we know, we never fall in love with a particular model, because we're literally shipping it, and then we're probably gonna put it in maintenance and move to the next big generation of, of model architecture that has 2X the quality or performance. So, but I do think that 10 years down the road, by embracing AI throughout the, throughout The software development process. AI will replace more and more of the software stack and will enable us to enable amazing experiences for our customers in both content creation, editing. And again, I just think AI will enable massive changes in what type of content will be created.
49:43 Speaker 01 Again, I spoke about the exponential need for digital photography content or digital publishing content. I think AI will enable hyper-personalization. This is a big problem we're seeing in enterprises right now where they have what we call the content velocity or the content supply chain problem. They want to communicate with their customers in a individualized or even by segment way. And it's too expensive or impossible to manage humans to create the content they need. So it's a huge bottleneck in the content creation process. So what we see generative AI enabling is this hyper-personalization where both large and small enterprises and actually anybody can tap into to generate infinite content that will be personalized to the viewer.
50:27 Speaker 01 Maybe even, I mean, hopefully to the segment of one. So when you visit a website, you will see content created for you. You will see images created for you that speak your language because we would know. And again, it will be opt-in. We will know what, how do you prefer information to be served, et cetera. So we see Creative Cloud evolving into enabling our customers become from visual designers to more like creative directors of this meta experiences that will be then hyper-personalizable and they will again enable brands and the small businesses to communicate differently with their constituents. And that's a really big change that I hope will happen.
51:08 Speaker 01 And of course, everything will be, human-computer interaction will be much more fluid, less convoluted, more accessible to larger audiences. That's what I hope for Creative Cloud in five, 10 years down the road. A more accessible AI at its core, but not even, maybe not even going to be mentioned.
51:32 Speaker 01 A creative suite that will enable content creation with much higher creativity and velocity.
51:39 Speaker 00 Awesome. Well, I will wrap with a line that I steal from Jacob, which is, I feel like we would be doing a disservice to Adobe AI if we were to keep you any longer, but we really appreciate you taking the time to chat with us and thank you so much. It was great to meet you.
51:54 Speaker 01 It was my pleasure and I'm going to continue to hear to unsupervised learning. Thank you to listen to unsupervised learning.
Previous Ep 15: Snorkel AI CEO Alex Ratner on What's Needed for Wider-Spread Enterprise AI Adoption Next Ep 17: Nomic AI Co-Founder Andriy Mulyar on "GPT-4-All", LLMs in Video Games, and Apple's AI Strategy Page updated October 26, 2026 · transcript and summary are machine-generated
