Building AI for Creators | Luma & Phota Labs

Yoko Li speaks with Luma's Head of Applied Research Matt Tancik and Phota Labs cofounder and CTO Zach Xia about how AI is changing creativity, photography, and the tools people use to make art. The conversation explores the evolving relationship between artists and AI, from image generation and personalization to creative workflows, controllability, and agentic design tools. They discuss personalization, photography, creative software, model design, evaluation, and why the future of creative tools may depend less on generating content and more on helping people express ideas they couldn't easily realize before. Along the way, they explore AI agents, interfaces, and how creators are already using these tools in unexpected ways.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I think the creativity is building a story. The tools alone aren't a story. Someone has to direct them.

• 0:05Speaker 01

  It's not about mastering those tools. It's about directing an agent who can use those tools to achieve your creativity. Generation AI has become so good. You can be sort of authentic to that moment while getting a little bit creative of stuff. So I just think a lot of photographers are having more fun post capturing than before.

• 0:23Speaker 00

  How do you make something unique with the tools you have access to now? There has to be something more than just text. If you go to, say, a studio and you say, "Make me a 10-second video about a dog jumping in the grass." They're never gonna take that deal. They're gonna want more specific. And I think AI tools are no different.

• 0:41Speaker 02

  Something I heard from someone today earlier was, "They assume AI to be slop. It's up to the humans to create something out of it." What are your thoughts?

• 0:51Speaker 01

  I feel like that's such a hard question.

• 0:54Erik Torenberg?

  Creative tools have always changed the way people make art. Photography changed painting. Digital software changed photography. And now AI is changing how images, videos, and entire creative workflows come together. The technology is improving at an incredible pace. But the most important ingredient in creative work may not be the model. It may still be the person behind it. In this episode, Yoko Li speaks with Matt Tensic of Luma and Zach Xia of Photo Labs about AI-generated imagery, creative workflows, personalization, and why they believe the future belongs not to the tools themselves, but to the people directing them.

• 1:34Speaker 02

  Today, we have Zach Xia, who is the co-founder and CTO of Photo Labs. He works on personalized AI and AI photography. We also have Matt Tensic, who is the head of applied research at Luma. He works on agentic systems, fundamental research, and he was also the co-creator of NERF. Thanks for coming on the pod. So maybe let me start with this question, Matt. Many years ago, maybe three years ago, you had this online talk about NERF. And how you started this online talk was, you posed a really interesting question, which is, "What is the role of the artist? And what is the role of technology?" Now, fast forward three years, which is a lifetime in AI. What is your answer then? And what's your answer now? Has it changed?

• 2:16Speaker 00

  Yeah, so if I recall correctly, my answer at that point was that... Really, the point of this AI technology is to act as a tool for artists to better execute on the creative vision that they have. And at the time, Nerf was providing an ability for people to create 3D assets in a way that didn't require tons and tons of manual effort.

• 2:37Speaker 02

  Right.

• 2:37Speaker 00

  Because when you're creating 3D scenes, creating those assets isn't really where the creative part is. It's how do you put them all together? How do you build a story around it? And that's where those tools were helpful.

• 2:47Speaker 02

  Right.

• 2:48Speaker 00

  I think now, there's just so many more tools that just allow you to do that, not just in 3D, but also in video, in images. And I think this thesis that these AI technologies can be used as tools is even more so than it was in the past.

• 3:01Speaker 02

  Yeah, interesting. And Zach, I know you work at Adobe and then you did a lot of research that later, you know, went into Lightroom, which is also a creator tool.

• 3:10Speaker 00

  Yes.

• 3:11Speaker 02

  Creators have been using. What's your view of this?

• 3:13Speaker 01

  Yeah, I think artists really are where their creative minds are. They have a lot of creativity and technology is sort of a way to help them express that kind of creativity. I think maybe in the past few years, people like artists were also spending a lot of time trying to mastering the workflows, the tools. I don't think that's necessarily where their creativities are. So now their tools have become easier to use, more expressive than before. It's just so much easier for them to express the creativity. But I still think technology cannot replace the creativity minds today.

• 3:43Speaker 02

  Interesting. What would you define as creativity here? Because what the models are creating, not creativity, and is what user kind of composing together, that's what we call creativity?

• 3:53Speaker 00

  I think the creativity is building a story. And there's so many different ways to build a story. And you use tools to help construct that.

• 4:01Speaker 02

  Yeah.

• 4:01Speaker 00

  And the tools alone aren't a story.

• 4:03Speaker 02

  Right.

• 4:04Speaker 00

  Someone has to direct them.

• 4:05Speaker 02

  Yeah.

• 4:05Speaker 01

  I really think that this directive mind is just so important. Like tools, everybody can use the tools, but the stuff that gets created from these tools are very different. And the difference of the things that are created, that's where the creativities are. People have different thoughts, different stories, or even just a simple thing they create, which doesn't necessarily have a very strong story behind it, but they use the tool differently than others to create that thing. I think that's where creativity.

• 4:30Speaker 02

  Yeah, so Matt and I, we had this discussion earlier. So Matt is also a photographer and painter in his spare time. So he takes a lot of pictures. It makes me wonder, what is like AI creativity when it comes to photography nowadays? Right? There's one school of thought where whatever you're capturing in the real world is the end result, and I don't want to go change the pixels. There's the other view, which is like camera is just one way to produce the reality. Other ways, it might be painting, like before the cameras came around. It might be nerve, wait, if you want something 3D and then go scan all the objects. It might be a model. So what is your view here and how has the view changed?

• 5:13Speaker 01

  Yeah, I can maybe say a little bit about photography. I feel like photography has always been a very creative process. I'd say a lot of the creativity on photography was more on the capturing side before this AI boom. And that people were just trying to find the important moments, find the good landscape views and find the right angle, composition, use the right tool, wait for a very long time to take that photo. That's where the creativity is at. Photographers are really proud of that.

• 5:39Speaker 02

  They wait and wait for that moment to arrive.

• 5:42Speaker 01

  Exactly, exactly. But I think the reason that most of creativity was at capturing time is because the decisive moment, once that happened, afterwards, there's not so much you can do to it. I feel like that part has changed a lot in the past three years.

• 5:56Speaker 02

  Yeah.

• 5:57Speaker 01

  Generally, I have become so good, you can be sort of authentic to that moment while getting a little bit creative of stuff. So I just think a lot of photographers are having more fun post capturing than before.

• 6:08Speaker 02

  Interesting, yeah, super interesting. So I also know both of you start off your career doing PhD and doing a lot of very in-depth research. And then since you went into an industry, there's a lot of user-facing work and building the actual product, shipping the product, learning from the users. From our perspective, it seems like the research community, they care more about the benchmarking optimization part of it. And then there's art of building the product. How has it been like for both of you when it comes to transitioning from research to a product builder? What did you have to learn and where do you think the gap is? Maybe we can start with that.

• 6:45Speaker 00

  As a researcher, your goal is always to push the technology. And so wherever there's a hole in the technology, you push it. That doesn't necessarily align with what, say, a customer or an end user would actually be interested in. And so quite often, you see these things diverge. For example, in image generation, a artist probably doesn't care about generating paragraphs long of really dense text in some esoteric font.

• 7:08Speaker 02

  Right.

• 7:09Speaker 00

  Maybe some do, but most don't. But as a research question, that's extremely interesting. So you do have to decide, do you want to keep pushing the research in certain directions? And when do you take a step back and push the creative side?

• 7:19Speaker 02

  Yeah.

• 7:20Speaker 00

  Because in many cases, what an artist or a creative would want is actually very simplistic. And the tools already exist out there. And it's just a matter of piecing together in the right way.

• 7:28Speaker 02

  Yeah, like removing background itself is a boring task for most researchers, I guess.

• 7:34Speaker 00

  Yeah.

• 7:34Speaker 02

  But it's extremely useful for any creators out there.

• 7:38Speaker 00

  Or even just fixing up lighting of an image. It's something that also doesn't have a very concrete solution. So it's hard for, say, a researcher to optimize a metric against. But when it comes to a creative task, it's extremely useful.

• 7:50Speaker 02

  Yeah. And what about photography? I guess, like, curious, what's your learning there?

• 7:55Speaker 01

  Yeah, yeah. What I've learned, and I'm still learning, is that there's such a fine balance. Like, researchers, they get excited about new technologies. So they just, as Matt said, they just want to push it forward. Users, they want you to solve their specific problem they just ran into. Now, there's some balance that we need to do there. We can't just solve that they do their problem that users are creating. We also want to be, the technology needs to be ahead of users a little bit.

• 8:21Speaker 02

  Yeah.

• 8:22Speaker 01

  So that maybe you can reimagine their workflow.

• 8:24Speaker 02

  Right.

• 8:24Speaker 01

  Right? But at the same time, you want to make it balanced. You're still solving their problems. And you're not just producing a product that no one is going to use. So that's such a fine art. With photography, there's something like that. I think the beginning of this AI era will get a lot of pushback. Photographers, I just want to capture. I don't want to edit. Now they're slowly getting into, oh, it's fine to remove some distracting object in that image. It doesn't matter to me and that's still my creativity part. So people are transitioning as well. Technology needs to stay a little bit ahead of that spectrum.

• 8:59Speaker 02

  That's such a great point. It is a chicken and egg problem. You can't include a hundred percent of what users ask. A seasoned Photoshop user may ask you to reinvent a lasso tool.

• 9:09Speaker 01

  Yeah, exactly.

• 9:10Speaker 02

  And questions, do you want to do that? Is that the mental model you want your model to have there? Like, maybe the next abstraction is there's no lasso tool. Like, we're not implementing the world in the old Photoshop world.

• 9:21Speaker 01

  Yes.

• 9:21Speaker 02

  So I guess like when you work closely with the creators out there and as you're learning from them, what is the surprising thing about their workflow? Anything that you push your product out there but wasn't designed for, but had a lot of adoption that you were pleasantly surprised by?

• 9:38Speaker 00

  Yeah, I think one of the first things that comes to mind is that you tend to think that like an artist or creative has an end goal in mind, like they know exactly in their head what they want. But in reality, that's not true. Part of art is that iteration.

• 9:50Speaker 02

  Yeah.

• 9:51Speaker 00

  And so making sure the tools are able to handle that level of iteration is something extremely important.

• 9:56Speaker 02

  Yeah.

• 9:57Speaker 00

  And this also makes it very hard to like benchmark these things because they often isn't that ground sure in their head.

• 10:03Speaker 01

  Yeah, for us, it's a little bit specific to Photoshop. So like we have to allow the identity preservation, identity consistency. We set out to do it, we're more targeting at image editing of real photos that people want to preserve that in the image. What surprised me is that people are actually very willing to generate a new image, put them in different environments, generating AI headshot, generating AI video of themselves using Photoshop technology. So that kind of surprised me a little bit and people are getting very creative with that. So.

• 10:34Speaker 02

  The video use case is actually a surprising one. Do you want to explain it further?

• 10:39Speaker 01

  Yes, yes, yes. So we start, we, right now, our technology mostly just work on images. And but people are getting creative. People are like, oh, the video models are so good, so bad at, at the new preservation. What I can do is I, I'm gonna use photo to generate frames of those videos. And then I'm gonna use a video model to take those frames into a video. So that I didn't is good, something that we didn't think of at the beginning, but they figured out, so creators, yes.

• 11:06Speaker 02

  Yeah, interesting.

• 11:07Speaker 00

  I, I think one thing to chime in on that, um, I, I've seen it multiple times now where, uh, people find very interesting ways to use tools, uh, not just our tools, but our tools in addition to other tools that, you never expected to work. And for every reason it shouldn't work, but people find crazy ways to get these things to work.

• 11:26Speaker 02

  Do you have an example?

• 11:27Speaker 00

  Uh, there's so many, I, I would have to pick a little bit more.

• 11:30Speaker 02

  Yeah, um, I'll come back to prompt you.

• 11:34Speaker 02

  So, uh, I guess when it comes to the tool abstraction, um, what kind of tool? Like, I almost see the end user as type of agents who need to use tools. It's just you provide the tools. And sometimes they will use the agent, like their AI agent, to kind of leverage the, you know, tool calls as well. So there's a tools for humans and a tools for agents. How do you see those differ, um, in your, like, uh, worlds?

• 12:02Speaker 00

  Yeah, I, I think it's interesting to look at how creative tools have existed traditionally. You look at your Photoshop's, your illustrators, you look at your blenders, whatever modality you're interested in. If you hop into those programs, they're extremely complex, right? You can literally have a whole career, um, learning the ins and outs of these programs. Um, and so that's great, uh, for people who've had that time and that experience to invest in it. Where agents come along and the ability to like have agents work with tools is then the ability to,

• 12:37Speaker 00

  tune the tool to the user.

• 12:40Speaker 02

  Yeah.

• 12:40Speaker 00

  Some users, all they care about is, you know, relighting an image, removing a background. Some want extreme levels of control. So then the question is, how do you interact with an agent? Or whatever AI tool that you're working with that can choose that level of abstraction that makes sense for you?

• 12:56Speaker 01

  Yeah, I feel like, as we said at the beginning, creativity is about like directing something to achieve your creativity, your creative mind. I think this directing thing is so important. I think that's where sort of agent comes into place. Like, it's not about mastering those tools, it's about directing an agent who can use those tools to achieve your creativity. That's just so important. Another thing I feel like is some lesson that we learn in photography is that, you know, a lot of people edit photos, a lot of people just don't want to edit photos at all. They have creativity in mind, they're just, they don't want to do their work, or they're too busy to do that, but they still have a creative mind. So, if you have some kind of agent that can understand your creativity and can express your creativity

• 13:39Speaker 01

  without, like, very low effort from you to help you achieve that, I feel like that's even better.

• 13:44Speaker 02

  Yeah.

• 13:45Speaker 01

  Like, this kind of automatic or passive agents that run in the backcourt.

• 13:49Speaker 02

  Um, I guess just on this point, uh, my observation so far has been, at first, we have simplistic tools where users can go in and prompt, and they get an image, and that's it, one shot.

• 13:59Speaker 01

  Yes.

• 13:59Speaker 02

  And later, we start to get crazy nodes on, like, concrete workflow, and it's like, you can't even, like, understand it as a human. And then we start to have canvas, and it's more, uh, you know, it's not, um, you know, like, a, um, a graph that you can easily map out. It's more like a set of things you can parallelize once in a while. So, how do you think about the next interface for both, like, agents and humans? Um, like, would the agents be on the camp of using Adobe Photoshop? And humans just don't need to learn how to use it anymore, and just ask the agent how to do it? Or would it be a different interface altogether?

• 14:41Speaker 00

  I, I, I think it's false to assume that, uh, every user is gonna want the same thing.

• 14:46Speaker 02

  Exactly.

• 14:48Speaker 00

  You look at directors of movies and some directors are go super deep into the weeds of making sure that you know every glass is in the right place. Whereas other directors, they don't care about any of that. All they care about is like, oh, is the actor doing the line in the right way?

• 15:01Speaker 02

  I see.

• 15:01Speaker 00

  And I think all that is true in these sorts of creative tools too. And so you have to create an agent that can work with these different directors. The one that has that ability and that control to go in and manipulate these things at very fine-grained levels. Or the ones that maybe are more on the like, I don't know, the literary side or whatever it might be and just wants to talk and like, interact and have fun with the tool and see what comes out of it.

• 15:27Speaker 01

  Yeah, personalization is absolutely key here. Different people have different levels of acceptance of like complexive tools. And they also have different kinds of preference and taste. So yeah, it has to be personalized to them. And I think Matt is speaking about like the agent needs to understand the user really well before it's well about that. I also think the agent needs to have really good memory. So like, if I tell you once, shouldn't need to tell you again in the future. So like, I think that's also just really important.

• 15:57Speaker 02

  Yeah. I guess just to push on that point a little bit. Obviously, if you travel down the abstraction layers far down enough, you can build anything and everything with C and compiler. But it would be weird to assume that agents are gonna rebuild your creative tools for yourself every time. So there has to be somewhere in the layer you take like, draw a line in the sand and decide that this is the layer we're gonna operate on. How do you think about what that layer is for each of you?

• 16:27Speaker 00

  I don't know if I have a concrete answer because I think it's constantly changing. There's a lot of arguments for different sort of underlying representations.

• 16:34Speaker 02

  Right.

• 16:34Speaker 00

  And some make sense, in some cases others may sense and others. Everything from, you know, is it a 3D world that you're trying to model? Is it images that you're trying to model? Are you trying to break up the images into layers? And I'm not sure we really hit on as an industry what the right representation is.

• 16:49Speaker 01

  Yeah, yeah. And I think there's abstraction of what's a model, what's a tool, right? Like a tool can be a little bit more than a model. A model can do a lot of things, right? A lot, you need a lot of the tools, traditional tools to do. So I think that we're gonna see how that sort of roll out. I personally think model is replacing a lot of the old tools for not so precise control.

• 17:15Speaker 01

  But I feel like also at the same time, people are putting more precise control in back to the model. So it's a back and forth process, you know?

• 17:23Speaker 02

  Um, both of you have been doing a lot of model and app co-design. And then, uh, you know, serving your own like in-house trained model on the app you also built. The question is, um, how do you think about what good looks like when it comes to the generation? Cause there are so many different ways to judge that. And is it a software, is it a model, or is it something else?

• 17:48Speaker 00

  Yeah, um. Being a researcher is always a focus of how do we convert good into a metric that you can measure? Because then the agents and uh, the research teams can optimize against it. However, there's no way that that'll ever tell the full story. And so you do have to involve humans in the loop. During development, it's how do you bring in creatives and people whose taste that you're trying to mimic? Can come in and help advise in that way. But that also isn't, can't be the full story. Because uh, one person creating one video is gonna have a different vision than someone else. And so this is where the personalization design really becomes key. And the question is, how do you learn those uh, uh, what the user cares about?

• 18:31Speaker 00

  Maybe as they're using the app, maybe as they've provided additional context, that's the type of stuff they're building? Or based on the history of the stuff they've built in the past?

• 18:39Speaker 02

  Can you measure taste? I guess when you have the tastemakers come using the tool, they'll have different workflows. Of course, you can, you know, optimize for every one of them. But is there some way to measure what taste is for a particular tool?

• 18:55Speaker 00

  I think you can measure taste given some constraint, like given a population. You can maybe measure the taste of that population. But then you can also measure the taste of an individual user. And so depending on how coarse you get in that measurement, it just becomes a bit more muddled.

• 19:10Speaker 02

  Hmm, I see.

• 19:11Speaker 01

  Yeah, I think for, for us, evaluation, there is evaluation before you ship something. And there is evaluation after you ship something. So when evaluation before you ship something, all the times, and for us, with specific or, or characterization. So we really care about having each, each individual tester or user to say, are they happy with the result? It's very hard for us to come up with like a universal thing, for especially for very subjective things like.

• 19:37Speaker 01

  Taste, preference, and even for very personal thing like identity. We always ask the person to evaluate their own identity or the person to evaluate very close ones in their life, the identity of them. And then there's evaluation after you ship the product. I think especially for very subjective things, that's so important for personalization because the personalization technology is going to be applied to different users differently. Have we made the user happy? Are the user satisfied with the result they're getting? And can you collect that kind of feedback in an ambient way? That's just like non-intrusive way is just so important.

• 20:14Speaker 02

  This is such an interesting point. I was actually talking to someone earlier about their photo result. His view was happiness and just seeking are two different things. When he saw the result and he was like, this is exactly me. It's like the best model output, the likeness is a hundred percent. Violet to fat in there. So he's not happy with the result. So there is some kind of like, you know, filtering, like something that makes the user happy may not be the benchmark thing that, you know, like you might be straight. How are you guys solving that?

• 20:49Speaker 01

  Well, like at the end of the day, we think, you know, user is the one who is controlling things. You know, if they want them to look in a certain way in a photo, I think that's their choice and we need to empower them to do that. And again, this is a very subjective thing. Beautification has been a long, long time debate of the like the spectrum of it. Everyone has their own taste. Personalization is key here, you know?

• 21:16Speaker 02

  I imagine personalization is more of a scale problem compared to having empower everyone, everything to do things on the canvas. How do you think about evaluation when it comes to there are so many unlimited workflows one can do on the canvas?

• 21:31Speaker 00

  Yeah, I think somewhat similar. You have to look at how people ultimately use the tool. We can come up with as many guesses on how to evaluate before we release it. But at the end of the day, you have to see, once it's out there, are people using it? Where are people using it? Where are people getting frustrated? Where are they feeling like the level of control is sufficient? And where is it not sufficient? And trying to gather as many of the signals as possible across the different domains that people use the tool.

• 21:56Speaker 02

  Interesting, so this totally makes sense. I think of a very, I guess, like a broad tool as there's a distribution of use cases. When you launch the tool, there's obvious ones that you probably would have tested before launching. And there may or may not be a fat tail on the distribution. The question is, are there specific use cases you want to move from the fat tail to the head? Have you ever run into use cases like that?

• 22:22Speaker 00

  Yeah, I think one interesting one was seeing how brands would bring in safe brand guidelines. It's not something we really considered very much. But then, once they brought them in, you realized this was an extremely useful source of information for really getting through what a user is interested in. And so there's some questions there of not just how do you enable that pipeline for these safe brands or other companies to include that information?

• 22:49Speaker 02

  Yeah.

• 22:50Speaker 00

  But how do you also get that type of information from others who aren't used to working with those sorts of pipelines?

• 22:54Speaker 02

  Yeah, should say.

• 22:56Speaker 01

  Yeah, for us, there was some users were trapped kind of like, our current API only supports human and pets for identity. There's some users who are trying to hack our system so they can use it for personalization for product photography.

• 23:09Speaker 02

  Oh, interesting.

• 23:10Speaker 01

  And they're like, we want this. And so now that's on our roadmap, you know, we want, I mean, it makes sense. But so that's kind of an example where like, we didn't see that kind of use case. We first launched it and there's market pool. And then we just want to increase the priority on those, yeah.

• 23:26Speaker 02

  How do you see the difference between, I guess, human likeness and product likeness? I mean, our brain is involved to recognize human faces. So I imagine there's, you know, lower margin for error. But what about product? What's the learning there?

• 23:41Speaker 01

  Yeah, I mean, so first of all, there are a lot of similarities between these two. Uh, first is like everybody say, oh, the general foundation model is already very good at identity. Until you try to use it to generate image of yourself. Same thing with product photography. Oh, it's pretty good at generating this specific product. Until you are the product owner and you're trying to generate photo of that particular product, it just stops working. So that's a not obvious one. The difference is like. Identity is a little bit more, I would say, subjective and dynamic. It's really, really hard to put into words, what is identity, right? Like product, on the other hand, you know, you need to get the text right, you get the shape of the product right, get all of those right. So it's a little bit different there. And there unique challenges in both.

• 24:25Speaker 01

  For example, text rendering is so important in uh product photography.

• 24:29Speaker 02

  Right.

• 24:30Speaker 01

  Not so much in identity preservation, right?

• 24:32Speaker 02

  I imagine uh the question becomes, do you want the model to do it all? Or do you want the subsequent workflow to pick some of the things that your model gonna do? Just curious about both of your opinions on. What kind of task, when you run into it, you feel like it's something the model should solve? And what kind of task you're okay with having a workflow solving it?

• 24:52Speaker 00

  We've seen a trend in language models, uh where they've transitioned to thinking modes, right? If you use ChatGPT or one of these models nowadays without thinking, it just doesn't feel smart.

• 25:02Speaker 02

  Right.

• 25:03Speaker 00

  And I view that as kind of this in-between of it's like a workflow. But it's still using the core model. And I think creative tools can go in that direction too.

• 25:11Speaker 02

  Yeah.

• 25:12Speaker 00

  Where you you generate stuff, but you can evaluate it, you can build off of it. Uh, maybe the user's involved in that, maybe an agent's involved in that, but it helps you get to the final result.

• 25:22Speaker 02

  Got it.

• 25:22Speaker 01

  Yeah, I think there's a separation of model versus technology. Like you, in terms of personalization, it doesn't matter. It's like a general model solves everyone's problem. As long as the technology is the same, we can apply the same technology to different to different users. Um, but again, you want to make that tool that or the model so that you can unlock the creativity in people. So they can figure out the best workflows to solve a particular problem that you haven't even thought about before.

• 25:48Speaker 02

  Yeah. How do you see controllability? I mean, phase one of the AI is we generate things and we just keep re-prompting the model for the thing we want. It's almost like a lottery. Phase two is that we put more control either on the model level or on the systems level to make that work. What does it mean for AI-created creativity tool to have like better controllability?

• 26:14Speaker 00

  Yeah, I think it's an open question and there's different ways you can add control. But the short answer is you definitely need to add control. I think that was the number one complaint from anyone who's actually trying to use these tools for any sort of professional or to replace any part of their existing creative workflow is that you don't want to operate with text. There has to be something more than just text.

• 26:37Speaker 00

  Or if it is text, you need that level of control, which can be very difficult in text. And so we've experimented a lot. Most recently, we've been doing a lot of work in the video to video space because we think that's like a very good way to start incorporating pretty precise controls, especially in the time dimension. In the spatial dimension, I think there's a lot of interesting things with being able to scribble, being able to like point to certain regions. And so adding more and more of these controls on top of the model so that the user can do, while also opening up these controls for an agent so you can instruct an agent to do these sorts of edits.

• 27:13Speaker 01

  Yeah, I think it's also about helping the user to say what they want. I think a lot of the time, the model doesn't do exactly what you want. It's because of like the user's prompt or whatever is a little bit ambiguous. And it's also, so other than the, you need to tell the model what to do. I feel like this is the underexploited research direction is if the model can tell you what it needs to get your inputs, to actively asking for inputs before it does something. I feel like that's also really important because like when you are working with a creative professional as an amateur, a lot of the times they ask you questions and you can tell them your opinions, your preferences, and they sort of achieve their preferences and opinions.

• 27:56Speaker 01

  I feel like that mostly today is almost a one-way street. It's like the user tell the model what to do.

• 28:02Speaker 00

  I think that's a really good point. If you go to say a studio and you say, make me a 10-second video about a dog jumping in the grass. They're never going to take that deal. They're going to want more specific.

• 28:17Speaker 00

  Yeah, you have to figure out how to apply that in the AI.

• 28:20Speaker 02

  Interesting. Do you think model app code design has benefits when it comes to introducing more controllability? What's your view there?

• 28:31Speaker 00

  I think so. I think being able to have the users interact in your app and work through their workflow in your app, you learn which things a user actually cares about. You learn which steps do they have to redo multiple times and why maybe they chose the step that they ended up choosing. And that can help a lot with model design. Right.

• 28:50Speaker 01

  Yeah, I think the other direction is also really important is like your product can also help educate the user, what's the best way to sort of use their model. I feel like we're seeing that all that was like coding and stuff, right? Like, you know, people are used to like to do everything like VS Code or whatever. And now like call code, just in the terminal, you can, you can do, do that code. And it's the best ways to sort of use a model because you cut, cut through a lot of other things. I still feel, feel like app and model code design has that advantage too.

• 29:18Speaker 02

  Yeah, when I think about AI creative tools, the first thing that comes to mind is for me is it's very visual, just being very visual animals. And now we're used to working with agents. It feels like, almost like agents are, and are aliens who don't perceive the world the same way that we did, or we do. So I'm curious, like, is there a world where creative tools become primarily headless? What would that look like if you, you know, leverage, want to leverage agent type looks?

• 29:52Speaker 00

  Yeah, I, I think, again, going back to like, if you were to work with a studio, you are talking to an agent, right? You're talking to someone who is working with you, but they're not talking back to you in text. They're showing you examples, they're walking through, hey, do you like these ideas? Do you like these ideas? And I think that's the direction you'll see these AI creative tools go. Yeah.

• 30:11Speaker 01

  Yeah, like today, when I, when I try to brainstorm something and, you know, you try to write a code to, to like generate HTML page, showing me the graph, graphics of it, I want just a sketch out of it. I like, that's probably the best way to iterate, right? Like, a sort of image is more than a solid words. I like, so that's, that's just so important. On the other hand, I feel like people haven't figured out a way, what's the best way to feed the visual information to a model? I still feel like models today is a little bit blind. I mean, it's getting better, like having high level semantic understanding. But visual is just such a, I would say.

• 30:52Speaker 01

  Uh redundant information, there's like pixels that are very redundant. So unlike languages, so how do you sort of give the model that kind of information? So the model is not blind anymore, which is so important. I I don't think we have figured out that yet.

• 31:07Speaker 02

  That is so interesting. It does remind me of like the importance of having the right representation uh in the creative tool. I mean, Matt, you worked on Nerf and before that people were on meshes. And now there's a spectrum on how efficient a representation is on Nerf. And uh, you know, like Gaussian slide is great in details, but it's really hard to work with. Cause it's not like a beautiful mathematic function. How do you think about representation or different kinds of representations for, you know, feature iteration or the design tools you want to make?

• 31:37Speaker 00

  I I I think the representation question ultimately comes down to the control question too. It's what do you want to be able to modify it in the future?

• 31:46Speaker 02

  Yeah.

• 31:46Speaker 00

  Uh if you're designing a poster and you know you're going to be changing the text on it, you probably want a representation that's not pixel level or changing that text.

• 31:54Speaker 02

  Yeah.

• 31:54Speaker 00

  Um if you're just modifying pictures, then pixels make sense. If you're uh navigating 3D scenes, then for for say movies, you may want a 3D representation. So I don't think there's necessarily one answer. Um it's whatever makes sense with a given level of control.

• 32:11Speaker 01

  I think that's so important. I think right now a lot of people are more focusing on what's the best representation for the model. But as Matt said, I think the consumer of those representation is not just the model, it's also the human. So like, can you represent it in a way that human can understand? And then be able to control that representation, do stuff to it, like, yeah.

• 32:31Speaker 02

  Um one thing I am curious about for both of you is how do you develop the intuition for, you know, building for the creators as former researchers? But Matt, I know you are also an artist in your own way. So like, how does that affect what kind of research or what kind of product you ship?

• 32:48Speaker 00

  I think it sort of gives you some insight in the control aspect. Um like one of the first things I remember being drilled in during sort of art classes when I was younger is uh the blank canvas problem.

• 32:59Speaker 02

  Yeah.

• 33:01Speaker 00

  There's a tendency of wanting to, well, getting stuck by that, but then once starting very meticulously, like doing one section at a time.

• 33:09Speaker 02

  Right.

• 33:09Speaker 00

  But in reality, it's better to just sort of go at it, just try things out, mess with it until you get to the thing that you're ultimately happy with.

• 33:16Speaker 02

  Right.

• 33:17Speaker 00

  And I think AI tools are no different. The iteration cycle is quite fast, but you really want to iterate. It's not just put in a single prompt to get a final answer out. You want to iterate on top of that. So that definitely impacts how to think about building the tools. And then I think another aspect is figuring out what language to describe what works or doesn't work in the tool. Because this can be very important for evaluation.

• 33:40Speaker 02

  Right.

• 33:40Speaker 00

  Where, I mean, throughout art over the, in the past, however many years, like people have built out these ways to describe composition, lighting, et cetera. So having that knowledge and incorporating it into the design of the models, I think can be very helpful.

• 33:59Speaker 01

  Talking, yeah, talking to users is really important. We talk to our users, we feel like, oh, we definitely recognize the blank canvas problem. Like when you ask the user, how do you want to improve this photo? They don't know. When you make some edits to that photo, now they have opinions on like whether they like edits or not.

• 34:15Speaker 02

  Is that what they don't like?

• 34:16Speaker 01

  They know what they don't like. Exactly. So, you know, talking to users is really, really important. And I also think having a very good understanding of the technology so that you can make this fine balance of, you know, staying ahead of the users. I think it's also really important to build that kind of intuition, not just from the user perspective, but also from the technology perspective.

• 34:38Speaker 02

  How do you see the process versus end results? Just because in art, if your human is creating art, a lot of it goes into the process of creating it. And then final result is when you decide the process come to a point of your satisfaction. How do you compare that with how agents create art?

• 34:58Speaker 00

  I think in many ways it can be quite similar. And I think even when agents create art, there's still the human involved in that process. Because that process is the part that's on for artists.

• 35:10Speaker 00

  You look at like breakdowns, VFX breakdowns, and you see how many different steps and layers went into the final pixels.

• 35:19Speaker 02

  Yeah.

• 35:19Speaker 00

  And it's fun. And I think people who do that kind of work find it very exciting to kind of see how everything breaks down. I think the way things break down in the future with AI tools will look very different, but it'll still exist. And so it'll be this kind of back and forth between agents doing some parts of the work, but then humans also ultimately tying it all together.

• 35:37Speaker 01

  Yeah, and this may not directly answer your question, but there's always this debate of like, okay, we're working on AI photography. Is photography gonna die? Regular photography gonna die in the future? I don't think so. There are a lot of people who are doing photography, not just because they enjoy the result, they enjoy the process so much. Right? Like even digital cameras have become so good nowadays. A lot of people are still taking films. Film is such an interesting process, right? Like it has this unique thing of you have to capture that moment, otherwise it's gone. And you don't see the result right away. You have to develop the film. So a lot of it is interesting. So I think people are still enjoying those processes. But at the same time, there are people who just want the results. And you know, how do we empower them with the AI creative tools to make it easy for them, to unlock their creativity?

• 36:22Speaker 01

  I think that's also where I'm excited about the AI touring for creatives.

• 36:25Speaker 02

  There's definitely a lot of latent demand. So the people who did not have the money to buy a very fancy camera to go out there and take pictures, and they also never took a film developing class ever. Like they found their medium. Like here's how we see the history too. Like the traditional, you know, creative tools, they were all there, but new entrants, you know, they come on the market, they targeted users who were not users of their traditional tool. They will want to learn, but there's an easier way for them to get there and get the results. How have you observed where that latent demand come from? Like in photography, who are the people who are, you know, I don't have money to buy a camera, but I, yeah, I want some sort of results from photo.

• 37:11Speaker 01

  Yeah, yeah, I think photography is such a, it's not just art. It's also a document of our record of your important moments in life. So that's very different and a lot of people, they want a good recording of their moments, their lives. They may not necessarily have the skills to take that photo, they may not have the equipment to take that photo. And for them, they can only, in the past, they tried to buy a fancier iPhone, maybe the latest generation iPhone. So they hopefully they get a better camera out of it. But they don't have the skills to do editing. Those are the latent demand that we try to address, you know, we really feel like, whether it's AI model or AI agents, creative tools, that we can develop to make that easy for them or even do the job for them.

• 37:53Speaker 01

  So that they can get this good photos of their important moments in their life. Um, that's what we want to do.

• 37:59Speaker 02

  Yeah, so one thing um we had in a firm, which is like we have photo shoots for people's headshots. Uh, and those like your premium and you know, you invited a photographer and the whole studio and lighting and makeup. Um, and recently I've found that there's more and more folks who are, who are just like, I couldn't make it to the photo shoot. This way. But I'm going to generate it on photo, which you know, it's like a very interesting segment of market where you need a physical presence to take a photo, either that's a product or a human or something else. Have you seen that too, like on the new photo users?

• 38:35Speaker 01

  Yes, yes, uh, I think, and also like photo, photo shoot is uh, if you think about it, it's a setup environment anyway, right? Like you're setting up the lighting, you're, you, you go there and you try to take the photo is, and, and most of the time you don't have all control of everything just because of the physical limit.

• 38:52Speaker 02

  Yeah.

• 38:52Speaker 01

  Um, when you're doing it in a generative ways, there's so much thing, so many things that you can do. You can change the lighting the way you want, you can change the angle, you can even change your makeup if you want. Uh, all of this, I think, is just so important and people are finding that those use cases, that for sure.

• 39:06Speaker 02

  Yeah, I would love to hear from uh actual photographers' point of view too. Cause Matt, you take a lot of photos. So I guess like, how do you think about this old versus new world of AI photography?

• 39:17Speaker 01

  Yeah, I, it's fun, I almost viewed, uh, there's definitely like the overlap where you can.

• 39:24Speaker 00

  Do them together, but then there's also this very interesting where you're like, truth is very different medium. When you start doing things that are purely in the AI world, you can start thinking about what would an image look like? Maybe halfway across the world or in specific studio lighting situations that you would never actually be able to do. But you think it's like a cool idea or image. I find myself sometimes going and like seeing a cool image out there and trying to think, how would I actually replicate this in AI? What are the right terms to use? And you kind of go through a rabbit hole to get to that point. So it unlocks like a new area that I haven't been able to explore before.

• 40:03Speaker 02

  Yeah. How do you think about the role of traditional tools in the new world of AI created tools? Are the traditional tools becoming infrastructure? Where, you know, they're always gonna be here and now we just train agents to know how to use them for, you know, bits and pieces? Or are they getting rebuilt?

• 40:23Speaker 00

  I think at the end of the day, all these things are just tools. And so it really just comes down to whomever has some vision and they want to execute on that vision. What tool is right for them? For some people, it'll always be Photoshop. For probably newer generation, it won't be. And so it'll just be kind of dependent on the person. And as you mentioned, maybe it'll be agents that'll be interacting with these tools. So I think it's hard to say. I think these things will always exist to some extent, but it'll find different uses by different users.

• 40:51Speaker 02

  Yeah.

• 40:52Speaker 01

  Yeah, I really like this framing is it's not like we have this new problem of like, oh, very new tools, very old tools. I feel like the tools have always been iterating in the past. Like there's always new tools come out, whether it's AI based or not AI based. There's always new tools come out. Just we're probably at a stage where the tools are iterating so fast. So I do think some tools, you may not need them in the future anymore. They're gonna be replaced, but a lot of tools will still exist. And even the AI models that we have today, they're gonna be replaced as well, like new models come out. So just an iterative process, you know?

• 41:25Speaker 02

  Yeah. What is the benefit of having your own model as part of what you put in the toolings that you build? Like, and is there a benefit in open sourcing the model?

• 41:38Speaker 00

  So I think one of the biggest benefits of having your own model is the ability to build off of it. At the end of the day, your product has a certain set of users and you want to do the best thing you can for those users. The best way to do that is by customizing it, maybe making adjustments to the model that don't make sense for, say, a general open source model, but makes sense for your product. And that's one thing that is not always easy to do on top of existing models and may require some specialization that only you can provide in your own models.

• 42:12Speaker 02

  Right.

• 42:13Speaker 01

  Yeah, for us, a little bit different. We don't directly do foundation model. We mostly work on personalization. The reason is because there are so many cool companies out there like Luma. We really think customization is something the user should own, like that's the user's model. You build a user's model for them, but at the end of the day, they're using that model. They have full control of that. So like for that particular thing, like we want to build a technology to enable them to do that, but they own the model. And we want them to be able to combine that with any foundation model they like to use, right? It could be a specialized model that Luma specifically built for like a particular use case, for example, right? So and to really disentangle this personalization versus foundation is sort of the goal that we have.

• 42:56Speaker 02

  With the advancement of AI tools, what do you think will separate the best artists from everyone else in the next couple of years?

• 43:07Speaker 00

  I can say from my experience already, it's very obvious when people use our tools or not even our tools, any of the AI tools. You can tell pretty quickly someone who has, who's just throwing in a prompt and getting a result out versus someone who has thought more about like the whole holistic story. I always get amazed at what very creative artists can do with these tools. Many times you see things that just don't seem like they would have been possible, given the tools capabilities. But artists have figured out how to make it possible. And so I think that's really the thing that separates it. It's how do you make something unique with the tools you have access to now?

• 43:48Speaker 01

  Yeah, I I think, I mean, it's it's hard to say there's good artists and bad artists, but like they're probably good artists and average artists. I feel like with AI tools, they're definitely raising the lower lower bar of just just so that, you know, even someone like like me can create something pretty, pretty hit with the AI tools. But I also think that the gap between the very best artist and average artist is gonna be even bigger. It it exactly because what Matt said, you know, like there's so so much easier for people to express their creativity when they have a very good understanding of what they want to get, how they want to get there, and they can use these powerful tools to do that. They can achieve something they won't be able to achieve before. So the gap is gonna just get bigger.

• 44:30Speaker 01

  And of course, they also need to have very good understanding mastering all these new AI tools as well.

• 44:38Speaker 00

  I I I think maybe one final point. You've seen AI generations that are bad, right? Like, I think we've all seen them that are bad. I can guarantee the same tools that made those bad generations, they've also made generations that you would think are amazing.

• 44:49Speaker 01

  Exactly.

• 44:50Speaker 02

  Yeah. Something I heard from someone today earlier was they assume AI to be slop. Like, anything AI models have would be slop. It's up to the humans to create something out of it.

• 45:01Speaker 01

  Right.

• 45:02Speaker 02

  Just because if you're just like, say, I want a picture of a coffee as a like a cup of a coffee. The first generation is just mediocre. You can't have it in any stock image website. It's nothing unique or it's nothing personal.

• 45:16Speaker 01

  Right.

• 45:17Speaker 02

  So it really is up to the the user to kind of make it like theirs, to kind of put their mark and style on there. What are your thoughts on personal styles, like in in terms of personalization? Like, what constitutes the style? What constitutes personalization for the users?

• 45:36Speaker 01

  I I feel like that's such a hard question, like it's almost like it's. I almost feel like there's no good way to define it other than you can learn that from data. Um, so and it's not just like, okay, the user's existing data. It's also like when the user use it, what's their reaction to it? How did they like it or not? Because even if you ask the user what's their own style, I mean, they can put some keywords there, but it's not complete. So I feel like, you know, if the way to solve the problem is you really to kind of define their style and then model their style, they're probably not the approach I would take. Even from a research perspective, I feel like you look at the data, you find attribution that the data shows you, and then you probably have to figure out that style.

• 46:21Speaker 01

  And even at that time, it's still implicit because it's part of the model.

• 46:25Speaker 02

  Yeah, I see, interesting.

• 46:27Speaker 00

  I think it's also something that changes over time. Yes.

• 46:31Speaker 00

  Both as a person changes. Like, I know the stuff that I thought was interesting five years ago versus now is very different. But then also, you know, I'm working on one project one week and another project another week, and I inevitably need a different style. And so maybe that's how you define like the styles for specific tasks, and maybe personalization is sort of the holistic, like what you as a user tend to like. But yeah, I guess it's a bit mixed.

• 46:57Speaker 02

  Awesome. And with that, well, thank you so much for coming on the pod. This is such a fun discussion. Yeah, I really appreciate it.

• 47:04Speaker 01

  Yeah, thank you for having us.

• 47:08Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at A16Z.substack.com. Thanks again for listening, and I'll see you in the next episode.

• 47:33Erik Torenberg?

  As a reminder, the content here is for informational purposes only. It should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com/disclosures.