Generating product imagery at Shopify

Shopify recently released a Hugging Face space demonstrating very impressive results for replacing background scenes in product imagery. In this episode, we hear the backstory technical details about this work from Shopify’s Russ Maschmeyer. Along the way we discuss how to come up with clever AI solutions (without training your own model). Sponsors: Advent of GenAI Hackathon https://adventofgenai.com/ – Join us for a 7-day journey into the world of Generative AI with the Advent of GenAI Hackathon. Learn more here https://adventofgenai.com/! Traceroute https://deploy.equinix.com/traceroute/ – Listen and follow Season 3 of Traceroute starting November 2 on Apple, Spotify, or wherever you get your podcasts! Featuring: Russ Maschmeyer – GitHub https://github.com/StrangeNative, LinkedIn https://www.linkedin.com/in/russmaschmeyer, Mastodon https://sfba.social/@strangenative, X https://x.com/StrangeNativeChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Advent of GenAI Hackathon https://adventofgenai.com/Shopify’s HF Space for background replacement https://huggingface.co/spaces/Shopify/background-replacementShopify Magic https://www.shopify.com/magicUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:08Speaker 00

  Welcome to Practical AI. If you work with artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners for helping us bring you Practical AI each and every week: Fasi.com, Fly.io, and Typesense.org.

• 0:37Daniel Whitenack?

  Hello, it's your co-host Daniel here. Just wanted to let you know about an awesome event that's coming up this December of 2023. It's called the Advent of Gen AI. It's a hackathon that's being put on by Intel's Liftoff program and Prediction Guard, and it's going to be a seven-day journey into the world of generative AI. You're going to get access to some really cool hardware from Intel. You'll also get access to run prompts through the latest Open Access LLMs via Prediction Guard. And every day of the challenge, you'll get a new chance to show your generative AI skills and learn a bunch of cool stuff. So I encourage you to register. It's totally free.

• 1:22Daniel Whitenack?

  You can take part and learn all of the cool generative AI things that you're hearing about on this podcast. Find out more at adventofgenai.com. That's adventofgenai.com.

• 1:53Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder at Prediction Guard and I'm joined as always by Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris? Doing very well, Daniel. How's it going today? Oh, it's going great. You know, as our listeners know, or at least the ones that have listened to a lot of our shows, my wife owns a business, which is an e-commerce business. Quite a business. And next week, as we're recording this, for those listening at a future date, next week is Thanksgiving here in the US, which means next Friday, basically, well, it's already sort of started, but next Friday to that following week,

• 2:37Daniel Whitenack?

  Black Friday, Cyber Monday is a huge retail and e-commerce extravaganza in our world. And so I'm really excited because leading up into that today, we've got the expert with us, Russ Maschmeyer from Shopify, who is project lead for Spatial Commerce and Magic Labs at Shopify. Welcome, Russ.

• 3:01Speaker 02

  Hey, thanks, guys. I'm super excited to be here. It's been cool to follow along with the podcast and I'm just super stoked to chat with you guys today.

• 3:08Daniel Whitenack?

  Yeah, for sure. Well, I'm, I'm coming into this like super excited because over the past, well, it's been 10 years, my wife's business, and at least so nine of those years, they've been on Shopify, which means I have been in Shopify. I've dug into all the data behind. I've worked with the Shopify API. I've built chatbots on top of Shopify to like sign up wholesale customers. I've like dug into the like liquid code on the site. So I'm like all about whatever I can learn from you today and hear about what's going on at Shopify. I am super excited. I didn't even know he was that much into it, Russ. I'm, you know.

• 3:54Daniel Whitenack?

  Well, when you're the husband of an e-commerce entrepreneur and you're also a data scientist, occasionally favors are asked and. I'm feeling very third wheel now. I just wanted you to know. Well, over that time, it's been cool to see how Shopify has added so many amazing features and is really powering a lot of huge brands, not only small brands, but larger brands. I'm sure you all are gearing up for a crate. First off, I just have to ask, like, what is the week leading up to Black Friday, Cyber Monday like at Shopify?

• 4:29Speaker 02

  You know, it's very busy, as you can imagine.

• 4:32Daniel Whitenack?

  Say, what a loaded question.

• 4:34Speaker 02

  It's the kickoff to the biggest shopping season of the year. Shopify powers just an enormous amount of that holiday shopping season. So you can imagine the teams internally are prepping for it. They are getting products like locked in place and just, you know, operating at their optimal maximal performances just to support the load that's coming in this upcoming weekend. But, you know, every year, we also launched this really cool live globe that's a 3D visualization of all the live data and perched and like orders happening all around the globe in real time. So you see like, you know, orders streaming around this globe. And so this year, I've been also helping to lead some of those efforts. I'm

• 5:17Speaker 02

  really excited for that to get its annual debut this year. You might, you might see some ideas we talked about today. Hope you're there.

• 5:24Daniel Whitenack?

  Cool. Yeah, it's like the live view of Santa going around the world at, at light speed.

• 5:30Speaker 02

  Totally the Santa map entrepreneurship.

• 5:34Daniel Whitenack?

  There's all sorts of interesting things that Shopify is doing, but specifically here, we're talking about AI and what you're doing in regards to that. Maybe as we jump into that, could you describe a little bit from a person that's embedded in this kind of e-commerce world and seeing what a lot of people are doing in various industries, various stores? How do you view the impact of AI on e-commerce specifically right now? And kind of where it's headed? Like, where's the, where are we seeing the biggest impact in terms of AI right now in e-commerce? And I know we're going to be talking about some of the recent things you've done, but kind of across the board, what, how does it look

• 6:18Daniel Whitenack?

  to you? And what are people thinking about?

• 6:21Speaker 02

  Yeah, I mean Shopify was pretty early in kind of this new wave of AI capability to say like, "Hey, whoa, like this is a completely new class of possibility for the tools that we make for merchants and the shopping experiences that our platform provides on the other end to shoppers." And Shopify is really just kind of here to make commerce accessible and entrepreneurship accessible to everyone. And we're really excited about these tools as a way to kind of further democratize entrepreneurship. There are so many things you have to create and produce and ideas to develop and knowledge to gain about markets and positioning and strategy and branding.

• 7:00Speaker 02

  There are so many tasks that entrepreneurs have to learn and develop along the road to building a successful business. And LLMs, generative AI are all incredibly powerful tools to help accelerate that learning curve for new merchants and to help them kind of get up that curve faster and build better businesses.

• 7:19Chris Benson?

  I'm curious, we all hear, you know, us consumers in the world, we hear about AI impacting retail and stuff. But for those of us who don't have as much of a view on it, can you kind of talk about how you've seen retail change in these years since AI is really, the last few years with AI really kind of getting in everywhere? What are some of the things that might surprise people? You know, they kind of know there's AI there in the background, but they don't really know how it plays or what it is. You know, surprise us a little bit, like what's something where I go, "Oh, wow, I didn't realize that."

• 7:51Speaker 02

  Well, you know, I mean, we started by just like adopting these tools in our engineering practice to begin with. We got some of the early previews of Copilot and started using that to help accelerate some of our development work early on. But really, the place where we've seen it have the biggest impact in the near term is on tools for merchants. You know, like when we think about who our core customers are at Shopify, it's the merchants who we power with our platform and enable them to do really creative, amazing things, you know, at the scale that they never maybe thought was possible for them, right? And AI is again like sort of a way to accelerate that work and give them more time back to, you know, instead of spending an hour and a half like trying to craft the perfect product description, because you're not totally sure exactly what makes a good product description.

• 8:35Speaker 02

  You know, last year at our winter edition, we shipped a really simple tool where you just like enter in like a couple of raw details about your product and hit the magic button and it just writes a well crafted narrative product description that speaks to product benefits and all the great standard practices of writing a good product description. And you get that in seconds versus an hour of human toil. And so the place where we've seen AI really have the biggest impact early on is just in accelerating the work that merchants are already doing and allowing them more.

• 9:06Daniel Whitenack?

  In, well, I guess it's e-commerce, but also like web content development. It's a very like multimodal thing, right? Like you've got these product descriptions, that's part of it. You've got product imagery, you've got website layout, you've got potentially ads and integration with like other platforms. Talk a little bit about like within that space, because there's so many, as you mentioned, there's so many tasks to address within that space. As Shopify kind of looks at the merchant experience, how have you narrowed down on the particular problem sets that merchants really want to hand off?

• 9:49Daniel Whitenack?

  Versus like those things I know also from just being in it, like, you know, marketing teams love to get in there and tweak things and like be part of the process, but they also really don't want to do certain things too, or things that are just kind of grunt work essentially.

• 10:06Chris Benson?

  That sounds like it's coming from experience right there.

• 10:08Daniel Whitenack?

  Yeah.

• 10:09Speaker 02

  Yeah.

• 10:11Speaker 02

  I mean, we, we have a word that we use at Shopify, you know, toil, this idea of like work that kind of has to be done, but isn't desirable work to do. And so we look for toil that merchants do. And so we spend an enormous amount of time sitting down with merchants, talking with them about how they use our platform, what they want more out of our platform, what they wish they could be doing with their business, what they are doing with their business. And from that, we've learned a ton about, you know, what are the ways that merchants would like to spend their time? And then what are the ways that they just kind of have to? Because that's the way the world works right now. And so I think the opportunity for us is to find those moments and to build tools, like particularly magic tools, into those spaces that just sort of like make that go away.

• 10:55Speaker 02

  And when we do that, what we hope is that merchants will take that extra time that they have, that hour that they got back not spending on that one product description, or that one blog post, or that one email headline, they're like, ah, should I use A or B? I don't know. And just give them a really easy tool to generate that content, make it really high quality, give them the control to adjust if needed, and then publish it really quickly.

• 11:20Daniel Whitenack?

  You already mentioned one of those, this sort of product description thing. Are there a couple other ones that you could kind of highlight just to give a sense of the breadth of how this technology is applicable in the space?

• 11:32Speaker 02

  So we've launched this suite of tools that we call Shopify Magic, right? It's our free suite of AI-enabled features across our whole Shopify admin. And these things crop up in a few different places. It can sort of help you take the power of your own data and make it work better for you. And we've applied that in places like email headline subject writing for marketing emails and things like that. We've leveraged it in the context of generating blog content. Obviously, product descriptions is another. And we're obviously really excited about some of the early work that we've also done in the image generation space. We recently released a hugging face space that I'm super excited to dig into more, I'm sure, a little bit later.

• 12:15Speaker 02

  You know, when you think about a storefront and the kind of content merchants need to produce, it really falls generally into one of two categories. It's either text, or it's images. We're really excited about both of these spaces and helping accelerate merchants there.

• 12:29Chris Benson?

  I know Daniel has used the tools a lot, but if you had someone who was a novice and they were getting into business, and let's say they're starting it now, so they haven't been doing it.

• 12:38Daniel Whitenack?

  It's a new store. Chris is gonna sell, what, like socks to fund your animal charity?

• 12:45Chris Benson?

  Raccoon socks. There you go.

• 12:46Daniel Whitenack?

  Raccoon socks. There you go. I like it.

• 12:48Chris Benson?

  Christmasy raccoon socks. How's that?

• 12:51Speaker 02

  Will it keep the raccoons out of my trash cans? Will the socks, is that what they do?

• 12:55Chris Benson?

  If you want them to, that's no problem. For those who are going, what just happened on the show? In the world away from technology and AI, I'm a wildlife rehabber and I, right now I have 20 raccoons at my house. So that's what that's all about.

• 13:10Speaker 02

  It's a full Christmas party.

• 13:12Chris Benson?

  It's a full, yeah, well, it's quite a Christmas party. You put 20 raccoons loose in a room. Oh, boy. Yeah. Okay. So back to my, back to my new store that I just opened up. I'm excited. I don't have Daniel's depth of experience at this. What are all of the amazing things? I'm either by myself or I don't have a lot of help. Everyone's tossed me to the wolves. I've come to Shopify because I know you have all these, these magical tools. Can you tell me a little bit about that experience from a merchant's standpoint? Like on day one, what am I getting into? How should I think about it a little bit? And what are those? How do those AI tools directly impact what I want to start doing today?

• 13:51Speaker 02

  Yeah, totally. Well, I mean, I think from a merchant's perspective, if they were to log into the admin today, I don't think they'd be overwhelmed with like the amount of AI tools, you know, sort of all over the, I think today we really started with a focused approach that feels super seamless and integrated into just the activities that merchants are already doing. For example, product descriptions, let's say, you know, you're a merchant, you've just started building your storefront. You're super excited to like get that up, put a great face out on the web. And you're starting to build out your product catalog. You're starting to think about, you know, how do I merchandise my products? How do I talk about them? You're a new merchant, you haven't really done this before. You don't know what the best practices are for product descriptions.

• 14:34Speaker 02

  Or, you know, if you want to create some SEO content to kind of market your brand and product expertise in the space, you can go into your product detail editing page for a product you want to add, just drag and drop your images. One of the really cool things, and I'll say this because I'm also product lead for Spatial Commerce, is you can also drag and drop 3D models into that image bin and it'll handle it beautifully. So just some cool stuff coming down the line.

• 14:59Chris Benson?

  Raccoon 3D models. That's awesome.

• 15:02Daniel Whitenack?

  This is gonna be an awesome site, Chris. Looking forward to it.

• 15:05Speaker 02

  If you've got 3D models, drop them in there, too. Those will display on your product detail page on your web storefront. And then when you get to that, like, that challenge of like, okay, now I've got to write a product description. Oh, gosh, I haven't thought this through. I'm not really a copywriter. You know, I went to business school and maybe I can write things, but like, I don't know what, what's a good product description? What does that even look like? And I could go and spend, you know, an hour, two hours doing Google searches and combing through results and sort of like collating my own idea of what makes a good practice for product descriptions. Or I could just click on that lovely little sparkle button after entering in like, oh, it's white. It's, you know, these dimensions, it's got these materials and just like, boom.

• 15:47Speaker 02

  And you've got this incredible, you know, text description of your product. It pulls from your product title. So like, you know, if you've mentioned that it's like this kind of product for this category, it gathers all that context initially and then brings that to bear on the description that it writes. You'll have the ability to pick what tone you want that description to have. So we give the merchant some ability to kind of shape like, oh, do I want this to feel sophisticated? Do I want it to feel fun? Do I want it to feel like there's deep expertise behind this product description? And so I think those really simple tools just kind of placed seamlessly into the UI, exactly where the merchant is kind of doing these activities today anyway, is really kind of the powerful first step that we want to take.

• 16:32Speaker 02

  Introduce merchants to these new tools and then we'll expand from there in some pretty powerful ways.

• 16:42Speaker 04

  What's up, friends? AI continues to be integrated into every facet of our lives. And that remains true because you can now index your database with AI. You can write more code, become that 10x where you always wanted to be. And you can even draft a letter for a lease on an apartment or a new property. AI is everywhere and it might be time for us to start questioning, is AI our friend or our worst enemy? And that's the focus of the three-part season opener of the Awobony podcast called Trace RAP Podcast. You can listen and follow the new season of Trace Route starting November 2nd on Apple, Spotify, or wherever you get your podcasts. And this show is all about the humanity and the hardware that shapes our digital world. In every episode of Trace Route, a team of

• 17:24Speaker 04

  technologists seeks to untangle the complex question, who shapes the internet? Seasons one and two gave us a crucial understanding of the inner workings of technology while revealing the human element behind tech. And season three tackles not just AI questions, but also, how can we use technology to preserve the earth? Who influences the technology that gets made? And what happened to the flying cars we were promised? I think it's safe to say that the future of AI is both exciting and terrifying. So it's interesting to hear the perspectives of experts in the field. Listen and follow this new season of Trace Route starting November 2nd on Apple, Spotify, or wherever you get your podcasts.

• 18:14Daniel Whitenack?

  I think how we initially started chatting back and forth was at least partially because seeing this Hugging Face space, which is really cool that you all put up and I know got a lot of attention, partially in the merchant world, but also in the AI world around, you know, the community that's being built around open source AI tools on Hugging Face. And you have this space there that had to do with product photography. Before we go into the technology, the space, kind of how this works, could you kind of describe a little bit of the motivation behind this, this project? Because you mentioned product

• 18:56Daniel Whitenack?

  photography, but people might not, if they haven't been exposed to kind of e-commerce as much, or worked on their own e-commerce store, they might not realize kind of what product photography means and some of the challenges around it. So could you kind of set up the motivation for this before we hop into the technical pieces?

• 19:16Speaker 02

  Yeah, absolutely. So merchants spend an enormous amount of time and money generating visual media. That's like compelling. That gets people excited about their products, either the details in the design, or the lifestyle that it might afford, you know, whoever buys it. And these images are really core to what drives a lot of commerce online, whether it's advertising. Or whether it's building out an attractive storefront, like a web storefront, or whether it's appearing in various channels, you know, in different marketplaces as well. But not least of which is on the product detail page where somebody has landed a shopper and ostensibly they're interested in this product.

• 19:57Speaker 02

  And the job of those images in that context is to do the best job possible, painting a picture of what like that product looks like in somebody's life, as well as all of the details about the product. And early on, you know, last year, when Stable Diffusion and other open source image models started to land, I got really excited about a future where you could imagine merchants just being incredibly more agile and cost efficient in how they create these images. And so we started digging in pretty quickly. We played with Dreambooth as soon as that was available and Stable Diffusion. And we started to see like, actually, could we train a Dreambooth model that can encapsulate the concept of a product and recreate it in high fidelity

• 20:43Speaker 02

  over and over and over again? That's like the dream, right? And we're getting closer and closer to that, but we weren't quite there yet. But some of those early explorations proved beneficial to understanding the space, understanding the technology and thinking a little bit more deeply about some simpler ways that we might be able to bring this to market in the near term. When you think about the opportunity for image gen in commerce, I mean, it's massive, right? And the ability, the promise of being able to recreate your product in high fidelity in any scenario is kind of the dream. You know, you could imagine at requesting any kind of lifestyle or product detail image and just in seconds, getting that out the other end to use in your storefront or to use in blog content or to use in advertisements about your product. And that's incredibly powerful because commerce is always changing.

• 21:29Speaker 02

  Taste is always changing. Seasonality is a huge piece of commerce and thinking about, how do you merchandise and market your products differently in the spring versus the fall? And keeping up with the amount of imagery just required to drive that, that part of your business is really challenging. I think the reason that we got really excited is that we saw an opportunity to take the existing imagery that merchants had, either from past photo shoots or from humble, like at-home photography with their kind of mobile cameras set up on their kitchen counter, or whatever they might have access to, and give them a tool that could not really change any pixel of the product itself, but otherwise completely reinvent the reality

• 22:13Speaker 02

  around that product. And so we started to work on, you've seen a lot of examples of this out in the market, but I think the key problem that we saw with a lot of these early examples of this, where you sort of, you do object segmentation to select the product and keep it sacred. You don't touch any of the pixels that you sort of guard and mask there, but then all the pixels in the background you reimagine with AI. And what we saw with most of these early tools, as I said, was that there was this real disjoint appearance between the product that got masked and safeguarded and the reality that was created around it. The camera angle, like it looked like, oh, well, this one was taken from above, but the original product image was from straight on. And there's no grounding shadows and there's no realistic reflections of the product in the environment.

• 22:58Speaker 02

  The pixels of the product and the pixels of the environment aren't speaking to each other. They don't know, one hand doesn't know what the other is doing. And so they can't knit those pixels, those moments of grounding around the product that really sell the illusion that it's part of this other reality. Those shadows, those ground reflections, seeing maybe some of the light of the scene hit the product object itself. And so we wanted to really tackle those grounding problems that we saw in a lot of these early examples. And so I'm happy to dig into all the technical details of kind of how into that, but that was really the opportunity that we saw was to begin to bring some of this magic to merchants really early before we're even yet to that perfect personalization

• 23:40Speaker 02

  of you upload a bunch of images of your product and now it produces them again perfectly at the other end. We can begin to bring really powerful tools to merchants in the space already, even with techniques like this.

• 23:51Chris Benson?

  And just as a point of clarification, when you say grounding in this case, you're kind of talking about that visual context going across different, as opposed to technical grounding with a model and such, just because we talk about both on the show.

• 24:04Speaker 02

  Yeah. No, it's a great clarification. Yes. And purely talking about sort of the visual aspects of the output image and making that product feel seated in the new reality in some visual way.

• 24:15Daniel Whitenack?

  Before we hop into more of the details about how you actually accomplish this, I'm wondering how you see the kind of state of open source generative models in comparison to maybe some of the other platforms that are out there that do enable amazing things, but not in an open source way. It sounds like, at least for your team, I don't know if it was personally important to you and your team to leverage some of this open technology, or it was just like, these things are openly available, they're licensed permissively for our use, and they're enabling things that we couldn't do before. How do you view the state of generative AI on the image side specifically?

• 24:57Daniel Whitenack?

  Because we've talked a lot in recent weeks about the text side of things and how maybe text generation models that are open compare in certain ways or other ways to close models. But I'm wondering, from a team that's actually used this kind of image generation models that are open and licensed permissively, what was that experience like for you? It sounds like this grounding element was one thing that you had to deal with, but what was it like for you generally to work through the details of getting the model, figuring out how to run it, figuring out how to scale it, maybe that sort of stuff? Could you describe a little bit of that process?

• 25:37Speaker 02

  This is a really early field, so we're still figuring out what the tools need to look like and how to work efficiently. We were working on some of these early ideas in a very falling-over-ourselves way in some notebooks, trying to collaborate and work together and just not seeing the pace that we wanted to see in our iteration speed. Our team works really quickly. We work on these three-week sprints to just very rapidly prototype and understand a new technology space and develop some kind of potentially useful concept there. And so, we needed a way to move faster. And Toby, the CEO at Shopify, is incredibly technically adept.

• 26:19Speaker 02

  He's an incredible developer in his own right and was really interested in some of the image gen work that we were doing in the early phases, and suggested that we pick up this new tool called Comfy UI, which is an open source tool. So, we're big fans of open source at Shopify. It's why we shared to Hugging Face, because we want to contribute back to that community. You can go take our pipeline and do something with it. It's up on Hugging Face. And so we're really excited about open source and obviously the capability of other providers as well. And you know, our objective is always to bring the best technology to our merchants, whether it's open source or buy a closed provider. So we're really excited about all contributors in the space and what tools we can build for merchants with them.

• 27:02Speaker 02

  So we focused a lot on SD in the early days and we were excited when Stable Diffusion XL launched. And that's actually the model that underpins our Hugging Face space. We've done a lot of work with Stable Diffusion in all of its iterations as we've explored this space and are excited to continue to work with it and obviously build amazing new stuff with it. But yeah, I mean, I think we used Comfy UI, we dug into it. I think what we loved about it is that it's this node-based UI. I come from the design world originally for product and UI design. And there was this much-loved tool originally from Apple, but it got hacked by a bunch of prototypers called Quartz Composer. It's a node-based interface with a bunch of like little modules that do little conversion jobs.

• 27:45Speaker 02

  And you can wire them together in these sort of larger machines and recompose and move things around really quickly and rewire them and change the sort of constant values and very quickly build these very complex computing machines in a visual way. And so for me and for our team, that was a really powerful tool for us to accelerate our process. We began building these machines, this pipeline that we ended up putting up on Hugging Face in Comfy UI and iterating there. And when we had it to a great place, we pulled that code into Hugging Face, you know, sort of rebuilt everything, ground up with the models hosted on Hugging Face and sort of encapsulated the pipeline there. But we were able to iterate super quickly and visually this way and see exactly what every piece of the machine was doing at each run.

• 28:30Chris Benson?

  It's really interesting because you're taking new capabilities in the AI space with a large business that's, you know, running and you're trying to, you're trying to kind of do the uptake while absorbing the technology at the same time. And as you pointed out, you know, your CEO brought Comfy UI to your attention. As you're doing these activities as a business owner in general, like the folks that are there. How do you decide to make investments in certain areas with these new technologies? And decide, because you know, there's the, the pull and push of, well, direct AI isn't our business. Our business is to make the best platform for all these merchants. And yet there's all these new capabilities out there, but they're not mature enough. You brought an example to bear a second ago.

• 29:14Chris Benson?

  That's a complex set of business processes to work through and, and figure out what's the right level. How does Spotify think about that? You and others there in terms of like, is this a step too far to go on a particular leap? Or, or this is appropriate, like comfy UI turned out to be? How do you make those choices?

• 29:33Speaker 02

  It's a jungle. And one of the tools that we've used is, is really our magic labs team. So early on, uh, at the end, well, actually rather at the end of 2022, as we began to see some of the rapid advancements in L L Ms begin to take shape and the product possibilities became clear. We started our early efforts around product descriptions and generating those on the fly for merchants. And early on, it was really about saying, okay, what are the things that this tool, this new technology is going to obviously be capable of, right? With maybe a little prompt engineering, you know, we'll figure it out. But like, what seems to be well within its grasp, but also of maximum time-saving value to merchants?

• 30:17Speaker 02

  And product descriptions was like that perfect Venn diagram out of the gate, right? It was just kind of obvious to everybody. It was like, oh, so much, we know, we already know so much toil is spent. Just creative writing, something that can probably be written pretty quickly. If you have the necessary context and the best practices and all of that stuff. So we got to work on that and we shipped that super quickly. I think we turned that around from concept and team assembly to launching it winter additions in about two months. And so it was just an incredible accelerated, you know, one of those moments, right? Where just the right people and the right technology and the right opportunity come together. And pretty rapidly out of that, we formed the magic team at Shopify to sort of help. Invest more deeply in these AI technologies and figure out all of the places and all of the ways.

• 31:01Speaker 02

  We wanted to leverage these new capabilities across the admin to help accelerate what merchants were doing. And so we've continued to work on a bunch of different ideas there, not least of which is some of the image gen work that we've done. And the way that we kind of work through this space, because there is so much going on, like every week, you've got to weed through at least a half dozen like groundbreaking papers all over the map. And so a big part of the process is connecting to that fire hose of what's happening so that you never lose sight of like a paper that might completely change how we think about serving our merchants. And then weeding through those and just sort of logging them as you go.

• 31:42Speaker 02

  Like I've got a Twitter bookmarks folder that's just so deep that I get back to periodically and sort of pull, pull things that sort of feel like they have remaining value out of and surface to the team and surface to the company and start discussions around. And within Magic Labs, our small team has been iterating on this three week cycle to just digest all of these new technologies, all of these capabilities. Every three weeks, we pick up a new one. We have no roadmap. We just, we have areas of curiosity and every three weeks, we look at what's out on the table in the world. And we say, what's the most exciting or potentially impactful thing for commerce and our merchants next? Based on what we see here and we pick what we want to work on within a day.

• 32:26Speaker 02

  And we're prototyping by, you know, day two or three, after having like picked up either a new piece of hardware or opened a, you know, some open source code on GitHub to get started. And we're prototyping and within three weeks, we've gotten to the end of that process. We've got a deliverable that like peruse either disperuse that something we hoped could be possible is actually not possible. And here are the reasons why. And here's now what we're looking for in the next iteration of this technology. Or Quite often, actually, there is a path here. Here's what it looks like. Here's how we we might shape this. And from there, tons of internal teams are eager and interested and hungry in sort of like, how do we think their products or how delivers these technologies for their particular

• 33:10Speaker 02

  challenge with merchants? We're really lucky to just be working in an organization that that gives us really fertile ground to just kind of bring these technologies and what we're learning about them to a really wide set of problems that all seem very tractable based on the trajectory in what we're seeing in the tech right now.

• 33:27Daniel Whitenack?

  Well, Russ, I really appreciated your perspective on kind of how how your team is thinking about processing a lot of these advancements that we're seeing in technology and tools so rapidly, which is definitely hard to keep up with. But I love how you're thinking about these short cycles of work and thinking about what what could be impactful. I'm wondering if we could kind of revisit this problem of grounding with these product images, because I think some people might really be interested in that. And I'm wondering, to start with that, could you just rephrase the kind of main problem of this grounding for people that are kind of new to this?

• 34:06Daniel Whitenack?

  And walk us into like, how you identified this problem and thought about coming up with a solution to it? Because I could see a whole spectrum of things here, right? Like, there's a hierarchy of ways to do this. Everything from, oh, we just need to make our prompt better to we need to retrain a model from scratch that's Shopify Stable Diffusion or Shopify GPT for for this, right?

• 34:32Speaker 02

  Shopify Fusion.

• 34:34Daniel Whitenack?

  And, yeah, exactly. Shopify Fusion. Yeah, obviously, that ladder of the spectrum. I think there's very few people that get to that level of the hierarchy when they're solving these problems. But it is hard sometimes for people to parse out. Where along that spectrum from playing with your prompts to maybe chaining, to like creating some preprocessing, post-processing, to fine-tuning, to training your own model, where is it reasonable for us to land on that spectrum? That's something for people that's really, in my experience, hard for them to parse out. So how did that work out for you all in this case? Maybe starting with rephrasing that problem of grounding and then getting into how you started thinking about how you might solve it.

• 35:17Speaker 02

  From the highest level, it's like if you're a merchant, right? And you're just starting out and you've got some products that you're really excited about. Either you've sourced them from like a really great provider or you make them yourself. You don't have all the resources that, you know, somebody with operating business and scale and lots of employees and tons of capital can deploy, you know, to build a business, you know, by hiring contractors and all those things. And if you're this merchant and you've got, you know, you can take photos at home or you've got maybe like some photos from a previous shoot, you paid your friend to do and they're pretty good, but they're not quite like helping your brand sing. You're looking for something to help you get over that. You're looking for that tool that's going to help take the, the media that you have

• 36:00Speaker 02

  and turn it into the media that you want. And your first thought is like, oh my gosh, AI, like of course, AI is going to unlock that. This was our first thought. It's like, well, if we can just train a model to know exactly what your product looks like, great, we can just create it over and over. And again, we're, we're getting closer to that, but we're not quite there. And short of that, like we're looking for ways to help merchants still realize this elevate, this creative elevation of the creative materials that they do have, right? And it turns out that a lot of merchants have pretty good photography. It's, it's almost there, right? Either because they took it at home with their mobile camera and they just don't have a whole lighting studio set up or, or they're just not sure like how best to art direct an image so that it feels tantalizing to look at and drives purchase behavior.

• 36:44Speaker 02

  And so we saw a path where, you know, you can of course crop out and save the product pixels from the original image that you took and keep those sacred and eliminate the challenge of getting AI to recreate the product again, which is a very specific thing, right? It's got details, it's got your logo on it. And AI has a hard time holding on to some of those details at times. But it can be fantastic at creating the background, right? The not centerpiece of your image to create a new elevated environment around it. And so we saw an opportunity to kind of take that path and give merchants an early tool as personalization matures, as we get to that point eventually,

• 37:25Speaker 02

  that can begin to help them unlock some of the value in their existing image medias, their humble kitchen countertop photography. And so by building this pipeline where we're able to hold and keep your product pixels intact and not change those. We keep those details intact. And yet we can magically create this world around it. When we started this journey, what we saw was, okay, we were like, okay, great. We'll take ControlNet and with stable diffusion and we'll combine these things. We'll use the depth of your original image and then we'll just ask it for a new background and it'll come out the other end and it'll be great. But what happens when you sort of segment out your product image and keep ControlNet from really understanding what's in there? So it doesn't change anything.

• 38:08Speaker 02

  It begins to lose an understanding of how to fill in the details around that object to make it look like it's a part of the environment that it's been creating. So it loses its shadows. It loses its tabletop surface reflections because it's actually kind of forgotten in a weird way that there's even an object there at all. So it doesn't, in whatever way you can say an AI model thinks, it doesn't have the triggers to generate those ground shadows and surface reflections in the scene it creates around the product. And so that was the first key obstacle that we saw when we started moving down this pipeline of trying to help merchants take their existing product photos and just kind of create new, rich realities around them. So we had to solve that grounding problem. There were no shadows. There were no good ground reflections.

• 38:53Speaker 02

  The camera angles were off, like you'd get a kind of tabletop scene background for a front on, you know, product photo and it's just, it looks wrong. You see it immediately, of course. And so we started tinkering and trying to figure out how to get that to work. And it actually turned out to be kind of a multivariate approach. We had to think about prompting. We had to think about how do we structure a good prompt just so that we get a good result, even without all the fancy stuff we want to do in the in-between, right? And it turns out, one of the key things we learned was that you need to start with a declaration of what your foreground object is, what your product is in the shot. And if you can get a really good description of that, then your prompt is already starting out in a really good grounded spot.

• 39:36Speaker 02

  Obviously, adding stylistic language like commercial product photography, you know, high quality, all those sort of like little tricks of early image models that will eventually like pass away. But, you know, we injected a few of those into the prompt as well. You start with that product description and then the next key line has to be some kind of grounding description of how it's been placed in the environment. Without that description, you don't get those shadows, you don't get those table reflections, even with all the cool additional sort of support for that functionality we've built in. And so you need that grounding. And then finally, you can describe the scene that you want in that background. It's a great description.

• 40:12Chris Benson?

  It really is. Um, I'm really enjoying this, but I wanted to ask a couple of questions to make sure. It sounds like when the way you were starting, when you're kind of talking about the product pixels and pulling those out, almost in my mind, I was almost thinking of it in an old fashioned way, like, like a Photoshop mask or something where you're masking out the product. And then you're trying to bring all the goodness of the contextual understanding of the models in. The thing that I think surprised me in there was, was kind of if you talk about that, that initial masking, I wasn't surprised when you talked about finding the description for the background and everything, but I was a little bit about the thing being masked, if you will, the, the product itself. How do you think about that? If you're, um, as you're going through the process and you're saying, I need that description,

• 40:55Chris Benson?

  like, could you describe that step a little bit? Cause I was, I'm trying to kind of really grok that one, but it sounds really interesting to me.

• 41:02Speaker 02

  I think it's probably helpful to work backward from really what we deliver to stable diffusion as a model to generate the output that we get from it in the end. And then kind of work back really, okay, well then how do we assemble all of that input to then get it in the mall, right?

• 41:15Chris Benson?

  Sure.

• 41:16Speaker 02

  So at the end of our pipeline, once we processed all the prompts you've put in and the image you, you uploaded of your original product, all that stuff, really what we're delivering to stable diffusion in the end is a masked depth map of your original product.

• 41:31Speaker 02

  Add a little bit of a bloom at the very bottom, where it might make connection with the scene, with the original scene around it.

• 41:38Daniel Whitenack?

  Could that be like a shadow when you say that?

• 41:40Speaker 02

  Kind of like, yeah, sort of like a little bit of that shadow. Maybe, you know, if it's, if it's a table reflection, you'll get a little bit of that table reflection. And what we found was that that little hack is just enough context. That, that little like gradient of additional depth info as you leave the product pixels is just the right amount of grounding information that stable diffusion and ControlNet need to be like, oh, there's a shadow there. Oh, and I see the angle of the table is this way. And oh, I see the, you know, the camera angle is kind of like this. And all of that together collectively gives stable diffusion the context that it needs to then paint a grounded scene around that product in high fidelity. And of course, we're generating a new product

• 42:25Speaker 02

  in that resulting image, but we do a composite in the end. And those pixels, because we're using depth ControlNet, adhere very closely to the original product pixels. So when we do the paste over in the end, you never see the sort of like hallucinated product pixels in the background.

• 42:40Daniel Whitenack?

  It's so interesting. I think what, one question that would be really interesting for our listeners and for me, from a selfish perspective is like, how does one, because I think a lot of people play with these models. They can pull down and, you know, figure out ControlNet reasonable enough. But then this like connection of this like little hack, as you described it, in some ways, after you have something like that, it like, it seems simple enough to describe like why that would work. And it's like a cool hack. But to get to it, it's like, how do you come up with that? I think is what is in a lot of people's mind. And some of it, for me, I know, like, a lot of times I banging my head against the wall one day and like I sleep on it.

• 43:24Daniel Whitenack?

  And in the shower in the morning, it's just like, whatever that, that idea comes. But I'm curious, from your perspective, from your team's perspective, how did that happen? And what sort of environment exists that would promote this sort of hacking of, because you're not retraining a whole model here. You're kind of using what, what is off the shelf, but using it in an extremely powerful way, but in a very creative way that is creative, not in the sense of training a new model, but creative in the sense of how you're using the existing model, which I think is really intriguing.

• 43:55Speaker 02

  This really was just kind of the perfect workshop product where we had just a bunch of like brilliant people who kind of understood these models enough, had played with them enough, knew and had seen enough of what they were capable of from different demos and other things to to have a real opinion about what was possible and kind of what wasn't. And when we started the journey and started building the machine, right? And trying to figure out what is this, how should this work? How do all the pieces fit together? We knew that there were hundreds of these little amazing AI machines that could be plugged into and turned into bigger machines that do even more powerful things. And it's just about figuring out like, what's the sequence? What are the pieces? What are the core problems? And it's just, how do you get to that iteration speed?

• 44:40Speaker 02

  Where you can try something? It's why I fell in with web dev, like in the early, early days of like web standards and 2.0. Like you could code something and see it immediately. Okay, that didn't work. Go back, code it again, see it immediately. Okay, no, that didn't work. Go back, code it immediately, see it again. And getting into that state, and that's really what the open source tool Comfy UI really unlocked for us was, you know, and GPUs still take a few seconds to deliver images. So it wasn't perfect, a rapid fire iteration, but way faster than trying to do it all remotely and trying to, you know, Comfy UI dramatically accelerated our ability to kind of like build this more complex machine because it was so easy to configure and reconfigure and try a thing and wire it a different way. And then that didn't work and wired a different way.

• 45:24Speaker 02

  You see the results, you're like, oh, wait, that's new, but different, but not what I want. But isn't that interesting? And then, oh, maybe I have a hunch about why that happened. And like you pull that back into something else and you, now you unlocked something, not because you had some like amazing insight, but just because you've tried enough stuff and you've seen enough weirdness. And then like, there was something there. That was weird. That shouldn't have happened and something surprised me and I want to understand it. And that's how it just like the, you know, it just unfolds that way. And eventually you're like, you start to connect all these little discoveries you make as you're like, why did that happen? Why did that happen? Why did that happen? And sooner or later you end up with something that works. It's kind of magic.

• 46:02Chris Benson?

  It is a kind of magic. As a Queen fan, that fit right in there as well.

• 46:08Chris Benson?

  I'm still almost stuck on that creative epiphany that you had a moment ago. That's really, I found that really interesting that you came up on that. As you're looking at this set of technologies evolving over the years ahead, as the organization is maturing with these technologies, and you have this amazing creative capability in your humans, in your organization, that can use these tools, where's all of this going? As we wind up this conversation, what, you know, how, how are you thinking about the future? What are you excited about that? What do you not have yet that you wish you had in your fingers right now? How's your thinking about that?

• 46:45Speaker 02

  I hate when people think about shopping, they don't always like jump to think about technology. But if you think about how technology has impacted commerce over the years, commerce and our culture around it and how it works is always inherently tied to like the wave of technology that we're experiencing. You know, whether you're talking about IBM cash register adding machines, or whether you're talking about mass media and the creation of kind of mega brands, or you're talking about the evolution to the internet and sort of the democratization of commerce and connecting with niche audiences. The culture around commerce always evolves around technology.

• 47:26Speaker 02

  It's why I'm so excited to be working at the intersection of these new technologies at a company like Shopify, because, you know, they are so directly related. And as we kind of think about the future of technology and sort of where this is all going, I get really excited about AI being an incredible driver of personalization in commerce. You know, when I go to some of my favorite stores, the person behind the desk like knows me, they recognize me, they remember what I bought last time. We have a conversation about it, like I can ask questions of the new line or the new products, or ask them to help me find stuff that I might really enjoy based on what they know I've bought in the past. And that's all an experience that I can get at an in-person store today, because like

• 48:10Speaker 02

  the person there knows me. I'm really excited about a future where our online commerce experiences become a little bit more like that. Where we visit an online store and it knows who we are and it helps us find the stuff that we'll be most interested in. And even, you know, really exciting things like being able to visualize myself in different clothes I might want to buy live in a browser. Like that kind of stuff is in the future, out ahead of us. And so I'm really excited about a future where AI helps bring these kinds of personalized, one-to-one customized shopping experiences to merchants and helps them bring that to their shoppers.

• 48:49Daniel Whitenack?

  That's awesome. Well, I'm, I'm definitely looking forward to seeing the, the things that your team comes up with and moving towards the future. And just really appreciate you taking time out of what must be an incredibly busy week leading up to Black Friday, Cyber Monday at Shopify. But yeah, thank you so much for the work you and your team are doing, Russ. Hope to, hope to have you on a future show to see some of those things you just mentioned become reality. Thanks so much for joining us.

• 49:17Speaker 02

  Awesome. Thanks, Chris. Thanks, Daniel. Really appreciate it.

• 49:28Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a long time listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.