Sidekick is an AI Shopify expert

Today, Chris explores Shopify Magic and other AI offerings with Mike Tamir, Distinguished ML Engineer and Head of Machine Learning, and Matt Colyer, Director of Product Management for Sidekick. They talk about how Shopify uses generative AI and LLMs to enhance their products, and they take a deeper dive into Sidekick, a first-of-its-kind, AI-enabled commerce assistant that understands a merchant’s business (products, orders, customers) and has been trained to know all about Shopify. Sponsors: Fly.io https://fly.io/ – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun https://fly.io/speedrun/ to get started in minutes. Timescale https://www.timescale.com – Purpose-built performance for AI Build RAG, search, and AI agents on the cloud and with PostgreSQL and purpose-built extensions for AI: pgvector, pgvectorscale, and pgai. Eight Sleep https://www.eightsleep.com/changelog – Up to $600 off Pod 4 Ultra — Go to eightsleep.com/changelog https://www.eightsleep.com/changelog and use the code CHANGELOG. You can try it for free for 30 days - but we’re confident you will not want to return it (we love ours). Once you experience AI-optimized sleep, you’ll wonder how you ever slept without it. Currently shipping to: United States, Canada, United Kingdom, Europe, and Australia. Featuring: Mike Tamir – LinkedIn https://www.linkedin.com/in/miketamirMatt Colyer – LinkedIn https://www.linkedin.com/in/mattcolyer, X https://x.com/mcolyerChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: Shopify https://shopify.comShopify’s Winter Edition ’25 http://shopify.com/editions/winter2025Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep, technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at fly.io. Launch your AI apps in five minutes or less. Learn how at fly.io.

• 0:35Speaker 01

  What's up, friends? I'm here with Kurt Mackey, co-founder and CEO of Fly. As you know, we love Fly. That is the home of changelog.com. But Kurt, I want to know how you explain Fly to developers. Do you tell them a story first? How do you do it?

• 0:48Speaker 02

  I kind of change how I explain it based on, almost like the generation of developer I'm talking to. So like, for me, I build and shipped apps on Heroku, which if you've never used Heroku is roughly like building and shipping an app on Vercel today. It's just it's 2024 instead of 2008 or whatever. And what frustrated me about doing that was I didn't, I got stuck. You can build and ship a Rails app with a Postgres on Heroku, the same way you can build and ship a Next.js app on Vercel. But as soon as you want to do something interesting, like as soon as you want to, at the time, I think one of the things I ran into is like, I wanted to add what used to be like, kind of the basis for Elasticsearch. I want to do full text search in my applications. You kind of hit this wall with something like Heroku where you can't really

• 1:29Speaker 02

  do that. I think lately we've seen it with like, people wanting to add LLMs kind of inference stuff to their applications on Vercel or Heroku or Cloudflare, whoever these days, they've, they've started like, releasing abstractions that sort of let you do this. But I can't just run the model I'd run locally on these black box platforms that are very specialized. For the people my age, it's always like, oh, Heroku was great, but I outgrew it. And one of the things that I felt like I should be able to do when I was using Heroku was like, run my app close to people in Tokyo, for users that were in Tokyo. And that was never possible. For modern generation devs, it's, it's a lot more Vercel based. It's a lot like, Vercel is great, right up until you hit one of their hard line boundaries, and then you're kind of stuck. There's the other one, we've had someone within the company, I can't remember the name of this game, but the

• 2:14Speaker 02

  tagline was like, five minutes to start forever to master. That's sort of how we're pitching Fly. It's like, you can get an app going in five minutes, but there's so much depth to the platform that you're never going to run out of things you can do with it.

• 2:25Speaker 01

  So unlike AWS, or Heroku, or Vercel, which are all great platforms, the cool thing we love here at Changelab most about Fly is that no matter what we want to do on the platform, we have primitives, we have abilities, and we as developers can charge our own mission on Fly. It is a no limits platform built for developers, and we think you should try it out. Go to fly.io to learn more, launch your app in five minutes. Too easy. Once again, fly.io.

• 3:16Chris Benson?

  Welcome to another edition of the Practical AI Podcast. This is Chris Benson. I am going solo today. Daniel's not able to join me, but we have two guests today, and I would like to introduce you to Mike Tamir, who is distinguished machine learning engineer and head of machine learning at Shopify, as well as his colleague, Mike Collier, who is the director of product management for Sidekick. Gentlemen, welcome to the show.

• 3:43Speaker 04

  Thanks, Chris.

• 3:43Chris Benson?

  Yeah, thanks for having us. Glad to have you on board. I know you guys are doing a lot of cool stuff in the AI space. And so, thank you both for joining to cover the different aspects of it. For those who may be joining, who probably have heard of Shopify, but may not be users, or may not be intimately familiar, can you guys talk a little bit, before we dive into all the good, the AI goodness, could you guys talk a little bit about what Shopify is as a company? And kind of how you see the space that you're in? What need are you fulfilling? Just some of the general understanding of your business before we get into the AI stuff.

• 4:18Speaker 04

  Yeah, I can jump in and then maybe Mike can add in, sprinkle in some bits.

• 4:22Chris Benson?

  Sure.

• 4:22Speaker 04

  Yeah, Shopify is one of those incredible companies that you might not be aware of, but you've probably used it, even if you aren't specifically aware of it. So our mission is to create the retail operating system behind your favorite brands. So you can think about it, it's like, there's the chain restaurants you go to, right? All around your hometown, but then there's that, your favorite coffee shop, that's run by like a local operator, right? And like, it's just, there's something better about that coffee, right? And so- Shopify's goal on the internet is to enable those kinds of operators to have a successful business out there. And so we power many brands online. Some of the ones that are out there that are more famous are like Drake, Patel, Jim Shark, Heinz, just to name a few. And so like some of the world's biggest brands are on there, but also some of those entrepreneurs

• 5:07Speaker 04

  that are in your local area are too.

• 5:10Chris Benson?

  Can you describe like what sets Shopify apart from other, you know, like other processing? I mean, it has a very distinctive brand, I know, but can you share a little bit about what, what makes it distinct from other things like credit card processors and, you know, cart processors? I know you guys kind of have your own distinctive way of doing things.

• 5:28Speaker 04

  Yeah, I mean, it's a full soup to nuts kind of solution. So like we do e-commerce. So like we can help you build your site. We can help you create merchandising around the products that you offer. We also offer payment solutions. So you don't have to set up a separate credit card processor, but if you have one, you can bring it too. I think that's actually one of the most powerful things about the Shopify platform is that it's got a very extensive developer ecosystem. And so many of our merchants install apps from our partners to do specific things. So if you've got a specific shipping provider, you can use their app for it. If you've got a specific email provider, you can use your app for it. For many of these categories, we also have solutions, but the ecosystem is definitely the richest part.

• 6:08Chris Benson?

  It sounds like you also do lots of different market segments from some of the large brands that you just talked about down to, I know, when I've, when I've come across Shopify in the past, it's been in the context of, of kind of smaller business and e-commerce and midsize business and things like that. So it sounds like you guys hit quite an array of different customer segments with technology solutions.

• 6:30Speaker 04

  Yeah, I mean, maybe, Mike, do you want to chime in? I feel like you work on more of them than I do these days.

• 6:35Speaker 05

  If you think about what, you know, all the things that Matt described, you know, establishing a website, payment processing, that's the infrastructure for, and something you could provide for a large established brand, all the way down to a smaller brand. More and more, especially in recent years, what Shopify has been focusing on is not just providing you with. The website, but also providing you with the, with kind of the tools for growth. And this is where AI and the focus that we've had on moving to machine learning and AI has really made itself apparent. So for instance, um, you know, we, we have our, our shop app where a new merchant who has no track record of sales or history can join that app.

• 7:17Speaker 05

  And when we search, if we understand the backend of machine learning and understanding, you know, how to do the retrieval and the ranking well, we can reveal a new fresh merchant to a customer right away.

• 7:30Chris Benson?

  Very cool.

• 7:30Speaker 04

  To get back on what Mike was saying, I think it's like a great example of how you've worked in the AI component of it. Um, but like one of the core values of Shopify is that we keep merchants on the cutting edge. So like we view our mission to like understand technology at a really deep level and always be out there scouring the best and then figure out how we can apply it to our merchants businesses, right? And so like Mike was talking about, how can we help our merchants grow, right? And so how can we apply these machine learning models or these AI techniques and really help these people grow their business?

• 8:00Chris Benson?

  With you guys having that technology infrastructure that you've been supporting all these businesses with and everything, at what point did you start thinking about the fact that there were these AI technologies that are, you know, been on the rise in recent years? What was the, the turning point for the company where you started seriously looking at AI as, as a supporting factor in the business model? You know, what made you say, I see an opportunity to go help our customer, get done the things that we're trying to do that we've been doing for years? What was that turning point? Kind of how did that come about? And how did you start thinking about AI in the business?

• 8:36Speaker 05

  I mean, knowing the timeline, you know, I think that the turning point was, was, you know, we might be the, the chicken and not the egg, so to speak. Um, in that, you know, Shopify. Maybe historically was not as invested in machine learning and AI. But by 2022, I'd become interested in that and has certainly massively redirected our forces. And the work that Matt and I joined and have focused on over the last several years.

• 9:04Chris Benson?

  And can you talk a little bit about that vision as you talk about, you know, you guys kind of coming into the company at that point? And carrying that forward? How do you think about your, the, the mission, if you will, that you're, you're doing? How do you, how do you contextualize it in terms of how you want to carry it forward? And how you're going to serve your customers with that effort?

• 9:22Speaker 04

  I'll give you my product manager-y answer to this.

• 9:25Chris Benson?

  That sounds fine.

• 9:26Speaker 04

  It was interesting hearing Mike's science answer to this. I think it's about finding out what's out there in the world. So to be quite honest, like in that time period, like if, if you all recall, it feels like, I don't know, a million years ago, but ChatGPT didn't exist like three years ago, right?

• 9:41Chris Benson?

  That's right.

• 9:41Speaker 04

  And we forget that that was a world, but it was. And so like, I think we got enamored, like the folks that work at Shopify with that technology, just as much as everybody else did. I definitely remember there was a peak ChatGPT moment where my mom was telling me about how she got it to write at a poem. And so I think the whole world was just captivated by the fact that we had computers that could, you know, write us stories. And so I think, I think that's kind of the culture of Shopify, right? Like we're all tinkerers and we like to build and we're all there getting messages from our moms about how they can write stories that they couldn't write before. And they're like, naturally you're like, hmm, maybe, maybe there's a way to apply this to commerce, right?

• 10:16Chris Benson?

  I'm curious, as you talk about that, I like, I like the mom story because that literally holds true with me. I have a mom, she's long since retired, but she, she was a technologist. And so, yeah, we have those moments where she's like, well, I'm going to go try that thing out and do that. I'm kind of curious, as these new technologies were coming about and you guys are coming into the company and kind of carrying it forward. There were a lot of choices that you guys had to make, you know, in terms of like, there's obviously, we talked about ChatGPT, there's open source. There's a whole bunch of different approaches to how you're going to support your customers with different technologies and different ways of, of addressing. What was your thinking both on the technical and on the product side in terms of how you might do that?

• 11:01Speaker 05

  I would describe our approach as as unembarrassed with how pragmatic we are on these issues, you know, whether it's open source or one of the commodity providers for L L Ms. We tend to gravitate to whatever works and and we do keep multiple threads of experiments with all of these different options, technological options, open for solving every problem. I think that that pans out with what we currently have in production and active is a nice array of of not just all of the foundation models that you might think, the commodity options that that are out there, but also being pretty aggressive with how we use the open source versions that are out there.

• 11:43Chris Benson?

  As we've talked about that, and you know, you bring up open source there and we've talked about, you know, productized offerings such as chat G P T through A P I and stuff. Going into this, Ren, before you got to the point now where there are a number of of things that I know we're about to talk about, how did you strategically, like from a strategic standpoint, how did you parse that? How did you say, we have this, you know, challenge in that we're a big, successful technology company, we're moving into the brave new world of A I and you had to kind of like figure out what do you want to do with foundation models? Are you hosting your own? Are you are you going to go to A P Is? How do you think about that? Like from and and not just from a technical standpoint, but from a

• 12:28Chris Benson?

  the perspective of serving your customers and stuff, like what are the what? How do you see the strengths and the weaknesses of different of different perspectives there? If you could share a little bit about your insight as you had to to analyze on behalf of your own business?

• 12:41Speaker 04

  Echoing what Mike said, it's like, I think it's it's early innings in this industry, right? And so like at the beginning of an industry, it's like, there's a lot of change and it happens very, very quickly. And so I think it's hard to like pick any one solution. Like, I think what people thought at the beginning of this, like a new wave of technology is like, they're like, maybe fine tuning was the answer, right? And it's like, as model and context windows will be small forever and it will be expensive and like. I think it's blown everybody's expectations out of the water at how much, how quickly, how things have gotten less expensive. Like, I don't think anybody could have predicted that. And so like, I think it's just such a world of abundance, both in like the operational costs, but also the other thing I would say that's unpredicted is that the number of

• 13:23Speaker 04

  solutions out there is just unparalleled. Like, I don't know, without the leaking of the llama waits, I forget back in like March of, I don't know, was it 23 or something? I don't know that we'd have the open source community that we have today, but like, since that kind of Cambrian explosion of open source models, it's been like crazy, like some of the innovation that's out there. And so like, I think, I would like to think that we had a master plan. We're like, oh yes, we saw exactly, we were going to use these commodity models and then move to the open source models. But like, the reality of it is always messier than the historical written version, I think. And so I think the short answer to all of that is that we try a bunch of stuff, right? And I think the people that went in this space are the people that try the most things the fastest. And so I'd say that's our overall strategy is we try a lot of different things. And then, like Mike said, like, we have a rigorous way of determining which is the best, but

• 14:08Speaker 04

  that frequently changes. Like, one approach that worked three months ago, like, I don't, I can't even count how many times on Sidekick, we've replaced the core underpinnings of it because we found a new approach that's better.

• 14:18Chris Benson?

  I'm not at all surprised by that, and I appreciate that answer because I think that's a challenge that everyone out there, a lot of folks listening to this are facing as well, is it is moving so fast and what was expensive yesterday is plunging in cost as new things are on the rise and such. And so I like your, your notion of, of experiment very fast, fail fast, I guess, in the process on your experiments so that you know what's working for you at least today until the next thing hits you tomorrow. Have there been any, any growing pains that they come to mind that you can share that have been kind of like, oh, you know, where you plant your, your palm on your, your forehead and say, I wish I could have seen that coming?

• 15:03Speaker 05

  So this is something that, that I think the entire community has, has been working out in real time. Like Matt said, you can try different things and see. Which strategy works the best, or which tactic works best, or what combination of those tactics works best? And change all the time, because we do know that new models are going to keep being released. Large commodity ones, smaller open source ones, large open source ones. And different, we're learning all the time in the research, what sorts of tactics with, whether it's fine tuning, or using long prompts, or combinations of domain adaptation. That's all going to be in flux, and we should believe that that's going to be in flux for a very long time. And so if you have a mandate of, hey, we're open to anything, and we'll use what works.

• 15:47Speaker 05

  We have to have a definition of what working looks like. And that means, and Matt's gonna laugh, because this is something that we've worked on quite a bit. You have to have your eval system dialed in. And that's the sort of work that, you know, we're moving into different formats for how you might eval, especially with unstructured text generation, for figuring out, this was a good answer, this was a bad answer, and being able to measure it in various ways. And there's all sorts of creative solutions, depending on the context. But making sure that we have a way of measuring that, versus saying, two anecdotes is enough for me to think that all these ones are white. That's an important part of the process, if we want to be pragmatic about our solutioning.

• 16:29Speaker 04

  Yeah, we have like a, I guess, a story on the team. We talk about it being the dark forest. And I think we're all grasping, we wish we had our GPS-enabled phones of pinpointing us on the map of how you get out of the dark forest. But I think what we've had to settle for with eval is like a compass, if you will, like the metaphor here. Is that it's like, everybody wishes we knew absolutely, like, this is how good the system is, right? But like, what we've kind of settled for instead, that's more practical, is like, well, we know A is better than B, right? And so if you just make enough decisions in aggregate, where A is always better than B, you eventually find yourself out of the forest, right? But we wish we knew how long it would take. The question's always like, well, when are you gonna find your way out? Or it's like, well, we don't know, we're just gonna keep making the best next decision we can.

• 17:22Speaker 01

  Okay, friends. I'm the good friend of mine, Aftar Swethan from Timescale. They are positioning Postgres for everything from IOT, sensors, AI, dev tools, crypto, and finance apps. So Aftar, help me understand why Timescale feels Postgres is most well-positioned to be the database for AI applications.

• 17:41Speaker 06

  It's the most popular database according to the Stack Overflow Developer Survey. And Postgres, one of the distinguishing characteristics is that it's extensible. And so you can extend it for use cases beyond just relational and transactional data for use cases like time series and analytics. That's kind of where Timescale, the company, started, as well as now more recently, vector search and vector storage, which are super impactful for applications like RAG, recommendation systems, and even AI agents, which we're seeing more and more of those things today.

• 18:10Speaker 01

  Yeah, Postgres is super powerful. It's well-loved by developers. I feel like more devs, because they know it, it can enable more developers to become AI developers, AI engineers, and build AI apps.

• 18:23Speaker 06

  From our side, we think Postgres is really the no-brainer choice. You don't have to manage a different database. You don't have to deal with data synchronization and data isolation because you have like three different systems and three different sources of truth. And one area where we've done work in is around the performance and scalability. So we've built an extension called PG Vector Scale that enhances the performance and scalability of Postgres so that you can use it with confidence for large-scale AI applications like RAG and agents and search. And then also, another area is, coming back to something that you said, enabling more and more developers to make the jump into building AI applications and become AI engineers using the expertise that they already have. And so that's where we built the PG AI extension that brings LLMs to Postgres to enable things

• 19:07Speaker 06

  like LLM reasoning on your Postgres data as well as embedding creation. And for all those reasons, I think when you're building an AI application, you don't have to use something new. You can just use Postgres.

• 19:18Speaker 01

  Well, friends, learn how Timescale is making Postgres powerful. Over 3 million Timescale databases power IoT, sensors, AI, DevTools, crypto and finance applications, and they do it all on Postgres. Timescale uses Postgres for everything, and now you can too. Learn more at timescale.com. Again, timescale.com.

• 19:58Chris Benson?

  So guys, I love the fact that your notion of eval and finding your way back out of the forest. You, with that in mind, you have a lot of different products that you work with. And as you're bringing these new technologies in, and you're doing these evals, and you're trying to find your way out of the black forest through that, and managing across multiple areas there, how does that look for you? How do you unify different products so that you can effectively serve your customers with these technologies? And how do you make all those Legos come together in a usable way?

• 20:39Speaker 05

  There are certain genres of problems, and it seems like one or more strategies for eval will be appropriate for each genre. So let me give you an example. With search, and evaluating search quality, there's been some good research that shows that LLMs tend to be better at rank ordering or labeling relevant, not relevant of a product to a query at a certain resolution. In fact, there's research that shows that the LLMs are better than a human at doing this. And you might ask yourself, why is that? Why, why, you know, you're a human, Chris. You, you searched for a white flower dress. And you're going to click on one of those, right?

• 21:24Speaker 05

  One of those is going to be the right answer for that query for you. And then I might search for a white flower dress. And you searched for one because you wanted a dress with a white flower. And I didn't want a dress with a white flower. I wanted a white dress with colorful flowers. And both of these are completely legitimate answers to that query. And what we're seeing here is it's actually just a sampling problem. If you think about it, there's a distribution of appropriate products matching any query. And so every time we ask a human, we're getting a sample from that distribution. But if that distribution is very flat, then we're going to sample across a wide variety of different answers. And so what we've done with L L Ms in this is, you know, you can think about this as an analogy.

• 22:08Speaker 05

  In the morning, I like fill in the blank, right? And there's a lot of good answers to that. I like to exercise. I like coffee. I like breakfast. I like orange juice, whatever it is, right? There's a lot of ways of completing that sentence and they're all legitimate. It's just that it's a very flat distribution. And with language, what we do is we just overwhelm with sample after sample after sample after sample. So that we can fill in that whole distribution. In the query product genre, it takes too much time and too much cost to fill in that distribution at in mass that way. Right until you, until you get into implicit feedback. So you have to find another solution.

• 22:48Speaker 05

  And this is one of the reasons why when you're using an L L M and replacing a typical terker from filling in those answers, you get better results. Now, something important is something that Matt and I have seen in other contexts. You can't do this ungrounded. You can't just have the robots grade the robots and then, you know, hope for the best. You have to have different expert supervision to ground those answers. Whether it's in a search context, a personalization context, in a more of a chat context, like the sidekick product. You have to have that grounding. And once you inject that kind of like course correction. Then you can kind of get the best of both worlds.

• 23:29Chris Benson?

  Could you talk a little bit about the the different products? You talked about query, you talked about personalization. But how are there any others there that you say are kind of very prominent in your world that you're thinking about applying L L Ms or other A I technologies to?

• 23:45Speaker 04

  So we've got several products that are A I enabled or magic enabled Shopify. So sidekick is kind of the main one, which we'll talk more about. But to give you a general idea, it's a tool that helps merchants find their way around Shopify, but also answer questions about their business. So you can think of it as like the co-founder they wish they had. That's available 24 7 and isn't judgmental. So that's kind of like the sidekick idea. And then we've got a variety of other ones as well. So we have a lot of imagery. So it turns out shopping like people want to see what the thing is before they purchase it. Not terribly surprising. And one of the things that merchants often want to be able to do before they've scaled up to a whole team that, you know, has a studio and a photographer and the rest of it. Is they want to enhance the pictures that they do have, right?

• 24:29Speaker 04

  So like at the scale that they're at, this is where technology, again, bringing back like the best from the frontier and making it accessible to all of our merchants is exciting. So like, there is technology that's out there now that you can essentially describe what you want to do to an image. You're like, hey, my background's a little bit messy. Can you replace it with a studio background instead? Because like, we all know that the nice white studio background that looks like the objects floating in space, right? And it's like, you can do that in real life. It's just really hard and expensive and you have to know what you're doing. And there are very few people who know how to do that well. And so it turns out we've created models that can do it fairly well as well. And so bringing that technology back, that's one of the products we do offer integrated into Shopify today is background generation.

• 25:10Speaker 04

  So merchants can import an image that they already have, replace the background with something that's more on brand. Like say they want to set their coffee to the background of like a jungle, right? They can like place it on a table in front of a jungle if that's what they would prefer. Or they could do it into the to the void of the white space. So lots of exciting opportunities with that. Another area that we've been investing in is that we have a product called inbox, which allows our merchants to talk with the buyers that they have on their site. And if a buyer has a question like, hey, what's your return policy? Or like, where's my order at? They can interact with the merchants through inbox. And so one of the things that we're offering today is that we look at all the merchants' policies and all the other things that they've given to us. And then can help formulate answers for those common questions, right? It's like, well, what's your return policy? It's like, well,

• 25:54Speaker 04

  We're pretty sure that this is the answer. And then we can suggest that to a merchant who then says, yep, that's right. Or if that's not right, they can adjust it to be correct and then send it. And so merchants love that because it saves them time for answering a lot of those repetitive questions. And then those ones that are a little bit harder, they can write them themselves, just as they would before. And then the last one that I'm thinking about is like, or again, going back to that product and merchandising kind of task is that oftentimes merchants are uploading a lot of these at the same time. Right. And so like, they don't always capture all the metadata in the first go. And so this is again, where we created models that actually can help with that. So like, if you upload an image of a white flower dress, we have a model that can actually understand what that picture is and suggest that, hey, maybe this is a white flower dress.

• 26:37Speaker 04

  And this should be categorized under dresses and like under cotton, right? And like, maybe it can suggest, oh, it turns out if you upload multiple different colors, it's like, well, maybe you want to create product variance. And so that's some of the other technology that we're kind of working on today is like using the data that we have from merchants, enabling them to more expressively describe their products through our sites.

• 26:56Chris Benson?

  With Inbox, is Inbox part of Sidekick or is it adjacent or parallel to Sidekick and the way, the way that you see it?

• 27:03Speaker 04

  No, it's a completely separate offering. So merchants have to choose to install, again, going back to kind of the platform thing we talked about earlier. They choose to install the Inbox app, which Shopify builds. And then within the Inbox app, you can choose to use this behavior or not.

• 27:16Chris Benson?

  You mentioned also the ecosystem a little while ago, and I'm curious how, as you have created these new AI-enabled products, how has the ecosystem been plugging into that? Is that something that's possible? What kind of interactions do they have together?

• 27:32Speaker 04

  So today we have a very extensive API through GraphQL that we expose, like that data we just talked about, right? The categorization. So whatever the merchant decides, like we make a recommendation, they say, yes, that's correct. This dress is actually a dress. Once they save that change, that information is then available through that product description API.

• 27:50Chris Benson?

  I see.

• 27:50Speaker 05

  Matt covered a nice breadth of, you know, generative text, generative images, product understanding, all of which are kind of adjacent to, or, you know, image generation is sort of a different algorithm under, under the hood. There's also a direction where we can think about something that I've, it's been a gift that keeps on giving for, for all of my career is that the sorts of machine learning techniques that work with text. Often also work with commerce. And so this goes back to old fashioned matrix factorization for recommender engines was also useful for understanding text. RNNs useful for looking at sequences or sentences, you know, before transformers took over. Good for text also.

• 28:35Chris Benson?

  Boy, you're you're taking me way back. We we did many whole shows on RNNs and that seems like the Stone Age now.

• 28:40Speaker 05

  Yeah, the Stone Age of the 20 teens, right?

• 28:43Chris Benson?

  That's right.

• 28:44Speaker 05

  That also, you know, a lot of the techniques, you know, it was you could always just peek at what you're doing in language and come up with a cool idea for e-commerce and vice versa. And so this has not stopped. I mean, transformers have kind of taken over everything, but you know, there are not quite transformer architectures, but heavy attention method transformer like architectures that can look at sequences of behaviors of merchants, of buyers, the people that are shopping with our merchants. Those are sequences too, and can be processed in an analogous way in order to understand what is next step on the journey for a merchant and how can we help them get to that journey? What are the likely ways we can simulate that?

• 29:28Speaker 05

  And that's been sort of one of our frontier, cutting edge areas that we've been applying in.

• 29:35Speaker 04

  Actually, Mike's answer reminded me. I want to add one more product that always slips in mind here. And it also blends with the ecosystem thing that we talked about. So as I'm sure you've talked with other folks on the show about one of the exciting parts about LLMs is the ability to write code. And we talked about the GraphQL API. And so one of the other exciting applications that we've done is for our developer ecosystem is enhancing our developer docs in the way that like we now have an integrated tool that assists developers in writing code. You like described like, hey, I'm looking to find the product category. Can you write me the query to do it? And it will literally write you the GraphQL query. You can copy and paste that, put your right in your application. And so I think we're still in the early days of figuring out how like it's it's such a

• 30:18Speaker 04

  dramatic shift for engineering to like figure out how we apply these LLMs. And it's exciting to see these new applications to like existing documentation sites and just unlock the power and making it more straightforward to develop apps.

• 30:48Speaker 01

  What's up friends? I love my 8Sleep. Check them out, 8sleep.com. I've never slept better. And you know, I love biohacking. I love sleep science. And this is all about sleep science, mixed with AI to keep you at your best while you sleep. This technology is pushing the boundaries of what's possible in our bedrooms. Let me tell you about 8Sleep and their cutting-edge Pod4Ultra. So what exactly is the Pod? Imagine a high-tech mattress cover that you can easily add to any bed. But this isn't just any cover. It's packed with sensors, heating and cooling elements, and it's all controlled by sophisticated AI algorithms.

• 31:30Speaker 01

  It's like having a sleep lab, a smart thermostat, and a personal sleep coach, all rolled into one single device. And the Pod uses a network of sensors to track a wide array of biometrics while you sleep. It tracks sleep stages, heart rate variability, respiratory rate, temperature, and more. And the really cool part is this, it does all this without you having to wear any devices. The accuracy of this thing rivals what you would get in a professional sleep lab. Now, let me tell you about my personal favorite thing, autopilot recap. Every day, my 8Sleep tells me what my autopilot did for me to help me sleep better at night. Here's what it said last night. "Last night, autopilot made adjustments to boost your REM sleep by 62%."

• 32:15Speaker 01

  Wow, 62%. That means that it updated and changed my temperature to cool to warm, and helped me fine-tune exactly where I wanted to be with precision temperature control to get to that maximum REM sleep. And sleep is the most important function we do every single day. As you can probably tell, I'm a massive fan of my 8Sleep, and I think you should get one. So, go to 8sleep.com/changelog. And right now, they have an awesome deal for Black Friday, going from November 11th through December 14th. The discount code CHANGELOG will get you Up to $600 off the Puff4Ultra when you bundle it. Again, the code to use is CHANGELOG, and that's from November 11th through December 14th.

• 33:00Speaker 01

  Once again, that's eightsleep.com/CHANGelog. I know you'll love it. I sleep on this thing every night, and I absolutely love it. It's a game changer, and it's going to change your game. Once again, eightsleep.com/CHANGelog.

• 33:28Chris Benson?

  Going back to something that you had mentioned earlier in the conversation, you had talked about magic and magic-enabled and stuff. Could you tell me a little bit about that? I may have misunderstood. Is that a product or a supporting technology that you guys are using?

• 33:43Speaker 04

  I mean, it's kind of the way we refer to things. We should have done a better job of explaining it. So all the things we just talked about, we would consider part of the magic brand at Shopify. So it's like the product taxonomy stuff, the text generation, the sidekick, the image generation in the background. So like, those are all magic features that Shopify offers.

• 34:03Chris Benson?

  Gotcha. So it's kind of the AI-enabling brand that's around all these things.

• 34:09Speaker 04

  Yep.

• 34:09Chris Benson?

  So as you, you know, we've kind of talked a little bit about Sidekick, and we've gone through. There was another thing I always wanted to ask you about, and that was how your current array of kind of AI-enabled capabilities that we've been talking about, how are you thinking about that going forward? Where are you looking at? How are you, you know, are you going to add any more in there that are announcable yet? Or maybe at least allutable to? How are you thinking about kind of where you're at today versus kind of some of the things you might be doing in, you know, in the fairly near future and stuff? And we'll get into farther future a little bit later, but.

• 34:46Speaker 04

  I can't answer today, unfortunately, other announcements that are coming, but I think we could talk about generalities of like what's interesting, right?

• 34:52Chris Benson?

  Fair enough.

• 34:52Speaker 04

  That Mike talked about of like, you know, applying old techniques in new ways around commerce-specific things, like predictions and customization around that are interesting. I'd say like, me personally, I think the thing that I get excited about is like the other modalities that are out there. So going back to that reference from earlier, like ChatGPT was cool. I feel like I had a second bout of ChatGPT when the voice mode came out. I don't know if you've played with it, Chris, but it's like absolutely incredible.

• 35:21Chris Benson?

  During the typical day, I have an ongoing, I probably, this is really terrible that I would say this, but I probably talk to ChatGPT more than I talk to my wife.

• 35:32Speaker 04

  Well, keep that between us.

• 35:34Chris Benson?

  Thank goodness she doesn't want to hear me anymore than she already does. So she won't hear this on the show. So, but yes, I have an ongoing conversation about a plethora of topics. So, which beckons back to the fact that this is moving so fast. And you guys are, as you guys are having to kind of match your customer needs with products that support, with the technologies that are driving that forward. What are some of the things that you're thinking about now for maybe as you go forward into the future? And more specifically, how are you thinking about handling the risks associated with changing technology? Right now, we've talked a little bit about constant experimentation and everything,

• 36:14Chris Benson?

  but there's also a point where you kind of have to make investments in different directions and trade offs and stuff like that. Other than the experimentation of that to support as this increasing line of capabilities that you guys are offering, how are you thinking about that risk directions and stuff? Do you think that commercial product offerings, for instance, one topic that comes up all the time on the show? Do you think open source is going to overcome that and kind of take over? Since things are slowing a little bit on the frontier models in terms of the gains they're making? And open source seems to be catching up faster. How are you guys thinking about problems like that as you're dealing with these business issues in your company?

• 36:51Speaker 04

  I think it will likely be hybrid. So like, I know we talked to kind of like our strategy is like everything all the time. So I can't imagine a world where the commercial offerings completely take over. And I really, I don't know if I could imagine a world where open source entirely takes over either. And I think that's probably a good thing for the world. Like, I think that's what drives the innovation, right? It's like a competition between the two. And there are some things that one is good at and the other is not. So like I can't imagine worlds where we aren't using both at Shopify.

• 37:18Chris Benson?

  Can you talk a little bit about kind of how you see the strengths and weaknesses, recognizing it may change tomorrow, given how fast things are moving. But like, when you look at that kind of what we might go with a commercial offering like ChatGPT or one of the other several biggest competitors versus the open source and probably the foundation infrastructure that you guys will have. How do you guys know where to go? Like, how do you know to go to ChatGPT as an API versus using a foundation model you're storing in your infrastructure?

• 37:46Speaker 04

  Well, I think it goes back to Mike's favorite point from earlier of evals. We've got to have our compass because without the compass, we're lost. So that's how we answer which one. But I think the other part of like, you're asking like, which one is good at what at this point. And so like, I think in general, the way to frame it is like, open source, the power is in the control. It's like, you're guaranteed to run this exact model with this exact set of training data with this exact outcome. So it's very predictable. You have way more control over the training process and like the post-training process and like, you just get a lot more knobs, right? But also, with great power comes great responsibility, right? Like, there's a cost to operating all those knobs and knowing what the correct values are for that. So for problems that you have pretty fully defined and you know exactly how you want to do it, it's like, open source is great, right? The

• 38:31Speaker 04

  commercial models, fewer knobs. But the thing that's great about that is like, you know, the defaults out of the box usually work pretty well, right? And so like, I think if you're looking at things that are early on in prototyping, like the commercial models work great, right? They can get you from zero to one real quick. And then when you get to that one, you realize they're like, oh, well, I want to get to 2.0. And like, sometimes it necessitates a need to that shift to an open source model to get that extra control out of it.

• 38:55Speaker 05

  There's kind of this question of what size of a problem are you trying to solve? If you're trying to solve a, you know, a central, do everything, you know, the founder that you would, the co-founder that you wish you had model, we're going to need to pull out all the guns, right? And really, put everything we can into making this, leveraging as much power as we, as we can, right? And the question is just what is the most powerful for this task at this time? There's another side of things where maybe we're trying to solve problems that aren't supersized problems, but they're more manageable problems, right? Or maybe we need to do it at scale. And, you know, of course, the commercial models are getting faster and faster and cheaper and cheaper.

• 39:37Speaker 05

  But when you need to do something at scale, it might be worthwhile to distill a model from some patterns and then run it at scale. We have billions of products, if you look over our entire history. Doing that at scale for the product that Matt described, where we understand all the different attributes and the taxonomy and we normalize the description of those products, that's a true engineering feat that we need to work on. And that may not be a great idea to send that to GPT-01, right?

• 40:08Chris Benson?

  Absolutely. One of the question, I'm guessing, Mike, this is coming to you. The last two, three years, we've been so focused on LLMs and generative AI capabilities. And I know in general, the industry is starting to kind of also kind of pull back and look at some of the other things, things we used to talk about, other technologies in the AI space we talked about a lot. It seems that some industries, things like reinforcement learning and CNNs and things like that, depending on the industry, I think in my own experiences, I've talked to different people, some find utility in these other architectures with other purposes and some don't. How are you guys? Are you really primarily focused on

• 40:49Chris Benson?

  LLMs and generative? Or do you have use cases where some of the other technologies that we haven't talked about as much lately, but are still very much out there in industry? Are they coming into play for you guys?

• 41:00Speaker 05

  Yeah, so this may be where we appeal to technology. I need one more layer, right? At a base level, any neural net is a universal approximator, right? And so if we have enough data, there is a big enough neural net that will solve, just an MLP, right?

• 41:15Chris Benson?

  Sure.

• 41:16Speaker 05

  A fully connected neural network. And so the way I tend to think about it, whether it's a CNN or a heavy attention model or an RNN, whatever it is, all that's really in the business of doing is making it so that, even though there is a number of neurons, it might be way too many neurons, right? And we might need way too much data in order to do that. And all of these are really just tactics for reducing the amount of data that we need in order to approximate the patterns that we want. Right now, for sure, heavy attention models, whether it's traditional transformers or evolutions, like you might see of the original 2017, attention is all you need transformer architecture to what you see in Llama, these are kind of like tweaks and they're still very

• 42:01Speaker 05

  Multi-head attention focus. There are other techniques like the one I described for e-commerce that are making substantial changes, like removing the softmax out of multi-head attention, which is a, you know, sort of like having the sigmoid as our activation function 10 years ago. It was just a mistake and a sociological mistake at that, right? So seeing small changes like that, that maybe move us out of transformer architectures, I think that that we're definitely in a, in an era where that makes sense. There's also kind of combinations of things, right? So you mentioned reinforcement learning, there's GNN architectures, and these are actually compatible with vision transformers for planning and reinforcement learning,

• 42:41Speaker 05

  you know, using transformers for your aggregation functions for a graph neural network. And so it's not an either or, it's that now we have another tool for either, in the former case, modeling the world so that we can do a good job at our Q learning and our, at our, in our policy learning, or do a good job in capturing the right information when we have a graph structure of how we've organized the different kinds of nodes, the different kinds of, in our case, merchants and products and buyers. That being said, you know, there's another way of taking your question, which is, look at when are transformers going to be done and are they gone already?

• 43:18Chris Benson?

  That was on my mind as well, actually, because we've been, everyone's talking about, okay, what does the post-transformer world look like? So, yes.

• 43:26Speaker 05

  I think I can say this, and I say this to my students pretty religiously, I'm going to make this claim with full understanding that you should never make a prediction that will be falsified in your lifetime. I'm pretty sure transformers is not the last architecture out there. It seemed for a decade that CNN was almost synonymous with vision since 2012, right? And now it's not. And if you would have asked me in the late 20-teens if I thought it was, I'd probably say the same thing. It's a heavy thing, you know, it's a heavy bet. CNN seemed to be the top of the hill and they seem to do such a good job with image classification.

• 44:08Speaker 05

  It's hard to imagine what will replace that, but probably it's not the last chapter of the story. And I think that you could probably say the same thing for transformers.

• 44:16Chris Benson?

  So with the, a little bit ironically, and you've sort of kind of covered this territory a little bit with that last answer, Mike. From each of you, we usually finish the show really wanting to get perspectives from, from our guests on kind of what the future looks like. And with each of you addressing kind of different areas, you probably have somewhat different answers based on your focus and stuff. And, and Mike, recognizing that you, you've already kind of touched a little bit on the future, but I'm actually, despite your, your comment about not making predictions that might prove falsified in your lifetime, I'm going to ask you both to kind of do that. If you're looking out, and I'll let you kind of decide on what timeframe works for you.

• 44:59Chris Benson?

  But maybe, you know, beyond the short term, waxing poetic a little possibly, and trying to say, you know, what would you, what do you think you're going to see? What, what do you want to see? And how might your various jobs and how your company serves customers, you know, how do you see this fast moving, there are twists and turns all along the way that catch us all by surprise? How do you see that playing out from each of you? Matt, if you could, if you could lead off and then Mike, I'll come back to you for that.

• 45:28Speaker 04

  I think what's most exciting about this, I mean, when I grew up, I remember when we first got the internet and like, it was like the first ISP out there and there was like a dial-up modem and there was like a BBS. Like it was just, that was like the first wave of technology to me. And like, that's how I got into this field. And then I feel like the mobile revolution caught me by surprise. Like, I think at the moment, like I knew when the first iPhone came out, I was like, I need to have one of those. But what I didn't expect was like how much the world would change after that. And it feels like this time around, I wasn't a big believer in web three. I was like, what is this web three business? But like, I feel like this is again, that same kind of shift. So I'm just going to ignore web three. I think this is the real web three. It's AI. And so like, how does this play out, right? Like, I don't think what's different this time is that like,

• 46:11Speaker 04

  for the last, I don't know, 70 years that we've had computers, like, we as humans have had to conform to like how computers work. Like, at first, we wrote assembly code. At first, we wrote like, literally bits. Then we wrote assembly code. Then we're like, well, maybe we should have languages. And it's like, okay, so we're slowly crawling there. And then the next revolution is like, oh, we should have point and click. And so these boxes. And so now we have a world where everybody spends, you know, eight hours a day clicking on little colored boxes and then typing in to other colored boxes, like, characters on a keyboard. And I think what's fascinating to me is that we've become, we've shaped who we are to conform to how computers work today. But I think this point in time, and I'm sorry, I'm giving it 10 years out from now.

• 46:53Chris Benson?

  No, it's fine.

• 46:53Speaker 04

  All that's gonna change. Like, I think that whole, like, all these browsers that we click buttons on to like, set settings, like, all that is gonna go away. I think it's gonna be that we interact with, like, an agent or some amorphous entity that it's like, and instead of, you know, listing all the steps of like, first, search for this, then click on this link, then do this thing. It's like, be like, I would like to buy a box of toothpaste. And it's like, the agent's like, great, do you want to buy one for, that ships tomorrow? Or do you want one that's like, cheaper but ships next week? And you're like, the cheaper one. And then it's like, done. Right? You didn't fill out a credit card form, you didn't click through 10 sites, you didn't do any of that. So like, I'm gonna put my bet on that. It's like, I just think the web will change again. Like, I think we've gotten so used to SaaS and all these models. And like, I'm just, I don't know how long it's gonna take. That's why I'm like,

• 47:38Speaker 04

  10 years out, I don't know. It might be three years out, but it might also be 20. But I don't think we're gonna be typing in boxes in 20 years from now.

• 47:45Chris Benson?

  Good answer. Mike, back to you.

• 47:47Speaker 05

  Yeah, I think I'm gonna flank this from both directions. So I spent a little bit of time in the self-driving space years ago. And this was during the, I think, the maximum hype period for self-driving, where, you know, everyone I talked to said, oh, well, people won't even need to drive in five years. And this was more than five years ago. And on the one hand, just across the bay, Waymo is giving rides to people, right? Not at scale yet, but it is, right? For sure, you know, a lot of people said my son would never need to get a license and he's about to get his driver's permit, right? So, so just to draw an analogy, I think it's reasonable to expect what Matt is expecting

• 48:32Speaker 05

  of having kind of like a self-driving assistant that can do that. I have all the respect for Matt in the world because he was careful about how he's gonna do the time, not five years or whatever. There's probably going to be a little bit of, of difficulty smoothing down the edges for that. And luckily, you know, a crash with a, with a self-driving assistant is far less dangerous than a self-driving car. So it could be that we Indeed. We, we get imperfect models there.

• 48:59Speaker 04

  I'll take an extra five boxes of toothpaste. That seems like

• 49:01Speaker 05

  Right. So that's, that's one side of, of flanking it is, um, you know, we very well will, right? There will be a self-driving moment for, for these assistants and, um, how long out is, is completely on the same page with Matt, that that's, you know, we're gonna hit some bumps before we actually get there. One thing that I, I feel very confident of is that we are going to change the way we organize and access and utilize information. This is going to be a forcing function that we really haven't seen since early search days for the internet, which is also a way of just completely transforming how we organize and access information.

• 49:42Speaker 05

  And, uh, you know, a lot of people you, you will talk to will already say that they go to their favorite L L M first before they go to a search experience. Um, and there are also a whole host of product and, and interface and questions like that about what's gonna be the best way of doing this. But, you know, there's also, um, um, once again, piggybacking on something that, that Matt said, you know, it is incredibly significant that we are now speaking in the same language, quite literally, when we want to access and refine the information we're looking for. And that's something that's really never happened before. Well said.

• 50:23Chris Benson?

  Well, gentlemen, thank you very much for coming on the show. It was really interesting. I learned a lot and thanks for sharing your perspectives going forward. I hope you guys will will come back as things evolve and you have more things that you want to share with the audience. Thanks for coming on. I'd be happy to. Thanks, Chris. Thanks, Chris.

• 50:49Speaker 00

  All right, that is our show for this week. If you haven't checked out our Change Log newsletter, head to changelog.com/news. There you'll find 29 reasons, yes, 29 reasons why you should subscribe. I'll tell you reason number 17, you might actually start looking forward to Mondays.

• 51:10Speaker 07

  Sounds like somebody's got a case of the Mondays.

• 51:13Speaker 00

  28 more reasons are waiting for you at changelog.com/news. Thanks again to our partners at fly.io, to Breakmaster Cylinder for the Beats, and to you for listening. That is all for now, but we'll talk to you again next time.