The Copilot for Ecommerce with Shopify VP of Core Product Glen Coates

Building an ecommerce business is hard – it requires merchants to have a wealth of skills: technical, logistics, marketing, pricing, vendor management, finance and analytics. That’s why Shopify is releasing new AI features that help merchants tackle things like product descriptions, marketing suggestions and search. Today on No Priors, Glen Coates, the VP of core product at Shopify (and former founder of b2b wholesale platform Handshake), joins Sarah and Elad. They talk about the releases from Shopify Editions, why they are deploying “copilot” rather than “autopilot,” AI innovation-at-scale, how to change the basement of a house while people are living in it, and building a leadership team of entrepreneurs. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @glencoates Shopify Editions | AI Section of Shopify Editions Show Notes: (0:00) Background (2:22) Calling a “Code Red” at Shopify (4:04) Integrating acquisitions, entrepreneurial leaders (12:15) AI adoption (15:51) Deciding when to ship AI products, evaluations (17:33) Shopify’s risk orientation (18:50) Changing the core Shopify data model, enabling AI features (26:05) What’s missing from LLMs for merchants (28:47) Most interesting AI developments in the industry (33:22) What users want from LLMs and search (38:20) No Priors social

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Hi listeners and welcome to another episode of No Priors. Today, we're joined by Glenn Coats, the VP of Product at Shopify, where he leads the core developer platform, including all of their AI products that we'll get into today. Before he was a leader at Shopify, Glenn was the founder of Handshake, a B2B e-commerce platform that was acquired by Shopify in 2019. We're excited to talk about how AI is changing e-commerce and entrepreneurship, as well as innovation at scale and leading at Shopify. Welcome Glenn.

• 0:33Speaker 01

  Thanks for having me.

• 0:34Sarah Guo?

  So lots of fun stuff today. Let's definitely cover your personal story quickly, since you're also a former founder, one of several at Shopify now. Can you give us some of your background?

• 0:42Speaker 01

  I was a comp site grad. I spent the early part of my career in video games. And then I had a weird left step in my career, where I moved to San Diego in 2008 to run the US operations of an eco-friendly shopping bag company. You know, like you take your own bags to the store, that whole thing. Anyway, this company did a whole bunch of wholesale. They sold a lot of bags through stores. And through that, I ended up getting to the idea for Handshake. And Handshake was initially a very like, sales rep-focused B2B e-commerce product, but then became both for sales reps and for customers to buy online. So Handshake eventually became basically a B2B-only version of Shopify, like a wholesale-only version of Shopify.

• 1:27Speaker 01

  So it's not hard to join the dots from there to, after, you know, building Handshake for better part of nine years here in New York, the opportunity came up to join forces with Shopify. And then I've been at Shopify for almost, it'll be five years in May. I started out very much focused on B2B and wholesale, which was obviously the point of the acquisition. I spent about a year doing that. I then moved on to focus, I ran a code red on checkout in 2020, which was the first year of the pandemic. And then since the end of that, I've been leading the kind of core product group at Shopify, which is, you could think of it as all of the built-ins of Shopify.

• 2:11Speaker 01

  The online store, the checkout, the back office, the developer platform, the app store. And yeah, that's what I do at Shopify.

• 2:21Sarah Guo?

  Lots of, lots of good stuff. What's a code red look like?

• 2:24Speaker 01

  A code red is when Toby sends an email to the entire company saying this thing is the number one priority and he means it.

• 2:33Speaker 01

  From an actual operational perspective, what that really means is if the team that's working on this code red asks you to help, please drop what you're doing and help. This is the number one priority. So that's how they work, but usually a code red is a symptom of some other much more systemic problem that's gotten you to that point. At least in my case, checkout code red in 2020 was, hey, the checkout is failing in, I don't know, three, four, five different ways and the checkout is a pretty important part of Shopify, obviously. So let's go fix that. So we, me and, you know, somewhere between, I think at its peak, it was probably two, 300 people working on various parts of the problem. So we all like scrambled for

• 3:18Speaker 01

  a year and like did what it took to fix those issues. But then at the end of that year, Toby took a step back and said, okay, well, why did that happen? Like, how did we get to the point where those problems even happened in the first place? And then that led to some of the reorganization of the company around less, like there used to be 12 to 15 of these kind of fairly small fractured business units. And now there is actually only like three or four, which is actually truer to what the product is. But of course, each of those units is bigger than the ones that were before. So it actually requires you to lead in a slightly different way, what, once you start grouping that many, much of the product and the people together.

• 4:04Speaker 02

  I guess like two related questions from your background. I guess number one is since you were acquired into Shopify, one of the things I've been very impressed with by the company is a degree to which it's been able to retain and grow the roles of founders that it's acquired. Because as far as I understand it, the majority of the management team at this point was acquired in at some point, or at least a very large fraction. And then secondly, as you buy a lot of things, you end up with a lot of disparate products, disparate platforms, disparate integrations, and all sorts of things can go right or wrong based on that. And so I was curious a little bit about the broader concept of integration. Number one, from a team and individual perspective as a founder coming in, and why is it such a good home for founders? And then number two, like, once you acquire in the tech and the product, how do you integrate that as well?

• 4:45Speaker 01

  Yeah. It's really notable. So like, when I look at the, so the sort of exact team of Shopify contains a number of founders, some of whom were acquired in and some of whom just were founders. And when I look at like my management team, the people who report to me, who each of these sort of nine or 10 people run R and D orgs that are like a couple hundred people each, and almost like a very high percentage of those people are also founders, right? And so you ask, okay, well, why are all these people ending up essentially running the main parts of the product of Shopify? And Shopify is a product-first company, so that's sort of the same thing as saying running Shopify.

• 5:24Speaker 01

  I think a lot of this comes from Toby himself, who has extremely strong opinions and has, I think, learned for the better to no longer be shy about like saying, I want to aim in this direction, right? I think there was like a big culture of bottoms-up decision-making, power to the edges, like whatever you want to call that school of management philosophy, which is like, you know, delegate and empower. I think a lot of former founders are actually coming back to the place where they say, hey, what makes, what am I uniquely good at? I'm uniquely good at like having a point of view, being willing to smash my head against

• 6:07Speaker 01

  a wall and like do whatever it takes to manifest that point of view in the world through building things, building teams, you know, saying the hard thing at the right moment. And I think Toby has kind of re-embraced that, right? In the past few years of saying like, hey, if I'm good at one thing, it's being extremely opinionated, and it seems that in this domain, at least for Toby, in this domain of like the internet and commerce, I'm right more often than I'm not. Like, I have a higher batting average than usual. Therefore, if I can help accelerate this company through like being decisive, that's going to be good. And so.

• 6:46Speaker 01

  He sort of sets the example and a lot of us, you know, see that and we're like, hey, we remember being founders. We remember what it's like to have all bucks stop with you. And if there's a way that we can apply that level of decisiveness here and accelerate teams through like the otherwise like, you know, designed by committee bullshit that kills a lot of companies, then like that's a net positive, right?

• 7:12Speaker 02

  The other piece of buying in companies is sort of integrating different systems and platforms. And often you see people do one of two things. They either let things run independently. That was WhatsApp for a long time at Meta, for example. Or you integrate in all the infrastructure and usually that makes things more performant. You can have features that cross over, et cetera. But the flip side of it is sometimes you see these giant projects that just stop a company from functioning, right? Like at Google, there was a famous sort of identity layer that was built and for a year and a half, people just didn't ship things, right? In other areas. And so how do you all think about integrating and acquired companies and acquired technologies? But more generally, as you build and build and build and build, there's a big munch of stuff. How do you sort of consolidate that down in a performant way that's actually speedy to execute?

• 7:55Speaker 02

  And then I guess later on, maybe we could talk about whether AI has played a role in that or not.

• 7:59Speaker 01

  Yeah, it's a really good question. The answer is it depends, like most problems. Usually when you encounter these problems, you sort of have to look at the stack. You usually don't want duplicates at the same layer of the stack, right?

• 8:17Speaker 01

  And the funny thing about this is sometimes you actually don't notice the two things are in fact at the same layer of the stack until much later on, right? So as an example, inside of Shopify, there is a, there's the checkout, which takes, you know, someone's shopping cart and says, okay, let's take the business rules of the store and convert that to an agreement for a sale with taxes and duties and shipping and all that stuff. So that's one engine. And then at different points in Shopify's history, other things that have been built that are sort of like adjacent to that, but not quite the same thing, right?

• 8:58Speaker 01

  So like a thing for creating like an invoice or a thing for like even editing an order after it's been placed. And a lot of these things kind of get built because you start with a small problem. Example, I just need to be able to add a discount to an order after it's already been placed because you know, someone calls up and says, hey, I forgot to put the discount code on, can you please enter it, right? And so someone starts a project over here, which is, oh, we're just going to support editing discounts onto orders. And you don't realize that what you're actually doing is creating a second version of the negotiation engine, right? It's just that you started with a very, very small part of it. And then that thing kind of grows for a few years. And you're like, oh, it's not just editing discounts in, it's also editing the shipping

• 9:43Speaker 01

  costs, it's also editing another product onto the order that they forgot to add. And before you realize it, you're like, oh my God, we're reinventing the checkout over here, right? And by the way, in the meantime, all the teams that work at Shopify are finding themselves having to build to support both of these things, right? So someone builds a new feature and they're like, I'm going to add, I don't know, international pricing. And then that team is like, oh shit, I have to build it into the checkout, but I also have to build it into that order editing thing. Oh my God, and I also have to build it into that invoicing thing. And now it's like to your point, right? It's like, okay, now every, the thing that should have taken a month to build is now

• 10:25Speaker 01

  taking six months to build because the layer of the stack that's underneath me has five things where there should just be one, right? And so sometimes it's hard to know what to do, right? Because sometimes the right answer is let's collapse those five things in the stack into a single thing, which is it sounds like what Google tried to do. But then you're like, oh my God, it's like trying to take the foundations of a house and like rearrange them while the house is still on top. And like, that's, oh God, this is going to be really hard because I can't just make everyone move out of the house, right? The real answer to how do you solve this problem is, one is, Um, developing extremely good intuitions for when you are accidentally creating duplicates at the same layer of the stack. So you just never get into the problem in the first place.

• 11:09Speaker 01

  And then the second part is being willing to actually notice when you have created the duplication and eat the vegetables of saying like, oh shit, whoops, we did it. Okay, now it's time to eat the vegetables and collapse these things together because if we don't, every team that exists above us in the stack is going to pay this duplication cost like in perpetuity. And the problem is it's not even just your R and D teams that pay the cost. The saddest thing is that usually it's a combination of your R and D teams and your customers paying the cost. Because the reality is that they use, not everyone will do the thing of building the five versions. They'll probably build like the top two or three most important ones.

• 11:54Speaker 01

  And then all the customers who depend on numbers four and five will go to use the thing. And they'll be like, oh, this is super weird. This thing doesn't work. Why doesn't, why is this thing so different to the other thing? I thought they were the same thing. And then they look under the covers and they're like, oh, I know why it's different. Because actually this part of the stack is all fucked up.

• 12:12Speaker 02

  Have, have you approached that from an AI perspective? So I think like Shopify was a very, very early adopter of AI. And in general, whenever there's like a really sharp technical founder driving a company, they've been an earlier adopter, right? And Toby obviously fits that mold. But it's an area where I think you kind of have to play around with it, experiment, et cetera, to really understand the capability set. But at the same time, to your point, you want kind of centralized infrastructure approaches. So have you thought about how to do that adoption as well as, how, how have you thought about that in the context of launches like Sidekick and Shopify Magic and things like that?

• 12:42Speaker 01

  I mean, yes, this is the, obviously the space where everything's moving the fastest, right? Like the world of invoicing is not moving at the pace of the world of AI. Right? Depends on why you're invoicing that. Yeah.

• 12:58Speaker 01

  Yeah, it's been an extremely interesting last 12 months of like looking at where can we apply different models to parts of Shopify? I think the high level idea is, you know, Shopify's real focus and real mission is enabling entrepreneurs, right? And the meta strategy of everything we do is, how do you simplify the process of starting and running a business so that more people who would otherwise not get over the hump, get over the hump and have an opportunity to try building whatever that business is? And like the historical strategy for that Shopify was build really easy to use software.

• 13:43Speaker 01

  Use software that is super simple when you start and it kind of reveals complexity as you need it, like as you business grows and you have harder problems, like the tools sort of like appear at the right time. I mean, the world is littered with the corpses of, you know, the companies that tried to serve SMB and enterprise at the same time. And this is kind of the hard problem of Shopify. How do you be simple, but also powerful and scale up? AI is this really interesting moment where it's like, okay, well Shopify built its position in the market largely by being the easiest to use thing as you build a business and scale, but in imperative mode, right? It's the, I'm going to give you all these switches and we're going to give you the

• 14:25Speaker 01

  best switches and we're going to show you the switches at the right moment in time. But these, this is the best switches, right? And AI brings us into the world of like, okay, well, what if we can give you the, the driver in the car who knows how all these switches work and can help you along without having to learn all the switches? It's probably one of the most powerful opportunities for more people who would otherwise be daunted by having to learn the switches, even though we try to make the switches as good as possible. There's all these people who can't quite figure out the switches, but they could talk to the, they could talk to the co-pilot and they could explain what

• 15:07Speaker 01

  switches they want hit, but they just can't do it themselves. And so I think we have an amazing opportunity to increase like the amount of entrepreneurs in the world who actually get a shot, right? And so that's the kind of headline. And then from there you go into, okay, well, which models do we apply to which problems? You know, you look at individual problems like, okay, how many people get stumped just writing the descriptions of their products? Cause they're like, they're not good copywriters and they're like, oh, this is embarrassing. I can't write a good description for my candle. I'm not going to launch my store, you know? And there's so many instances like this where it's like, okay, well, can you give someone that extra bit of skill that gets them over the hump where they're ready to hit the big green launch button, you know? Yeah, that's really cool.

• 15:50Sarah Guo?

  Glenn, it's, it's really exciting how many obvious places there are to apply that to make it easier for the merchant or the would be merchant. But I think one of the things that has struck me that you and Toby and the team has done is actually shipped a lot of that quickly when a lot of very large organizations are like, oh, that makes sense. The outputs are non-deterministic by nature. We have a process for measuring and evaluating quality of our products. Generally that process does not apply. Like, how did you get to this is good enough and we can ship it?

• 16:23Speaker 01

  Well, well, one of the principles that we currently hold, this might change in the future if confidence intervals go high enough. But one of the principles that we have for all of the Shopify magic features today is that they're allowed to propose changes, but not commit changes, right? So they can like generate text, but they're not going to save it without you actually reading it and saving it. We might suggest a reply for the user that's, you know, someone writes in like, hey, what's your shipping policy? We can like suggest the reply based on us knowing what's in the store and like running that through an LLM. But you have to hit enter, right? And so one of the things that like human is in the loop essentially right now is one of the one of the ways that we're kind of mitigating risk here.

• 17:06Speaker 01

  And obviously, human in the loop is great because it gives you the feedback cycle. You actually get a three-part signal. You get which suggestions are accepted clean, which suggestions were accepted, but then with minor edits, and then which suggestions were outright rejected. And that's an amazing loop to be able to improve things through. So we're always getting better, but that sort of human in the loop, human must click save thing is a is a big part of the strategy.

• 17:32Sarah Guo?

  Do you think that Shopify has a different like risk orientation than other companies of its size? You still need guardrails and even if there's suggestions, like it's not, it's not perfect, right?

• 17:44Speaker 01

  Correct, yeah. I think Toby, Toby and Kaz and myself were all like fairly risk hungry people. I mean, I think that's just a little bit of founder culture is like you want to take risks. Like you don't really want to be safe. You get bored, you get annoyed when things get too safe. But there's always this balance of like, we like taking risks that are risks to us. We don't like taking risks that could hurt, like break someone's business. And so if there's a way to do a thing that's like very risky for us, but but is the risk is mitigated for the actual merchant, like we're very for that. And I think that's where that kind of human in the loop, human must click save thing is is part of that, right?

• 18:28Speaker 01

  And who knows, like maybe a year from now, maybe two years from now, like we get to like, you know, hey, you're, you seem to be accepting our suggestions without edits 99% of the time. Do you want to just go into like full auto mode? Like, maybe we get there eventually. We're not there yet, but like you can sort of see that the hill climbing will eventually take us somewhere close to that, right?

• 18:49Sarah Guo?

  Can you just describe for our listeners, um, some of the stuff that you think is coolest to add additions? If that's, um, you know, image gen or semantic search or or even any of the foundations work?

• 18:59Speaker 01

  The the foundations work is the thing that I'm most excited about, but I'm a commerce nerd. So like this is going to seem like very boring to the audience, but like just humor me for a sec.

• 19:07Sarah Guo?

  It's a pretty nerdy audience, so it depends.

• 19:09Speaker 01

  Yeah, the data model that is the most, most, most central to Shopify is the products data model, right? Like, how do you represent the products that you're selling? And Shopify has for, you know, basically like 10, 15 years had. The product's data model hasn't changed that much. It's like fairly simplistic. It's actually an amazing testament to how big of a company and how big of a, you know, the fact that you can get to like 10% of all US e-commerce on like a fairly simplistic data model is actually kind of amazing. But, you know, like Shopify has been a little bit on the weak side for dealing with like very large and complex products. So products that have a lot of options, a lot of colors, a lot of sizes.

• 19:50Speaker 01

  When it gets complex, we don't do as well. And so we're updating our data model with support for just a much larger number of variants. And again, this is the nerdy part. This is one of these crazy things about tech. The reason this is so hard is because it forces you to go from un-paginated to paginated APIs. When you start making the variant counts very large, and it's just a breaking, it's just a breaking change for like the entire app ecosystem. And it's just breaking changes are hard, right? So I'm really excited about just the fact that we're unlocking that data model. We've actually also embedded in, not just like, hey, you can have more variants and more options, but there's also now a standard product taxonomy that comes with standard categories and standard attributes.

• 20:35Speaker 01

  So that like when a merchant creates like a T-shirt, we're going to auto recognize, okay, this is in the standard category T-shirts. And that category comes with all of these standard attributes. And we're going to use AI to, one, detect what category it should be in, two, detect what the values should be. So we're going to order to, we're going to try and guess, okay, seems like the colors here are green and yellow based on the images you uploaded. Seems like it's made out of cotton based on the description you had. And the great thing about that is when that merchant takes those products out to their own storefront, all of that data is going to make the search experience better. But more importantly, when they take those products out to like Google and Facebook and Amazon and all those places,

• 21:17Speaker 01

  having that structured metadata around their products makes them way, way, way more discoverable, which is going to lead to them ranking higher in search results and basically making more money. So this is one of those weird things where like getting the data model right and having the data quality be very high actually causes these very like important effects up funnel for the app, for the business. So that's really cool. Um, uh, we are also, uh, releasing, um, the first version of our, uh, image, uh, image editing with AI thing. Um, so you can take product images, you can replace backgrounds. Basically, you can take, uh, uh, product imagery and get to a very, very, very professional, uh,

• 22:01Speaker 01

  standard, uh, on it without having to go through like very expensive photo shoots. Basically, a lot of what we do is like, how do you help someone who's like actually just, you know, my mom at home trying to start a business? How do you help that person look like as professional as like the world's largest companies? And then, um, yeah, what else am I really excited about in the addition? I think you just mentioned something, Sarah. Now I'm having a mental blank.

• 22:26Sarah Guo?

  Uh, I mean, I just have like, I'm a search nerd. And so I think like relate. Yeah.

• 22:31Speaker 01

  So here's a, here's the crazy thing. So I just learned what LBD means.

• 22:36Sarah Guo?

  Like a little black dress.

• 22:37Speaker 01

  Exactly. I didn't know what LBD meant until recently. Um, but, uh, very soon Shopify stores will actually understand the meanings of things as you. So today's Shopify search is very, very literal keyword searchy. There's a video of me going around internally being like infuriated when I go to, there's a store that I actually buy things from and I went on there and searched for, um,

• 23:00Sarah Guo?

  LBD.

• 23:01Speaker 01

  Not LBD. I searched for a sweater and because that store, uh, only lists their thing as sweatshirt, I got like zero search results. I was like, I can't, this cannot still be true in 2023. Like this is insane that this is happening. So that's a really easy example, but like, you know, if you go to a store now and they haven't put little black dress in any of their keywords, if you search for LBD, it'll actually do the right thing. You can even do things like, um, I saw a crazy example from the team yesterday. They were like, um, Christmas themed shoes. And it would actually correctly go and find all the red and green shoes in the store.

• 23:40Speaker 01

  So that's kind of an edge case example, but it's like, you know, searching for things like, um, something to wear to a wedding, something to wear to the beach. Will actually do the right thing, whereas before would only would never do the right thing unless those keywords happen to be in the product descriptions, which will never happen. So that's, I think, again, that's one of those things where it's like, how can this search experience on someone's tiny little storefront actually be as smart as like what you would get on Google?

• 24:05Speaker 02

  That's really cool. And are you doing like rag or embeddings or specific technical approaches like to that or?

• 24:10Speaker 01

  Yeah. Yeah. Yeah. So a lot of what we have to do is figure out what the right embeddings are to use, how to fine tune the models. Like obviously there's categories in Shopify that are, um, more like apparel, fashion, gift, homewares. So spending the time to, um, assemble the data sets and do the, uh, fine tunings and pick the right embedding models that are best for this category. But it's also been really interesting with like, especially some of the multimodal models that have emerged in the last, I don't know, two, three months.

• 24:46Speaker 01

  Experimenting with how much, um, how much weight to give, like the text descriptions versus the images versus the, um, taxonomy attributes and figuring out what mix of those things generates the best results is, I mean, we're still working on it, honestly, but it's a pretty cool and exciting space.

• 25:04Speaker 02

  That's really cool. Yeah. Cause I guess like, uh, a lot of people increasingly I feel are adopting GPTV or the vision model, which allows you to upload images or understand or interrogate them. And so it seems like there's a lot of information just resident in like visual imagery that people aren't really making use of it. Now you can translate into. A textual understanding and therefore tie that into everything else that's associated with the merchants. That sounds very exciting.

• 25:25Speaker 01

  And it's it's an amazing thing for the buyer experience. I mean, like, to your point, like, the buyer experience is really cool when you search for things to wear to a wedding and it does the right thing. But actually, think about it from a merchant experience point of view. If you're like, hey, I've been running a store for five years and I've got like 10,000 products in there. And now you want me to go back and backfill like 10,000 products worth of five attributes for a product like Jesus Christ. Being able to actually apply the models to do something that turns out to be like a 90% correct guess is an amazing way to help bootstrap people into like the current moment, you know?

• 26:04Speaker 02

  What do you think is still missing from a capabilities perspective to substantiate your perfect AI world for Shopify?

• 26:13Speaker 01

  Well, look, I mean, I think the the exciting and frustrating thing about L L Ms right now is that you can get an L L M like agent thing, like a sidekick-esque thing. You can get it to like 75% in like 10 minutes. And then it's like this brutal hill climb to get it to like 95% like over time, right? It's just like, it's like catnip for like hackers because they're like, oh, my God, I just 10 minutes. I got to a thing that's like pretty good. And then you're like, yeah, wait, buddy, wait, wait for the next bit. The next bit's real interesting, you know?

• 26:47Speaker 02

  Yeah, the last 5% I think is the next five years of tech or something. So.

• 26:51Speaker 01

  Right, right, right. And and the irony is, is like, you know, people talk about, they're like, oh, yeah, this L L Ms hallucinating right now. And you're like, yeah, yeah, you know, that's its job, right? You know that every time it emits a token, it's hallucinating, right? It's just that you like some of the hallucinations more than the others, right? And so I think that's that's been the challenge for us. Like, it's even in the name, right? Like, Sidekick is is supposed to feel like your companion on the journey, like your coach, your assistant, your the person who's maybe seen seen this movie before a little bit. Getting it from 75% to 95% is is the process we're working through. And it's and it's so exciting because, you know, we're doing stuff on our side where we improve the training

• 27:36Speaker 01

  sets, you know, we're building more and more conversations with feedback from users that help us dial it in more. But then every five seconds, you look around and like, there's a new model that you can. Take all your training set and all your evals and then it might be another step function. So it's kind of innovation really rapidly happening both inside and outside our R and D team. And you really know, you never know what we each week is going to bring, really, you know?

• 28:04Sarah Guo?

  And you can be pretty optimistic in that, like, especially if you look at any one of these problems. I think there's a lot of focus on the core model capability, and there should be. But like, you're like, oh, well, embeddings models are getting better, right? And and people are working on like synthetic data generation tools and working on tooling for the entire RAG pipeline. And so I think it's, yes, like, you hit the the pain of reality of like, oh, no, I have to go beyond the demo. But a lot of other people are working on some enabling stuff. So I still expected a lot of the experiences to get better fast, counting on it from Shopify.

• 28:38Speaker 01

  Yes. Yeah, yeah. Well, I mean, we're at the cutting edge, you know?

• 28:42Speaker 02

  Yeah. Is there anything external to Shopify that you think is especially interesting right now in the AI world? Be it startups or things that people are working on or projects?

• 28:52Speaker 01

  This is literally every single person has probably said this, but I think the like, the rabbit thing at CES was pretty interesting. I mean, I'm literally wearing the shirt right now. Like, I'm a teenage engineering nerd. So like, I was I was hyped on the hardware. But I think one of the interesting problems with like, LLM based applications and agents in particular is like, what's the actual interface they're interacting with, right? Like, in the case of Sidekick, right? There's a couple different places. Like, what is Sidekick using? Right? Is Sidekick actually using the admin API under the hood? Or is Sidekick actually reaching into the pixels of the web app and clicking

• 29:36Speaker 01

  around in the web app and doing stuff, right? Like, that's a pretty important question. Like, what is the interface that the agent is actually learning and interacting with?

• 29:48Speaker 01

  And I thought the really interesting part of the Rabbit presentation was that they decided to treat the actual GUI as the interface and to try and like have a model that became very good at interacting with essentially web apps. If it actually works, the strategic brilliance of that is they instantly have the world's largest app store because the world's largest app store is just the web, right? Yeah. Now, who knows if it'll actually work, but it's an incredibly interesting strategy.

• 30:17Sarah Guo?

  I was talking to Jesse from Rabbit about this and I was like, it's a very, it's a controversial, interesting strategic decision to say like, I'm going to interact with the applications themselves versus use APIs. And his last company tried to unify actions on APIs and his view was very much like, well, there are implications on the, like, as you said, reach and ability to use different types of data and like whether or not you own your destiny if you're relying on those interfaces. And so, as you said, if it works, that's super interesting.

• 30:50Speaker 01

  Yeah, and it's, even philosophically and from first principles, it sort of rings true to me, right? Because, you know, a lot of these experiences are, especially with, like, when I actually think about the Sidekick UI, the Sidekick actually runs Sidecar in the admin next to you, right? Even the name Sidekick is like, hey, I'm here with you, you know, we're both here together and we're both using this Shopify thing to try and build your business, right? And so the idea that Sidekick would literally be interacting with the pixels in the same way that you are, it's sort of almost like smells right in a weird way.

• 31:30Speaker 01

  And, I mean, obviously, the nice thing about it is as long as the model can actually learn the UI, then every time you ship updates to the UI, if the model's smart enough, like, it's just like, oh yeah, my smart buddy keeps, it also can use this new button that just appeared today, right? So yeah, I think it's an interesting approach. We'll wait and see what actually happens. I also know people in the industry who are like, yeah, yeah, it's a cool idea, it's going to be insanely hard to pull off, but who knows? Again, this thing changes every week. The other thing I think is kind of interesting, which is actually like Shopify relevant, is I guess I think, I'm curious what you guys think about this, but like the sort of...

• 32:10Speaker 01

  The the triangle of like perplexity, Google, ChatGPT of like, it's basically like LLM native search and like ChatGPT really good at the LLM interface thing, not amazing at search yet. Google amazing at search, not really figured out the LLM thing yet. Perplexity is kind of squarely in the middle and saying like, we think LLM native search is a really interesting like hill in the middle of this thing. I'm sort of watching this from the sidelines and kind of interested to see where it plays out. I think embedded in that strategic tussle is the question of what do people actually want from search? Do they want hard facts?

• 32:55Speaker 01

  Do they want opinions? How much lossiness are they willing to tolerate in order to get the compression of the LLM took 10 search results and summarized them for me, but maybe they hallucinated the summarization, right? It's it's a really interesting place. Like, what do people actually want, you know?

• 33:16Speaker 02

  Yeah, I think it's a really interesting open question. And related to that, too, you start, I've I've started to see the degradation of some performance of some other really early players in the market where there's a lot of qualifications or safety, or you start to see the LLM go out to try and gather information. And it just really, in some cases, either slows down or makes the information worse because to some extent, the reason you're interrogating an LLM is you're, you want to get to an answer. You don't want a web search or you don't want that, you know, poor synthesis of bad data from the web. You want the good synthesis of bad data from the web. And so, yeah, it's been fascinating to watch. And to your point, I think a lot of people have started to adopt perplexity for that specific use case because

• 33:57Speaker 02

  it has that middle ground of some form of IR plus LLM in a traditional sense. So, yeah, I agree with you. It's it's going to be fascinating to watch how all the directions this goes. And I think the reality is people almost forgot that people from search, they actually want an answer. They don't, they don't want to do a search process. They want to get to a result in many cases or a list of results. It's almost like people forgot the user need in some sense.

• 34:19Speaker 01

  Yeah. And it's funny when like, I mean, some searches are so specific, but some are almost like. Shared knowledge of humanity. So it's like one, one thing I found myself doing a lot like a year ago. Um. When I was looking at, okay, how can we improve search at Shopify, both on the storefronts, but also on the shop app? And just kind of playing with this idea, one of the tests that I found myself running over and over again. So a- at the time, I actually had uh this happen. I had a, a friend's uh daughter. It was her birthday and I was like, she was, I think she was six at the time. And I was like, I want to buy a great classic book for a six-year-old girl.

• 35:01Speaker 01

  And as an experiment, I would go to Google and I would like say, classic children's book for six-year-old girl. And like the search results you would back, get back on Google Shopping would just be like terrible. Like absolute disaster, right? And then you would go to Amazon type, same thing, and you would get like nonsense back, right? You went to ChatGPT and you said, give me ten great children's books for a six-year-old girl. What would come back would actually be amazing. Like it would be like really like the best ten recommendations, right? And so I was like, huh, there's something here, right? Now, like ChatGPT, terrible if you ask it for, hey, I want a pair of Nike AF1s in size ten point five in a store in New York City.

• 35:43Speaker 01

  But really, really, really good at the general knowledge question of like great children's books, you know?

• 35:50Speaker 02

  Yeah, it's funny because I worked at Google years ago. And then when I was at Twitter, one of the teams that worked for me initially was search. And um, you know, you tend to segment these things as you undoubtedly do at Shopify into uh types of search, right? Is it navigational? Is it certain types of information that you're looking for? Is it something else? And so, you know, you could imagine that in the LLM world, you effectively want to do almost like one boxes like you have at Google, where you trigger off a certain types of keywords or phrases. And then therefore you end up with a different result and you should be able to, especially have integrations with search engines, be able to serve the, I want these specific Nikes and where should I get them, right? Because that's almost like a form of navigational query in some sense for a commerce objective versus uh just give me some knowledge. So it feels to me like a lot of stuff will come as long as people don't actually think that LLM has to do everything.

• 36:36Speaker 02

  Unless it can also function as a router or something. So it's really fascinating to think about.

• 36:40Speaker 01

  Yeah, and then that's when you get into the world of like, things that are like matters of fact and things that are matters of opinion, right? Like, is this shoe a Nike is not a matter of opinion, right? Whereas is this book great for a six-year-old girl? That's a matter of opinion. And how do you construct search systems that can actually sit correctly at the midpoint of the world of facts and the world of opinions, right?

• 37:05Sarah Guo?

  I think also going back to Alad's point on use cases, like one version of this, like from a scenarios of the future perspective is it fragments, right? Sometimes I want facts and sometimes I want opinions, especially opinions like my own already. I mean, like a date on how people consume media definitely looks like that, right?

• 37:26Speaker 01

  Yes.

• 37:26Sarah Guo?

  And so I think it's very easy. I mean, I was an investor in a search company called Neva. I think very highly of the perplexity team. I think it's very easy to look at the chokehold on distribution that Google has through its partners and say like, that's very hard, but we're at this very, I think, special moment where the technology and perhaps the like user behavior, the expectation, if we take out like slices of it, where the expectation is very different and get people to think about pieces of search. Because it's not at all clear to me it should have been one market or it's permanently one market. Right? You guys probably think about owning commerce search and so.

• 38:04Speaker 01

  Right, but even within commerce, there's so many different, like the way people search for clothes and the way they search for industrial parts are like not the same thing at all, right? So it's even within that world, there's so much nuance of the way that people express what they're looking for.

• 38:23Sarah Guo?

  Glenn, awesome. Thank you for being here. That was great.

• 38:26Speaker 01

  Thanks, guys. Appreciate it.

• 38:27Speaker 02

  Good to see you. Thanks so much.

• 38:29Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.