AI for search at Etsy

We have all used web and product search technologies for quite some time, but how do they actually work and how is AI impacting search? Andrew Stanton from Etsy joins us to dive into AI-based search methods and to talk about neuroevolution. He also gives us an introduction to Rust for production ML/AI and explains how that community is developing. Sponsors: DigitalOcean https://do.co/changelog – The simplest cloud platform for developers and teams Whether you’re running one virtual machine or ten thousand, makes managing your infrastructure too easy. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Andrew Stanton – GitHub https://github.com/Refefer, LinkedIn https://www.linkedin.com/in/andrew-stanton-00b9207, X https://x.com/RefeferChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Yahoo learning to rank challenge http://proceedings.mlr.press/v14/chapelle11a/chapelle11a.pdfNEAT approach to neuroevolution https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologiesOpenAI evolutionary strategies for Atari https://openai.com/blog/evolution-strategies/Rust https://www.rust-lang.org/PyO3, Rust binding for the Python interpreter https://github.com/PyO3/pyo3Buzzsaw: A System for High Speed Feature Engineering https://mlsys.org/Conferences/2019/doc/2018/141.pdfLinkedIn at KDD https://engineering.linkedin.com/research/linkedin-kdd-2019Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode Cloud Servers. Head to linode.com/changelog. This episode is brought to you by Digital Ocean. Guess what? Digital Ocean recently added MySQL and Redis to their list of managed databases. Their full managed databases lineup now includes the three most popular databases out there for developers: Postgres, MySQL, and Redis. Eliminate the complexity involved in managing, scaling, and securing your database infrastructure, and instead get back to focusing on building value for your users.

• 0:41Speaker 00

  Learn more and get started for free with a $50 credit at do.co/changelog. Again, do.co/changelog.

• 0:59Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slag with us around various topics of the show at changelog.com/community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:24Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitehead. I'm a data scientist with S I L International. And I'm joined as always by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are you doing this week, Chris? Hey, I'm doing fine, Daniel. What's up? Not, not much. Busy day. It's submission day for ACL. And I'm trying to get something ready. We'll see if I actually make it. By the time this goes live, I will have either failed or not made the deadline. And ACL is for the listeners? Yeah, that's the computa- it's a large computational linguistics conference, but it's kind of one of the larger-

• 2:10Daniel Whitenack?

  Natural language processing, community research conferences. And so there's like EMNLP and then there's ACL. And there's larger, you know, like right now started maybe today when we're recording is the start of NEREPs, which is another large AI research conference. And hopefully I'll be live streaming some of that later and trying to keep up because I'm not there. But yeah, it's one of those, those sorts of conferences. So we'll see if I, if I make it.

• 2:39Chris Benson?

  We'll make it through. I gotta say, I had one of the coolest weeks last week I've ever had. I started at Carnegie Mellon University. We had, there was a big conference on future of AI and STEM in society. I got to do a breakout on AI and ethics and such in STEM and what things that could be done. That was really a cool conversation, solved all sorts of world problems right there.

• 2:59Daniel Whitenack?

  I bet so.

• 2:59Chris Benson?

  Yeah, I got to sit on a panel called protecting AI from threats. And the guy beside me was General Cartwright, who used to be the Vice Chairman of the Joint Chiefs of Staff. And he just had brilliant insights. He's not an AI person, the way we are, but was just really impressed with what he had to say. And then did an opening keynote in Philadelphia later on on ethics and AI. And finally, we finished out the week, listeners will probably recognize this. We had the championship for Alpha Pilot, which we had a previous episode on in Austin, Texas. And we handed out a $1 million prize to Team MavLab. They're from Holland. And so it was a pretty cool week.

• 3:36Daniel Whitenack?

  Yeah, that sounds extremely eventful. And I imagine that in the midst of all of that travel and logistics and all of those things, you were utilizing some form of search in some way to, to manage your life.

• 3:52Chris Benson?

  I might have been.

• 3:53Daniel Whitenack?

  Yeah, today on the show, we have Andrew Stanton with us, who is a Staff Product Manager of Search Ranking and Platform at Etsy. And I'm, I'm excited to talk with Andrew about search, but also some other things. And also, this is the first episode, I think, where I told my wife, who was coming on the show. And she recognized, you know, obviously Etsy and was pretty, pretty psyched that I was talking to someone from Etsy. So we're all excited to, to talk to you. Welcome, Andrew.

• 4:23Speaker 03

  Thank you so much for having me.

• 4:24Daniel Whitenack?

  Yeah, definitely. So maybe to start us out, if you could just give us a little bit of your background, how you got into AI, ML-related things and search and eventually ended up at Etsy?

• 4:37Speaker 03

  Yeah, great question. So I've been kind of blessed to be working with machine learning and search on and off for about 15 years at this point. And the irony is I actually never intended to go into either. When I was in school, I was much more interested in distributed systems. And the funny thing was, is that, you know, as our data grows, I kept running face-first into places where we needed to have more sophisticated search. We needed to have better predictive performance than kind of standard heuristics. So when I was in undergrad, I was actually working full-time for AOL at the same time. And the big focus at that point was working for an online radio show of all things. But it ended up boiling down to this kind of predictive problem where we

• 5:18Speaker 03

  were trying to understand how, basically, our listeners would kind of tune in from around the world. And so that was my first kind of face plant into linear regression and kind of time series prediction. When I left, I ended up moving into something called entity recognition, which is this kind of process of trying to understand from unstructured data, the different types of entities that might be represented in it. It could be people, it could be companies, it could be any type of entity that might be useful. And then building kind of this typo-resistant search on top of it. And that was also probably my first real interaction with extremely big data. We were dealing with billions and billions of records. And so how you build kind of performance

• 6:02Speaker 03

  search on top of this entity recognition system, which is constantly ingesting, you know, hundreds of millions of records per day, turned out to be kind of this scratch and itch sweet spot for me. From there, we worked on a bunch of different problems, ended up in a startup called Blackbird Technologies, where we were working on e-commerce search in the B2B space. And our big kind of value add was being able to leverage multimodal deep learning to basically tease apart a lot of these products that companies had to provide a better search. Experience on top of it. We were acquired by Etsy back in 2016 and I've been making my home ever since.

• 6:40Daniel Whitenack?

  And that that sort of multimodal side of search, when you mean that you're meaning like images and text sort of thing?

• 6:49Speaker 03

  That's exactly right.

• 6:50Daniel Whitenack?

  Okay. All right. So like products, if you're searching products on a website, there's obviously product photography, right?

• 6:57Speaker 03

  Mm hmm.

• 6:58Daniel Whitenack?

  I guess that could factor in somehow.

• 7:00Speaker 03

  Yeah, I would say a good example would be think about something like Craigslist or Facebook Marketplace. You have kind of an image and then you have maybe a sentence or two about what that item is. But somehow you have to understand that when a potential buyer comes in and they type in kind of this highly specific queries such as colors and materials and other types of attributes. You have to take this this very unstructured piece of information and convert it into something which is both relevant and searchable.

• 7:27Chris Benson?

  So kind of wondering as we start to dive in and we're talking about search right off the bat. Before we get fully into into what Etsy is doing with search, can you kind of talk a little bit about what types of search problems are out there? We tend to use the the word search in all sorts of different contexts, you know, and there's full text search, web page search, product search, you name it. Can you kind of give us an idea of kind of the overall landscape of what search problems look like and and how they're related, if at all?

• 7:54Speaker 03

  Yeah, that's that's a really good question. There's really, I would say, maybe three major areas of search. There's information search, which you know, from something like Google, you go and you type CNN. 99% of the time you're intended to go to CNN.com, maybe you find the Wikipedia page is number two, but largely you're searching to find pieces of information. The second type is probably e-commerce search, which I'm most familiar with. Amazon, Walmart, Alibaba, et cetera, et cetera. How do you match these buyers who are oftentimes giving these very vague queries like jewelry and trying to understand, you know, what are the latent factors that are actually interesting to the buyer? The third, which has kind of grown up much more recently, is probably question and answering, some kind of the Stack Overflow problem,

• 8:37Speaker 03

  where you are asking questions, much more natural language, but you're trying to tease out this this kind of community aspect of retrieval where the intent is not necessarily on finding a single piece of information, but perhaps finding a collection of pieces of information. And the domain is a little bit more NLP heavy.

• 8:57Daniel Whitenack?

  So in those different areas, I mean, obviously, e-commerce developed at a certain time and certain things like Stack Overflow probably have been more popular of recent times. Looking back over the history of search, you know, when did machine learning and AI start being applied to these types of search problems? Was it always applied or did it kind of start out as rule-based algorithms and like, oh, you know, this thing includes this word this many more times than this other thing and so it's ranked higher? I'm assuming some of those things started earlier. When did AI start being applied to search?

• 9:37Speaker 03

  It's always been somewhat applied, right? So you can think about kind of the nexus as originally starting with catalogs. You had a bunch of records you wanted to retrieve across these kind of old library systems. And I don't know if you remember, but you used to have to give these kind of Boolean queries and then these kind of quasi-rankers built into it. It's for machine learning, you know, we've always tried to use and try to understand relevancy. And so we've had things like TFIDF and BM25, which have kind of existed for several decades at this point.

• 10:08Daniel Whitenack?

  And those are like methods that are based on counting instances of tokens and working off of tokens that are in certain samples and not other samples and that sort of thing.

• 10:20Speaker 03

  Correct. Oftentimes, it's statistical-based. The idea is that rarer tokens contain more information is kind of the primary motivator for it. And they were really the first attempts at kind of understanding relevancy inside of kind of free text search. Facets and filters. So there was a company that really started popularizing this in the e-commerce space called Endeka that was, I believe, late 90s, early 2000s.

• 10:45Daniel Whitenack?

  I used to work with them.

• 10:46Speaker 03

  Oh, wonderful. So you know them well. They kind of innovated on things such as filters and facets and really kind of scaled out the initial, you know, ecosystem for e-commerce search. As it came to machine learning, I would say probably the biggest step functions in terms of improvements were kind of the learning to rank work in the early 2000s, late 90s, where we started to apply, you know, basic machine learning problems such as support vector machines, which in the case is SVM rank, to kind of improve relevancy across a whole bunch of different signals. And at that point, it moved away from these kind of generative type models such as TFIDF to these more discriminative type models.

• 11:26Daniel Whitenack?

  Awesome. Yeah. And has I guess there been a lot of momentum in new like deep neural network, unsupervised, all these sorts of like kind of hype things that are happening now? Has that impacted the search world a lot?

• 11:43Speaker 03

  Oh, so much.

• 11:45Daniel Whitenack?

  Okay. Yeah. Tell us, tell us more.

• 11:48Speaker 03

  Oh goodness. Okay. Where to start? So I would say search is really an interesting kind of problem space because it's really kind of a confluence of a bunch of different technologies. You can think of the most, a pretty standard stack is looking like something like a solo or an elastic search where you index all your documents. You retrieve some type of candidate set based on the input query and other conditionals like filters. And then you re-rank them a bunch of times and then spit out some output to the end user. We have innovated in the industry across every single one of those elements. Learning to rank models have improved. The recent hotness in deep learning has really started to have an effect in search in the form of things such as neural IR. So this idea that

• 12:32Speaker 03

  you can build these massive models, these massive neural nets, which know how to translate from the query space to the document space and just replace the retrieval systems that were historically just tech space matching.

• 12:46Chris Benson?

  So I'm kind of curious, you know, what types of data get involved? And when you're building out a machine learning model these days and you know, what data is relevant? Where do you go for your data for search? It's just not something that I'm familiar with. And I was, I was rather curious.

• 13:02Daniel Whitenack?

  Yeah. I was thinking while Chris was saying that, like, are there data sets? Like you're talking about learning to rank and like going from query to document. Are there like existing data sets that, you know, are kind of standard in that? Or is it still like a lot of people using, you have to kind of build up your own internal data sets and that sort of thing?

• 13:22Speaker 03

  I would say it's a bit of both. There's definitely learning to rank data sets out there. So the Yahoo Learning to Rank Challenges from the mid 2000s, you have, Microsoft had a bunch of different learning to rank data sets over the years, such as the Web 10K and the Web 30K. You have Trek and you have a whole bunch of these kind of historical data sets. The problem with them is that they were all universally kind of overfitted to kind of the information piece, right? So that kind of web search element. And the problem is, is that we've learned that search inside e-commerce is actually quite a bit different. So when it comes to taking a lot of these, these kind of benchmark data sets and applying them, they don't necessarily translate well from one domain to another.

• 14:06Speaker 03

  So most companies will, will build up their own data sets internally and they will apply a variety of different methods, some of which might be state of the art in kind of the traditional information sense, some which might be bespoke to their own needs.

• 14:22Chris Benson?

  So I'm kind of curious, you know, can you kind of tell us a little bit about why search is relevant to Etsy? Just to cover a little bit about the tie-in on why it is that you're doing that? What does Etsy need search for in that way?

• 14:37Speaker 03

  So Etsy has over 60 million results and most of them, a good portion of them are handmade, customized, one of a kind. We have a big vintage basis where you only have an example of one. And so unlike Google or Bing or Gandex or any of these kind of massive search engines where you're constantly returning a top result, we have this constant turnover in inventory and our inventory is just growing every day. The other problem with Etsy is that we don't have SKUs. So unlike Amazon who's able to leverage a lot of the structured data that's provided to them by the manufacturer, we basically rely on our sellers to figure it out and our machine learning algorithms to try to tease apart the different pieces

• 15:18Speaker 03

  in there into the type of information which is actually useful to the buyer. To give you an idea of why search is needed, like before, you know, back in the late 90s, inventories were so small, you could properly navigate through, you'd go through a dropdown box. At the top and click on jewelry and then you'd you'd see all 200 items that were for sale. If you go on Etsy right now and you type in jewelry, you get 18 million results. There's no human out there that is going to go through 18 million results. And so that's where ranking and search and personalization, all these different elements kind of come together to try to hone down that 18 million total result set to something that's actually digestible by the buyer.

• 15:57Daniel Whitenack?

  Yeah, and and I was just thinking while you were describing things, it seems almost like there's there's so many outliers in Etsy in the sense that like I just searched for, for example, I searched for R2-D2 because uh you know, we've been watching Star Wars stuff recently and I see like R2-D2 like gift pack of like planting pots and then I see a R2-D2 like chalk bag for rock climbing, which if I'm just thinking like an R2-D2 rock climbing chalk bag seems like extremely outlier to me in terms of like products that that you could create some rules around, right? It seems super challenging.

• 16:42Speaker 03

  It's it's a really good point. Etsy, we have a lot of very niche niches. And that's really spectacular for the buyer too, because that means there is likely something out there for you that almost feels like it was made specifically for you. For that one person, that Star Wars lover who likes to get up on rock walls, that that's a very applicable gift. And understanding when to surface those versus when not to is a big part of the challenge.

• 17:21Speaker 04

  This episode is brought to you by Brave. Big news from the Brave team, version 1.0 is official. That means our favorite open source, privacy focused, blazing fast browser is ready for primetime. Their brand new iOS app landed just in time for the announcement and the Brave team is celebrating by granting 8 million basic attention tokens to the community. That means when you download the iOS app, you get 20 bat absolutely free. Put it to good use by heading to changelog.com, hitting the triangle icon in the upper right hand corner and flipping us a tip.

• 18:08Daniel Whitenack?

  So it was mentioned to us that Etsy is using neuroevolution for search and I guess if you could kind of tell us a little bit about what is neuroevolution and kind of what does that mean? It's a new term to me and so I'm kind of curious not only to understand what it is, but how it relates to search.

• 18:28Speaker 03

  Yeah, so neuroevolution has been kind of a moving definition. It originally started, at least the first time I heard about it.

• 18:35Daniel Whitenack?

  Been evolving?

• 18:36Speaker 03

  It is about evolving. It's combining kind of these evolutionary algorithms. You might have remembered them as things like genetic algorithms from back in the early 2000s, but really neuroevolution is kind of combining these kind of evolutionary algorithms to neural nets effectively. I first came aware of it from a project called NEAT, which I believe was mid 2000s. And the idea was is that it could actually evolve both network structures as well as the weights associated with those neural nets to solve these kind of black box problems.

• 19:11Daniel Whitenack?

  I know jargon is often a point of confusion, but we've talked before on the program about meta learning and sort of learning to learn and different things involved with that. Is that sort of how evolutionary algorithms are being applied to neural nets? You mentioned kind of learning architecture and weights and other things? Or is evolutionary algorithms, is that a sort of different piece of the puzzle?

• 19:38Speaker 03

  Meta learning is much more about, as you say, learning to learn, right? So you can either figure out ways to learn optimizers, to train models, or you can learn parameter weights which make fine tuning on those models a lot faster, such as the case of mammal and reptile. Neuroevolution is more of like a competitor to things like stochastic gradient descent, I would say. It's more of a way of learning models based on these kind of beliefs of populations of answers that can kind of compete with each other and based on a very... Rough estimation of say Darwin, where the best survive, the candidates in the population which ended performing better, end up persisting through multiple generations of work.

• 20:21Speaker 03

  So I'd say it's more common to think of it as kind of more of a learning paradigm. It became a little bit more popular recently, back in, I think just 2017, OpenAI published this paper on how they applied this one particular technique from neural evolution called evolutionary strategies to train agents in reinforcement learning. And they applied it to the standard Atari data sets. And they found out that it was actually very competitive. And so this field, which was much more popular in the early 2000s, that kind of got backburnered when neural nets started really taking off. When I guess 2013, when AlexNet showed that they were actually useful for solving these big problems. We're starting to see resurgence because much of the same reasons

• 21:04Speaker 03

  that neural nets have become successful, neural evolution has become successful. The computation is finally there.

• 21:10Chris Benson?

  For some clarification, can you talk a little bit about, you mentioned, you know, kind of as a replacement for stochastic gradient descent. Could you actually kind of talk about where you might use neural evolution instead of that? Because obviously, as a lot of our listeners and certainly myself, having come into this, we're very familiar with stochastic gradient descent. And can you kind of say where it would be productive to consider neural evolution to replace it in kind of a use case?

• 21:35Speaker 03

  Sure, absolutely. And I can, I can speak specifically to that's his use case. So whenever you can compute a gradient, it's almost always better to use SGD. The problem that you have is that there, there are a number of domains where it's very difficult to compute the gradient of the actual objective function. And you think of reinforcement learning, right? We have this environment where we send in some actions and we get some rewards. And then we get an updated state. But there's, there's no real closed form mathematical equation we can use to try to understand where to step the policy next. And so most of the, the policy gradient methods and Q learning and all of those are really trying to do sort of credit assignment and figure out ways to kind of compute gradients which improve the model.

• 22:17Speaker 03

  Neural evolution is really nice because it makes very little assumption about the Relying objective function. In fact, all it really needs is to be able to know what you're putting into your your fitness function and get some type of fitness score out of it. Where the higher the fitness score is, the better the model is, or the the input spaces that you pass into it. So anytime you have like a situation where it's very difficult to compute the gradient and you need to do it based on sampling or some other form of estimation, it can be quite competitive.

• 22:49Daniel Whitenack?

  In those those scenarios where it might be hard to compute a gradient, is that typically when like when you have such like in in the problem with Etsy, I'm I'm trying to connect this to the search problem with Etsy. It's be is that because there's there's so much diversity in your data set between like query and product match or rank? Where, you know, there's not kind of recognized categories of things, but there's so much diversity. Is that what produces that sort of scenario?

• 23:19Speaker 03

  So in the case of Etsy, one of the challenges we have is that we're a two sided marketplace, which really means we have two customers. We have buyers and we have sellers. And one of the places where things like evolution algorithms had been very successful is in this this particular subfield called multi-objective optimization. And the idea is that relevancy is only one of the factors that go into a healthy marketplace. Uh, let me give you a hypothetical. So imagine that you want to make your your seller successful. Now you have this kind of problem, right? You have sellers who've been on the site for a long time and they're relying on a consistent form of income from the marketplace. But at the same time, you want to make sure that new success, new sellers are also successful.

• 24:00Speaker 03

  And so you need to kind of expose them artificially higher in the rankings to make sure that they can do it. Now, those two needs are naturally in conflict with each other because there's only so much space for search.

• 24:12Daniel Whitenack?

  Yeah, you can't optimize one because you would necessarily kill the other.

• 24:16Speaker 03

  That's exactly right. And so that's called the the rate between trade off between those two objectives is called kind of the Pareto frontier, the Pareto efficiency of the actual problem. And it turns out that it's really hard to do when you start combining a lot of these. Different objectives in there and especially when you start boiling down to things which require relevance to be considered as well, which is a very hard gradient to compute in the best of times. And so you have this kind of black box function which have all these different factors which have these trade offs between them. How can you in a very principled way? Train a model that's able to kind of adjust for those trade offs and kind of learn an optimal balance between all of them.

• 25:00Chris Benson?

  So I'm kind of thinking when you are implementing this and I'm still kind of very focused on how neuro evolution can be implemented in a kind of a practical way, what kind of challenges? Did you find yourself facing in implementing that into your algorithm versus some of the more traditional approaches that might have been more obvious?

• 25:20Speaker 03

  Yeah, it's a good question. I think there were a couple of different challenges. First, neuro evolution is many things, but it is not computationally efficient. Because you're basically relying on sampling. Kind of basically you have like a model, let's say you have parameters of that model. The way you can kind of estimate a gradient step is by sampling slight perturbations of the parameters around that model space and trying to intelligently combine them into kind of a gradient. Which hopefully improves the model's performance. Unfortunately, we've learned from stereotypes of research in zeroth order optimization that as the dimensions of the model increases, you need about a square of that in terms of samples to accurately measure that.

• 26:06Speaker 03

  And so as your model gets bigger, you need to spend more and more time in that exploratory space. And that gets really expensive. Now. Where you can kind of mitigate some of that is by using more efficient languages. Being smarter about the size of your space, being smarter about the type of algorithms that you're using to combine them. So one of these is something called evolutionary strategies and it's actually pretty good. You can combine it with these these second order approximators like Adam or RMS prop or momentum to kind of take advantage of some of the work inside of the classic stochastic gradient sense base.

• 26:45Speaker 03

  To speed up optimization, but it really becomes a question of how do you maintain the efficiency of the search? At the same time, get the results that you're hoping for.

• 26:55Daniel Whitenack?

  So before we move on to some of those things that you just mentioned, like the language, which I particularly want to follow up on, I was wondering if you could just kind of give an update as far as, you know, how did this end up working? Did it improve things by leaps and bounds? Was it marginal? And what are your thoughts in terms of after doing these neuro evolution experiments? What's next in terms of upgrading search at Etsy? Or do you feel like there's other things maybe that are more important to focus on now?

• 27:25Speaker 03

  Yeah, great question. So the way we decide integrated, I mentioned before kind of a rough topology of what a search stack looks like. You have an information retrieval system like solar elastic search, where you get some candidate sets back. And then you go through this cascade ranking system, where you're constantly re-ranking and refining down the result set. And that means you can go from simple models, which are very fast, but are operating on a fairly large candidate set, down to expensive models, which are slow, but are operating on a much smaller one. We put it at the very end. We call it the kind of the business intelligence layer. And it allows us to kind of incorporate both, you know, beliefs or priors about what would be beneficial for the marketplace, but apply it at the end of the ranking. So we're always getting the best possible relevance we can out of our systems,

• 28:09Speaker 03

  but we're adjusting the ordering at the end to try to influence these other factors. From online experiments, it worked about as well as we could have asked for. It is somewhat funny. There were trade offs as well as we find the metrics that we're optimizing for. It's one of those funny things where you almost have to be like a lawyer when you're writing the type of fitness functions for these things to evolve, because it will follow the letter of the law, but it will do it in kind of weird ways. So for example, we were optimizing relevancy at K, right? So precision at K, where you want to get the item that was purchased in the top 10 results. And what we were finding is that it would oftentimes put that purchase at the 10th position. Even though that's not what we actually wanted,

• 28:54Speaker 03

  we wanted to move it higher up, because as far as it cared, all it needed to do is get into that top 10 position, and it was able to make balances up there. So it'll do what you say, not necessarily what you want it to do.

• 29:05Daniel Whitenack?

  Yeah, so it sounds like maybe like part of the future is really now that you have some of these things implemented, really exploring the the policies that you put in place and the strategies that you're using to kind of rein these things in. Is that right?

• 29:20Speaker 03

  That's exactly right. A lot more work on metrics, a lot more work on kind of understanding what the trade offs in the marketplace are.

• 29:27Chris Benson?

  So I know that one of the things that we talked about earlier on in the conversation that I've been kind of waiting to get to, because I'm pretty fascinated with it, is is that you guys are using Rust in your in your line. You've mentioned it a couple of times. And I know kind of Daniel and I are both interested in that. We're both actually gophers and you know, there's a friendly competition a little bit culturally between the Rust and the the Go people. We certainly have a great respect for each other. I'd love to understand how you're using Rust in productizing your machine learning systems.

• 29:59Speaker 03

  Oh, it's a it's a wonderful question. So for folks who don't know, Rust is a language that was developed by Mozilla. And Mozilla, this was back in, I guess, 2010, they had a problem. Mozilla is most well known for its browser Firefox. And every week, it felt like there was some type of security flaw that was being found that they had to release a patch for. And they started looking at kind of the core reasons behind. And they realized that these kind of low level systems languages, which browsers like Chrome and Firefox and Internet Explorer and all those are written in, are really not optimized for solving these kind of common problems that you run into. That can, you know, result in things like buffer overflows, or use after free, or null pointer dereferencing, all

• 30:42Speaker 03

  these kind of problems that you might run into and practice when you're writing in a language like C or C++. So they got together, and they started looking around that kind of modern programming language theory. And they kind of picked and choose some of the best pieces from languages in the ML space, such as Haskell and OCaml, as well as practical pieces such as, you know, ALG-based systems such as C and C++ and try to combine it together with really strong static analysis to produce a language which was both extremely fast and a suitable replacement for C and C++ in systems language, but at the same time, kind of had the static analysis you needed to write safe and efficient code.

• 31:25Chris Benson?

  So just as a quick follow-up, could you describe a little bit about how one would apply Rust in that environment? Does it basically replace the software architectures that are wrapping your machine learning pipeline? Or how does that work? Where does it fit into the overall architecture?

• 31:47Speaker 03

  So I'll talk about more generally, then I'll talk about the Etsy specific case. More generally, the kind of frameworks that most people are accustomed to using are actually written in Python. But the lie about Python is that none of the fast bits are actually written in Python. It's all indexing into C and C++ or Scython, or in some cases, Fortran. And what Python really becomes is this kind of domain-specific language for gluing these together. The ones that are probably most familiar to everyone on the show are scikit-learn, TensorFlow, PyTorch, LightGVM, XGBoost. All of those have the core performance pieces written in C and C++. And they also aren't immune to these problems.

• 32:30Speaker 03

  And so you can actually look and find TensorFlow's how to release patches, because they also, by nature of being in C and C++, have these problems with safety and reliability. So the place where Rust tends to have the biggest benefit is by replacing those backend components with a safer, faster language. And we've had a lot of work recently done in this space to make it a little bit easier to integrate with Python. So there's a project called Py3 out there, which makes it much simpler to interface between the backend and the frontend. And it applies to Etsy. So in the learning to rank space, we have to do a lot of feature engineering. The state-of-the-art is still gradient boosting models for the most part, and that means

• 33:13Speaker 03

  that a lot of the benefits you get from neural nets, that feature engineering piece being deferred to the algorithm, you have to do manually. And we were running into this case where every night we were training hundreds of millions or billions of records, and we were trying to plumb them through a whole bunch of different features, and it was taking an exorbitant amount of time. The second piece that's really challenging in the search space is that the machine learning algorithms are traditionally written in Python, but solar and elastic and these kind of core inverted index based systems are actually written in languages like Java. And what we really didn't want to have to do is write feature engineering twice. So do something like the hashing trick in Python and then have to port that same

• 33:56Speaker 03

  implementation over to something like Java to get the models trained in Python and then deployed on Java. And so we were really looking for a language which would allow us to kind of embed it in both Python and Java at the same time. And that kind of puts some restrictions. So you mentioned that you both are Gophers. One of the problems with Go is that it actually has managed memory. And Java and Python don't necessarily work particularly well with managed memory while managing their own memory. And so those types of constraints made it kind of hone down the number of opportunities we had. And we were mostly focused on trying to find one where we felt we could be productive quickly, but at the same time didn't have to pay a performance penalty.

• 34:36Daniel Whitenack?

  And with that, as you looked into Rust for those particular problems, like you're talking about with feature engineering, but also considered like maybe some of the neural evolution things that you were exploring, I'm assuming that some of those, you know, fundamental or foundational papers, like the OpenAI paper that you mentioned, if you went and looked at the implementation, maybe the model is implemented in, you know, Python and in PyTorch or TensorFlow or something like that. How does that piece fit in along with the sort of feature engineering, preprocessing stuff? Are you taking models kind of from the one frameworks and then doing a lot of the feature engineering and that sort of thing with Rust?

• 35:21Daniel Whitenack?

  And how does that play together?

• 35:22Speaker 03

  So we really have two main systems that are written in Rust. Both are powering, you know, hundreds of billions of predictions a day. So our first one was BuzzSaw, which we wrote a paper on back in 2018. And it really is kind of the backbone of how we do feature engineering at Etsy at this point. We pump a lot of data through it, it can scale across clusters, we can embed it inside Python and inside the Java. And it's really nice because when you're when you're training models, especially in the search space, you want to make sure that what you're training against doesn't change. And so you can imagine that by adjusting the implementation of, say, TFIDF just slightly, you can actually have these changes to the prediction space. But because we're able to ship the library to both our cluster compute, which does the preprocessing,

• 36:07Speaker 03

  and run that exact same code in our Python-based learning to rank prediction services, we don't have to worry about that gap in terms of implementation. As for the neuroevolution space, as I mentioned before, it's not super sample efficient. So there's been a lot of work around trying to figure out how to scale up these systems. And one of the way we were very successful in doing that is by moving the neuroevolution pieces down into Rust rather than from Python. And so when we originally prototyped this out in Python, it was, it worked. It was slow, but it worked. But by moving it into Rust, the main core implementation, we were able to speed it up by some hundred x and reduce the memory footprint of it

• 36:49Speaker 03

  and just scale up both of our data as well as the size of the models.

• 36:53Daniel Whitenack?

  Was that sort of reimplementation overhead? Was that high or did you find it going fairly smoothly? And I'm not sure about, you know, maybe you had experienced Rust people and that sort of thing, or was it more like you have experienced AI, like Python people, and they're kind of dipping into Rust?

• 37:11Speaker 03

  I would say it's more the latter than the former. Rust is a new language by the the general timelines of language. I mean, C++ came out in 1985, I believe, and Java was 98. Python was 1990. So there's been a lot of time to kind of bake engineers there. Rust only hit 1.0, I believe, back in 2015. So it's still new and there's still kind of this community building that's going on. So most of our developers are coming from, I know Python, I know a little bit of C++. You know, I had to deal with Java in school, but that's largely my experience in languages that I use on a day to day. From an implementation perspective, swapping from Python to Rust, there was some cognitive overhead in terms of

• 37:54Speaker 03

  How to work with static analysis, learning how to use these kind of more advanced features that come in the language to your benefit. And then there's just the the nature that when you're writing code in Python, you're really gearing for kind of a prototyping space. You're not really necessarily thinking about performance. When you move to a language like Rust and you're trying to do it to squeak out performance, you have to think about things such as memory allocation, using SIMD, potentially using CUDA, and how those all kind of play in to building robust systems.

• 38:24Chris Benson?

  So I'm kind of curious, you're recognizing that Rust is still a very new language and kind of, you know, scratching that itch, you know, of the languages that it's replacing. As you're looking further into using it in the kind of AI ML space, do you think that the community, though it be small today, is likely to grow and develop going forward? Do you think it's a substantial enough use case for Rust to really kind of blossom in that area?

• 38:51Speaker 03

  I do. I don't think we're necessarily all the way there yet, but I think there's a bunch of indicators that are really positive for it. First, big companies are starting to depend on it. So Dropbox, for example, uses Rust in their storage layer because they need reliability. Facebook came out earlier this year with their cryptocurrency, which is written in Rust because they need the security. Microsoft, just a few days ago, published a result of a project where they were trying to replace core pieces of their Windows 10 code with Rust because they got tired of security flaws. So we're kind of seeing a lot of big companies starting to adopt it, at least in prototyping cases, and more seriously in the case of companies like Facebook.

• 39:33Speaker 03

  I think that it is going to be really valuable because the one thing that I've noticed for us is that there's really kind of three payments that you have to make, right? When it comes to building machine learning, there's the cost of development, there's the cost of running, and then there's the cost of maintenance. And Python does a great job at the cost of development, right? We have this rich ecosystem filled with the pet-like libraries out there. You can just pip install them, they just kind of work. You can prototype an idea very quickly. But it's not an efficient language, so that cost to run starts adding up, especially when you have to clone it over hundreds of machines to train these very large models. And the final piece, and this is the more insidious one, is the cost of maintenance. What is the cost of failure in your systems?

• 40:18Speaker 03

  And when we think about things like production, Whenever our learning to rank models go down, we start losing money really quickly. We have 18 million results for jewelry, right? Turns out that there are better answers in there than others. And so we need to be very cognizant of what that long-term effect actually is when it comes to building out quote-unquote production.

• 40:38Daniel Whitenack?

  Yeah, and I mean, that's something that's definitely resonates with with us like we like we mentioned, you know, having worked a little bit in Go. I'm really curious to, you know, start dipping into Rust a little bit and get my hands dirty. Are there good places like, let's say, for people that have some experience doing Python stuff in machine learning and AI? What are some good ways for them to start getting into Rust a little bit? Are there kind of tutorials or machine learning parallels that they could go to out there in terms of common machine learning problems and that sort of thing?

• 41:16Speaker 03

  Yeah, it's a really good question. So I will say this about Rust is that I feel like in marketing right now, but the truth is, is that it's the nicest community that I've ever run into. So the IRC channels, the Rust subreddit, just generally the way Mozilla has run the community is the most welcoming I've I've ever seen.

• 41:34Daniel Whitenack?

  Yeah, with Mozilla behind it, I think one could expect some good things in that sense.

• 41:39Speaker 03

  And they've done a really good job of kind of baking it and kind of, I heard a really interesting quote recently, which I think does kind of resonate. You know, Google built Go for Google, right? They didn't necessarily build it for the community, whereas it really does feel like Mozilla kind of built Rust for the community. And it just so happened to also work extremely well for them. And that really is kind of permeated through all layers of it. And so you can find good books on Rust on kind of adopting it from other languages. You can find a lot of GitHub repos. I know there's not much value in popularity contests, but it's one, I believe, the most loved language on Stack Overflow for like five years running now. So there's a lot of joy kind of involved in that space. And there's lots of people very eager to help you with your problems.

• 42:24Daniel Whitenack?

  Great. Well, as we come nearer to a close here in our conversation, we've talked about search, we talked about evolutionary algorithms, we've talked about REST. I'd be curious just to hear about, you know, what you're excited about in terms of the search space and how AI is influencing that space as we look forward. You know, what are the biggest open problems that you think, you know, are really interesting that people are working on? Or what are you just excited about kind of over the next years as the community grows?

• 42:58Speaker 03

  Yeah, that's a wonderful question. So we've never lived in a more exciting time for search. We have this kind of openness now in industry in terms of publishing, you know, state of the art. You have Alibaba's been making a lot of work recently and then the classics like MSR and Google and other folks who are really publishing great research on how to both build production-grade systems and how to kind of push the state of the art in terms of retrieval. At KDD this year, one of the big workshops was from LinkedIn and they published this fabulous deck on how they've kind of integrated machine learning at every level of their search platform. So using things like GANs and using BERT to

• 43:39Speaker 03

  kind of tease apart the NLP pieces. Then you have great papers talking about how folks like Amazon are embedding billion parameter neural nets inside of their information retrieval stacks. So these real production problems that folks like Chinese e-commerce companies deal with, such as Singles Day, right? How do you handle the scale of those systems? And one of the things that I've been starting to see as a trend over the last few years is that the blending between lines of where machine learning starts and distributed systems and systems engineering starts is starting to be a little bit fuzzier. And it turns out that the best search systems are really going to incorporate techniques from both worlds

• 44:23Speaker 03

  into the code that's being built, rather than kind of having them segregated apart. So all the improvements that we get out of conferences like NERIFs and ICLR and SIGIR and all these wonderful conferences, we're finding they're making their way faster into search to actually solve real problems.

• 44:41Daniel Whitenack?

  Awesome. That's great to hear. And I definitely resonate with a lot of what you said, like I mentioned, and I certainly hope that we do see some of those trends. And of course, we'll keep looking for great things coming out of Etsy and what you're contributing to search. And really appreciate you, you know, releasing your findings and other things, like you said, with BuzzSaw and other things. So yeah, great work. And thank you so much for taking time to talk with us. I really appreciate being here. Thank you.

• 45:14Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Leno cloud servers. Head to leno.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop

• 45:56Speaker 00

  in your email address, get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.