Reinforcement Learning for search

Hamish from Sajari blows our mind with a great discussion about AI in search. In particular, he talks about Sajari’s quest for performant AI implementations and extensive use of Reinforcement Learning (RL). We’ve been wanting to make this one happen for a while, and it was well worth the wait. Featuring: Hamish Ogilvy – X https://x.com/hamishogilvyChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Sajari https://www.sajari.com/Blog post: “Reinforcement Learning Assisted Search Ranking” https://www.sajari.com/blog/reinforcement-learning-assisted-search-ranking/Blog post: “Query Understanding 101” https://www.sajari.com/blog/query-understanding-101/Blog post: “The Inevitable Collision of Search and AI Tech” https://www.sajari.com/blog/search-ai-tech/Special offer from Sajari for Changelog listeners https://try.sajari.com/changelogUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  You can take reinforcement learning from an AI standpoint, or you can take reinforcement from a statistics standpoint. So, you know, people would probably be familiar with things like Optimizely that use bandit algorithms for optimization. And so that's one of our favorite techniques is actually a Bayesian bandit style reinforcement learning. And the reason for that is that it just works with very little overhead as opposed to building, you know, these big, complex machine learning models where anything can happen. And yes, you can probably do a lot better in a very fixed use case. But if you have broad use cases and lots of different variables changing, it's a different story.

• 0:39Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:56Speaker 02

  Whether you're working on a personal project or managing enterprise infrastructure, you deserve simple, affordable and accessible cloud computing solutions so you can take your project to the next level. Simplify your life with Linode's Linux VMs to develop, deploy and scale your applications faster and easier. Get started on Linode today with $100 in free credit for our listeners. You can find all the details at linode.com/changelog. Or if you're not at your desk, just text CHANGELOG to 474747 and get instant access to that 100 bucks. Linode has 11 global data centers and provides 24/7/365 human support with no tiers or handoffs, regardless of your plan size. In addition to shared and dedicated compute instances,

• 1:41Speaker 02

  you can use that $100 credit on S3 compatible object storage, manage Kubernetes and more. Visit linode.com/changelog and click on the Create Free Account button to get started, or just text CHANGELOG to 474747. Get started today on Linode.

• 2:05Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:32Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whiteknecht. I am a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 2:51Chris Benson?

  I am doing very well, Daniel, enjoying this beautiful fall weather. And how's it going with you?

• 2:56Daniel Whitenack?

  Going good. It's a busy week. There's a conference paper deadline tomorrow that I'm trying to meet. So last night, and I'm guessing tonight, will be a little bit restless for me.

• 3:10Chris Benson?

  I'll bet. I mean, has anyone out there ever actually submitted their conference paper before like this final moment?

• 3:17Daniel Whitenack?

  Actual day of?

• 3:19Chris Benson?

  Yeah.

• 3:19Daniel Whitenack?

  I mean, when you have to make like three LaTeX tables that like each one, you know, the tweaking of those adds like each one adds like a day or something.

• 3:30Chris Benson?

  Yeah, I'm terrible about that. I'm like, yeah, coming down to the final hours.

• 3:35Daniel Whitenack?

  Yep, but it's a good time. I enjoy it. And we're collaborating with some interesting people. So it's, it's good to do that and good exercise to kind of organize all your thoughts into that paper format. It's, it's kind of a good. But yeah, what's going on in your world?

• 3:52Chris Benson?

  I am actually trying to do as little AI as possible right now. The weather has been just too nice outside. And if I even look at my laptop, my wife gives me a scathing look like, you idiot, what are you doing?

• 4:04Daniel Whitenack?

  Are we not like hiking?

• 4:05Chris Benson?

  Exactly. Look outside the window. Yeah, I spent most of my, so we're trying to get some, some outdoor time in when we're not working right now.

• 4:12Daniel Whitenack?

  Awesome. Awesome. Well, Chris, I don't know if you remember, actually, this is kind of a long story, but part of the reason why this podcast exists is because I met our editors, I met them at GopherCon, I believe it was in 2016. So I do a little bit of Go programming, as our listeners probably know. And I met Adam Stokoviac, and actually, I think it was just Adam there, who is editor in chief at The Change Log. And that's actually what got us started talking about doing an AI podcast. Well, at that same conference, I met our guest today

• 4:53Daniel Whitenack?

  and have been following their company for some time and using some of their packages that they've open source. And they're just doing really cool stuff. And so today, we're joined by Hamish Ogilvy, who is a founder at Sajari, talking to us all the way from Australia, I believe. How are you doing, Hamish?

• 5:11Speaker 00

  I'm good. I'm in San Francisco, though.

• 5:13Daniel Whitenack?

  Oh, really? Did you move?

• 5:16Speaker 00

  I did. I did. I had to move across. Yeah.

• 5:20Chris Benson?

  So from our perspective, it's earlier today rather than tomorrow.

• 5:24Speaker 00

  That's right.

• 5:25Daniel Whitenack?

  There you go.

• 5:25Speaker 00

  That's right. Yeah.

• 5:27Daniel Whitenack?

  Awesome.

• 5:28Speaker 00

  Not Tuesday yet.

• 5:29Daniel Whitenack?

  So is the whole company now in San Francisco?

• 5:32Speaker 00

  No, most of our guys are in Sydney still. We have a few people in Vietnam, New York, Ohio. And then there's a few people here in the Bay Area as well.

• 5:40Daniel Whitenack?

  Yeah. So were you all doing kind of remote workers before it was cool and necessary to do remote workers now in this world?

• 5:50Speaker 00

  I'd like to say we were, but we weren't. We were actually, our engineering, we always had in one location, which was in Sydney. And I was probably the exception to the rule. But I came across to build our team up in the US. But we always had an in-house, like no remote engineers, because some of the stuff that we do is so complicated that we often fill whiteboards and argue for two hours at a time before we make a plan to do anything. And so COVID's been really interesting, because that's kind of flipped everything. And everybody's working from home, although it's not too bad in Sydney at the moment. The guys still get into the office. Yeah, we've, we've talking about reducing office space and everyone's working remote. So it's, it's definitely a different world.

• 6:35Daniel Whitenack?

  Yeah, I would say there's a lot of things about working remote that can be really efficient. But that sort of like trying to get a new effort underway and like brainstorming and whiteboard time, it's pretty incredible when, when you're in person and I'm not sure that I've experienced anything of that sort of nature in, in a remote setting.

• 6:57Chris Benson?

  There's a business opportunity right there.

• 6:59Daniel Whitenack?

  Yeah, I'm sure there's like virtual, like

• 7:02Chris Benson?

  Remote team whiteboarding that's actually really amazingly wonderful, which pardoning, you know, those that are already in that space. I, I haven't experienced it either, you know?

• 7:11Daniel Whitenack?

  I think I'd have to have some type of special surface to write on or something because then otherwise drawing with my mouse, it'd just look like something a four-year-old would draw or something.

• 7:23Speaker 00

  Even if you go past that, you've still got time changes and stuff for us as well.

• 7:27Daniel Whitenack?

  Oh yeah, sure. That's rough.

• 7:28Speaker 00

  Yeah, it's brutal whichever way you look at it, but it's definitely not solved, that's for sure.

• 7:34Daniel Whitenack?

  Well, um, as interesting as that topic is, I think you and I have talked a bit about what you're doing at Sajari and, and some of your work there, but I actually don't know your kind of personal background and how you got into the space and, you know, eventually ended up founding Sajari. So maybe you could give us a little bit of that story.

• 7:53Speaker 00

  Yeah, it's, um, I'll try to give you the short version because it's probably a long story, but my background was actually originally in physics. I got a PhD in physics and

• 8:01Daniel Whitenack?

  Score another one for the physicists.

• 8:03Chris Benson?

  No kidding. You guys are just all over the place.

• 8:06Daniel Whitenack?

  Yeah, we have a few on here occasionally.

• 8:09Chris Benson?

  We're broken out and physicists have gone to AI. What's going on with that?

• 8:13Speaker 00

  I know. It's a, it's a common path. It's, it's funny, but, uh, I was one of the, the rare few when I first kind of made that path into the space. And then now I think the university's got full programs set up around transitioning people down that, that line. Well, yeah, so I did PhD in physics, used to design lasers, um, so some of my las- lasers are still used for surgery and skin treatments and and other things, but I shipped out into analytics. Worked in analytics for probably four or five years and then jumped out to start the business. So, yeah, different route.

• 8:48Daniel Whitenack?

  Maybe you could just, uh, uh, uh, cause I'm interested in kind of the backstory of of Sajari a bit. Um, but maybe before we get into that, maybe you could just say kind of in a short blurb, what Sajari is, what it's trying to do.

• 9:01Speaker 00

  Yeah, so we're basically trying to offer machine learning based search as a service, but we want it to be really, really fast. And so I guess one of the trade offs that people make is speed for accuracy. So we're trying to kind of mesh the two of those things together in in in a short way.

• 9:18Daniel Whitenack?

  Cool, yeah. And how did that idea, how did the sort of um AI and search, but also the performance side of that? How did that kind of creep up as something that you thought was a space that you wanted to live in?

• 9:32Speaker 00

  Yeah, so it's really interesting, but such kind of interested me even back during my PhD, cause I was doing something really obscure. That was very difficult to find literature on and you'd you'd go to search for things and type in. You know, if you typed in two keywords, you get the phone book. If you type in three, you get very very few results. You type in four words, you get nothing. And that's just a space because it was very small and it it always seemed really bizarre to me because I'd written a bunch of papers. I I had, you know, enormous volumes of context in in text that I'd actually written. And yet I couldn't use that to actually search and find something on context. And so

• 10:15Speaker 00

  that kind of sparked my brain, I guess you would say, because it always seemed like we should be able to search more conceptually. And then the university actually spun out some of the research that I was doing during my PhD into a company. And one of the first things that they did was an IP literature search and then the patent emerged all over again, the the IP search company was basically, they had a um a spreadsheet and they said you keep. Filling it up with queries until you keep getting the same results. And so it just looking at that was kind of crazy. And so fast forward, I I kind of ticked along in the background and chatted to to some friends about it and um played around the background and couldn't really get any of the technology out there to do what we wanted to do.

• 10:58Speaker 00

  And kind of saw that there was this convergence of, you know, machine learning was starting to come up. Search technology was basically mostly built from around the nineties. It hadn't changed too much in approach. It's changed a lot uh more recently. But you could kind of see that there was this convergence of smart machine learning and information retrieval. And so we kind of we jumped on that train and been following it ever since. I think the fast side of things, uh when we're talking previously about making things really fast. I don't know if you guys saw, but there was a study a few years ago, Amazon, they intentionally added blocks of a hundred milliseconds delay in their e-commerce queries and saw a drop of one percent

• 11:42Speaker 00

  of revenue for every hundred milliseconds delay they introduced. I think Google did the same thing and saw a twenty percent drop in click through rates and things like that. So there's this huge correlation between speed and performance. But then, you know, a lot of the things that you do in AI can be quite hard to make performant. And so there's just these interesting challenges where you're trying to balance.

• 12:04Daniel Whitenack?

  That's definitely true. And it's like, you'll read about this cool model and you want to make it work, but then you read about like, oh, how do you make it performant? Oh, you like get like this like GPU server.

• 12:16Speaker 00

  Take the smart stuff.

• 12:17Daniel Whitenack?

  Yeah, that like just like brute force it, which not all of us have the have the capability to do. Maybe some fortunate people do.

• 12:25Speaker 00

  Yeah.

• 12:25Chris Benson?

  So as we kind of get going here, and and I know Daniel's known you for a while, but I'm kind of, as I'm learning as a as a novice here. Is like, can you differentiate a little bit more uh or or talk about kind of the specific areas that you're addressing versus a Google? So like, um, you know, with obviously AI being a topic that's in search kind of broadly at this point.

• 12:46Speaker 00

  Yeah.

• 12:46Chris Benson?

  What areas are you guys really addressing? And you know, how does that differentiate you from the others that are out there?

• 12:51Speaker 00

  Yeah, so interestingly, we jumped into site search around twenty fifteen when we were first kicking things off. And the reason that we we did that was because Google was talking about getting out of site search. And I don't mean searching the public web, but I mean searching your, you know, your internal website.

• 13:09Daniel Whitenack?

  Right.

• 13:09Speaker 00

  And so they retracted back and sunsetted the Google search appliances, those bright yellow boxes that people used to put in their in their data centers. What a different world it is, but uh they also got rid of their their product where they were doing that as a service and basically said you could use ad-supported or find another product, basically. And so we saw an opportunity there and it was a it was a good testbed to start in because all of the data was public already. So you didn't have security implications and privacy. So it was kind of an easy area to get started and and I've done so much in the background in um marketing analytics and site tagging and things like that, that we saw that we could basically automate that that entire workflow.

• 13:54Speaker 00

  And so that was the origins of the company. And then now as we move forward, we're kind of going where the transactional value of search is highest, which is e-commerce search, where you're just seeing that you can, you know, you make small changes and they mean millions of dollars and you can measure it. Uh which is very different to searching a a website where people they they don't care as much, if you know what I mean.

• 14:18Daniel Whitenack?

  I'm kind of curious on the site search side of of things i- in terms of how big and how many like results a site needs to have to like really like until this really starts becoming a problem. And I was wondering if that's at all related to as well, like you can put in a lot of design work in terms of making your site easy to navigate. But it seems like you're, if you're able to search something and have it come up sort of instantly and it was a good result, then it also maybe reduces the burden there. So um could you talk to that a little bit in terms of like?

• 14:54Chris Benson?

  The scale that we're talking about and sort of when this type of thing becomes a really big problem?

• 15:01Speaker 00

  Yeah, I mean, I think it's interesting because remember websites back in the day had these enormous navigational structures and, you know, there's five hundred links on a page and then now you go to sites and there's basically nothing. And so we've had this interesting move where everything's gone to mobile and people have an expectation that they can just search and find what they want. And so you find that you don't need that much content before it actually becomes useful. And the second aspect to this is that in user testing, we've actually had customers that have come to us and they said in user testing, the first thing people do if there's a search icon, they go to the search icon. They don't want to be bothered, you know, going through five different nested navigation structures to find what they want if there's a search box there, but it has to work.

• 15:44Chris Benson?

  I was just gonna say, I've noticed that, um, especially like I will go through other uh, other avenues myself, but I do notice like my wife, she goes to the search box for absolutely everything. I mean, like everything, uh, and like almost doesn't use navigation other than that.

• 16:02Speaker 00

  And that's user testing showing that more and more and more. It's just becoming a, a base level expectation, which is, is kind of interesting. But the other side to that is that you actually get people's intent. So we talk about it in terms of intent analytics, but like a cable TV company that's a customer, for instance, cancel is their biggest search. And when people search for cancel, they may happen to know their account ID so they can automatically do things like feed retention programs. You can't do these things as easily as if people are navigating around because you don't know um, specifically what their intention is, but if they type it in, you got a pretty good idea that you know what it is. And so the other thing that we see is that if you have searches that are by far and away

• 16:45Speaker 00

  bigger than anything else, then they probably should be navigation components. And so there's a whole suite of sort of this, um, I guess it's a workflow way you can just continually improve user experience.

• 17:00Chris Benson?

  I was gonna say, it almost informs the UX of the site itself, so you can evolve it with the results from the search. That's great.

• 17:07Speaker 00

  That's right. Yeah.

• 17:08Daniel Whitenack?

  It's really interesting. I wasn't thinking along that vein. It's almost like, like you're treating the search box, because I've worked a bit in chat and other things, and it's like, you're treating that query almost as like a chatbot input. Not that the user is wanting a chat response, but you're trying to detect like potentially cities and those sorts of things and track those over time. That's super interesting.

• 17:33Speaker 00

  Yeah.

• 17:33Daniel Whitenack?

  Because users' website data, a lot of times, is something that they hold near and dear and don't want to like, they don't want to give their, their data out, or, you know, all of that sort of stuff. So especially like usage type things. So has that been an interesting challenge in terms of like bringing AI to someone's site, or maybe their own infrastructure versus like them, or maybe it's not, maybe they reach out to Sajari Service and like as an API or something? How does that balance work?

• 18:09Speaker 00

  Yeah, so we do get a lot of companies that ask to host our stuff internally, and we don't do it at the moment. We've just chosen to offer a service for now, which is tough, because we know that we could do some really interesting things. And there's been banks and defense and investment companies that have, you know, treasure troves of data, and they want to be able to actually start doing more interesting things with it, but there's no way they're going to ship it out to third-party providers. And so we haven't gone down that route yet, but we do keep it in mind for, for future, for sure. Yeah.

• 18:45Chris Benson?

  As Daniel was asking you that, and I was listening to you talking, it made me think, I suddenly realized that we all probably assume that we know search really, really well over the years, but I know it's evolving very rapidly. It just occurred to me that I probably don't know search nearly as well as I think I do. So actually, if you would take just a moment and maybe kind of give us a landscape of how searches have evolved, what is search right now as we are at the end of 2020? And, you know, what, what is different from the way we might have thought of it a few years ago, you know, when we were originally thinking we were learning it so well?

• 19:22Speaker 00

  Wow, where to begin?

• 19:23Chris Benson?

  I know. It's wide open. You can go anywhere you want on it.

• 19:26Speaker 00

  Yeah, it's really interesting because when you see things like A I and machine learning or where they can apply in search. It's not just one area, it's many, many different areas that that they can apply. And so I think like historically. You know, we saw search engines split away from databases because you wanted to be able to run these really long tail language based queries. That used to just absolutely crush databases. And so search engines enabled you to do basically. Write out indexes that were immutable that you could have very high query concurrency on. But then that came with trade offs where you couldn't update the data as fast. And then. We fast forward through the years and machine learning became more and more involved in search. And so.

• 20:10Speaker 00

  I guess we saw a transition to learn to rank models where people could. Um, re-rank a set of results with machine learning and and that started becoming popular. We've got N L P, uh, entity extraction and things like that. So if you go to a e-commerce site and you search for size fourteen. Black shoes, then you should get size fourteen black shoes, but most often you don't. Which is really interesting. And so there's this sort of. Language extraction aspect to it. And then you've got things like reinforcement learning, which we're big fans of to. If, say, you don't want to build a model and and one of the challenges that we have is that we have hundreds of customers. And so.

• 20:53Speaker 00

  We can't go and build a custom model for somebody paying on a credit card. It just, it's not, it does not economical. So. How do you improve performance for them without the um added overhead? And so. Reinforcement learning is is one of the ways that we see that um work really well. So there's a lot, I mean. It's it's a hard question to to answer, really. It's so broad.

• 21:19Daniel Whitenack?

  Yeah, it's interesting to me. Um, of course, we've talked about reinforcement learning a few times on the podcast and. You know, I think particularly in relation to robotics. Or or maybe automation or autonomous vehicles and that sort of thing. So I think maybe people might not be as familiar with the idea of reinforcement learning in search or also like marketing analytics type things, which I've also heard. So you said you were big fans of reinforcement learning. I know there's still some people out there that are like, you know, maybe think that it's not quite there in terms of like being practical yet or something like that.

• 22:04Daniel Whitenack?

  How has your experience been over the years? When, when did you start thinking that this is really practical for our use cases and has the tooling around it to make it useful, I guess?

• 22:16Chris Benson?

  And we may want to define it along the way, by the way, just for those who have joined who aren't familiar with it.

• 22:20Speaker 00

  Yeah, that's the interesting thing in defining it. I mean, you can take reinforcement learning from an AI standpoint, or you can take reinforcement from a statistics standpoint. So, you know, people would probably be familiar with things like Optimizely that use bandit algorithms for optimization. And so that's one of our favorite techniques is actually a Bayesian bandit style reinforcement learning. And the reason for that is that it just works with very little overhead as opposed to building, you know, these big complex machine learning models where things you, anything can happen. And yes, you can probably do a lot better in a very fixed use case, but if you have

• 23:00Speaker 00

  broad use cases and, you know, lots of different variables changing, it's a different story.

• 23:07Daniel Whitenack?

  Yeah. Yeah. So could you kind of describe, I guess, then like from your perspective, where reinforcement learning is specifically applicable to, to search? Like what sort of tasks are, because I think most people, when they think of search, they think of, oh, I put in a query and then I get a result. But of course you're doing much more sophisticated analytics and, you know, intent mapping and routing. So where does it fit into that puzzle from your perspective in terms of tasks?

• 23:35Speaker 00

  Yeah. So, I mean, I can give you an example, say with COVID-19 this year, you, if people went pre-COVID and searched for face masks, that, you know, you can imagine women sitting at home in front of the TV with the face masks on and then COVID hits and everybody's looking for N95 respirators. How do you evolve? Like when that change started to happen, how do you know when someone's searching for face masks? Do they want the face masks of the past or do they want an N95 respirator?

• 24:04Chris Benson?

  That's a, that's an example where.

• 24:05Daniel Whitenack?

  It, it took me a second to get there. I was like, I don't think about that first type of face mask. Talking about.

• 24:13Chris Benson?

  It took me a moment too.

• 24:14Daniel Whitenack?

  You're talking about like the, like the, the goop you put on one's face.

• 24:19Chris Benson?

  The green face, right? With the towel.

• 24:21Daniel Whitenack?

  Yeah, yeah. Sorry.

• 24:23Chris Benson?

  Mud mask or, yeah.

• 24:26Daniel Whitenack?

  No, no, I, I had my own search difficulty in, in my mind.

• 24:31Chris Benson?

  Context, right? Daniel, we've really shown, you know, we're not thinking expansively enough. Shame on us.

• 24:38Speaker 00

  It's a common problem. And I mean, I guess it alludes to the same problem that you have in search context. What is the context and text in particular is highly ambiguous. You can use the situation to help you. You can use past queries. So you can see if there's a pattern in what people are doing. But overall, it's, it's ambiguous. And this is where keyword search kind of falls down. And another example that we saw quite a, quite a lot of is, um, as seen on TV. Uh, so if you type TV, you could possibly match on things that say as seen on TV. Because they have TV in the, in the text, but obviously there'd be poor results. But how do you know that?

• 25:18Speaker 00

  A word is a word. And so one of the interesting things is that AI and word embeddings and other things have been able to transition that out into vectors. And so you can start to actually see the context that something's used in to understand what the meaning is. So when you search for TV, you get TVs. You don't get as seen on TV. And so there's a whole host of interesting ways to try to embed that context into the way that the, the indexes work. But going back to reinforcement learning, the reinforcement learning is really interesting because it's trying to look for the maximum reward. And so in the case that we mentioned before, the N95 had a high purchase rate.

• 26:01Speaker 00

  And so that's rewarding the algorithm to move more towards that and optimize the results. So the medical masks, so the cosmetic, um, style masks, and that happens in a relatively short timeframe. And so then you have trade offs in timeframe, you know, if you put a six month timeframe, it wouldn't have changed. You put a two week timeframe on, it would change, but you've got less data, so you've got less certainty. So there there's a whole host of different things that that play into how that would work. But yeah.

• 26:32Daniel Whitenack?

  Yeah.

• 26:32Speaker 00

  It works really well.

• 26:33Daniel Whitenack?

  I I feel like I'm almost asking it as a tangent, but I was just curious as we're talking about reinforcement learning. Just out of curiosity, are there other architectures that you're finding very common? Or is reinforcement learning really kind of the dominant architecture, given the use case that you're addressing?

• 26:48Speaker 00

  It's really interesting. I mean, I think learn to rank style stuff is probably more popular, I feel like. But reinforcement learning, I know a lot of people have picked up more reinforcement learning, particularly recently. But Microsoft were big pioneers of that. There's a whole host of research articles on Microsoft. I think they have a whole unit dedicated to to reinforcement learning. Yeah, but it's for us, that's always been one of our main things. And it's probably getting more and more popular because it's it's easier than learn to rank models and such, for sure.

• 27:22Daniel Whitenack?

  So one of the things that even in like my workshops that I teach, and this topic comes up kind of a lot. I I like searching around for like tooling for reinforcement learning and and all of that. And there's like some things that have come onto the scene, they seem like they're sort of rapidly advancing. Have you found yourself having to sort of roll your own internal kind of infrastructure around reinforcement learning?

• 27:51Speaker 00

  We rolled our own for performance benefits and it's it's probably quite different. But we actually bake some of that information into the indexes themselves.

• 28:01Daniel Whitenack?

  That makes sense.

• 28:02Speaker 00

  Yeah, yeah, it it does, except the implementation is very difficult. Because you you actually have to be able to live at it, the reverse indexes, which historically in search is a is a no no. Basically, I mean, search indexes have been immutable, particularly the scoring. And I think it was a few years ago, LinkedIn, they released a few different libraries around actually being able to update the scores for intersections in reverse indexes in place. And part of the uh logic behind that was they don't have to rebuild the entire index every time they do it. And so. It's not a common thing though. Um. It's it's one of those things that if you're used to immutable index structures,

• 28:45Speaker 00

  you would probably not even try to do this because it would seem so foreign. But because we built our own index from the ground up, we were thinking about this from day one, basically. So, yeah, probably makes it a little bit different, I guess.

• 28:57Daniel Whitenack?

  Yeah. And for those out there that may be a little bit more new to search, when you're immutable indexes and uh and reverse indexes, could you just explain sort of briefly how that fits into search and how it, you know, would interact with a model or something like that?

• 29:16Speaker 00

  Yeah, so a reverse index is basically like when you read the back of a book and you have a list of uh terms that link to the pages that those terms actually exist on. And uh it's very similar, but the the links are to actual document IDs. So for any given term, you can tell which documents that term appeared in, where it appeared in the document. And we also include things like the context of how it occurred in the document as well. That's what reverse indexes do. And what was the second part, Dan? I forgot.

• 29:47Daniel Whitenack?

  Yeah, so just like why in general these would be immutable and like why maybe it's necessary that they not be in the case of applying these more advanced techniques?

• 30:00Speaker 00

  Yeah, so the beautiful thing about immutability is that you can have as many readers as you want, basically because you know the data is not going to change. So as soon as you have mutable indexes, you have to manage concurrent reads and writes, which has a whole heap of locking and other implications that can affect performance and other things. So originally, when, you know, search indexes were first done, they were immutable because of uh speed and performance reasons. And remember back a while ago, disk space was very expensive, CPU is very expensive. And so if you could compress the information down to as small as you could possibly get, that had huge advantages.

• 30:41Speaker 00

  And so historically, search indexes are very well compressed. And they can do that because they know exactly what's in the data set before they write it out into an index. So I'll give you an example, um, delta encoding. If you know the maximum minimum number in a list of numbers, then you can massively cut down the size of the the storage for those numbers because you know that you can basically add an offset to recreate the number. And so there's all these compression techniques that allow them to get the indexes very small. When you're working with mutable indexes, you can't do that the same way because

• 31:22Speaker 00

  you may change a number and then that would change the the boundaries for how things are encoded. And so you just have a a different set of trade offs that you have to deal with. From an algorithmic standpoint, it works okay today with things like Elastic and Solr because people have built in this sort of differential. Uh, when you delete something or when you edit something, they don't actually make that uh, deletion or edit straight away. They buffer them and then re-merge them out to disk. So you kind of buffer up the changes and then you pay for them in one big hit as they're re-merged out to disk. And so just different trade offs there, but from an algorithmic standpoint, that's why

• 32:04Speaker 00

  I learned to rank is typically a second, third, however many passes where you get an initial set of results and then you you basically re-rank them with a model externally. And so the model you can update more frequently and it's totally decoupled from the way that the indexes work. So we kind of went a different route where we said we wanted to have the indexes more involved in that initial ranking process. We wanted to have a better initial ranking process. So it's just a different, different way of solving the problem.

• 32:51Speaker 02

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 33:43Chris Benson?

  So I guess to get very practical, given that we are Practical AI, I wanted to ask, if I am in charge of an organization's website and we are trying to figure out what to do that and I decide, hey, I want to put your toolset into my website to be able to apply search, what does that workflow look like to me as that practitioner who has to actually get it up and running? What am I experiencing? What do I need to know? What kind of prerequisites are there that I need to be thinking about? Can you describe that whole process of how you go implement it?

• 34:19Speaker 00

  Yeah, so there's basically two ways that we implement. The easiest way is just to buy a crawl. So you point it at a website, tell it what to touch, what not to touch, and then we go and crawl that information that's public. You can also link it into internets and things like that. But basically, it's a crawl process. The positive side of that is that it's very easy, very low overhead, no code basically to implement it. The downside is that crawling is a kind of a black art. It's amazingly hard to keep everything up to date and in sequence. You can't, you you you have to know when somebody changed a particular piece of content and then to go and recrawl it, for instance, and so that's difficult.

• 34:59Speaker 00

  So in applications like e-commerce, you find that people wanna have things uh more tightly coupled to their their data. And so in those cases, you have integrations. So we have integrations for things like Shopify, for instance. You can use an API, so you can custom load your data in, which gives you a lot more control. And so some of the e-commerce customers we have do over five hundred product updates a second. And they have, they're updating things like inventory in real time. The performance of things in real time and and other such things. And even the other day, I went to buy a jacket online and it wouldn't go through. It kept telling me I could buy it, but it wouldn't go through.

• 35:39Speaker 00

  And then I just basically gave up and then I went back in twenty minutes later or a half hour later and the jacket was actually sold out. And so you you could tell that the the search and the the front end was not aware that it was sold out. And so it was it was kind of letting me, but then when it went through the purchase, it it wasn't. And so there's this sort of the need for real time data integration is is much more important with things like e-commerce.

• 36:05Chris Benson?

  So would it be fair to kind of, eh given those integrations and given the fact that, you know, that that's a whole bunch of different dependencies that you have to to deal with and those integrations, eh I mean, would you almost think of an implementation sort of as a system of systems where you, you know, based on whatever that user is needing? And as you address that, obviously, you're you've going to, you've talked about the speed and all the things that you've done to make your service fantastic. How do you keep these integrations that are depending on APIs beyond your control from impacting, you know, how, you know, if if something does go down, them thinking that is is your service as opposed to another? How how do you manage that process?

• 36:45Speaker 00

  It's a really good question. It's one that we think about all the time. And at the moment, we're kind of taking this approach of middleware where we kind of, we have a translation layer that that does the connection with the external services, but then once it's inside, it's all kind of the same for us. So it's it's much easier for us to have a consistent view and see when something connecting to external is broken. But there are interesting implications as well, because say things like we we're trying to generate a a decent ranking algorithm for you out of the box. So if we see that you have something like a size or a color, then we're trying to add in the N L P automatically to be able to pick that out.

• 37:28Speaker 00

  So that if somebody does come along and search for size fourteen black shoes, it will automatically map exactly to your categories. And so you will get that exact result. And so there's interesting implications here. If you come along and then you get rid of sizes tomorrow, I don't I don't see why you would do that, then what's the implication there? So there's not only the connecting the data, but there's also how it's inferred in search is is another problem there. And so one of the things that we're trying to do is actually run algorithms in the background and we do quite a bit of backtesting as well. So if you come up with a new algorithm, you can actually look behind the scenes and we can backtest to see would that have given you a a greater return on historical data?

• 38:13Speaker 00

  And what we're looking to do moving forward is actually be making changes in the background, backtesting them for people automatically and then reporting back to them to say, hey, we found a new algorithm that is actually better for you. If you press this button, you can put it into an A B test and then, you know, move it into to life. And eventually I would like to get to a point where we're actually surfacing and allowing people externally to come in and write better algorithms within our system. So it's quite complicated.

• 38:44Daniel Whitenack?

  Yeah, and coming from a company that does manage all of these different models and config for all sorts of different clients, do you have any kind of practical advice and wisdom for i- in terms of model management and sort of managing like um or or maybe even automating some of the training and and updating of models while not also

• 39:11Daniel Whitenack?

  Losing the ability to debug when things go wrong. Any advice there?

• 39:16Speaker 00

  Yeah, I would say version everything and make it part of your entire workflow so that that flows all the way through to the analytics. And so we record analytics for every single key pressed. We know exactly what that flowed through, where it hit, how long it took, everything. And so versioning in that is critically important to making things work. And yeah, that's probably the thing that stands out the most.

• 39:45Daniel Whitenack?

  Do you have to do something like very meta and like search your own search analytics or something to like be able to like find things?

• 39:54Speaker 00

  Basically.

• 39:55Daniel Whitenack?

  Yeah.

• 39:56Speaker 00

  Basically, yeah.

• 39:57Daniel Whitenack?

  Yeah, that's funny.

• 39:59Chris Benson?

  I'm curious, as you look forward, you know, we're kind of going through this time over the last couple of years where voice has become a big mode of making queries instead of everything being through a keyboard um or touch. And and as we look forward and there are other modes of interaction that are on the horizon, how does that change how you think about the process of search on the front end? And maybe, you know, given the fact that not all modes of interaction yield the same information at the same place, does that change how you have to architect your solution to accommodate those? Uh, a- any thoughts on on kind of where you're going with that?

• 40:37Speaker 00

  Definitely. And it raises a really important point because when people use voice queries, they add a lot more context. Google went through this period where the query length, the average query length got shorter and shorter and shorter. And then now we're in a period where it's getting longer and longer and longer. And that is definitely due to the technology, like being able to take longer queries, understand the context. And like when at the start of this, when we were talking about academic search, you know, three or four keywords and you get no results. That doesn't happen today. And that's because there's a better understanding of how to map that text through into the indexes. And so I think as voice gets more and more, you're going to see these sort of hybrid models and.

• 41:23Speaker 00

  Keywords aren't going away. If you search for very specific things, then you wanna get very specific things. And people, if they don't see those specific things, they're less likely to click on the result. So it kind of feeds back into itself, but then you have the hybrid of turning things into vectors and understanding them with models. And so you'll you'll probably see that different queries, some will optimize more to the old style and some will optimize more to the newer style. Yeah, if that makes sense. I don't know if I explained that well, but.

• 41:53Daniel Whitenack?

  It does. No, you did. That was good. So you you emphasized e-commerce a couple of times as you were discussing use cases. And I know that this year in particular has just been like insane in terms of the growth of of e-commerce. And I'm wondering if like that's shifted, like some of the priorities within Sajari in terms of like having a lot more inbound requests for for that type of search. What have you seen in terms of the impact that that kind of whole segment, you know, the explosion of that segment of the industry has, how how has that affected your growth and the company in general?

• 42:35Speaker 00

  Yeah, it's really interesting. I mean, I wish we went to e-commerce earlier than what we did. Uh, in a way, we we always plan to and it it's a natural area for us because we have so much focus on improving the the transaction rate. And so we always talk about the flywheel of improvement. How do you get continuous improvement in search that drives more revenue, et cetera? And I wish we switched into e-commerce sooner, but this year's been incredible. I think it's about three hundred and fifty percent growth in e-commerce queries from e-commerce customers. I mean, they're just growing at such a rapid rate. It's crazy. There's definitely demand there, but then there's also fatigue in e-commerce because some of them have just been trying to keep the machine working.

• 43:20Speaker 00

  I don't know if you you guys wouldn't have seen this, but in in Australia, Kmart brand had a virtual queue. Because they got so busy, you would actually say, you know, you're fifth in the queue. You have to wait three minutes before you can shop online, which is, it it seems kind of crazy for today. But you have to remember that a lot of these companies are still operating on some legacy uh systems that don't scale. And so they've had to deal with those. They've had to deal with, you know, increase in in inventory and all sorts of other problems. And so it's top of mind, but they're also busy, if you know what I mean.

• 43:58Chris Benson?

  I do. I guess, you know, as you're looking forward, you know, as we're kind of winding up here and you're kind of thinking about the future of your company, how you're implementing these algorithms and you're busy tracking the future of AI on one side and maybe the AI and kind of more specifically, the future of search on the other, whether it includes AI or the other non-AI components of it, that's a lot to track together. What are you expecting and hoping to see? And how do you think that will will impact your own service as you develop this over the next few years? What's your whole future outlook at this point?

• 44:35Speaker 00

  Yeah, I think it's a really good question. And in terms of keeping up with AI, it's almost impossible. I mean, you guys do this blog regularly and you probably still struggle.

• 44:45Chris Benson?

  Part of the reason why we do this podcast.

• 44:48Speaker 00

  Yeah, it's just, it's moving so fast. And I think like for us, being able to be pluggable is really important because I think that the AI aspect has to be able to evolve on its own. And from the search side, we're trying to get better at, we have the mutable indexes, like I mentioned before, and we're working on being able to distribute them globally so that wherever you you're searching, it's right by where you are. But that means you also need to distribute models and other things. So there's, there's a whole host of challenges from a dev up standpoint, systems engineering that you need to deal with as well. So I think plugability is one of the big things. Don't, don't try to keep up or stay ahead because you're going to struggle.

• 45:33Chris Benson?

  Awesome. Well, thank you so much for enlightening us on, on so many things. And it's great to connect again. I know I've been wanting to make this conversation happen for, for quite some time. So really happy to connect and hope that someday we're able to see each other again in person. But thanks for chatting with us virtually. It was a real pleasure and looking forward to seeing what Sajari does in the future.

• 45:57Speaker 00

  Thanks, Dan. And thanks, Chris. Great chat.

• 46:00Chris Benson?

  Thank you. Great.

• 46:04Speaker 02

  If you enjoy Practical AI, we would enjoy a five-star review on Apple Podcasts, a blog post in response to something said on the show, and/or a recommendation to a friend or colleague. Those word-of-mouth recommendations really do make a difference. Practical AI is hosted by Chris Benson and Daniel Whitehead. It is produced by Jared Santo, with music by the mysterious Breakmaster Cylinder. Thanks again to our partners who support this show's existence. Shout out to Fastly, Linode, and Robar. That's all we have for you today. We'll talk to you again next week.