Recommender systems and high-frequency trading

David Sweet, author of “Tuning Up: From A/B testing to Bayesian optimization”, introduces Dan and Chris to system tuning, and takes them from A/B testing to response surface methodology, contextual bandit, and finally bayesian optimization. Along the way, we get fascinating insights into recommender systems and high-frequency trading! Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogRudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: David Sweet – LinkedIn https://www.linkedin.com/in/dsweet99, X https://x.com/phinance99Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Books “Experimentation for Engineers” by David Sweet https://www.manning.com/books/experimentation-for-engineersTuning Up | GitHub https://github.com/tuningup/tuningupManning 40% discount code: podpracticalAI19Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Noticing the similarities in the workflow between a quantitative trader and an ML engineer on a recommender system is what kind of gave me the idea for the book. Oh. I thought, hey, you know, this is, you can describe this all as, you know, one instruction book for an engineer and they both could use the same instruction book.

• 0:18Speaker 01

  Being with for change log is provided by Fastly. Learn more at fastly.com. Our feature flags are powered by Launch Darkly. Check them out at launchdarkly.com. And we're hosted on Leno cloud servers. Get $100 in hosting credit at leno.com/changelog. This episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification, practice exams, and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning,

• 1:02Speaker 01

  AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then check out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at O'Reilly.com/changelog. Again, O'Reilly.com/changelog.

• 1:40Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter here at Practical AI.

• 2:06Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with S I L International. And I'm joined, as always, by my co-host, Chris Benson, who is a principal emerging technology strategist at Lockheed Martin. How are you doing, Chris?

• 2:23Chris Benson?

  I am doing very well. How's it going today, Daniel?

• 2:25Daniel Whitenack?

  It's going great. I feel like it's been a productive week. Got made some progress on some projects and have some new collaborations going. And that's always fun. So what about you?

• 2:38Chris Benson?

  Same kind of thing. Been just yesterday was like a 15 hour day for work.

• 2:43Daniel Whitenack?

  Oh, geez.

• 2:43Chris Benson?

  There's all sorts of stuff in the air.

• 2:46Daniel Whitenack?

  Lots of emerging technologies.

• 2:48Chris Benson?

  Emerging technologies, you know? And we were talking in the pre-show about something that I have such a deep appreciation for AI right now, because as I think I've mentioned before, I'm doing flying lessons. And I'm in this ancient 1973 airplane. And every time I'm in it, I'm just like, I'm thinking about us. I'm thinking about deep learning and all the things that we're able to do. And just wishing that, because I need some help, man. I need some help.

• 3:14Daniel Whitenack?

  Yeah.

• 3:14Chris Benson?

  Okay. I'm just saying.

• 3:16Daniel Whitenack?

  Did we not have an episode about autonomous flying races and that sort of thing?

• 3:21Chris Benson?

  We did.

• 3:21Daniel Whitenack?

  Wasn't that a thing? Oh, yeah, you did like, you were like TV host.

• 3:25Chris Benson?

  Yeah.

• 3:25Daniel Whitenack?

  Or like autonomous flying thing or something, right?

• 3:29Chris Benson?

  We had a fellow named Keith Lynn, who was running the show. And he came on, it was AlphaPilot a while back and we were doing.

• 3:35Daniel Whitenack?

  That's right, yeah.

• 3:36Chris Benson?

  So I just want to say, I am doing the opposite of AlphaPilot. And really wishing that some of that amazing technology was available to me. So I'm just saying, I just wanted to come to the show. Because we're Practical AI and saying, there are moments in my life, I need a little bit more Practical AI to help me out.

• 3:52Daniel Whitenack?

  Yeah, yeah.

• 3:53Chris Benson?

  There we go.

• 3:54Daniel Whitenack?

  Hopefully that technology filters down eventually.

• 3:57Chris Benson?

  Absolutely. I'm counting on it.

• 3:59Daniel Whitenack?

  I'm excited today. Occasionally on the podcast, we get a chance to talk to someone who has authored a new book or updated a book or something like that. And of course, I always love that because, you know, authors have put so much thought into the topics that they're passionate about. So it's really good to get their feedback and views on things. Today, we have with us David Sweet. Who is the author of a new book called Tuning Up: From A/B Testing to Bayesian Optimization. Welcome, David.

• 4:32Speaker 00

  Thank you. It's great to be here.

• 4:34Daniel Whitenack?

  Yeah. Maybe before we get into what tuning up means from your perspective, could you just give us a little bit of a background about yourself and how you got into doing what you're doing now?

• 4:45Speaker 00

  Sure. I started out getting a PhD in physics, it was theoretical and computational. So I did a lot of-

• 4:53Daniel Whitenack?

  Another physics guy.

• 4:54Chris Benson?

  I know.

• 4:55Daniel Whitenack?

  I love that. I don't know what percentage we have on the show, but it is a high percentage.

• 5:00Chris Benson?

  And Daniel is a physics guy, too.

• 5:02Daniel Whitenack?

  Yeah.

• 5:02Chris Benson?

  I'm feeling very left out at this point.

• 5:05Speaker 00

  Physicists get spread far and wide. It's a popular subject. And I guess the world's only got room for so many professional physicists. So yeah, so doing a lot of computer programming kind of lent itself to an industrial job. And either in kind of in finance or in technology, and I guess I was an East Coaster from New York. So that kind of drew me towards finance a little bit. But ultimately, I wanted to go into industry because I really wanted to build things that kind of work. Like, kind of like, most engineers kind of have this story. They grew up as a little kid working with Legos or I just had something called an erector set or some other kind of building toy like this.

• 5:48Speaker 00

  The fun of research for me was that I got to do something new. But I felt like there was like something missing in it. Like, the end is you get a paper. And it's wonderful because you get to kind of communicate with the world, but you don't get to kind of wind it up and watch it go, the way like the toys I built and all those. So I still had some of that desire and that's kind of what brought me into industry and finance in particular.

• 6:09Daniel Whitenack?

  So I'm curious, when I was in physics, it was like this urban myth that like, you know, there was a way to go from physics to finance. And, you know, there were these like mythical physics people that made that transition. But it was very unclear to me at the time, like, how does that happen? And like, of course, I, as a physics person, didn't really know anything about finance. And while I still don't really, but yeah, I'm curious, how was that transition for you? And was that something that like was on your mind for a while?

• 6:42Speaker 00

  I can say there are two factors there that mattered. One was that there had been people from my group who had gone into finance before. And so that kind of set the stage for headhunters to come back to the group or recruiters to come back to the group and look for, look for people. So I got a call from a recruiter and he talked to me about what, you know, what kind of jobs there were in finance for physicists and what, you know, the one, the one in particular he had in mind, but he had kind of others in, in backup as well. And that's what really sparked my interest. I had an interest in the finance from the perspective of personal finance, investing. Like I had bought Netscape the day after an IPO. That was my first trade. And of course it was Netscape.

• 7:23Daniel Whitenack?

  Nice.

• 7:23Speaker 00

  It did really well, but it was like, it was very, it was very exciting. It was wow, you know, you can buy something and it can go up really quickly. And so most investments, the most, most purchases or trades don't go like that, but it was a, you know, it had that, had the impact that it had. So that was the first step. The, um, it was that there's people in the group, but then the second step was just going, I went on an interview just to sort of see what, what I thought about it. And the company was a small company with lots of computing resources. And the job was to build a strategy that traded a portfolio autonomously, more or less, you know, almost, you know, almost completely autonomously. And, um, and that was very exciting. That was that, that really was like right up my alley was I, you'd get to do, I get to do coding, which is something I enjoyed doing for a long time.

• 8:08Speaker 00

  You get to do math, building, building the models, but then you get this kind of final piece of satisfaction of just kind of watching it go do its thing. And it's, it's, it's pretty cool.

• 8:17Daniel Whitenack?

  So did you, did you reach that sort of end to end state?

• 8:21Speaker 00

  Well, I was, you know, junior, a junior person at the time. So the, the people who had founded the company had built a strategy. It was a single strategy. So my job was to work on this and, and, and improve it and enhance it. And, and, you know, actually I got to learn how, how that kind of thing worked and at the same time make some contribution.

• 8:34Daniel Whitenack?

  So what, what happened after that? Did you end up staying in, in finance for some time or how did things kind of progress from there?

• 8:42Speaker 00

  Yeah. So I was in finance for, for a long time. My, um, my next step after that was to go and try to build my own trading strategy. And it was at a time when, you know, day trading and intraday trading was something people were doing. Lots of people were doing by hand very successfully. It was right after the NASDAQ.com boom. So I got, I partnered up with a buddy from my research group in grad school. And we just started building. We just went on the web and went to the websites for the exchanges and just figured out how to do this and built a, you know, a small but working trading strategy. And it was super exciting, in part because we got to build our own thing, but in part, or build our own strategy, but in part also because we

• 9:24Speaker 00

  were building our own company at the same time, you know, for the two of us. It was interesting that it was a totally different kind of experience from anyone had had up to that point. Up to that point, I'd always, and I think a lot of people have this experience going through school, is you rely on your practice to give you confidence that you can do a particular thing that you've learned. When you go to build a business, you have to kind of build this meta-confidence that you can figure out how to do new things, which is nerve-wracking, but super exhilarating. It was a lot of, it was a lot of fun.

• 9:56Chris Benson?

  Cool. That sounds cool. So I can't help but ask, you know, being the podcast that we are, and having, you know, you tackled the finance side of things, you know, what is the state of AI? Because I don't think we've had anyone on the show in that specific arena. AI in the world of finance and trading, how do you see it? What do you think of it at this point? What's the state?

• 10:17Speaker 00

  Sure. What I see from talking to, I have colleagues, you know, from the past that have kind of spread out all over to different companies. From like early in my career, people progressed and they go different directions, but we keep in touch. And what I see people doing quite a bit in short-term trading strategies, well, I'll back that up, long-term too, but for different reasons, is I see three things. And one is I see linear models with lots of regressors and automatic feature selection and kind of like pre-deep learning. You may call it linear machine learning, right? So you're building linear models, but not interactively and carefully building, checking each regressor, but putting

• 10:59Speaker 00

  in lots of signals and automatically turning them into a compact model. I always think of Hasty and Tipshirani, they have this book out and this whole methodology, you build a large dictionary of signals and use sparsity and that one. So I see that. I see And deep learning used for interpreting maybe alternative data sets. Sometimes you have short term trading where you get lots and lots of samples. And you need to take something like the order book, which I maybe, and I'll probably all your listeners are familiar with this, but like it's a complex data structure. And so to turn it into something to something you can predict with, you have to do a lot of feature engineering and you can either do it manually or you can use some kind of automated feature, you know, representation learning.

• 11:41Speaker 00

  The other thing I see a lot of is optimization of a simulation. Now, this is like old school, this is like nineteen, maybe seventies on, maybe even earlier, depending on how you want to call a simulation. So optimization of simulation is like a, is an old school engineering technique. You build a model, you build a simulator for it and you want to run a controller and controllers are hard to solve. There's no quick and easy solution like a linear regression for a controller. So you run lots and lots of simulations and you see which parameters work best and there are all kinds of ways to solve this. And so that's a lot of what trading is. I mean, that's sort of like the, the end state of building a trading strategy right before you go and put it online and then. Yeah, then there's a whole other, other state.

• 12:24Speaker 00

  But um, so I think those are the three big AI related things. And in terms of simulation optimization, today you might see evolutionary algorithms where, you know, when I first started out, you might just have like one parameter at a time and sweep through 10 values and make sure it looks pretty good to the eye on the graph. And then repeat that process with each new parameter. Now, you know, it's more automated, black boxy.

• 12:47Daniel Whitenack?

  Do you feel like on that front, that in the sort of trading space, there's been a sort of a time in which people are considering and trying all sorts of new things as related to these new types of models that are, that are coming out? Or do you feel like it's been pretty progressive and incremental in terms of the same sort of, of strategy, but with maybe a different model like you're talking about? Or are people trying, you know, like I'm thinking of things like reinforcement learning or other things to, you know, try sort of whole new approaches to, to the problem?

• 13:25Speaker 00

  This is a good question. I have, as you were asking, I came up with four different answers. I'm going to go.

• 13:30Daniel Whitenack?

  Awesome.

• 13:32Speaker 00

  Talk about them. Maybe I'll try to do them from most interesting to least, but the. One thing, there's this interesting cultural dynamic in finance, which is in trading. I shouldn't say finance, trading specifically, quantitative, I even narrow it down more, quantitative trading, where people, especially when they're new to the field, they want to come in and they want to try the latest and greatest algorithms and ideas and everything they've learned recently in school or from papers or whatnot, and make some money, and make, you know, build the magic machine that makes a ton of money. And on the other side, you've got people who've been doing it for a while, usually mentoring the newer people, who roll their eyes at every new thing and say, "Ah, I know, that's not going to work. Neural networks don't work, SVMs don't work," and sometimes they're right, sometimes they're wrong.

• 14:15Speaker 00

  I think if you say something's not going to work, you'll usually be right, but you just won't be productive. So it's one of the unfortunate aspects of the distribution of new, quality of new ideas in engineering. What I find is that I've seen people try, or I've been one of the others who've tried all kinds of things, basically, you know, everything that you, if you wanted to, just randomly throw out ideas, probably I've seen somebody try it. And some of the things stick, and some people figure out how to get things to work. The big problems with financial data are the signal to noise ratio is very low. The signals aren't just small, but they're competed away. The act of going and trading on signals, which your competitors are seeing as well,

• 15:00Speaker 00

  is squashing the signals. And so it creates this non-stationarity, where over time, your strategies become less and less tradable, sometimes very quickly. And so you constantly have to adapt and look for new ways to predict or to trade. One other thing that, you mentioned reinforcement learning, and that brought to mind, I don't think reinforcement learning is ready to just, you know, turn it on and get a usable answer out of. In finance, I haven't seen that, and I say that only, you know, I say it because it's hard, I feel like it's still cutting edge for solving this kind of problem. I see a lot of promise in offline reinforcement learning, what's been going on the past year or so, it's just amazing. And it's very much in line with like a machine learning replacement for the old, or an AI

• 15:45Speaker 00

  replacement, I'll say, for the old school simulation optimization. Like, how do you make that more automated? You know, more autonomous, even hyper automated, but get that next level of automation. So yeah, so it's a lot of promise, but I don't see, I haven't, you know, some people just kind of taking that out of the box and making it work. Contextual bandit, on the other hand, which is a limited subset of reinforcement learning problem. Not only do I think that that's useful, directly useful, but I think people in finance have been doing, doing it ad hoc for a long time anyway. You know, if not the most, you know, superficial way it could be done, you know, the way people understand it these days. I think since the beginning of my career, I've been doing things that kind of look to me like a contextual bandit. Yeah. What makes that easier than a full reinforcement learning problem is that you're only predicting the immediate reward.

• 16:30Speaker 00

  So you don't have to worry about your decision now affecting the state of the world for your decision later. And then have this compounding of state changes based on previous decisions. It's, that's a more, a more I I D sample, so to speak, to build your model with.

• 17:02Speaker 01

  This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. RudderStack can be easily integrated into your existing development processes. And because they're open source, you can see all their code. So you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com.

• 17:47Speaker 01

  Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 18:03Chris Benson?

  So David, I know, once upon a time, you were working at Instagram, right? And you were working on recommender systems. And so, kind of curious, like, what is your perception of the differences? You know, we've been talking about finance and applying AI ML techniques in finance versus the social media world, the recommenders that you were doing before. What are they alike in? And what differences have you experienced? You know, even though that you're applying some of the same algorithms potentially in both areas?

• 18:31Speaker 00

  Sure. That's actually, well, noticing the similarities in the workflow between a quantitative trader and an ML engineer on a recommender system is what kind of gave me the idea for the book.

• 18:44Chris Benson?

  Oh.

• 18:45Speaker 00

  I thought, hey, you know, this is, you can describe this all as, you know, one instruction book for, for an engineer, and they both could use the same instruction book. So what the workflow looks like typically is first comes ideation and implementation, right? So you get some idea of, like a hypothesis, I think if I did this, it'll make their system better, and you code it up. And the next thing you do is you check it offline. Now, in finance and trading, that would be like running a back test or simulation of, of trading. And recommender systems, you might do a data analysis, a data science kind of analysis of the data, or you might run survey, or, or have something kind of like a simulation. I mean, some, you know, you can read in papers that there are simulations for these kinds of systems. And then the final step is you put it online.

• 19:28Speaker 00

  Mike Tyson has this saying, something like, everyone has a plan until they get punched in the face.

• 19:35Chris Benson?

  Someone just quoted that to me in Slack yesterday, by the way. And so, it's really funny that, that you're mentioning that, because I came up with a plan, and that was the quote they gave me in Slack. So, anyway, not to throw you off.

• 19:50Speaker 00

  Everyone has a model until they put it in production. And then, you know, then, then things change. And so, it's this final step of, the trading means you go, and you go and you trade, and you say, wow, that doesn't look like the back, I made so much money in back tests, what's going on? Or, you know, recommender systems, and, yeah, I'll say recommender systems, you look like you've made a better prediction of whether someone's gonna retweet, or click like, or do whatever, and you put it online, and it didn't quite pan out the way you thought. You might not, your predictions might not even be as good as they looked in the model offline, that can happen too. So this final step, this experimentation, or the, yeah, either it's A/B testing, or it's some other type of experimentation method. Is I think com is super important and it's the common final piece in that.

• 20:32Speaker 00

  And for me, I feel like it's the hardest piece in a lot of ways because you have the least amount of data to work with. But it's the most satisfying piece because it's the most accurate. It's the real system. It's not a model. It's not a simulation. It's the real thing. You have this trade, you know, this trade off between the, um, the offline stuff is can be precise. It's very quick to iterate, but it's got bias, model bias, simulator bias. The live stuff is noisy, but it's accurate because you're really doing it. And so that's what I wanted to write about. And, um, so what I'm, I'm writing in a way that, you know, can be kind of read by quantitative traders or ML engineers.

• 21:08Daniel Whitenack?

  Yeah, that's awesome. It's definitely a, um, common scenario that people face. And I think a lot of people just don't know what to do when they get to that place because they put all their time and reading and focus on the sort of offline testing that is also important, right? We don't want to like get rid of that, but often people don't necessarily know what to do. Maybe before we get into like the details of some of that, could you clear up maybe just some jargon? So, you know, I've heard all sorts of like hypothesis testing, experimentations, like you use the term system tuning. Are those sort of all things kind of people use interchangeably to talk about this kind of online testing scenario?

• 21:52Speaker 00

  Yeah, I, I see them all as kind of stitched together as one topic. So the idea of tuning, um, I think, I think hearkens back to like, uh, an AM or FM radio that had knobs on it from like 1965 where, you know, you take the knob and you tune a little bit to the left and you hear static and you go to the right and you can, the signal gets better, but you go too far, it gets noisy again. So you tune your radio into, you know, 88.7, uh, you know, K rock or whatever, and you get to listen to your music. I saw that, so if you. If you analogize that and you say your engineered system, be it the recommender system or trading strategy, all the parameters in it are like knobs on a radio, but instead of one knob, you have maybe 10 knobs or 100 knobs, and you have to tune them all to get

• 22:33Speaker 00

  the thing to work its best, then you're tuning the system. But you might also call that experimental optimization, right? So you optimize these parameters with experiments. But it really comes down to more of like a complex, focused, and efficient, maybe, way of the experimental optimization is of doing experiments, like an A/B test would be like the most basic form of that. And you really use the same, a lot of the same techniques and mathematics and analysis, whatever, to do it.

• 23:02Chris Benson?

  You're making me feel old because, you know, in this context, A/B testing is so old. And I remember back when I felt it was, you know, new and hot. And so I'm just gonna sit here quietly at this point.

• 23:13Speaker 00

  Well, it hasn't changed that A/B testing is the most robust, reliable, and believable way to do things. So it hasn't gone away at all.

• 23:23Chris Benson?

  So maybe before we go on, could we just talk about what an A/B test is for those out there that

• 23:29Daniel Whitenack?

  That's a good point.

• 23:30Chris Benson?

  Maybe are, you know, wondering about this thing that we're all so hyped about?

• 23:35Speaker 00

  Sure. So A/B test is like the engineer's term for medical research, or might call randomized controlled trial, which might be a phrase more familiar to people in these pandemic days. But so the idea is you have some way of doing things. Let's say you have some way of recommending new posts to somebody on Twitter, you know, ranking the posts. And you have some system that does this. You come up with an idea to make this system better. And so A is what you'll call the original system, and B is what you'll call the system with your change applied to it. So now you have two systems slightly different. And you go and you run them live, and you see how the audience or the users respond to these two systems. So you have some, usually you have some kind of business metric, like

• 24:20Speaker 00

  how long does the person spend on Twitter? And how long do people spend on Twitter using version A? And how long do they spend on Twitter using version B? And you want to compare, and you want to see that they spend more time. Or, well, maybe you want to see they spend less time in this day and age. But typically, in a business, you'd want more attention from users.

• 24:37Daniel Whitenack?

  Yeah, yeah. And I guess in terms of how this intersects with the AI ML community, I mean, A and B testing could be something like you were talking about that can be done outside of, you know, just changing a user interface or testing a drug or, you know, something like that. But in terms of like the ML AI space, I think something people don't often maybe think about is like, I have this model, I deploy it, and now I want to work on like version two of the model. But how do I actually know that that model is going to perform better? You know, other than on the test set that I already have, how do I know that that model is going to perform better?

• 25:19Daniel Whitenack?

  So do you use A and B tests for those sorts of scenarios?

• 25:23Speaker 00

  Yes, absolutely. So what you need, like the A and B test is sort of like the least you can do. It's the gatekeeper so that you don't ruin your system, right? So if your B version doesn't perform better, you want to throw it away and don't let it be the new standard version of your system. And so what sounds like you're kind of alluding to is this problem people run into, we've talked about it even before, where you try something offline and by the metrics you use offline, the thing you tried, let's say, to be concrete, a machine learning model, right? And you've done prediction with, say, cross entropy. You've got a data set and your cross entropy has decreased. It's looking better than the old model, but you put this now, this model online, but it's online, it's part of a larger system.

• 26:07Speaker 00

  So this is why I like to differentiate between a prediction and control or supervised learning and we'll call reinforcement learning for lack of a better term. But these are kind of different ways of looking at this problem. So that in prediction, that's what you would do offline with via supervised learning or classifier or regressor, and maybe you predict a probability someone's going to retweet a tweet. And when you put that online, you might use that probability along with other probabilities of the probability of liking a tweet or the probability of quote tweeting a tweet or commenting on a tweet. So you have all these kinds of probabilities and combine them to

• 26:44Speaker 00

  Create an interaction ranking, the probability of engagement, probability of engaging with the tweets. And that's going to be the number you use to rank the tweets. You want to put the most engageable tweets first.

• 26:54Chris Benson?

  So just to connect the two, like if you're stuck with me back in 2005 doing A/B testing, and we're moving forward in time and we're starting to apply that, how do you make that transition? How do those integrate in? How do you, what's that path forward? In a practical sense, for starting to move forward on the calendar closer to the present day, and we're trying to, to implement that. I guess what I'm asking is, is that a direct replacement for A/B testing when you're starting to do recommender? What's the relationship between the two?

• 27:24Speaker 00

  I see. Okay. So where the A/B testing would come in is you'd run the recommender system with the old model and compare it to the new, the A to the B. And the, um, and so, so the A/B test would be an experiment you'd run. You might run it for a couple of weeks or for a month, and then see whether users are engaging more with your old ranking model or your new ranking model.

• 27:42Daniel Whitenack?

  Gotcha. And do you feel like this sort of mindset of experimentation is fairly, is embedded enough in the technology space that, you know, it's not hard to sort of convince the product owners to let you run experiments in, in production? And, you know, how do people perceive this sort of idea of experimenting online and in that sort of way?

• 28:06Chris Benson?

  The risk of it?

• 28:07Daniel Whitenack?

  Yeah.

• 28:08Chris Benson?

  Yeah.

• 28:08Speaker 00

  So it's my understanding that at the larger companies, this is the standard practice is everything gets tested, you know, into production. Nothing just gets, or typically things don't just get put into production just on a whim or based on domain knowledge or just based on the offline results, but they really experiment. Now, at the bigger companies, you know, there's lots of money at stake. Revenue can be high, so a small percentage change up or down can be a dollar significant, right? Or engagement or time spent can be high, and so a small change up or down can be significant. So these things are, they're all, they're all tested, the way I understand it. I would think, I think it's not an uncommon. Less common these days, but not totally uncommon for people to have a feeling that

• 28:52Speaker 00

  they, an intuition that they understand their domain and they understand whether something is going to be right or wrong, whether it's obviously going to, something is obviously going to be better. And it's so common to be proven wrong. It's, it's humbling. You know, a person, a person who goes through that, I wouldn't say is strange or arrogant. I would say they're typical. And I've done sort of an informal survey over the past 10 years of, of quant traders. I always ask them, what fraction of the ideas that you come up with after you've tested them actually work and, and, you know, end up in your system is a good idea. And almost everybody who's had, you know, had some experience says one in 10 and just with kind of like shaking their head. It's just the nature of

• 29:32Speaker 00

  the beast. With a complex system, just things can seem very reasonable, but simply be wrong. There are just too many dimensions, too many factors to keep in the human mind and you really need to just go and test.

• 29:57Speaker 04

  Change log plus plus is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and help sustain our production of practical AI into the future. Simply follow the change log plus plus link in your show notes or point your favorite web browser to changelog.com slash plus plus. Once again, that's changelog.com slash plus plus.

• 30:41Chris Benson?

  So I'm kind of curious, as we kind of started to get into some of the techniques, I was looking through your, your table of contents on your book and stuff, and you have all of these techniques, most of which I'm not familiar with. And I was wondering if you could take a moment and kind of, kind of tell us briefly what each is and what that relationship is between those. So that I'll understand. I'm super interested, especially when I see things like multi armed bandits and stuff. It's a

• 31:09Speaker 00

  Sure. So the book is laid out in the order, it really is laid out in the order in which I learned these things. But that also happens to be the order in which they appeared chronologically in the literature. So A/B testing has been around, you know, I have 200 years or something, 100 years when it really started to pick up. The next one is response surface methodology is maybe from the 50s or 60s. And so what response surface methodology does is it focuses in the experimentation technique on systems with continuous parameters. Right? So like a knob that you could tune continuously. And if you had a couple of these, if you tune them simultaneously, you can be more efficient than if you tune them one at a time. If you go one at a time, you might say, you might tune one and say, well, I've got the right value for this one. It's seven. And then you tune the other one, you say, oh, the right value for this is three.

• 31:54Speaker 00

  But then if you go back and tune the first one, you might find that six is a little better. And so you can bounce back and forth. And there's no guarantee that you'll ever find the single optimum for both. So if you do them together, you can get the right answer. So the response surface literally means a picture of the function, the surface of the function of like your business metric, like revenue, let's say, versus those two parameters. So if you imagine a 3D plot with one of those nice colored or wireframe surfaces, you know, and you're looking for the top of that, the highest peak in that surface, that's what response surface methodology is about.

• 32:25Chris Benson?

  Gotcha. Thank you.

• 32:26Speaker 00

  Yeah. So that kind of limits you in some sense. Like A/B testing, you could apply to any two choices, you know, should I use red or green? And there's no real surface between red and green that would make, well, if you do frequencies, there might be. But there's, you know, we'll think about this categorical, you know, red, green, blue, there's no surface you plot there. But multi-armed bandits, now they're interesting. They go back to, instead focus on systems where, again, like A/B testing, you compare any two options. They don't really take advantage of the continuity of a parameter. But you can pick, you know, any finite set of values and compare them. With multi-armed bandits, the real change from A/B testing is that you're focusing on maximizing whatever your business metric is.

• 33:08Speaker 00

  Let's say revenue, you want to make money while you're running the experiment. So with A/B testing, I feel like, I think of it as more, you really want the information. You want a reliable answer whether A is better than B, right? Because people are going to die if you get it wrong, like with drug testing. It's so that the stakes are really high, you really want to get the right answer. With multi-armed bandits, you want to make money and you can afford to be, you can afford to be, nobody's going to die if you get it wrong. Good thing and you can afford to change back and forth very quickly. So the typical way it works is you take up your set of, let's take a trading example. Let's say in trading, you're trading 100 stocks, right?

• 33:49Speaker 00

  You might say 50 of them to start out dedicate them to strategy version A and 50 to strategy version B. And as time comes in, as the returns come in and you see A doing a little better than B, you increase the number. Say your 60 of them go to A and 40 to B and you just keep scaling and 70 to A and 30 to B. Rather than waiting until the end of the experiment, keeping it 50 50 the whole time, you kind of scale it gradually toward the one that seems to be doing better. And as the data comes in and the error bars go down, you scale back and forth however you need until, you know, until the probability of being in, let's say, B is low, under 5% or 1%. And you get to decide that that percent as your threshold. So now you're talking about like practical or business significance rather than statistical significance as your threshold, your criteria.

• 34:30Daniel Whitenack?

  Is it ever the case when you're doing that sort of test and adjusting things on the fly? Let's say you have, you know, model A and model B and it it's sort of a mixed bag where, well, it turns out over that time, like actually the mixture of models was better than either model A by itself or model B by itself. Does that sort of scenario occur?

• 34:52Speaker 00

  Yes. And it could also be that a model A and B trading or running together can be worse than either A or B running alone. There can be interactions basically depending on what it is you're testing. And it doesn't just apply to trading. So, so, yeah, that can be, that can be complicated. So by that, that process can be complicated by these interactions. That being said, it would also be complicated if you're just running an A B test. The interactions would interfere. So you, you, you kind of have to, you have this assumption of IID, independent identically distributed samples in, in the analyses through these kinds of algorithms typically. But in reality, your samples are, you know, they're I nor I nor D. So, oh, they're, they're either I nor I.

• 35:34Daniel Whitenack?

  And that I guess brings me to my next question, which is, you know, typically we think of A B test, one thing or the other thing. Do some of these techniques work when you have more than two options? Like, let's say I have A, B, C, D, and E. Or is it better to just always sort of assume you're going to test one thing at a time?

• 35:55Speaker 00

  As a general, it's better to test, it's more efficient to test statistically, to test everything at once. Right? For the same reasons I was talking about with, alluded to with the two knobs, you know, it's possible that moving, you know, to the left and moving up in your space, moving to the left in your space is good, moving up your space is good, but moving left and up at the same time is even better. If it's that kind of space you're testing, then you should do the same thing with A B testing and test them, and test them all at once. The virtue in doing only two things at a time is that your test gets done faster. As you increase the number of things you're testing, the amount of data you need increases. So your test gets done faster, you have, you'll typically have less noise, simply because you'll have, your patients won't be tried as much. So you'll, you'll

• 36:40Speaker 00

  get, maybe you'll get the data you need. It'll be easier to make a decision. It's also easier to decide between two things than it is to decide between more than two. So you should communicate it to your team or the, or other teams. So there are benefits to doing the simpler thing, for sure. You know, but it's a trade off.

• 36:55Daniel Whitenack?

  Gotcha. So having asked you about your response service methodology and the contextual bandit, how does Bayesian optimization, you have that in the title. So, you know, I feel like we're, we're moving up toward that. How does that fit into the equation? What does it bring us?

• 37:10Speaker 00

  Bayesian optimization takes bits from, bits from responsive methodology and contextual bandits and multi-armed bandits, but it makes this a super automated by comparison. Responsive methodology is very manual. You make the surface, you look, you make a decision where to go. And Bayesian Optimization does the modeling for you, makes the decision about where to test next, what parameter values to test next for you. It's nice and there's this interesting thing. You've just probably heard the phrase exploitation versus exploration. And multi-armed bandits will do this. This, that's what they're doing when they're waiting towards the FA is doing better and they're adding more stocks to A and fewer stocks to B. They're taking advantage of the information that's come in so far. They're exploiting the information to make more money,

• 37:54Speaker 00

  but they're still testing B because they want to explore to bring the error bars down. And so Bayesian Optimization will do that kind of exploitation modeling and decision-making for you as well. And to top things off, it'll also work with categorical and discrete parameters. So it really brings kind of the rest of the book together into a single, the ideas from the rest of the book together into a single automated technique.

• 38:17Daniel Whitenack?

  Awesome. So I'm kind of curious, I guess, just bringing things way down to a practical level. What are the sorts of practical tools that you can use to do this sort of testing? Is it things that are available in like, you know, familiar toolsets maybe to data science or AI people like the Python ecosystem? Is there, is there common stuff out there that people use? And, and how easy is it to sort of set up these sorts of scenarios online and in a reasonable and robust way?

• 38:49Speaker 00

  Okay, so for A/B testing, there are lots of tools, both, you know, open source and commercial. And when I say open source, you'll probably find tools that you, you can make use of in R or in like the NumPy Python ecosystem. But there are also, there are also commercial products that can help you do this online. They were, you can sort of submit results to a web service or the other ways of accessing. So A/B testing is well-known. It's the most commonly used of all these techniques and there's lots of support for it out there. On the other end of the spectrum, Bayesian Optimization, there's a tool called, open source tool called X that's written and supported by Facebook. And that's, that's very good. That's what I would be the go-to, I would think. I suspect Google's got

• 39:34Speaker 00

  Something similar, although I'm not as familiar with it. I'm familiar with their with their paper where they use their optimizer to make better chocolate chip cookies. Vizier it's called. And I don't know for sure, but there's an open source version of that.

• 39:47Chris Benson?

  So I guess as we've kind of gone through some of the things you've covered, what are you anticipating going forward? So as, you know, clearly from early A B testing and stuff, as we started the kind of, you know, the progression and you've just taken us into Bayesian optimization and like, what are you thinking of for the future? What are you getting excited about working on or utilizing going forward? Do you have any sense of where the next steps might be, what might go?

• 40:16Speaker 00

  Yes. What I'm most excited about is offline reinforcement learning. So doing reinforcement learning from small amounts of data. And really the key to that, it's not so much the learning of the controller, you know, or the strategy or whatnot, but it's the learning of the simulator itself. That's where the real power is, right? If I had a simulator, a learned simulator that I could believe, I could optimize it in, you know, five different ways, one of which would be, you know, through a reinforcement learning technique like, you know, PPO or something more, you know, more recent than that. But learning the simulator from a small amount of data and understanding, you know, what results you can trust in production, which ones you can't, and, you know, where to go and maybe explore next to get data built to improve the simulator,

• 40:58Speaker 00

  these kinds of questions, being able to do that well could be super valuable. I mean, I know when you read about these kinds of things, the examples are often from from robots, robotics, they can be, they can provide a great visual that you can kind of get data pretty quickly, but they're industrial processes, I think, everywhere that so many things work via optimization of a simulator that I think this would be super valuable throughout society. Yeah, I'm very excited about it.

• 41:25Daniel Whitenack?

  Awesome. Before we end here, I definitely want to let people know to check out David's book, but also we have a discount code through Manning, 40% off, which is pretty spectacular. So you can use the code PODPRACTICALAI19. We'll put that in our show notes as well, but that'll get you 40% off. And I know I'm really excited to dig more into this book, but David, it's been super fascinating and I really appreciate you taking time and explaining some of this stuff to us. It's really good and it's important. And like I say, I think it's something that people need to think about more and often are perplexed by in

• 42:08Daniel Whitenack?

  many scenarios. And I'm sure your book could shed a lot of light on those situations. So thank you for your work on it and thank you for the discussion.

• 42:16Speaker 00

  Thanks a lot for having me. I really enjoyed this.

• 42:22Speaker 04

  Thank you for listening to Practical AI. If this is your first time, make sure you subscribe so you don't miss a thing. Head to practicalai.fm to subscribe or find us in Apple Podcasts, Spotify, or wherever you listen to podcasts. And if you get value from the show, please do share it with a friend or a colleague. We appreciate you spreading the word. Practical AI is hosted by Daniel White neck and Chris Benson. It's produced by Jared Santo and our music is provided by Breakmaster Cylinder. We are brought to you by some awesome sponsors. Shout out to Fastly, Linode, and Launch Darkly. That is our show. We hope you enjoyed it and we'll talk to you again next week.