Jeff Ma – Making Decisions with Data - [Invest Like the Best, EP.151]

My guest this week is Jeff Ma. Jeff was on the famous MIT Blackjack team from the book Bringing Down the House but has spent his career in an around fields of analytics and data science. He’s studied sports betting and analytics, built companies for analyzing human capital, and ran the data science and analytics group at Twitter. Here are links to his book, blog, and podcast. Our discussion is about a number of fascinating ways data is being used to make decisions in the worlds of sports and business. Please enjoy! For more episodes go to InvestorFieldGuide.com/podcast. Sign up for the book club, where you’ll get a full investor curriculum and then 3-4 suggestions every month at InvestorFieldGuide.com/bookclub. Follow Patrick on Twitter at @patrick_oshag Show Notes 1:20 - (First Question) – How quantitative analytics have evolved in sports and how they’re being used 4:26 – Best role of humans in the analysis process 8:38 – Sports that are most interesting to observe through analytics 10:26 – How does luck play into sports analysis 11:54 – Team analytics vs better analytics 12:38 – Concentration of success among sports betters and their moats 14:58 – Favorite lessons learned from professional gamblers 16:45 – How analytics got introduced into gambling 19:21 – Understanding one’s own biases 24:04 – How he became VP of analytics at Twitter 28:37 – Primary lessons from the work evaluating human capital and talent with analytics 28:59 – Niel Roberson Podcast Episode 31:40 – How to model people for success when hiring 33:29 – How to hire the right data scientists’ team 37:54 – Most interesting problems they tackled at twitter 42:31 – Responsibility of social platforms to police itself 45:34 – Areas that would interest him in the future as an investor 49:24 – Kindest thing anyone has done for Jeff 51:50 – Values instilled in him by his parents. Learn More For more episodes go to InvestorFieldGuide.com/podcast. Sign up for the book club, where you’ll get a full investor curriculum and then 3-4 suggestions every month at InvestorFieldGuide.com/bookclub Follow Patrick on twitter at @patrick_oshag

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Patrick O'Shaughnessy?

  I know firsthand how complex the tech stack is for asset management firms. And seemingly every new tool and data source makes the problem even worse, adding more complexity, more headcount and more risk. RidgeLine offers a better way forward. One unified platform that automates away the complexity across portfolio accounting, reconciliation, reporting, trading, compliance and more, all at scale. RidgeLine is revolutionizing investment management, helping ambitious firms scale faster, operate smarter and stay ahead of the curve. See what RidgeLine can unlock for your firm. Schedule a demo at ridgeline.ai.

• 0:36Patrick O'Shaughnessy?

  Hello and welcome, everyone. I'm Patrick O'Neal and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, methods, stories and of strategies that will help you better invest both your time and your money. You can learn more and stay up to date at investorfieldguide.com.

• 0:57Speaker 02

  Patrick O'Neal is the CEO of O'Neal Asset Management. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of O'Neal Asset Management. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of O'Neal Asset Management may maintain positions in the securities discussed in this podcast.

• 1:22Patrick O'Shaughnessy?

  My guest this week is Jeff Ma. Jeff was on the famous MIT blackjack team from the book Bringing Down the House, but spent his career in and around the fields of analytics and data science. He studied sports betting and analytics, built companies for analyzing human capital and ran the data science and analytics group at Twitter. You can find links to his book and podcast in the show notes. Our discussion today is on a number of fascinating ways that data is being used to make decisions in the worlds of sports and in business. Please enjoy.

• 1:52Patrick O'Shaughnessy?

  So like a Billy Bean Moneyball type story, or from a money-making standpoint, the horse racing in Hong Kong type story, typically always comes back to kind of quantitative analytics. So tell me a bit about how you think the state of the art is today. A lot of people will know Moneyball, but what has evolved the most? How have analytics and the sophistication of the users of those types of systems evolved?

• 2:13Speaker 03

  I would say at the sports level, not the sports betting level, but the sports level, like the Billy Bean level, the way that it's evolved the most is just acceptance. People believe in this. So in basketball, there are analytical strategies that you see on the court. I don't know if you know, but three is worth more than two. So teams that shoot more three pointers, they're more analytically driven, especially certain three pointers. The corner three, which is much closer, it's closer than the rest of the three. So people shoot it at a higher percentage. That's why you see so many people trying to shoot corner threes. And you see the best defensive teams trying to take away the corner three because it's a highly efficient shot. And across sports, I would say there is much more of an acceptance that you need some form of analytics to basically compete now,

• 2:56Speaker 03

  both from a player personnel standpoint and then an on-field strategy standpoint. I think the more interesting thing or the evolution as sports is going forward is going to be around data. And that's sort of like the classic analytics paradigm, which is that you can't have any strong analytical system unless you have good data. For a long time, sports data has been pretty bad. Generally, the data that's captured is the data that's easy to capture. It's the data that describes what's happening in the game. It's not what is ultimately the best thing to build out an analytical system to predict what's going on. With things like player tracking and with computer vision and all these different types of things, we're getting much better data to understand how to predict games.

• 3:40Speaker 03

  Specifically in baseball, originally you would have very, very, very old data that wasn't useful. But now they literally have cameras that are capturing launch angle and spin rates of the ball and velocity and exit velocity of a hit that allows you to build better bottoms-up models to predict what would happen in a game. Because think about baseball, if all you capture is there was a pitch and then there was a single. A single could be a ground ball that just happened to find a hole, or it could be a rocket that just landed short of, in Fanway Park, it could be a rocket that hits off the wall and should have been a home run in any other ballpark, but someone basically hit it right directly into the wall and the defender made a great play.

• 4:24Patrick O'Shaughnessy?

  Do you think a fair way to describe this would be more of a focus on the quality of the process of a given action versus the outcome of that action? Like the quality of a shot versus whether or not it went in?

• 4:33Speaker 03

  For sure. I mean, that's definitely a good way to look at it. The challenge with that, both the value and the challenge with that is that you're talking about almost having somewhat of a qualitative judgment. And so I think that's an interesting thing that sports is going towards, which is this idea of, and you see it in machine learning right now, human evaluation. How do you actually get humans to help interpret data at scale that helps you create a better data set to use to train models and things like that?

• 4:59Patrick O'Shaughnessy?

  Again, on the sports team level, what do you think is the best role of talented humans right now in the process? So it's not, I don't think any major league teams are operating purely on a model for selection. There's still humans involved. So what are the best humans getting better at or focusing on?

• 5:17Speaker 03

  I think the first wave was always player personnel. It's like, how do we make our player personnel decisions? And specifically, a sport like football where you see a team like the 49ers becoming very aggressive and different in terms of how they do contracts and things like that. So that's sort of an inefficiency, which is to understand, can you structure contracts in a way that are better for the team in terms of guaranteed money, in terms of how it hits the cap, in terms of all those things? So that's, I think, an area of huge opportunity or gain. I think that generally, on-field strategies in places like football and understanding game theory around run-pass mix, around play calling, that's a huge area for opportunity to understand, like the key to analytics. And so

• 6:00Speaker 03

  When we were working on Protray, the sports company, Bill Walsh was an investor and Bill Walsh was this sort of legendary coach for the 49ers, won four Super Bowls and pioneer. And I sat down with him in his office and I remember I was, specifically, we were trying to build out a model to evaluate the process, to evaluate plays. A six yard run is, means success on first down, but a six yard run on third and 20 is not successful, or second and 12 or something like that is probably not successful. So you have to evaluate on any given play, how many yards do I need to make it successful? And so we had done some numbers, basically looking at building a model of all the different

• 6:43Speaker 03

  plays and what yardage gains would actually put you in a better position in terms of winning. So that's how we evaluated it. And what we found was that it was for first and 10, it was something like a little over four yards, so four and a half yards was meant success. And so I just sat down and I asked Bill Walsh, and you guys are here, I got my Excel spreadsheet out and all this kind of stuff. And I'm like, Coach Walsh, on first and 10, what do you? He goes, oh, probably a little over four yards. Like, he thought for a minute and they said, a little over four yards. He's never looked at a spreadsheet or anything, but he completely had it all in his mind. And I met with Jerry West at the same time. And Jerry West, the first thing he said to me is, boy, I flew to Memphis to sit with him. And he said, boy, I got one thing to tell you first. And I said, what? He goes, I hate statistics. And I was like, oh, okay, this is gonna be great. This would be a great time to hang out in Memphis.

• 7:27Speaker 03

  But what he meant was that he hated the way people use statistics in basketball. And he even said it. He said, I hate that people think Allen Iverson's a great basketball player, even though he needs 35 shots to score 30 points. He hated the way. And so it was clear that there is, in these geniuses, and I'm sure you guys see this in the finance world, there are geniuses that don't need models, that just somehow have this inane ability to sort of look at situations and then extract value from them. But the rest of us need models. The rest of us need data. And so the key is almost unlocking what's in their mind via data and analytics.

• 8:01Patrick O'Shaughnessy?

  Yeah. I'm always interested in these analytics challenges, what the target or outcome variables are. So oftentimes, when you see really creative data science predictive modeling projects, it's creative because they've chosen a unique thing that they're modeling. So they're not just modeling a run, they're modeling something more specific in baseball or what have you. Do you see much of that happening in sports analytics where we're trying to predict more unique and different things than the obvious things?

• 8:26Speaker 03

  At the core, What you're talking about is people would say, oh, how do you predict wins? But what are the real? And it's always scoring. So scoring at some level is. And then what are the things that really are the most predictive for scoring? And I guess I'll go back to baseball again, because there's this concept even in baseball. If you do a bottoms up model, meaning trying to predict hits or trying to predict each at bat, even if you have, let's say, three hits in a row in one inning and then no hits in the next two versus three hits in one inning. It's just this idea of cluster lock is what they call it. And so I think what I would say is that there continues to be really great opportunities to evaluate sports as you break down what you're saying, trying to change outcome variables. Or, yeah.

• 9:10Patrick O'Shaughnessy?

  What are, maybe a quick survey of the sports that are most interesting to you through the analytics lens of the major sports, maybe from most to least?

• 9:20Speaker 03

  So baseball is the most interesting. They're all interesting in different ways. So baseball is very interesting because at the core, it's very solved. And because it's very solved, they've had to go to like very new ways, i.e. this data that I'm talking about. They've had to kind of reinvent themselves and go to the data. Basketball is interesting because it's the one that's probably getting the most attention now because of guys like Daryl Morey and because of just the general trend of what's happening in these front offices and the corner three. And it just fundamentally, analytics, I think, have fundamentally changed the game the most in basketball. Like it looks fundamentally different now than it looked even five years ago. And how different it looked now.

• 10:01Speaker 03

  Teams play faster, teams try to shoot more threes, teams understand they need to drive more, the mid-range. That is all analytics driven. I think football is probably, I would say, singularly the most interesting, simply because it's the one where there's the most green space to go out and figure this out because so much of what happens on field is just not really well understood. And even something like the modern day concept of not punting or never punting or never kicking field goals. You still see teams that are very smart or analytically driven, even the Patriots. You see them kick field goals in situations where all the math and all the analytics would tell you that it's a terrible decision to do so, but you still see them. Anytime you kick a field goal inside of five yards, so any field goal that's less than,

• 10:45Speaker 03

  say, 22 yards is probably a really bad idea. Almost inside of 10 yards is probably a really bad idea because not only are you giving up the ball. But you're going to end up giving up field position and you give up the opportunity to score seven points versus three points.

• 10:58Patrick O'Shaughnessy?

  What about sports like, I always picture that luck skill continuum where the more players and certain other attributes, the more luck helps determine the outcome of a game. So like football and soccer would be higher on luck versus tennis and golf where pretty much the best person wins. Luck is a smaller piece of it. So what about that end of the spectrum, sort of this individual sports?

• 11:18Speaker 03

  Yeah. Well, you brought up soccer, which is interesting because soccer, soccer and hockey are very interesting because of how little scoring there is. And so therefore there isn't a lot of, you would think soccer is probably a perfect example of what you're saying where you want to actually come up with how to predict something beyond scoring. You want to predict scoring chances or score that kind of thing. And there's actually a guy that's a good friend of mine named Ted Knutson, who started a company called Stats Bomb, that is actually really going back and training people to go back and watch games and create new data from watching games that allows them to do these things, model out more predictive things. In terms of individual sports, my partner on the podcast that I do about the process, Rufus Peabody, he's one of the, if not the number one golf

• 12:00Speaker 03

  better in the world. He's one of the top and he definitely is, for a long time was sort of looking at data that most people weren't looking at. That market has become much more efficient now, unfortunately for him. But modeling out individual performance, yeah, I would say that there's definitely a level of it that's easier because there aren't as many sort of confounding factors or composite things that happen. But I think even in golf, it has its challenges because courses are different and everything like that.

• 12:26Patrick O'Shaughnessy?

  Talk about the differences between how teams use data and analytics versus professional gamblers. So it sounds like for teams, it's mostly strategy on the court, player personnel decisions and maybe personnel development. Obviously that's not what betters care about. They care about outcomes of the game versus spreads or whatever. So what is different in terms of the two analytics?

• 12:46Speaker 03

  Yeah, I just think exactly what you said. It's the sort of motivation. I think that a better would not care about, there was a lot of similarities, I guess, because ultimately you're trying to predict winning. I think there are many times where there'll be very aligned, but there's also times where a better is much more mercenary in how they're thinking about things. Yeah, I guess teams also have to think a lot more holistically about the entire season and what they're trying to do, whereas betters are often just trying to think about one individual game.

• 13:12Patrick O'Shaughnessy?

  Concentrated, do you think these betting markets are in terms of who's making the profits? So I've talked to some people in the sports betting world that would say in any given sport, it's usually one or a small handful that basically take 100 plus percent of the profits because there's a lot of dumb money in sports gambling. Is that, do you think that's roughly fair?

• 13:29Speaker 03

  Yeah, it's very true. It's very true. I would say the amount of people that can consistently beat the market, i.e. become professional sports bettors is very small. I mean, it's in the neighborhood of 1% of the people that bet, maybe even less.

• 13:40Patrick O'Shaughnessy?

  And how much of a moat do those bettors tend to have around their edge versus the competition? How much turnover is there in that 1%?

• 13:49Speaker 03

  I think it's hard to answer that simply because it's changing so much. Like, I think I would have said to you, I don't think there's much of a moat, to be honest. And I think one of the reasons that there isn't much of a moat is because the top bettors are constantly changing what they're doing. They're not doing the same thing over and over again. And it's not like they've established something they're constantly having to reevaluate. I was talking to someone the other day about the NBA and I was saying, people that bet the NBA with these sort of static models right now are crazy to me because the game is changing so much every year because of analytics, because of rules, because of everything that's happening, even just because of the way the schedules change. Like, the schedule has changed a lot because they're trying to give players more rest. And even with that, players are getting more rest naturally.

• 14:33Speaker 03

  They're also sitting them out of games much more than they used to. So if you're building a model based on historical data, that takes into account when someone's rested and when someone's not. It's not going to work anymore because it's just very different in terms of how they do things.

• 14:46Patrick O'Shaughnessy?

  And do you think that basically 100% of the 1%, let's call them, are quants? Are using analytics and data science?

• 14:54Speaker 03

  I wouldn't say that. Some of them play the market and if you call that a quant, i.e. they understand market moves and there are, if you bet into the black market or into the legal markets, there can be big opportunities for pricing disparities. So there are some people that do that and I wouldn't necessarily call them quants. There are definitely a handful of people that can sort of read those market moves or can look for things like that that do well over time. It's basically just arbitrage at some level. But I would say the people that originate that sort of are. Fundamental versus technical, they're sort of always going to be analytically driven or create models because it's just too hard not to be.

• 15:31Patrick O'Shaughnessy?

  Do you have any favorite encounters or lessons learned from other professional gamblers across your set of interests?

• 15:39Speaker 03

  I mean, I think a lot of it is just around understanding how to overcome biases. And so this is something you and I had talked about a lot and the biases around loss aversion or short-term thinking, or even just the idea of process over results and being driven that way. I think whenever you see a really strong sports better, you're never like, oh, wow, he's the most strong analytical person I've ever seen, or I'd want to hire. You just see this fortitude and this ability to sort of go and overcome these big swings. Because if you're a really successful sports better, you've had some really bad swings also, where you've had to really doubt yourself and doubt everything that you've done and

• 16:20Speaker 03

  doubt your process and you have to have been able to get through that to become successful. So I'd say that that's probably the consistent thing that I see. And then, yeah, I mean, I think the other thing, and this is one of those things that when I think about my early background sort of in blackjack and card counting, you never had to question your model and you never had to question stationarity in the data. But in sports betting, you consistently have to. If you don't, you're a fool. And so with that, there's just more inherent doubt that is cast on you that you've got to overcome. In blackjack, you have a bad out, you have a bad session, and you feel pretty good the next time you're going to get them. It's okay. And in sports, when you go through one of these bad streaks, you worry.

• 17:03Patrick O'Shaughnessy?

  It's the exact same thing in public stock markets. It's, everything is evolving, changing, regulations change, rules change, players change, the competition changes. So it sounds identical, very non-stationary type outcomes. It's fascinating. In the early days, let's talk a little bit about that. So at MIT, maybe tell me the backstory for how this idea first came together. People know the movie 21 and the book, Ben Mezrich's book.

• 17:28Speaker 03

  So the premise behind the blackjack stuff was relatively simple. I mean, MIT has always sort of had this cultish type group that figured out blackjack, all the way back to Ed Thorp, who Was, I think, either a visiting professor or whatnot at MIT. And he actually discovered card counting on an old school IBM mainframe computer where he did simulations, where he understood that if you took all twos out of a deck of cards, what would that mean in terms of win probability? So he would win, he would do simulations. And essentially, the concept of card counting is not new. Blackjack is the only game in the casino that's subject to conditional probability, meaning what you see impacts what you're going to see.

• 18:09Speaker 03

  And therefore, it's like this perfect thing for data-driven thinking or data-driven strategies. So the MIT Blackjack team, my involvement essentially was there were these guys that had been card counters, but no longer could play because the casinos had banned them. And they were recruiting new people to play. And they would basically teach you how to play and then send you to Vegas and give you some money. And you'd win money and you basically split it with them. We had a group that played and we were quite successful. And when it was all over, I approached Ben Mezrich, I think it was in 2001. And Ben had written six books at that time, but his career was definitely at a crossroads. He had business school applications out and was contemplating not being a writer anymore. And I said, hey, Ben, I got a really great idea for your next book.

• 18:52Speaker 03

  And again, remember, this is 2001, so the World Series of Poker isn't even on television yet. I don't even know if big data is a term at this time. And it was just a really interesting moment in time. This is even before Moneyball. So this is like an interesting moment in time where we told this analytics story. So Michael Lewis is a friend of mine. And I talk a lot about The Blind Side being sort of like this amazing nature versus nurture book that was written right around the same time that Gladwell wrote Outliers and Talent is Overrated were written. And those were all these very prescriptive nature versus nurture books. But Blind Side was sort of, to me, the most important one because it was this nature versus nurture story that told it in just this incredibly fascinating and accessible way.

• 19:37Speaker 03

  In many ways, that was luckily what brought you down the house. The book that Ben wrote in '21, the story did is it told sort of this analytics story in a very non-prescriptive way. And gave sort of this opening for people to think or talk about gambling and card counting in an analytical way.

• 19:54Patrick O'Shaughnessy?

  One of the things I've seen you talk about as the maybe most interesting takeaway from that experience is understanding one's own biases. And with blackjack as one presentation, you literally, there's like a card that shows how, I can't remember what it's called, like a perfect strategy or something like that.

• 20:08Speaker 03

  Yeah, basic strategy.

• 20:08Patrick O'Shaughnessy?

  Basic strategy, it tells you exactly what you should do odds wise on every hand. And that a lot of people know this, but still then don't follow it. So maybe talk a little bit about your experience with bias.

• 20:18Speaker 03

  So that's a very simple example and a very good example. Essentially, the game of blackjack has been solved. You know, given what hand you have and what the dealer has as an up card, you know what to do and you can look at this card and this card's not secret. It's available on the internet and it's available in books and whatnot. And if you just do that, if you just go from blind intuition to all of a sudden just following this matrix, decision matrix, you go from losing roughly about 3% of the money you put on the table to about half a percent of the money. So you almost become an even player against the casino. Yet you will find people all the time that know basic strategy, but then when given a situation, will decide to go with their gut over what the data says.

• 21:03Speaker 03

  And it highlights a lot of biases that we have. One that I talk about a lot is this concept of omission bias, where people favor inaction over action when they think it may lead to their own harm. Meaning, I'd rather you do something that causes my demise than me cause my own demise. And so the classic example is if I have 15 and the dealer has a nine showing, the math 100% will tell me to hit. Because if they have 19, that's a winning hand, they're not going to have to do anything. And there's a good chance they have 19. And if I have 15, I'm not going to win unless they bust. So I need to take a card. But a lot of people won't want to take a card there because if they get a seven, an eight, a nine, a 10, a jack, a queen, a king, they're going to lose right away. So they'd rather just say, okay, well. I don't want to be the cause of my own demise.

• 21:47Speaker 03

  And so they basically decide not to hit in that situation and hope that the dealer flips a low card and has to take a card and bust. But if the dealer flips a 10 and wins, well, at least in that case, they're probably going to lose anyways. That's what they're thinking in their mind and they weren't the cause of their own demise. And then you also see these sort of ideas of, and I don't know what this bias is, but I always joke that one of the most dangerous human beings in the world is Malcolm Gladwell because he's such a great writer, but he can take these concepts that aren't always completely true. And because he's such a great writer can make us believe that they're true. And the notion of blinking and intuition over data that it's largely not true. There are some people out there that probably don't need data because they've had such pattern recognition in their lives.

• 22:32Speaker 03

  But even that, people don't understand, that's their own data that they have in their mind and their own models in their mind. But again, the idea, if you decide to just use blind intuition at blackjack and I use data, I'm going to do better than you over the long haul. And then another is classic results bias or whatever process of a result where if tonight, if you and I decided, hey, well, let's go to Atlantic City. And I don't know why we'd want to do that, but if we said that there's 15 versus nine, I'm standing behind you and say, Jeff, what am I supposed to do in this situation? If I say you're supposed to hit, if you get six to make 21, you're going to turn around and high five me or be best friends forever. If you get seven to make 22 and lose, you're going to turn around and go, they should never made a movie or a book about you. But that's the classic, the decision. I even saw, I was watching Monday night football last night and there was a fourth down that

• 23:17Speaker 03

  one of the coaches went for. And when he got it, I think Chris Collinsworth said something like, again, with a great decision. No, the decision was independent of the results. So I think these are some of the classic biases.

• 23:30Patrick O'Shaughnessy?

  Yeah. In betting and markets and edge, these things won't go away. They're human nature. And the question is, can enough smart participants overcome their bias and sort of eke out that edge that does exist? I think you could argue and people like Michael Mobison have argued in public stock markets that the dumb money, so to speak, has sort of been pushed out and they're just opted out of the game. They buy index funds instead of trying to beat the market. But in gambling, which is much more recreative. It seems like the runway could be much longer. No index fund of gambling.

• 23:59Speaker 03

  Yeah, I mean, the challenge is liquidity and the challenge is being able to actually deploy capital. Because even though there's a lot of dumb money there, there's no incentive for the market makers to allow smart money. The really good market makers, in this case, market makers are just the sportsbook operators. But the real sophisticated ones are trying to use the smart money to inform how they do pricing to try to increase their edge over the dumb money. But most of people aren't thinking about it like that. Most of the people are thinking recreationally, how do I make the most money off of people that don't know how to bet? And then I can make a higher margin than I would if I let smart money in.

• 24:37Patrick O'Shaughnessy?

  Tell me the story of how you ended up as the VP of Analytics and Data Science at Twitter.

• 24:42Speaker 03

  I had started a company that was in the world of analytics for people. So for human performance, specifically in this case, we were looking at the performance of software engineers. How do you evaluate their performance? Then how do you create sort of enterprise tools to better manage them or better understand them or better motivate them? And pretty early on, we realized that this was an enterprise play. Like, we're gonna have to sell into big companies. And I don't think any of us really had that in our DNA. And so Twitter had just started an engineering effectiveness team. And they were going to try to do a lot of the same things that we were already doing. So they decided to buy us and let us build a tool at Twitter.

• 25:23Speaker 03

  And we took the opportunity to join Twitter in a lot of ways because we were all fascinated by the company Twitter at that time. And personally, I've been a part of a few startups that have had success and we've sold to bigger companies. And this was the only time that I was really excited to join the acquiring company. And so originally, when I went in there, actually, it was funny because the original thought was that I was gonna go in and build a sports-only app. So it was a vertical app and it was an idea of can they have a multiple app strategy, especially on the consumption side? And literally a day before the deal was gonna close, Kevin Wheel, who was the VP of Product at the time, called me and he said, hey, Jeff, we've decided we don't really wanna do this vertical strategy anymore.

• 26:08Speaker 03

  Do you still wanna join? And we had literally been coming to the office, playing FIFA soccer all day because we knew that whatever we had done, we weren't going to continue building, we're going to build everything within Twitter. So at that point, the idea of restarting and killing this deal, it wasn't going to happen. So I said, yeah, I'll join. And originally, I was part of running a lot of product within the growth team. So using analytics to understand user growth and to understand how to grow more. But then Anthony Noto, who is now the CEO of SoFi, and at that time was the CFO of Twitter, he had started a small group of data scientists and really economists that were trying to look very much at the macro type analytics at Twitter to understand user

• 26:53Speaker 03

  growth and to report for IR and all that kind of stuff. And he asked me to come run this team. What's interesting is the guy that originally was running the team now runs analytics and data science at Stripe. It's like this amazing group of people that I was allowed to sort of be around. And I didn't have a real traditional background to lead sort of a data science analytics team. If you look at me right now, compared to most of the people in the valley that are doing this, most people in the valley are PhDs at some level and are much more academics. And I had come from this completely different background to do this. And the team that they asked me to lead at that time was small, but it was immensely talented. And to be able to work with those types of people and understand Twitter is this incredible, almost like a playground to do analytics.

• 27:37Speaker 03

  And you have this community of 300 some million people that are interacting in ways that people don't even ever understand. There's one country, I think it's in maybe in one of the stands, Kazakhstan or something like that, where they actually use Twitter like they would Uber, because Uber's not allowed there. So someone will tweet out that they're at a certain place and someone will come pick them up or something like that. And just that the use cases around Twitter are really incredible. And so to be able to be a part of that. And so Anthony asked me to come sort of run this team. And there was another guy over there by the name of Todd Morgenfeld, who now is the CFO of Pinterest, who I worked for. And he and I built out just this sort of incredible team at Twitter of data science and analysts.

• 28:20Speaker 03

  And eventually, we consolidated all the different teams. And so one of the things I think that is fascinating, because I know you have a lot of people that listen to this that are in this world, is understanding how to properly structure a data science and analytics team at a company at scale. And do you have them embedded? Do you have them in a central org? If you have them in a central org, it's the best way because professionally, they understand how they can move up and things like that. But there is the side idea of HRBP's or things like that, that they need to be part of these teams to really understand what's going on in these teams. It's an interesting thing, but we were able to create a very strong central team at Twitter. And so when I left, I had got promoted to Vice President of Data Science and Analytics there.

• 29:04Speaker 03

  And I was leading all the central data science and analytics, which was just a fascinating thing to be a part of.

• 29:09Patrick O'Shaughnessy?

  I've got 10x for questions, some human capital questions, some Twitter questions and some team questions, lots of interesting stuff there. So I guess we'll start at the beginning with the company you were building. 10xer is the name, I think. You're evaluating specifically software engineers, but human capital and talent using analytics. What were the primary lessons from that research exercise?

• 29:27Speaker 03

  The thought was, and actually, I started this company with someone that has been on your podcast, Neil Robertson. So Neil was very much the formative idea for this. And he said, Jeff, do you want to kind of run with this and bring your own sort of flair to this? And so the idea was that right now, we create a digital footprint every day. And knowledge workers especially create this digital footprint, meaning email or calendar or Salesforce. And this digital footprint is now accessible via API. So software engineers, it's things like Jira, Pivotal Tracker, GitHub, things like that. That data clearly means something, but what does it mean and what can you use it for? And so our goal was to figure out how you could use it. Now, I think what we really believed or hoped is that we could build a bottoms up software system,

• 30:09Speaker 03

  meaning individuals would want to use this because it would help their own performance. And it would help them understand and get Maslow's pyramid of needs. They'd get mastery, autonomy, and they'd understand purpose and things like that within their own performance via data. And the idea that professional athletes are motivated very much by knowing their own statistics and how to use that as ways to set goals. That was again, like all this kind of stuff. I mean, one sad lesson we learned is that people are a lot less self motivated than we believe. The amount of people that really want to do that is much smaller than we believed. And so therefore, what we found is that To do this and have it be effective, it would have to be more of a manager tool and have to be more top downwards mandated by managers for their individuals to use this.

• 30:52Speaker 03

  Now, another thing we learned is that anytime you look at one of these systems where work is being done, that isn't sort of active work, meaning like GitHub is active work. You're checking in code, you're things like that, but something like Jira or Pivotal Tracker, those are not active. Those are more, you have to like, teach someone what process they need to do to use this thing. And when you have that situation, a lot of times that hygiene of the data is bad. And so what we found is that even to start using that data, we would have to change the way that people did work so that their data was better for us to actually analyze, which is kind of a nightmare in itself.

• 31:28Patrick O'Shaughnessy?

  How much room do you think there is for this sort of thing today? So that was many years ago. Has anything changed?

• 31:34Speaker 03

  Literally every few months, I'll get an email from an investor friend or someone that'll basically say like, hey, these guys are trying to start a company similar to 10x or can you go talk to them and things like that? I mean, I think there's an opportunity, but I think a lot of it comes down to the acceptance of the community, similar to sports. There was a lot of analogies where I would sit with engineering managers or VPs and would feel literally like sitting and talking to these coaches or these general managers from sports where there wasn't a real acceptance yet of this. They're like, no, my best engineers, they do this, they're, you can't capture what they do with data. So I think that has to change and I don't think it's quite there yet. And then I think the data has to be better and understanding performance.

• 32:14Patrick O'Shaughnessy?

  I've never thought about it, but it's almost embarrassing how little predictive modeling or data is involved in any hiring process. It's entirely mostly qualitative feel and short, but an interesting problem is what are you trying to model? What does success mean for a hire? Seems like a hard problem.

• 32:30Speaker 03

  So that's really fascinating because at the core, what you just said is, How do we hire people? We hire people by interviewing. And what is the most bias that you could ever introduce? It's like you and me sitting across from each other. And there's this whole concept at Twitter where they made us do a lot of this unconscious bias training. And at first I was kind of like, oh. And then I went in there and I was like, man, this is really valuable to really be aware of even the concept. I definitely thought, oh, do I want to go have a beer with this person? That's a criteria for whether I want to hire them. But that's like sort of the ultimate in bias. Whether I want to have a beer with them doesn't really describe whether they're going to be good at their job. It describes whether they're like me. So it's a terrible thing that I probably was doing in an interview process.

• 33:13Speaker 03

  But yeah, I mean, it's, I think ultimately, when you think about, for me, when I think about interviewing people, a lot of it is just around sort of this intellectual horsepower, just this ability to sort of really think creatively and to be intellectually curious. Because the whole idea of matching skillsets and things like that, I don't really believe in that, especially in the world of startups and things like that, where you're constantly adapting or changing what you do. It's really much more around, hey, how do I find someone that can challenge me and can really think on their feet?

• 33:43Patrick O'Shaughnessy?

  Do you think there are reliable ways of assessing that in a repeatable way?

• 33:47Speaker 03

  I'm sure there are, but I don't know them. And I would say that I'm probably very much of a contradiction when it comes to sort of interviewing people, because I do believe that I have an intuition on people. And I do believe that very quickly, I know whether that person's going to be good or not, so.

• 34:01Patrick O'Shaughnessy?

  Fascinating. In building the data science team at Twitter, what lessons did you learn? So for, this is almost every company now is starting to hire a team that does this or individuals that do this. What are some important lessons to think about for people that are building or hiring data scientists?

• 34:15Speaker 03

  Hire scientists. And what I mean by that is nowadays, so many people are quote unquote, data scientists. And so there's a woman that worked for me at Twitter named Yuliana Pasco, and she was an economist, PhD from MIT, and had an undergraduate from Wellesley.

• 34:31Speaker 03

  Just so thoughtful in how she approached the world and how she questioned things. Sort of learned to be much more technical over time and was constantly challenging herself. But this idea of being very hypothesis driven and how you do data science and being, I had really strong people that were chemistry PhDs that had essentially, they go through their PhD program and they realize I don't want to go into academia. So let me go take this data science bootcamp where I learned how to do map reduce and data engineering and things like that. And I learned how to use R and Python and I learned how to program and whatnot. And maybe I do that for six months and then I can come in and take a job at like a Twitter or a Facebook. But because I have this unique approach to

• 35:12Speaker 03

  understanding how to interpret data for science, I can actually be much more useful than someone that has more of a math background or a computer science background and really has a lot more of the technical skills, but not necessarily a lot of the foundational skills of scientific method and things like that. So I'm always much more, when I see a resume of someone that came through a traditional PhD program in a science and social sciences to me are some of the most interesting ones because of just how hard it is to create the right environment. This woman Esther Duflo, I don't know if you've read about her, but she does a lot of real world, in the world experiments to understand causation. And it's just fascinating how they set up these sort of real world kind of experiments that

• 35:55Speaker 03

  I personally like, didn't really study any economics in college. And like, I think it was a huge miss by me. And I'm going to encourage, I have two sons, I'm going to encourage them to study economics as much as they can, because just the way that even the way that economics is involved from sort of classical economics to now like behavioral economics and all the work that Kahneman and Chiversky and all those guys have done, it's fascinating to understand how people think and interact.

• 36:19Patrick O'Shaughnessy?

  Yeah, it's really good advice and maybe even like an arbitrage of sorts because the data science math heavy engineers are in such, it's very hard to find these people and to hire them. And maybe people that haven't yet built up the technical skillset would be easier to hire. Do you think that that's true?

• 36:35Speaker 03

  There's definitely an opportunity where People that are, "Oh, my God, I just spent six years doing a PhD and I've decided I don't really want to do chemistry, or I really don't want to do X." I mean, first of all, just the selection bias of someone that decided to do a PhD and got through it and defended a thesis. And they've run teams before. They just don't even realize it. They've run little research teams, they've run their lab or whatever. So they have all these skills that they don't realize. And classically, their whole life, they've been thinking, "Yeah, I had never thought about the arbitrage opportunity, but I definitely think there is." And oftentimes, in the data science world, I haven't really been in it for like a year or a half when I left Twitter. But back in those days, there was a cookie cutter type thing that people were often looking for.

• 37:19Speaker 03

  And again, back to Yuliana, one of her big things was that she always wanted to look at different types of people to bring in, not the normal cookie cutter ones. And I think it was because she came from a very non-traditional background. So yeah, I hadn't thought about that, but that's probably very true. But you have to start with a core team or a couple of people that can help these people, because they won't be able to be, they're not going to feel comfortable in a situation. And that's another thing that I've learned about. In the entrepreneurial world where I come from, no one sweats whether they can do a job or not. They just figure it out. But these people that have these deep analytical minds, they sweat themselves whether they can actually do a job.

• 38:00Speaker 03

  They clearly can do it. They're brilliant. They're coming from a very narrow focus to going into something. And so the unknown for them is so scary that they lose a lot of... One of the guys that worked for me at Twitter used to say to me, "We decided, there was a group of my team that were incredible and that I was always pushing, and they were always rebelling against me when I pushed them." He said to me, "We've kind of decided that you are right about us, but we're never ready for you to be right about us." So.

• 38:25Patrick O'Shaughnessy?

  Fascinating. What were the most interesting problems that you were tackling at Twitter?

• 38:31Speaker 03

  Yeah, I mean, there are a few that were super interesting. I think one was simply around user growth and it was understanding. So I was at Twitter when we were struggling quite a bit with user growth and the stock had gone from, say, 50 to 14. I think it's close to where it bottomed out. And we needed to really rewrite the story. And if you think about all the companies in the world that had come before, Twitter that had gotten to a plateau or even like a start to, no one had recovered. Twitter is the only story of a company that's done so. And one of the things that was challenging at Twitter is that the top level metric that we sort of use, monthly active users, did not really indicate or wasn't really indicative of what a truly healthy Twitter

• 39:16Speaker 03

  user was. A healthy Twitter user does not come once a month. A Twitter user comes every day. From a standpoint of monetization, you know a lot more about a user that comes every day and you're able to provide more value to them through the advertising chain and they provide more value to advertisers. And so one of the things that we did there and that they have done is look much more at this concept of month of daily active users. And that's to the credit of all the people that work at Twitter. That's the number that they've been able to move quite a bit and has led very much to the success of Twitter over the last few years. But moving the goalposts as a company from monthly active users to daily active users and even doing the data science. There's a guy named Matt Miller who is out of Boston, came out of MIT's Media Lab

• 40:01Speaker 03

  and was at a company called Bluefin, worked with a guy by the name of Deb Roy. These guys did a lot of the foundational work where they tried to understand via modeling what true healthy usage of Twitter was. They created HMM to sort of do this. And what they ended up finding is that the healthiest users mapped really well to sort of daily active users. And so all this data science led to saying, hey, guys, let's really start using daily active users as a target metric. And the concept of how important target metrics are to incenting the right type of growth of the right type of behavior at a company, but constantly reevaluating those target markets. Because once you really start getting good at that, you need to reevaluate. ...them because probably your optimizations have caused you to almost bastardize that target

• 40:45Speaker 03

  metric. So that was some interesting work. We did really interesting work around the elections and around understanding Russian interference in the elections. I had a team that did a lot of the foundational work for what we reported to Congress and sort of all the different things that you could find within the network. And so it was interesting because so much of what we needed to do is was work in lockstep with sort of user research to understand both almost like an anecdotal versus a analytics level, what's happening and have those two feed in because the Russians are doing all this crazy stuff on Twitter where they're creating these fake accounts and creating coordinated efforts. And this is nothing that hasn't been in the news in

• 41:27Speaker 03

  terms of just the amount of stuff that they did for manipulation. But my team did a lot of the foundational work to sort of help figure that out. And that was fascinating. And then there was also some work that the team did around understanding how coordinated misinformation was happening on Twitter and literally how people were trying to create fake news and trying to get fake news to bubble up in surface areas where search algorithms or algorithms would drive what you saw. And they were clearly understanding the algorithm and using different types of coordinated attacks to try to move things up. And that's what caused Twitter, the work that my team did, caused Twitter to change the sort of algorithms around some of these surface areas

• 42:09Speaker 03

  where they were able to rank tweets or rank accounts in terms of how much you could trust them.

• 42:14Patrick O'Shaughnessy?

  What was the most interesting finding there in terms of the strategies being employed by people trying to manipulate the existing algorithm? What were they doing?

• 42:22Speaker 03

  Their behavior was around, I think a lot of it was around amplification versus creation. So a lot of these accounts were amplifying and not creating and people that were doing that. And one of the things that was interesting was we found that many of their behaviors were similar to behaviors that we actioned on. There was almost like, okay, certain things are a violation of terms of service. And then certain things are sort of in the gray area that aren't quite violations of terms of service, but are not potentially acceptable. And those things that ended up being in the gray area, which weren't violations of terms of service, but were clearly coordinated attacks, were the things that we found were probably the things that were leading to manipulation.

• 43:04Patrick O'Shaughnessy?

  What do you think the responsibility is, is a huge topic, of these social platforms to police themselves? I think it was just last week that Twitter said you can't pay for political ads. Facebook has said the opposite. What do you think the responsibility is?

• 43:17Speaker 03

  I think it's a hard question. And one of the things that's amazing about Twitter, and I loved my time there, is just how inspirational Jack is in many ways about being fundamentally principled in how he thinks about the world. There is not a day or a moment that Jack is not thinking about what is Twitter's role in the world. He's not thinking about how does Twitter make money. He's thinking about how does Twitter continue to shape public conversation in a meaningful and important way. So I think the decision he made ultimately to not have political ads was through very thoughtful process of what role should we play?

• 43:57Speaker 03

  His statement that reach is earned, not paid for, is a very interesting thing to say. This sort of ultimate democratization of reach Twitter provides is something that he was interested in. I think it's really challenging right now, and this is the whole deep fake thing. There's so many ways that you can fool people into believing things. And in many ways, it's not, this is not new, right? This is like, media's been biased since back when they said you didn't have to report both sides. And all of a sudden, now this has become a thing that technology has made even more challenging. So I think I'm in favor of it, as long as they're able to really... They're trying to crack down on many things. They have a whole team right now that works on election manipulation and tries to understand it, not just in the US, but all around the world.

• 44:42Speaker 03

  And the idea of sort of like this overcoming political ads, which clearly have some level of fake news attached to them, I think it's important. I think I'm in favor of it. I don't know. How do you feel about it?

• 44:54Patrick O'Shaughnessy?

  I don't know. Like you said, I think it's a really, really hard question. I think the idea of reach being earned is a fascinating idea. The only question I have is, is it easier to earn a wider reach through nefarious methods than through legitimate, high-quality methods? Meaning the whole idea of lies traveling faster than the truth? That if you can play to people's biases, or tell them what they believe already, that's one of the fastest ways to grow a reach? So then, are we implicitly giving all the power to people that can do that effectively? So, that's a concern. I don't know that that's true, but it's a very complicated question, as you say.

• 45:31Speaker 03

  Yeah, I mean, I think that that's one of the things that Twitter struggled with a lot when I was there, and we realized it was a problem, which was this echo chamber that you create by who you follow, and ultimately, it creates this almost self-fulfilling prophecy of what you believe. I think one of the things, as we're talking about this idea of reach being earned versus bought, there is a way that people can basically buy followers over time, or can use advertising to promote their own account over time. So even probably some of the reach that political people have has been bought at some level. So it's challenging, but you do have to draw the line, and I definitely applaud Jack for the decision he made.

• 46:05Patrick O'Shaughnessy?

  Yeah. You mentioned earlier, thinking about things where maybe today, there's not a lot of interesting things going on, but in five or 10 years, something's going to be big. With my kind of investor and business hat on now, which we haven't talked about a whole lot, are there any areas that, if you were just an investor, let's say, you think are underappreciated right now in terms of industries or ideas that are going to grow a lot in the next five to 10 years?

• 46:28Speaker 03

  Yeah, I mean, I think this is, in many ways, the journey that I'm going through right now on a personal level, because I'm trying to figure out what I do next with my life. And so, I think I told you, the last time I was at this stage was about a year and a half ago, and that's when I discovered your podcast. And I would sort of walk around in San Francisco, because it's really hard to get around the city now, and if I had to go from meeting to meeting, I'd just pop the podcast on. And I would sort of discover many things from your podcast around the way that people are using AI or machine learning. I mean, I think last year, you had much more of a focus on that, and it was interesting to think through. I mean, I think, obviously, the obvious answers are things like blockchain, and what blockchain enables-

• 47:12Speaker 03

  Obviously crypto gets way more attention than the underlying technology of blockchain, but the underlying technology of blockchain and tokenization of assets and tokenization of what were physical goods is interesting. I think the idea of the shared economy of renting and whatnot, I think we've only sort of touched the iceberg on that. The idea of delivery and just one idea is probably cities, how cities function and how that sort of idea, I don't really know. If I had something that I was really, really interested in, I'd probably go out. I guess one of the things I'm interested right now is the evolution of media. Okay. So if you think about Netflix or Hulu or Apple Plus and

• 47:57Speaker 03

  all this kind of stuff, they're all like quote unquote evolving media and Quibi, let's take Quibi for the Meg Whitman and Jeffrey Katzenberg company. These are all this sort of evolution of media, but they're not evolving media at all. You're literally watching the exact same thing you could have watched on your television 25 years ago. It's just better, better quality. It's better written. There's better actors. You know, even Quibi, from what I hear, the changes are going to be very not what you would call groundbreaking. They're going to be shot a little bit differently optimized to the phone. They're going to be shorter, that kind of thing. It'll be interesting to see like true interaction or true evolution of media, I think will be one of the things to watch because my guess is that there are

• 48:40Speaker 03

  going to be winners that come out of this world of the Netflix and the Hulus and the HBO Maxes, but it's going to be people that really embrace the idea of Silicon Valley meeting Hollywood. That's what everyone wants. Everyone talks about, but there isn't really an example yet of a company that's done that. I think where there really is evolved media or evolved media experience.

• 49:02Patrick O'Shaughnessy?

  Do you think that that looks like video games and traditional media sort of converging? It's just more interactive?

• 49:07Speaker 03

  Yeah, I think so. And I don't know what it looks like. So I don't want to sit on my high horse and say, I understand this. I just know that it isn't what it is right now. The idea millennials, none of us are millennials, but millennials and Gen Z like when I think about sports, I don't see a world where they are going to sit through a three and a half hour football game.

• 49:26Patrick O'Shaughnessy?

  It's already true. They're on their phone.

• 49:27Speaker 03

  And so how do you make that content or that experience the same? And a lot of times, the other problem is that people conflate something like esports with sports and it's not. It's a totally different thing. It's not like because I'm a sports fan, I'll also want to go watch League of Legends at HP Pavilion or something like that's not true. I think there is a real disconnect right now in terms of media consumption and media companies, ideal media consumption and media companies, what media companies believe it to be.

• 49:56Patrick O'Shaughnessy?

  Well, this has been awesome. So interesting, so different than other conversations I've had recently. And I think a lot of threads for people to pull on. You probably know my closing question for everybody, which is for the kindest thing that anyone's ever done for you.

• 50:08Speaker 03

  Yeah. So I'm going to make a little bit of a long answer on this because I have listened to your podcast and I've thought about this a lot. I think anyone that's born of immigrant parents who came here basically to make their kids have a better life, their answer has to be their parents. My dad and mom came here from Taiwan. They were originally from China when the communists took over. My dad, they never had cable in their lives. And if you talk about a great lesson about compounding. My dad was a professor. My mom was a nurse and estheticist, great jobs, but none of them are what you would consider to be jobs that would set you up for the rest of your life in terms of. But they never had cable. The only money they really ever spent was on our education. Both my sisters and I went to Philip Sacks or Academy and then to MIT.

• 50:50Speaker 03

  So that's not a cheap amount of money at that level. But yet, when my mom got sick, I sort of turned to my dad and I was like, hey dad, how are you doing financially? Are you okay? And he was like, well, you know, this is many years ago, but he's like, oh, I have about $5 million in my retirement account. And I was like, what the? And so I think clearly my dad and my mom would be my answer, but that's the easy answer. And the other answer is a guy by the name of Kevin Compton. So Kevin, Kevin is close with Sam Hinky and that he's the operating partner at Kleiner Perkins for the heyday and now runs a company called Radar Partners. I invited him to my wedding and he at the time owned the San Jose Sharks and was this tremendously passionate owner about the Sharks.

• 51:33Speaker 03

  The weekend that we were getting married down in Newport Beach. The Sharks had game three against the Detroit Red Wings in Detroit. And rather than going to that game, he came to my wedding and because the game was Friday night when he needed to be traveling, instead of flying, he drove from San Jose down to Newport Beach so he and his wife could listen to the game on the radio so they wouldn't miss it, but they could still make all the events in time and have it all timed perfectly. And that idea of him coming, and he's a guy that's always shown up for everything for me, both from a social and a professional level. It's a guy that has had all the success in the world. And they always say the classic thing of how you know about someone is how they treat someone that can't really do anything for them. Everything he's done for me has

• 52:18Speaker 03

  been out of kindness and out of love, so.

• 52:21Speaker 04

  Two incredible answers. I have one follow-up on your parents. So what would you say are the values that they most instilled in you in your upbringing?

• 52:30Speaker 03

  Yeah, it's interesting because when I think about my own life now, they say each generation, we're becoming more and more friends with our kids. And my parents, I love them to death, but we were not friends. They were my parents and they pushed me really hard. I mean, the obvious thing is education, but I think really the thing that they taught me the most is that in life, there are times you have to do things that you don't want to do. That's just life. And you do things because you know that you're supposed to do them and you know them because it's the right thing to do. And life isn't always fun and you have to do things that are hard. And that's, I think, the thing they taught me.

• 53:03Speaker 04

  Well, this has been one of my favorite recent conversations. So thanks for sharing the three interesting stages of your career. Can't wait to see what you do next and I hope to stay in touch. Thanks.

• 53:12Speaker 03

  Thank you.

• 53:14Speaker 04

  Hey, everyone. Patrick here again. To find more episodes of Invest Like the Best, go to investorfieldguide.com/podcast. If you're a book lover, you can also sign up for my book club at investorfieldguide.com/bookclub. After you sign up, you'll receive a full investor curriculum right away and then three to four suggestions of new books every month. You can also follow me on Twitter, at Patrick underscore Oshag, O-S-H-A-G.

• 53:42Speaker 04

  If you enjoy the show, please leave a quick review for us on iTunes, which will help more people discover Invest Like the Best. Thanks so much for listening. [Music]