Annie Duke - winning bets: decision science in venture capital

Annie Duke is a former World Series of Poker champion, bestselling author of Thinking in Bets and Quit, and decision science expert who advises Fortune 500 companies on strategic decision-making. In this episode of World of DaaS, Annie and Auren discuss: Cognitive biases in venture capitalLoss aversion and mental accountingOne-way versus two-way doorsReading people in poker and business Looking for more tech, data and venture capital intel? Head to worldofdaas.com for our podcast, newsletter...

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Auren Hoffman?

  Hello fellow data nerds. Welcome to World of DAS. I'm your host, Orin Hoffman, CEO of Incubate and GP of Flex Capital. Discover more episodes, get weekly data as a service news, original content, articles on data and more at worldofdaas.com. That's world of das, D-A-A-S, dot com. Hello fellow data nerds. My guest today is Annie Duke. Annie is a former professional poker player. She's won over $4 million in tournament winnings during her career and holds a World Series of Poker Gold bracelet from 2004. Annie's also the author of a number of bestselling books, including Thinking in Bets, which is one of my favorite books, and Quit: The Power of Knowing When to Walk Away. Annie, welcome to World of DAS.

• 0:45Speaker 01

  Thank you for having me. I'm very excited.

• 0:48Auren Hoffman?

  Very excited. Now, when you look at how venture capitalists evaluate deals, what are the big cognitive traps that you see in their decision process?

• 0:58Speaker 01

  I'm gonna try to bucket these. The first one is that I think that there's, in general, in venture, a lot of mystique around gut decision making. That's a little bit like what the Midas List is about. It's this idea of there's a magical investor who just has a really good eye for talent. And there's just a lot of reliance on that, almost as your decision making process, that this person who has such a good eye for talent or nose for value should really be driving the train on a decision about whether to invest. That's the first big bucket that I think is a really big trap. The second has to do

• 1:39Speaker 01

  with a lot of cross-influence and then this really big problem of, I think, pretty bad record-keeping, like accumulation of data around the decision point itself.

• 1:51Auren Hoffman?

  Because when you go back to your winners, you remember these people as being super charismatic. And when you go back to your losers, you remember them as stumbling and you just invested anyway. You're not sure why type of thing?

• 2:01Speaker 01

  There's also the problem of, when you're operating a world that's governed by power law, it's very easy to just say, well, that was just one of the losers in my portfolio. You're hoping for maybe three fund returners in a fund out of 25 to 40 investments. So, in a world where there's so much luck... It's really easy just to say, I lost that. That doesn't have anything to do with my decision process. That's just the way it is. That one that I won, I totally saw it. So I think that's a big bucket as well. And there's no way, if you're not keeping a really good record, creating a really good artifact of your decision at the time that you invest. I don't think there's a really good way to resolve that, to really look back and say, was that a good decision or not?

• 2:46Speaker 01

  And I think that's really hard. So that would be bucket number two.

• 2:49Auren Hoffman?

  Is part of the problem is just these decisions are very long range. You might not know for 10 years or so.

• 2:55Speaker 01

  That I would put in bucket number three is pretending like things are a long cycle when they're not. That's a great one. I wasn't going to bucket that one, but I'll just put that in one big bucket. Just sort of waving your hands and saying it's very long cycle, but it's actually not. Number three would be that I think there's not enough attention paid to follow on decisions in a variety of ways, and we can talk about that.

• 3:16Auren Hoffman?

  Let's talk about the follow on decisions a bit. They always do follow ons or they don't do super parada enough, or they're not as discerning. Or where do you think their decisions could be better?

• 3:28Speaker 01

  There's a few problems. People very often follow on at the next round, not necessarily as a value bet, but because they're trying to support their investors. And so I'm not talking about that. I'm not talking about a different bucket of following on, where you're following on for reasons that don't have to do with really trying to discern which companies in your portfolio are going to have the highest return or not. But you're doing it because you're being supportive of your founders. That's a totally different bucket, which I have no critique of. But there are lots of follow on decisions that are made that are not for that reason, that are actually value decisions, where you're actually trying to identify the companies that you have put in your portfolio that you believe are going to have the highest return. And so therefore you would like to put more money in them

• 4:13Speaker 01

  because you think the probability that they're going to be a fund returner is very high. What I think that I've found when I talk to venture firms that aren't my clients is that there's just a lot of, if point partner is pounding the table, then we're going to do it. I think people need to be more aware of some of the cognitive biases that can really get in the way of good decision making around these types of decisions. Number one is that everybody should familiarize themselves with endowment bias. So the endowment effect is that you like things that you own more than identical things that you don't own. That includes ideas. We've all heard where like you'll give some idea in a

• 4:53Speaker 01

  meeting and then somebody else who sort of poo-pooed your idea will say the same idea and they'll think it's great. And you're like, wait a minute, what happened there? It's true for ideas as well. So we just like things that we own much more than identical things that we don't own. So, simple example of that would be, I'm sure everybody at some point in their life has sold a car. When you look at the Kelley Blue Book, what you're supposed to be getting for the car, you're like, well, obviously I should get top end for this car. This is a great car. But if you were buying the identical car, you would be arguing for the bottom of Kelley Blue Book. There's no question that that's true. The science on this is very strong. It's well replicated that people just like stuff they own better than stuff they don't.

• 5:36Auren Hoffman?

  On the VC perspective, it seems like a lot of VCs, when, you know, let's say they're evaluating doing more money in the Series D and they were in the Series A, and it's already up 50 plus X, and it's already maybe potentially a fun returner for them. Sometimes I see the bias going the other way where they're like, oh, this is already a fun returner. I shouldn't put more eggs necessarily in this basket. Maybe even I should take some off the table here. I feel like much more likely they're going the other way at that point.

• 6:07Speaker 01

  It depends on where you're following on. Endowment is a problem, which is this idea of point partner pounding the table. One of the concerns that I always have is that point partner is probably the most endowed. As we think about that, I mean, obviously, the whole partnership has that company in their portfolio, but point partner is going to be the most endowed, which I think is just an interesting problem. The problem that you just talked about, and then there's a third problem that I'll get to. The first finding in this category is from Kahneman-Tversky 1979. So, Daniel Kahneman actually ended up eventually winning the Nobel Laureate for something called Prospect Theory. And this finding from 1979 was really a key point of Prospect Theory, and it has to do with

• 6:50Speaker 01

  loss aversion, sure, loss aversion, and then this idea of how we deal with gains. So, we've got Kahneman-Tversky 1979. They offer two propositions to participants in the study. So, the first one is, I can give you $100, or we can flip double or nothing.

• 7:10Speaker 01

  And you can win $200, you can win zero. And the question is, are people willing to take that bet? Obviously, the expected value is the same. It's $100 either way. But will people take that bet? And the answer is no, they will not. But when you say you owe me $100, or we can flip double or nothing, do you want to take that bet? Then the majority of people will take that bet. So notice that in both cases, the expected value is the same. The only thing that's different is whether you're in the gains, so I can give you money, or you're in the losses, so you can give me money.

• 7:43Auren Hoffman?

  Question for you on the first one, which I think is a simpler one for people to understand. What are the odds that people are willing to switch? Because 50/50, I'm taking 100 bucks for sure. But at 60/40, I'm probably willing to roll the dice.

• 7:58Speaker 01

  That's an interesting thing. So everybody has a different indifference point. But basically, the finding is if I say I can give you $100, you understand expected value. But what is your instinct about what most people would say? If I say you've got $100 in your hand, like I'm going to hand it to you, or we can flip double or nothing, you'll win 220 or win zero.

• 8:20Auren Hoffman?

  My expectation is they take the 100 on the 220. It would have to be probably over 300 for the average person to be able to. Now, $100, maybe not. Now, if you're talking $10,000, maybe at 100 bucks, they're like, sure, I'll take the 220 because maybe neither of them moves the needle for them.

• 8:36Speaker 01

  But they don't take the 220, not even then.

• 8:39Auren Hoffman?

  These are always with poor college students. But let's say with the average American, where like $10,000 really changes the game for them. They'll probably take the $10,000 rather than roll the dice for $22,000 or something.

• 8:51Speaker 01

  Of course, in the 100 versus 220, they're costing themselves $10 in that spot in order to not take the gamble. So basically, they're paying. To take the variance to zero. What they're trying to do is take the sure gain. So the pain of having been up 100 and going to zero would be too great. So now we can take the other side and say, you're going to have to give me $100 or we can flip 0 or 220. People take that bet. So notice that's $10 worse for them. So now they're paying $10 to keep the gamble on.

• 9:25Auren Hoffman?

  That just seems crazy to me. Assuming they could afford the 100 bucks. Now, if someone can't afford it, I could see how they would take the bet. Because, well, they're going bankrupt anyway, so they might as well take the gamble. Because if you're like, oh, it's a million dollars loss and they can't afford the million dollars and the gamble makes a lot more sense to me than, well, with 2.2 million, who cares? I'm already so much in debt, like, it doesn't matter. But 100 bucks doesn't make sense to me on that side.

• 9:49Speaker 01

  What's interesting there is that that's why that paper was so groundbreaking. Because prior to that, economists who could calculate the expected value were like, well, if you offer those two things, people won't behave differently. So let's go back to the original one. So what an economist would have said is that you have some sort of risk appetite. And so if you're willing to take the gamble when you're in the gains, you would also be willing to take the gamble when you're in the losses, because those are symmetrical problems.

• 10:18Auren Hoffman?

  It's not exactly true, because if you add a million dollars to your net worth, let's say your net worth is $10 million and you add $1 million to your net worth, your net worth only goes up by 9%. But if you take $1 million out of your net worth, your net worth goes down by 11%. You are actually way worse off to lose that $1 million.

• 10:38Speaker 01

  So we can control for that. What economists would have said prior to this is people have risk appetites and people are either risky or not. If they're willing to take the risk and flip the coin in the gain situation, they will also do that in the loss situation and vice versa. But it turns out that people are very different when it comes to the two propositions. They will not take that bet when they are winning, but they will take the bet when they're losing, which actually, interestingly enough, is the opposite of what you just said. Even if we take into account the 10 versus 11% or whatever, you would predict the opposite. When we're actually calculating, what would a rational actor do? This is not what they actually end up doing. They take the gamble when they're in the losses

• 11:22Speaker 01

  and they take the money when they're in the gains. So they refuse the gamble. What's happening there is that we are very averse to taking losses on paper and turning them into realized losses. And we seek to take gains on paper and turn them on to realized gains, even when it is irrational for us to do so.

• 11:44Auren Hoffman?

  The venture capitalist is very weird. So there's a scenario where, let's say you're going to write down a company to zero today. Or there's 100% probability you're going to write down that same company in five years to zero. You would think you'd want to do it as soon as possible from your IRR perspective or something. Or even simpler, let's say the company can give you back all their money today that you put in. So it's a 1X today. You put $1 million into the company, it'll give you back 1X today. Or the company will give you back 1X five years from now. You would think every venture capitalist would rather have the 1X today, but almost every venture capitalist would rather have the 1X five years from now because they're fundraising, there's stories on it. There's other types of things that they would prefer.

• 12:29Speaker 01

  It's not just that, though, it's this. So if I write a company down today, I am taking the loss. As long as that company exists on paper, I have not lost yet. So this is the exact same problem. It's the reason why people will flip the coin, even though they could lose 220. It's the exact same thing. Why are you keeping the risk on? Because if I write it down today, I'm just taking the risk. I'm like, I'm done. So why are you keeping the risk on? Because as long as you do that, you haven't actually realized the loss. That's the way that it feels to us. In venture capital, if you take a 1X return, that's a loss. You're just not going to do that because the frame in venture is 10X or better. So that's kind of the frame.

• 13:14Speaker 01

  Anything that's less than that, you're sort of considering a loss. Why are people taking money off the table at Series D with an amazing company that's already returned 50X? Well, because they're already in the gains. And we know that there's some probability that even if a company is marked at 50x, it can go to zero.

• 13:31Auren Hoffman?

  We've seen it happen many times.

• 13:33Speaker 01

  Exactly. If you don't continue to bet on that, and in fact, if you trade on the secondary, you're realizing that gain. You're taking all the risk out of it. And we like to do that when we're in the gains. This research became the cornerstone of prospect theory, which ended up winning Daniel Kahneman, the Nobel laureate in economics. But this is very basic human behavior. And what you're pointing out about these scenarios that you've offered up in venture is just a real life instantiation of prospect theory of what you see. And if people are interested, there's very interesting work by Richard Thaler, who's also a Nobel laureate in economics on mental accounting.

• 14:15Speaker 01

  And what I think is important to note is this. So one of the things that people will say to me is, but why would you behave that way toward one company? Isn't that the whole point of having a portfolio? And that's the base of portfolio theory. What Richard Thaler's work shows is that you don't think about it as a portfolio, though, when you're approaching a decision about a single company. If we put a company in our portfolio, we've now opened an account up, a mental account for that company. And we're tracking that company. We're not actually thinking across the portfolio. Because obviously, if we're thinking across the portfolio, you might not make the types of trades that you're talking about.

• 14:55Speaker 01

  You might be willing to take 1x today. You might be willing to write a company to zero today. You might be willing to do what you're rationally supposed to do at Series D, but you don't. If you're actually thinking about it in the context of the portfolio, but that's not the way we think about it. We think about it, we open an account for that thing. A classic example, and I think this is a pretty easy thought experiment for people to wrap their heads around, is you buy a stock at 50 and it's trading at 40. If you were approaching the stock fresh today, you would not buy it. Do you sell it? And the answer is no, because you want to get your money back. Well, that's silly, because it's like across all stocks and there's opportunity costs associated with that. You could put that $40 into another stock that has a higher

• 15:40Speaker 01

  expected value, but we won't do it. How do we know that this isn't a ledger problem, but it's actually a cognitive problem, mental accounting problem? Well, because of this scenario. You buy a stock at 50. It goes up to 75 and it's trading at 75 for a while. And now it goes down to 60. Do you sell? Notice that on your physical ledger here, you're $10 in the gains. But on your cognitive ledger, you're $15 in the losses because you were at 75. And people won't trade under those circumstances either. So what's interesting is, once you can clear the account, we'll tend to approach risk more rationally. But we have to clear the account in order to do it. In other words, if I do actually get forced to sell that stock or somehow I end up selling it,

• 16:24Speaker 01

  the next decision that I make about a new stock is going to be more rational than the decision I would make about that stock. There's some really fun research which shows, using casino players' tracking cards, that when people are playing slots and they get into a loss situation, that they'll really play a very long time. So they'll continue to go. But once they leave for the day.

• 16:45Auren Hoffman?

  Once they go to sleep and come back, then they're more rational again.

• 16:47Speaker 01

  Because it's a new account. So what's happening is that as they get in the losses and you sort of track them at the end of a session where they're in the losses, their bets get bigger, they play faster. They're really just taking on more risk. But then when they leave, the next time they come back, they start at their baseline again. Because they've closed that mental account.

• 17:06Auren Hoffman?

  And that is because of that mental account, not just because they're more wasted or something?

• 17:10Speaker 01

  No, no, no. This is a very large scale study.

• 17:14Auren Hoffman?

  If you think of a private equity person, their main job is just never to lose money. That's the worst thing that they could do. And there are some private equity firms that have never lost money on a deal. If you think of a venture capitalist, which is different, their biggest mistake they could make is not investing in that 50-bagger that comes across their desk. Because that 50-bagger may only come across their desk every few years. And they have to make sure that they invest in that particular thing. And if they don't do it, it's just a huge monumental mistake that they make. How do you square those when you're talking to people?

• 17:49Speaker 01

  This is true of all decision making. You need to figure out what type of error is worse. So what you're describing is a type one or a type two error. You've also heard it talked about as specificity versus sensitivity. That was something that got tossed around.

• 18:07Auren Hoffman?

  Or kind of a false positive versus false negative or something like that.

• 18:10Speaker 01

  Exactly. Let's think about it as sensitivity versus specificity to start. Let's imagine that we're testing people for COVID. We're way back when, we're back in 2020 in the pandemic and we test people for COVID. A sensitive test is always going to capture people who have COVID. What that means is that you're going to have a lot of false positives because it's going to be flashing a lot. It's gonna be like COVID, COVID, COVID all the time. Now, you're never going to miss somebody who has COVID in that scenario. Specificity would be that you're really trying to only catch the people who have COVID. In that particular case, we're going to miss a lot of people who have COVID. There's always a tension between the two.

• 18:53Speaker 01

  And it depends on a variety of factors, whether you're going to lean towards sensitivity or specificity.

• 19:02Auren Hoffman?

  If we're going to test everyone in the country and only 1% of people have COVID, we wouldn't want to test with even a 1% false positive rate because then literally 50% of the people who take the test would say they're positive when they're not.

• 19:17Speaker 01

  That would depend on how dangerous the disease was. In a case where you were 100% to die, if you had it, you would be very tolerant of false positives.

• 19:26Auren Hoffman?

  If that case, you could freak the other 50% out or something and they might do something crazy.

• 19:31Speaker 01

  That is true. Let's imagine something that's highly contagious and kills everybody who gets it. It's totally fine if you have a test that's really high on sensitivity. And low on specificity, but we don't want that for something that is lower down on the danger rate. One of the ways that I like to think about it is it's really good for mice out in the wild to index on sensitivity as opposed to specificity, because if they don't run away when a predator is coming, they're little tiny mice, they're gonna be dead. So it's better for them to just run away way too often.

• 20:10Auren Hoffman?

  Just assume the danger.

• 20:12Speaker 01

  Assume the danger, run away.

• 20:15Auren Hoffman?

  I'll do that, too. I'm not going through that brush if I think there's a lion there.

• 20:19Speaker 01

  That's totally fine. But if you're an elephant, so now maybe you're not just running away because you can fend off a lot of predators and there's a big energy cost to you to actually run away like that. That's something that actually is stuck in my brain. So mice, so let's bring it to like type one versus type two error. Mice are making a lot of type one errors. Lots of false positives. They're like blink, blink, blink, predator, predator, predator all the time. And they're running away a lot when they probably don't need to, but that is better for them because they're little tiny mice. Elephants are gonna make more type two errors, which is false negatives. But then again, they're very big. There's a big energy cost to running away and that kind of thing. So elephants and mice should

• 21:00Speaker 01

  actually be different in that way. Let's bring this back now to all decision making. With all decision making, what you have to ask yourself as you go into the decision is what's worse? Is the bigger error a false positive or is the bigger error a false negative? Because that's going to tell you where you're supposed to set your threshold for good enough. What's the point at which I've built enough conviction in order to be willing to invest in this? So if we take a long short fund, for example, they're gonna not want to have a lot of false positives. They've got a very small portfolio. They tend to hold a very small number of positions for a very long time.

• 21:41Auren Hoffman?

  And they can be choosy. They can pass on tons of deals. If it works out, it works out and they can learn from it.

• 21:47Speaker 01

  Yeah, if they miss something that makes a lot of money, it's fine because they can probably get something else that makes a lot of money. Like, it's not really a big deal. And so you're not really too worried about false negatives in that world. But in venture, you're very worried about false negatives because there's a huge penalty because of power law. There's a huge penalty to missing a winner. That's a ginormous penalty. So generally, what we can sort of think about there is that if we're really worried about false negatives, what that means is that we're going into it understanding that we're going to purposely end up with duds in our portfolio. We don't know which they're going to be.

• 22:24Auren Hoffman?

  You're willing to have a higher loss ratio essentially than two.

• 22:27Speaker 01

  We're going to have a much higher loss ratio, but we're tolerating that and we're going to construct our portfolio to withstand that. If you look at some long short funds or Berkshire Hathaway or whatever, they have a very small number of positions, whereas venture firms have a very large number of positions.

• 22:43Auren Hoffman?

  Though even most venture firms in a given fund, they may only have 20 positions or something like that. Do you think they should have 50 to 100 positions or more in those funds to deal with that?

• 22:54Speaker 01

  There's a trade off. Let's take it to the extreme. If I invest in everything, I will never miss anything. So remember, just because we're saying we're not really tolerant about false negatives, it doesn't mean that there aren't going to be false negatives. If you're investing well, two things are always going to be true. You're going to pass on something that wins and you're going to invest in something that loses. And I can always guarantee that. The worst mistakes that you can make, I think, as an investor, there's two of them. One is looking at the bottom performers of your portfolio, drawing conclusions that are way too big, given the amount of luck. And starting to change your investment process somehow looking forward to prune the bottom range of your portfolio, I just think that's a huge error.

• 23:39Speaker 01

  Not that you can't learn something from your losers, but it's probably a relatively small sample size and there's a lot of luck. And if you reran it 100 times, those might be the top of your portfolio. There's a bunch of stuff that can happen there. That's one big mistake. And then the other big mistake you can make is trying to make it so you literally never have a false negative. Then you're just not going to be choosy at all. And what's your point on earth? Why are you a steward of somebody else's capital?

• 24:04Auren Hoffman?

  And also, it's be impossible to invest in anything anyway, because of any meaningful amount. No one has like a trillion dollars.

• 24:10Speaker 01

  The whole point, the whole reason that LPs are investing in you is because they feel like you have good taste.

• 24:15Auren Hoffman?

  One of my things I struggle with is just bet sizing. Because you could just say a scenario where imagine that same seed fund, they just put a million dollars in everything and it's just simple, or they own 5% of everything, whatever they decide, and that's how they bet size. Or there's something where you're being a little bit more discerning. I'm putting more risk in this. I'm having my normal allocation, this other thing. How should funds think about that?

• 24:39Speaker 01

  Your mileage may vary. I don't want people to take this as investment advice from me. I just want people to take this as where, because of my training and cognitive science, my head goes with that. In early-stage venture, you're making decisions under extreme uncertainty. That uncertainty comes from two places. One is you know very little. I mean, if you think about an angel investor, hey, I have a pack of gum and idea. There is no product-market fit.

• 25:07Auren Hoffman?

  Yeah, you may know something about the founders. You've worked with them before. You have some sort of sense that they're quality people or something.

• 25:14Speaker 01

  Exactly. So you really don't know anything and there's just a whole lot of luck. If you reran Uber 100 times, I don't know how many times it ends up being Uber, but it's probably low. For every single company, you're not going to do it 100 times out of 100. As much as the people who founded Uber would like to think that they would win 100 times out of 100, I would certainly short that position.

• 25:36Auren Hoffman?

  I would say if you actually look at it at a founder level, and let's say that founder has the ability to found many companies over time, I think that there is a much higher correlation with success. My guess is Travis Kalanick would maybe not be at the same wealthy as if you ran the simulation a million times, but he probably would be pretty successful in most scenarios. If Uber didn't work, he'd do something else or something where we're like, he's just that type of guy who would be pretty successful.

• 26:07Speaker 01

  That's true, although there's a lot of path dependence there. If you do well in one company, you're more likely to have opportunities in another company. But regardless, we've got hidden information in luck. Here's the thing, if he doesn't do well in Uber, you don't know, he may choose something less risky.

• 26:21Auren Hoffman?

  That's true. He might become a product manager at Facebook or something.

• 26:25Speaker 01

  Exactly. We don't know what's gonna happen. So this is where I think about what is our cognitive acuity to be able to distinguish one thing from another. I actually like to analogize it to choosing something off a menu. So you'll see people all the time who are looking at a menu and they're like, I can't decide, do I want the chicken or the fish? They're quizzing everybody at the table and they're asking the waitstaff and they're looking up Yelp reviews, I don't know. But they're having a really hard time.

• 26:56Auren Hoffman?

  Or even like what movie to watch tonight.

• 26:58Speaker 01

  Yeah, exactly. What should I wear? What should I eat? What vacation should I go on?

• 27:02Auren Hoffman?

  It's like the optimizer personality.

• 27:05Speaker 01

  They're trying to optimize, exactly. They finally somehow decide on something like they order the chicken and then it comes back dry and they immediately think they made a mistake. The last time I checked, when they were trying to order off the menu, they hadn't eaten those dishes before. I'm assuming it's a new restaurant. So they don't really know anything. So they're not omniscient and they don't have a time machine. Is that really a mistake? They narrowed it down to two things they really like and then they made a choice. So one of the things I like to point out to people is it's not like you're having trouble choosing between what's a food that you hate.

• 27:39Auren Hoffman?

  I don't like olives.

• 27:40Speaker 01

  Great. So it's not like you're having trouble choosing between the chicken pasta and the puttanesca. That is not a problem for you. Now, the reason why that's not a problem for you is because they're so far apart. That one of them does not meet your threshold for good enough. So we can think about this threshold for good enough. And what does that threshold look like? How do we figure out when two options are good enough? When we say to ourselves, if this were the only thing that I could order, would I be fine ordering it? And if the answer is yes for the chicken and yes for the fish, you're kind of done with the process at that point. They're both good enough. I can really flip a coin between the two and it doesn't matter. By the way, even if you could tell the difference between the two,

• 28:25Speaker 01

  what a waste of freaking time because they're both already good enough. You're totally fine ordering either of them off the menu. Now, how do we decide the threshold for good enough? It relates back to that idea of false positives versus false negatives. The more tolerant we are of false negatives, the lower our threshold for good enough. The less tolerant we are of false negatives, the higher our threshold is for good enough. Okay, so let's just tie that up in a bow. So how does this relate to like position sizing in really early stage venture? I don't know. You get to something that's good enough. Can you really tell the difference between the two?

• 28:57Auren Hoffman?

  Probably not. So maybe I should just do the same thing in everything.

• 29:01Speaker 01

  My personal feeling is that you have a goal and you can decide what the goal is. One of two ways, you can decide that you want a certain amount of ownership in a company or you only want to invest a certain amount in each company. Whatever, I don't care. You be you. But naturally what's going to happen is that you're going to end up with different sized bets regardless, because you can't always get what you want. You can't always get your allocation. And then what you should actually think about is this. Okay, so I've got my allocation. Now I know more about the company than the market does. And then you probably can discern later. That's not what everybody does. And other people do sort of think about big bets, medium sized bets, small sized bets. That's fine. I just personally am a little bit skeptical about it.

• 29:44Speaker 01

  People are welcome to go look on science on this issue of how well can you discern the difference between things that are really similar to each other? And I think it's just really easy for us to tell ourselves stories about that. And I also think it's honestly a little bit of a waste of time. That's just me. No financial advice here. Thank you very much.

• 30:02Auren Hoffman?

  You mentioned flipping a coin. What do you think about flipping a coin about a very big decision? Like I have friends and they flip a coin about all the big decisions they make if they're not sure about it. They even flip the coin of should they have a fourth kid? Something like that monumental, they flipped a coin on.

• 30:18Speaker 01

  If they feel like it's 50 50, absolutely.

• 30:21Auren Hoffman?

  Yeah, why not? That's the way to make the decision.

• 30:24Speaker 01

  Here's a very big decision. I've got a week of vacation time. Should I go to Paris or Rome? Big decision. Last time I checked, nobody has trouble choosing between Paris and Gary, Indiana. Oh, Paris and Gary, Indiana. I'll just go to Paris. But Paris and Rome, that's actually a really hard decision. But the thing is that I say hard decision because it feels really hard to people, but it's actually really easy. I think it's a big unlock. When a decision is hard, that means it's easy. When it's hard in the sense that, ooh, I really can't decide between these two things, then just flip a coin.

• 30:54Auren Hoffman?

  Or two job offers. Do you think it'd be good to both?

• 30:57Speaker 01

  Don't spend two months on TripAdvisor. What are you doing? You don't have a time machine. You can't go and see what it's going to be like when you're there. Stop with the two months worth of research on that.

• 31:08Auren Hoffman?

  But what about the things like, should I marry this person? Should I have another kid? Things that are maybe more of one way doors.

• 31:16Speaker 01

  Flip a coin.

• 31:17Auren Hoffman?

  All right, I love it.

• 31:19Speaker 01

  First of all, a kid is a one way door. Marriage is not a one way door. I actually don't like the one way door, two way door analogy. And I'll tell you why. Having a kid is a one way door. Death is a one way door. That's it. Those are the two one way doors. Everything else is a two way door, but some of them are more expensive.

• 31:41Auren Hoffman?

  Some of them are more expensive to get out of.

• 31:44Speaker 01

  That's right.

• 31:45Auren Hoffman?

  Some doors are a dollar to get, and some doors are billion dollars to get out.

• 31:50Speaker 01

  What I would love people to do is when it's more expensive to open that door, go back through, when it's more expensive, you should take more time with the decision and you should have a higher threshold for getting to the coin flip. There's two ways that we can think about. One is how expensive is it to reverse? How expensive it is to quit the thing that I'm doing? And the other one is what's the long-term impact if I get a bad outcome? Those are the two things that we want to think about. Let's think about hiring an intern versus hiring a CFO. I hire an intern and they don't work out. It probably has no impact on my bottom line in a year. And gosh, it's super easy to reverse at almost no cost to me. I'm just like, sorry, you didn't work out. Go back to college.

• 32:35Speaker 01

  That's it. So my threshold for getting to good enough should be really low in that situation. I should have a couple of checklist items that have to do with education, recommendations, whatever, I can decide what's on that checklist. And then I should end up with 20 interns that would be fine. And then I can put their name in a hat and I could pick one and just move on with my life, which is not what people do. They spend weeks trying to sort through the pile of intern resumes, figuring out who to hire. But now let's think about CFO. If I get a bad outcome with the CFO, long-term, it's probably going to have an impact on my bottom line and it's actually quite hard to unwind. It's probably pretty expensive. Certainly severance is a bigger issue. There are big cultural impacts to the organization.

• 33:20Speaker 01

  I have to go through a big, long search again. That's going to be expensive. There's a whole bunch of stuff that goes on with that. So my threshold for getting to good enough with the CFO is going to be higher, which means I have to spend more time on the decision. But in the end, even for one-way door decisions like having a child, you decide what the threshold is. Do I need to be leaning 60, 40 to do it? Do I need to be 50 50? So you decide whatever that is and then whatever if you decide that it's 50 50, maybe you go for it or maybe you flip a coin. I don't really care.

• 33:51Auren Hoffman?

  On the hiring thing, let's say you're hiring for a CFO. The classic thing is like the secretary's problem where you have N number of people in the pool. You first look at N over E of them and then you hire the first person after that that is better than the first set that's there. Do you subscribe to that or you gotta look at a few first? Do you know what you want or?

• 34:13Speaker 01

  It used to be called the Sultan's Daughter problem. And then at some point it was the secretary problem, but I think you're supposed to just call it an apartment hunting problem now. I'll give you the original Sultan's Daughter problem. It's important for understanding what the difference is because there's a difference between the apartment problem and the Sultan's Daughter problem that I think is very important to point out. So in the Sultan's Daughter problem, it goes like this. So a Sultan is gonna find a suitor for his daughter and he selected 100 suitors who are suitable for his daughter. All of them are suitable. They're a prince or they're whatever.

• 34:50Auren Hoffman?

  They're all over a certain bar.

• 34:51Speaker 01

  No peasants. So the daughter is gonna meet each of them. And if she rejects one, they're rejected forever. That's the end of that. And then she's gonna have to choose one at some point. There's some math around this, but it turns out that you should meet 37% of them.

• 35:10Auren Hoffman?

  Whereas 100 over E, which would be 37.

• 35:13Speaker 01

  Yep. The first person that's better than the ones you met before, at least as good as the best person you met before, you should lock that window. So here are the issues that don't apply very well to real life here. In real life, you learn as you go before you choose the next person. So generally, it's not like, here's 100, go for it.

• 35:34Auren Hoffman?

  You don't even know what the N is.

• 35:35Speaker 01

  Exactly. So if I'm dating somebody, I go on a date and I learn things about my own preferences and myself, and then that informs my choice of the next person I go on a date with. So that's the first thing that's different. The second thing that's different is that in real life, I can often go back to options that I've rejected in the past, which I cannot do here. I do think that we need to take that into account. But understanding this idea that you should do some sampling before you choose, I think that that's a really valuable idea to understand. But I don't want you to say, how many people do I think I'm going to meet? Then I'm going to reject everybody.

• 36:16Speaker 01

  It's a very interesting problem, but it's not a good isomorph. So if we go back to the apartment hunting problem, you could, I suppose, say, I'm going to look at 10 apartments and then I'm going to choose after four. You could do that, but you have to remember that you don't need to just pick 10 apartments. You can go look at a few apartments and say, I thought that I was going to like this, but actually it turns out I don't like this neighborhood very much, or I actually do want a third bedroom or whatever. And then that's going to inform your choices about the next apartment than you look at number one. And even if you didn't do that, you can't say, you know what? I want to go back to the first apartment.

• 36:58Auren Hoffman?

  Sometimes you can't because they go.

• 37:00Speaker 01

  Sure. Sometimes you can't because they go, but sometimes you can. And you should always sort of be taking that into account. With hiring, I really do approach it that way because you're getting candidates and then you're like, maybe I should change my job description if this is the pool that I'm getting. As you start to talk to people, you realize you should have specified different things or it's attracting a whole set of people who aren't a good fit. There you're learning as you go and you could change your job description. For example, you could change your screening process.

• 37:30Auren Hoffman?

  In poker, you've talked a lot about things like mathematical decision making, but you also in your poker career, I think, talked a lot about the importance of being able to read people. I'm personally terrible at reading people. What are one or two things that somebody like me can do to get better?

• 37:45Speaker 01

  In poker, the bulk of the work that you're doing, assuming that you're not playing just game theory optimal. Let's assume that's not what's happening, but you're actually playing an exploitative game. I just want to be very clear, I'm using exploit in the game theory sense, not in the manipulating.

• 38:05Auren Hoffman?

  We're amongst data nerds.

• 38:06Speaker 01

  I know.

• 38:07Auren Hoffman?

  We know what you're talking about. Yeah.

• 38:08Speaker 01

  I just always feel like I need to say that. But somebody has tendencies which you can exploit. And so you're not playing game theory optimal in that you're not following Nash equilibrium or anything like that. You're like, I know that Orin calls too much. I'm not going to bluff him very much, but I'm going to be what's called a nuts peddler. I'm going to start running the nuts into him because I know he's going to call me every time. You'll change your play based on what the other person is doing. When you're playing cash game poker, a lot of what you're doing is actually trying to create a model that's better than the general base rates for the player that you're playing against. Given that I know nothing about you, I'm going to assume you're going to enter 20% of the pots that you play in a nine-handed game.

• 38:54Speaker 01

  This is going to be your range for opening hands and things like that. So there's a whole bunch of things where I'm going in with like a general model, given reference class of poker player. I could have some general, easy ways to actually get a little bit more refined even before I know you. So if you're a retiree at 10 a.m. on a Wednesday morning, that's a different reference class that I'm going to be pulling from before I've ever seen you play a hand. Then if it's 2:00 in the morning on a Saturday and you're drunk. So I have different reference classes, but that's going to be my starting point. That's going to be my prior. And then what I'm trying to do as I watch you play is update my priors to refine my model of you personally.

• 39:32Auren Hoffman?

  But just with data, not about reading them?

• 39:35Speaker 01

  Just with data.

• 39:36Auren Hoffman?

  I could do that with the data. My problem is reading the person.

• 39:40Speaker 01

  Then there's a reading people part. So you're doing that just with data, but that's actually really important to do. One of the mistakes that a lot of people make is they think that other people play just like them. We don't want to do that because people are different. And I want to be able to see if there are some holes in your game. And if I can find those holes, then I can exploit them and I can win more money that way. So that's a nice thing to do is build a model of your opponent. Where does the reading come into play? When I'm playing poker, always what I'm trying to do. Is to narrow down the range of hands that you are holding. I can't see your cards, but what's going to be different between a really great poker player

• 40:19Speaker 01

  and an amateur is that the possibilities of hands that I think you have is going to be narrower and more accurate than what an amateur might think you have. It's not that I'm going to think that you have every two-hand combination because I'm going to be like, well, he plays a low percentage of his hands and he's opened in first position and he doesn't bluff a lot. And so now I've got you narrowed down pretty tight. I'm going to be better at doing that. Now, I'm going to do a lot of that just from the data work. Have I built a really good model of you? But some of that is going to be from reading my opponents. And when that's going to be the most important, and you're going to hear this as an echo from what we talked about before, is when it's a really close call.

• 41:04Speaker 01

  Remember when I said, what's the threshold for getting to good enough? If the price I'm getting from the pot is really big, I should not spend a lot of time trying to read you because I don't actually want to convince myself to fold because I'm totally fine with investing there, knowing that I may end up with a false positive. Why? Because maybe I'm getting 10 to 1 from the pot. I'm making money if I win 10% of the time, if I'm getting 10 to 1, like I'm making money. So I don't want to have false precision there. I don't want to be thinking that I know more than I do. And so I'm not going to bother reading you or anything. I'm just going to go, I'm getting 10 to 1 from the pot. Can I win 10% of the time? Sure. Okay. And I'm going to call.

• 41:50Speaker 01

  But when it gets close, so when you're in a situation where, say, you're getting 2 to 1 from the pot and you're like, well, I have to win this 33% of the time and this is pretty close. Now, reading your opponent can actually get you to fall onto one side of the fence or the other. And that's where it becomes really incredibly important. So how do you read your opponent? Number one is to recognize that when people are being stared at, they don't like it. And so you can get a lot of what are called false tells when you're just like staring somebody down at the table because they're just uncomfortable. Would you want to be stared at really hard by somebody? No.

• 42:24Auren Hoffman?

  Definitely not. I'm the kind of guy that looks at my shoes.

• 42:27Speaker 01

  Exactly. So you might start doing some stuff that I connect to your hand. That's really actually just about general discomfort that's being created by it's a long time that I've been sitting there and staring at you. So generally what you want to understand is that whatever physically you're getting from somebody, you're going to want to get relatively early on. And that's generally going to be better. It's not that someone might not give you something later on. It's just a lot noisier when that happens. So what am I going to be able to see? There's a whole bunch of signs of discomfort and they're signs of happiness. As an example, interestingly enough, at the poker table, when someone's moving their knee up and down, it's actually a sign of nerves. It's a sign of a good hand. It's like a dog wagging its tail.

• 43:12Auren Hoffman?

  That was like a Sam Bankman Freed thing with his knee going up and down all the time.

• 43:16Speaker 01

  A lot of settings, like a social setting, it might be a sign of nerves or being nervous or anxious or something like that.

• 43:22Auren Hoffman?

  If you see it and it's not normally the case, then okay, maybe they've got a good hand. But now everyone knows that. So now everyone just like bounces their knee up and down all the time.

• 43:31Speaker 01

  That's true. Now they'll bounce their knee up and down. There's a whole bunch of stuff that has to do with the mouth. There's these self-soothing behaviors. So if you see someone ever rubbing their hand on their fingers, or if they're moving their tongue on the inside of their cheek, that usually means that they're bluffing or they're uncomfortable with their hand because it's called a self-soothing behavior. So with a man, if they have a beard, don't do this, but if they're like rubbing the back of their neck, or sometimes they'll be sitting at the table and they'll be rubbing their fingers, or if they're going like on the inside of their cheek or something like that.

• 44:00Auren Hoffman?

  But again, now everyone knows that. I assume if you know that, other people know that. So now they're faking it or?

• 44:04Speaker 01

  Well, it's true, except that.

• 44:06Auren Hoffman?

  Except it's hard to fake.

• 44:07Speaker 01

  Yeah, because involuntary actions look really different than voluntary ones. You can also look at something called hooding. Have you ever seen someone? And you kind of always instinctively know they're lying. When they say the thing and they have this really long blink and then they open their eyes, that's called hooding. But there's actually a great guy who talks a lot about these body languages. His name is Joe Navarro.

• 44:29Auren Hoffman?

  Read his book, yeah.

• 44:30Speaker 01

  He's got great work on this idea of how you actually read people.

• 44:34Auren Hoffman?

  Interesting. I just like assume everything people tell me is 100% true and it's generally worked out pretty well. Is there any other advice you would have for someone like me who just assumes everything you tell me is correct?

• 44:48Speaker 01

  I think most people are telling you the truth. I agree. What we need to understand though, is that whatever their interpretation is, is not necessarily true.

• 44:57Auren Hoffman?

  It's their belief of it.

• 44:59Speaker 01

  But I think people talk a lot about misinformation online. There's work by Duncan Watts, which shows that misinformation versus what we call like misleading information is 46 to one, misleading information, 46 times the problem. The misleading isn't people purposely trying to mislead you. They're telling you some data.

• 45:19Auren Hoffman?

  From their side of the story.

• 45:21Speaker 01

  An example, you see a lot of people talking about crime statistics online a lot. And when people tell you that New York is more dangerous than Macon, Georgia, they think they're telling you the truth because they're like, I don't know how many murders were in New York last year, maybe 350 or something like that. Well, let's look it up. I have the interwebs. Let's see.

• 45:41Auren Hoffman?

  I like this. We're doing live reporting here.

• 45:44Speaker 01

  Yeah.

• 45:45Auren Hoffman?

  And safety is always this hard thing because it's not just the number of murders. It's all the other crime and it's just how you feel and you feel safe walking around and do people stare at you? Does someone just yell at you and that makes you feel unsafe, even though there may not be a crime to yell at somebody? So these things are very, very psychological, I imagine.

• 46:05Speaker 01

  Oh, it doesn't tell you for all of 2024, which is sad.

• 46:10Auren Hoffman?

  The interwebs are a problem right there. I think part of what you're saying is that these things are emotional. They're not just data driven.

• 46:17Speaker 01

  Let's make up some numbers. This is what I'm actually saying. So generally when they tell you that New York is less safe than Macon, Georgia, it's that New York had 350 murders and Macon, Georgia had 90. And they're like, see? New York is more dangerous. Those people are not lying to you.

• 46:37Auren Hoffman?

  And New York is like gazillion times the population.

• 46:40Speaker 01

  There's denominator neglect going on there. I think that there's all sorts of places where the denominator hides in the shadows. If I said back to them, oh yeah, well, Georgia is more dangerous than New York City, they would understand there was a denominator problem right away. Because they'd be like, wait, Georgia's a state and New York City is a city. But just so people know, because I have looked it up before, because Macon, Georgia is actually more dangerous per capita than New York City.

• 47:06Auren Hoffman?

  Good. I've never been to Macon, but now I know.

• 47:08Speaker 01

  You're not likely to get murdered in either of them, but people get that reversed. Now, look, I really don't think that the people who tell you that are lying.

• 47:16Auren Hoffman?

  But by the way, even when people use real data, they'll say it's more dangerous to be in the city of Baltimore than it is to be a US soldier in Afghanistan or something. And well, maybe that's technically true, but I could probably avoid the danger more in Baltimore than I could if I was a soldier in Afghanistan.

• 47:33Speaker 01

  They're generally making the mistake of telling you counts of things as opposed to, well, how many total people were exposed, how many people ended up dying, that kind of stuff. So one of my favorite examples of this actually comes from the Washington Post. I read an article in 2022 and the title was COVID is no longer a pandemic of the unvaccinated. So that was the title of the article. So I thought, oh, this is interesting. I'm going to go read this. At that time, the Washington Post definitely had a pro-vaccine bias. So I was like, oh, there must be some very interesting data behind this very interesting headline. And so I went and looked at it. It was that in the month of August of 2022, 58% of the

• 48:16Speaker 01

  people who died of COVID were vaccinated.

• 48:20Auren Hoffman?

  But I assume a very high percentage of the population was vaccinated by that point.

• 48:24Speaker 01

  Well, you wouldn't know from the article because that was the question I asked and it wasn't in the article. Now, I think we can agree that that reporter was not trying to lie. And if you go fact check that number, by the way, in August of 2022, were 58% of the people who died of COVID vaccinated, that will come up fact check, yes. So I asked the same question you did. Wait a minute, what percentage of the population is vaccinated? And I went and looked it up. It wasn't in the article. And at that time, 80% of the population was vaccinated. So I was like, oh. Well, that seems like vaccines are pretty effective. I don't know. Also, I guess you need to do some age matching because there's a lot more vaccinated people who are old than young, and old people are much more likely to die than young people.

• 49:07Speaker 01

  So it'd probably be good to do some age matching. I'm lazy. I wasn't actually going to do that, but I found a blog from someone who had done it. I was very lucky. And it turned out in the month of August, it was five times better to be vaccinated than unvaccinated. Now, the thing is, I have no truck with anybody who doesn't get vaccinated for COVID. That's your own choice. The problem is that you're trying to make a choice that's data-driven, and not everybody is Orin, who's going to go, wait a minute, how many people are vaccinated? Maybe I should do some age matching, whatever. So you're just looking at, here's a newspaper, they're not trying to lie to you, they're drawing a conclusion from this data that is totally misleading, and now you're going to make a decision based

• 49:49Speaker 01

  on that data, whether to get vaccinated or not. My whole thing is, I just want you to make a decision where you actually understand how you're supposed to make sense of the data, and then you can decide for yourself. I'm 20, I don't have a lot of risk, or whatever. Like, I don't care. You be you. Make your own decision about it. It has nothing to do with me. But I think that this is generally the problem. I think that mostly people are telling you the truth, at least what they believe the truth is, but you aren't taking enough responsibility in terms of your own decision-making to say, well, wait, how do I interpret the information that I've just been told? And I think we just stop at someone telling us a fact wrapped in their interpretation, and then we don't actually say, well, wait, what more are we supposed to ask here?

• 50:34Auren Hoffman?

  Really interesting. This has been awesome. And last question we ask all of our guests, what conventional wisdom or advice do you think is generally bad advice?

• 50:43Speaker 01

  Trust your gut. That's gotta be the worst. If the decision doesn't matter, it doesn't have any long-term consequences, it's a super inexpensive door to go back through. Sure. Trust your gut. You're gonna end up turning away from a lot of skids that you should be turning into. Sometimes your gut's pretty good, but if you don't make explicit how your gut is modeling a problem, you're gonna just miss a lot of errors.

• 51:09Auren Hoffman?

  I never trust my gut to do something, but I almost always trust my gut to not do something. So if it said to hire somebody, I wouldn't trust my gut. But if it said, don't hire this person, I would definitely trust my gut. Where do I have it wrong there?

• 51:26Speaker 01

  If you've gone through a really good decision process, and then your gut or intuition is like, hmm, I think that's a really good reason to go back and examine. When someone has a lot of experience, sometimes there's something that they may have missed and their gut is putting up an alarm bell. Blah, I'm a little nervous about this. And I think that then you should go back and look, because I think the thing that you should always strive to be able to do is actually give the real rationale for why your gut feels that way. I just think that's really important. Now, if you want to have a rule that your gut can't make you do something, but it can stop you from doing something, that might be okay, except what if you're like

• 52:08Speaker 01

  super risk averse? Now you're gonna make a lot of really bad decisions on that rule. What I like to do is think before I face a decision, what are the things that I care about? What are my values? What's the threshold for deciding? I try to sort of think about all of those things in advance. Then I go through all the work of doing it. And then if there's something going ping, ping, ping, ping, ping, I'm gonna go back and look at this work again. What more information do I need to know in order to understand? Because I'm gonna say, well, why is my gut feeling like that? And then I'm gonna go from there.

• 52:40Auren Hoffman?

  It's been great. Thank you, Annie Duke, for joining us on World of DaaS. I follow you at Annie Duke on X. I definitely encourage our listeners to engage you there. This has been a ton of fun.

• 52:50Speaker 01

  Well, thank you for having me. This was a super fun conversation.

• 52:56Auren Hoffman?

  Thanks for listening. If you haven't already done so, please subscribe to World of DAS wherever you consume your podcasts, YouTube, Apple, Spotify, and more. And please help us get discovered by leaving a review. And check out worldofdaas.com. That's worldofdas, D-A-A-S, dot com. And of course, connect with me on Twitter, at Auren, that's A-U-R-E-N. Would absolutely love to hear from you.