Best of Pre-Mortem Analysis: Gary Klein

This week's Summer Series covers the pre-mortem analysis with its founder, Gary Klein. Gary discovered pre-mortems while working with fighter pilots and wrote a white paper applying the tool to investment analysis alongside Paul Johnson and Paul Sonkin. Please enjoy this discussion of the proper use of pre-mortem analysis with Gary, Paul, and Paul from 2019. Learn more about Capital Allocators University: https://www.capitalallocators.com/capital-allocators-university/ Try ALEX by Admired Leadership: tryalex.admiredleadership.com http://tryalex.admiredleadership.com/ Learn More Follow Ted on Twitter at @tseides https://twitter.com/tseides?lang=en or LinkedIn https://www.linkedin.com/in/tedseides/ Subscribe to the mailing list https://capitalallocators.com/ Access Transcript with Premium Membership https://capitalallocators.com/signup Editing and post-production work for this episode was provided by The Podcast Consultant (⁠ https://thepodcastconsultant.com/https://thepodcastconsultant.com https://thepodcastconsultant.com/⁠ https://thepodcastconsultant.com/)

• 0:06Speaker 00

  I'm Ted Seides and this is Capital Allocators. For the second year, our summer series will share some of the best episodes of the past curated from our favorites and yours. You can pull down many of the great ones on our Spotify playlists. This series pushes them to your regular listening feed. Our 2026 summer series focuses on improving the investment process. Over the next six weeks, we'll share 10 gems from interdisciplinary skillsets that will help you level up your investing across decision making, interviewing, storytelling, negotiations, and habit formation.

• 0:48Speaker 00

  This week's summer series covers the pre-mortem analysis with its founder, Gary Klein. Gary discovered pre-mortems while working with fighter pilots and wrote a white paper applying the tool to investment analysis alongside Paul Johnson and Paul Sonkin.

• 1:04Speaker 01

  Before we get to the interview, we're excited to introduce the next chapter of Capital Allocators University, an online on-demand curriculum with an annual in-person gathering for investment professionals committed to continuous learning. One of the great ironies of investing is that the skills that matter most over a long career are often the ones least formally taught. Things like decision making, communication, leadership, time management, interviewing, and building relationships. That's why we've evolved Capital Allocators University into an online on-demand experience, making those lessons accessible to investment professionals everywhere. The best careers don't compound through knowledge alone. They compound through knowledge and relationships. If you'd like to explore the curriculum or

• 1:49Speaker 01

  someone on your team comes to mind, visit the link in our show notes or the university page on our website. Thanks so much for spreading the word about the next chapter of Capital Allocators University.

• 1:59Speaker 00

  Capital Allocators is brought to you by AlphaSense. Here's something for you. Most AI tools today are very good at sounding right, but can you actually trace it back to a filing, transcript, or specific passage that drove the answer? Or are you just trusting the confidence of the output? For allocators, that's not a minor concern. A missed filing, incorrect source, or context that gets lost somewhere in the retrieval chain aren't edge cases. They're how decisions go wrong. AlphaSense is the AI platform built specifically for

• 2:39Speaker 00

  this. They own the content, over 500 million curated documents from broker research and expert transcripts to filings and earnings calls. And they own the retrieval layer on top of it. That means every answer can link back to an exact verifiable source, because the answer is only as good as what's underneath it. And with AlphaSense, you know exactly what that is. See it for yourself. Try a free trial at alphasense.com/capital. That's alphasense.com with a hyphen in the middle slash capital. Capital Allocators is also brought to you by Morningstar.

• 3:20Speaker 00

  In a sea of data, you need to know what you can trust. Morningstar's data, analytics, and research helps bring clarity across public and private markets, connecting independent research and data into the tools and AI platforms you use every day, so insights are easier to spot and act on. Morningstar, nothing in your way. Visit morningstar.com/gameon to learn more. Capital Allocators is also brought to you by Juniper Square, the operations partner trusted by the leading private market GPs. One theme that keeps coming up on the show is GPs that build the strongest and most durable relationships with LPs are the ones

• 4:04Speaker 00

  that have both strong returns and strong operations. AI raises the stakes on operations, and having Juniper Square as your operations partner helps ensure your data is organized, accurate, and accessible, so you can unlock operational alpha. That's why thousands of GPs trust Juniper Square. Learn more at junipersquare.com/capitalallocators. Please enjoy this discussion of the proper use of pre-mortem analysis in investing with Gary Klein, Paul Johnson, and Paul Sonken from 2019. Gentlemen, thanks for joining me. Nobody wants to respond. There's too many people in the room, huh?

• 4:47Speaker 02

  Ted, it's always a pleasure. Thank you for inviting us.

• 4:51Speaker 00

  I know Paul and Paul have been on the show before, so I think it'd be fun, given that they were bringing you to the table for this paper we're going to talk about. Why don't you go through your background and areas of interest?

• 5:03Speaker 03

  Sure. So I'm a cognitive psychologist. My primary area of interest is expertise. How do people become highly skilled? And that started in the mid-'70s, and when all of a sudden, the Arab oil embargo hit, and you had all these pilots. I was working as a research psychologist for the Air Force at the time. All these pilots used to fly as much as they wanted. Now they couldn't because jet fuel was too expensive. And so they had to learn in simulators, which they always had considered as toys, but now they couldn't anymore. And so my job was to figure out what is expertise, and how do you develop it, and how do you develop it most quickly?

• 5:38Speaker 00

  And how'd you first get interested in that field?

• 5:40Speaker 03

  It was a requirement of the job. We had to help these pilots. They were sort of desperate to find out, how do we design simulators that are going to allow us to become as skilled as we used to, because we can't fly nearly as much.

• 5:53Speaker 00

  And what did you find in that initial research?

• 5:56Speaker 03

  So I'll tell you a story. I was doing a project and was down at Tyndall Air Force Base, and there were these two fighter pilots, F-106 pilots. And we were working on a simulator during the day. And at night, one of the evenings, we went to a bar, because that's where they usually went after we finished in the day. And they said, can we ask you a tough question? We're supposed to be developing training. And there's a system called instructional system design. And it's very procedural, and takes a task, break it down into the components, and that's the way you teach it. And I said, okay. And they said, we're the ISD specialists for our group. And I said, okay, that's nice. And we're building the IOD specs for how to perform a lot of our flying tasks. And I said, okay, so what's

• 6:40Speaker 03

  the problem? And they looked around to make sure nobody was overhearing them. And they said, well, we've written all the specs, but we don't follow any of those rules. That's not the way we fly. So I said, okay, just write down the rules you follow. And they said, we don't follow any rules. We just know what to do. And that's what I've been hearing time and time again from firefighters and others, that there's a belief that you just follow rules and procedures. For complex tasks and ambiguous situations with lots of uncertainty, there's no set of rules that are sufficient. And expertise is not about just incorporating all the rules. Expertise is about building tacit knowledge, and that's

• 7:25Speaker 03

  the core. That's the primary thing that I've learned, is how to go after tacit knowledge.

• 7:29Speaker 00

  We hear a lot about checklists and Atul Gawande's checklist manifestos. How do you blend the necessary tasks with the intuition that comes on top of that?

• 7:38Speaker 03

  Right. And that's an important part, is that it is a blend. And Gawande comes up with these great checklists, but in the book, he describes how much work it is to come up with a reasonable checklist. And you can't have too many items. You can't make it too complicated. And so there's a real art to coming up with a checklist. How do you blend the other part? You break people out of the mindset. My job is simply to learn the rules and learn the procedures. And a lot of people do have that belief. And you've got to get them out of that mindset and get them into accepting that life is too complicated, and they are going to be encountering situations that none of the checklists have prepared

• 8:19Speaker 03

  them for, and they better be adaptive and resilient in order to handle them.

• 8:24Speaker 00

  If we go back, then, to you're doing this research to try to distill how to help these fighter pilots in simulators, and then you learn that, well, the real way that they do it, the real way they develop expertise is in their heads. What do you do with that as a conclusion to research?

• 8:42Speaker 03

  Well, there was no conclusion for the work with the fighter pilots, because we were building simulators. So we just had to make the simulators.

• 8:51Speaker 03

  As effective as possible. Now, the fighter pilots and the whole community had the belief you had to make the simulator look as much as possible like the real thing. That was wrong. It doesn't have to look like the real thing, and you don't have to waste a lot of money on that. You have to have really good training scenarios that put people through their paces. And so that was the big aha from that project. But where the expertise work came in was much later, when I started studying decision making. And up to that point, there was a belief that a rational decision maker lays out all the options, lays out all the evaluation dimensions, rates each option on each dimension and so forth. When's the last time you ever, I mean, people sometimes do it, but it's

• 9:33Speaker 03

  very rare. And it's not an adaptive strategy. But that's what the field thought was the gold standard. Because the field had mostly studied college sophomores handling unfamiliar tasks in laboratory environments. And so the field hadn't studied experts. And so because of my interest and expertise, I was able to look at decision making and say, where's the expertise here? And that was the formulation for my recognition prime decision model. And the discovery we made is that people don't generate lots of options and use lots of standard evaluation dimensions. They use their experience to size up situations

• 10:12Speaker 03

  very quickly. And to identify what's going on, what's likely to happen next, what can I accomplish? And what should I do about it? And then to evaluate it, that's the intuitive part. But there's two parts to the model. The evaluation part was mental stimulation. If this is what I think I should do in this situation, let me imagine doing it in this context, and see if it works. If it works, I'm fine. If it almost works, I can improve it. And if it doesn't work, then I'll go to the next option in my list. And so we came up with a different model of decision making. And that's probably the accomplishment in my career that I'm the most pleased with.

• 10:52Speaker 00

  As you discussed that and looked at different applications, were there areas where it was highly effective, and others where it's less effective?

• 11:01Speaker 03

  We haven't found many areas where it's not effective. And we looked, we said, we're not teaching this strategy. This strategy describes what people naturally do. So we said, maybe it doesn't get applied when there's less time pressure. So we looked at non-time pressured situations. We looked at designers. And they still were predominantly using a recognition strategy. And then we said, maybe if you don't have much experience to begin with, it's not a strategy you'll use. So we, we looked at novices, we looked at soldiers at Fort Knox, tank commanders who hadn't really had much experience in tanks. And we said, what about them? They were the group we've seen that did not use recognition

• 11:42Speaker 03

  decision making. They did look at options. They tried to identify the best option from a set. It didn't help them. They still did a terrible job. But they had no basis and expertise to use recognition strategies.

• 11:54Speaker 00

  Once you identified that the recognition strategies are the way that people go about making decisions, how did you then study different ways that people can improve their decision processes?

• 12:05Speaker 03

  That's where it really gets important, isn't it? What can you do about it? And so for maybe 10 or 20 years, we were looking for, how do we turn this into training and to preparation so that we can help people become better decision makers? And we came up with nothing. And so it was very discouraging. And then about six or seven years ago, I was exposed to a technique that we call the shadow box, which is a technique that we've been using it for the last five years. And it wasn't developed by me or my team, developed by a guy named Neil Heinz, who's a retired New York Fire Department Battalion Commander.

• 12:47Speaker 03

  And he had developed this in the wake of the 9/11 attacks. And how do people handle situations that no training could have prepared them for, because nobody could foresee them? And the way shadow box works, it's a scenario based approach. So I give you a scenario, and we walk through the scenario. And then at a certain point, I'd stop and I'd say, okay, Ted, here's four options of what you can do at this point. Rank order them, and write down the rationale why you rank ordered them. But it's not just about the options. We may go a little bit further, and then I'd stop and I'd say, okay, Ted, here's three goals that you could be pursuing. Rank order them in terms of what's the most important to achieve to the least, and write down your reason. Then we might go a little bit further,

• 13:31Speaker 03

  and I'd say, here's five different kinds of signals and pieces of information that you might pursue. Rank order which you would go for, and why. And on the side, we'd have a set of three to five experts who've gone through the same scenario. They've done their ranking. They've written down their rationale. So after you, you come up with your ranking, and we say, here's what the experts ranked. And you want yours to match the experts. But more important, you look at what the experts saw. So this is really how to see the world through the eyes of experts. But the experts aren't there, because we already collected their data. So we don't have to gather them and schedule them or anything like that.

• 14:12Speaker 03

  And you see what the experts noticed in the same scenario you read. And you see the inferences they drew, and all of the warnings that they spotted, and things like that, that you just went right past and you didn't notice. So that's how shadow box works.

• 14:26Speaker 02

  Think about how you could revolutionize the Harvard Business School case studies. And it has enormous implications for training, security analysts. And Ted, you can see why, from what Gary's been talking about, that I think that, and that's why I was attracted to Gary's work, because it's directly applicable in so many different ways to the jobs that portfolio managers and security analysts do.

• 14:51Speaker 00

  You've now set up this framework with the shadow box, where someone can go through a specific scenario, and then have an expert way of assessing it. Is that just a specific training tool for certain scenarios? And where have you seen that applied?

• 15:08Speaker 03

  Okay, so we've applied it to panel operators in petrochemical plants. We've applied it with caseworkers doing child protective services. We just did a project on systems integration specialists, trying to get younger engineers up to speed faster in that area. I'm going to be talking to a hedge fund. They're interested in, in shadow box. So we've applied it in a variety of areas.

• 15:36Speaker 00

  Interesting. And what are some of the other tools that you've seen along the way?

• 15:41Speaker 03

  We have a method for cognitive after action reviews.

• 15:44Speaker 00

  Cognitive after action reviews.

• 15:47Speaker 03

  Right. So it's called CARGO, cognitive after action review guide for observers. And the idea here is, a lot of times, after a situation, after you've had something go well, or go poorly, you do an after action review. And frequently, the after action review is, you did X, and you should have done Y. If you had done Y, things would have turned out better. But it doesn't get into the cognitive part. When you chose X, what were you seeing? What were you interpreting? In hindsight, what could you have been looking at? And so trying to get into the person's head to see what they were considering at the moment that they made their judgment and decision, rather than just saying, X

• 16:30Speaker 03

  is better than Y. So that's, that's a method that we've been using now. Then there's also the pre-mortem method that we developed, gosh, about 30 years ago. And it's probably the best known of all the techniques that we're using.

• 16:46Speaker 00

  What was the first insight in pre-mortems?

• 16:49Speaker 03

  So let me just tell you what the pre-mortem is. So the pre-mortem is sort of a risk assessment method. And the idea is to try to anticipate what could possibly go wrong with a plan. And when I had my company, I had my research company for about 27 years till I sold it in 2005. And we had grown to 37 people by that time. And at a certain point, we said, we have a lot of projects that go well, but not all of them. And some of them don't go well. And we do a post-mortem after it's finished, and it may help us in the future, but it doesn't help us with the project that finished. Why don't we try to

• 17:29Speaker 03

  move that upfront to the kickoff meeting? And the analogy would be like in medicine, in healthcare. You do a post-mortem after a patient dies to find out why. And then the physician can discover what killed the patient, and let the family members know so they also gain. And if it's something interesting, they can write it up. So everybody benefits from a post-mortem, except for the patient, because the patient is dead.

• 17:56Speaker 03

  So the idea of a pre-mortem is let's anticipate what could happen at the very beginning of the project. So we started doing it with our own projects. We would add this, and it doesn't take very long. We had added to the end of our kickoff meetings for a new project. And then in around 1993, we had an Air Force customer that wanted us to lead an effort to build a complex piece of software for them. And there were a number of Air Force engineers and there were other team members who had algorithms that were going to be included in this tool. And we said, okay, at the end of this kickoff meeting, we're going to do a pre-mortem because we had really gotten to like the method. And our contract monitor

• 18:39Speaker 03

  said, absolutely not. He hated the idea. He said, it's going to be too depressing. I don't want to do anything that might reduce people's confidence and enthusiasm. And we said, this is the way we do things. You know, you hired us to run the project. This is the way we do it. So we have the meeting, the meeting goes for a couple of hours, and it's the end, and we're going to do the pre-mortem. And we said, all right, now let's go through the process. And the way the pre-mortem works is you have a plan. Everybody knows what the plan is, and it's the team that's done the plan and is going to do the execution. They're all primed. They're all ready to spring into action. And if I was doing a pre-mortem with

• 19:20Speaker 03

  you, I'd say, okay, now, now let's just step back a second. And I'm looking into a crystal ball, and I might sort of imagine there's a crystal ball on the table in front of us all. I'm looking in a crystal ball, and it's now six months in the future, maybe a year. Oh, I don't like what I see. I see a disaster. This plan has failed. It's been a complete and embarrassing failure. And the people on the team don't even make eye contact when they pass each other in the hall. It's been a really big disaster. All right, we know the plan has failed. Now, everybody around the table, take two minutes and write down

• 20:04Speaker 03

  all the reasons why this plan failed. And everybody starts writing madly. And then at the end of two minutes, I stop them, and I say, now, let's find out what people wrote. And I go around the table, and I start with the project lead to see what that person came up with. And then we just continue around the table like that until we get all of the items that people had. And that was fairly depressing. So we added another step after we finished compiling it, which is, now that we've seen all the things that might go wrong, that could go wrong, everybody take another two minutes and write down what you individually could do to prevent some of these things from occurring. And so people go through

• 20:49Speaker 03

  that, and that sort of is a mitigating factor. So we did the pre-mortem on this project, this Air Force project. And we got to one person. He was, I think, a captain. He was the most junior person at the table. His specialty was information technology. And we got to him. He hadn't said a word during the entire meeting. And we said, what do you have? And he said, well, I'm looking at the algorithms that people are describing, and they're really powerful algorithms to do the job. And they run on a supercomputer, and they take a day or two. And we're talking about putting this in the field, like during Desert Storm or something like that, to be used by a history major who graduated from

• 21:32Speaker 03

  Northwestern and is now in the reserves, and he's now thrust into this job. And at that point, he might have had a 486 machine or something like that. It just won't work. The software won't run on that machine. And everybody looked at him and realized how ridiculous the quest had been up to that point. And had we pursued the vision that was the sponsor's vision, it would have been a disaster. And instead, people said, you know, I have a back-of-the-envelope approach that doesn't require a supercomputer. And it's not as good as a supercomputer, but it's like about 90%. And we said, that's what we want.

• 22:12Speaker 00

  Interesting. So there were a couple of things you mentioned as you laid out how the pre-mortem works. I want to see if you can talk about the significance of it. The first is, you said there's two minutes for everybody to go around and write down. Is that just because you just threw out two minutes? Or is there significance to that timeframe?

• 22:29Speaker 03

  Trial and error. When we did this for a number of projects in my company, one minute was leaving people frustrated because people were still writing when they said, time is up. When we tried three minutes, a lot of people would be finished, and they'd be twiddling their thumbs. And it would reduce the energy of the room. So two minutes seemed to be the sweet spot.

• 22:48Speaker 00

  Okay. And then you said, when people start discussing what they wrote down, the leader comes first.

• 22:54Speaker 03

  Right. And the reason for that is people don't know how candid to be. And the leader is going to set an example. And if the leader picks something that's frivolous or trivial, that tells people not to take it seriously. So it's really important for the leader to come up with a real problem that hadn't been discussed before.

• 23:17Speaker 00

  Now, if you were conducting a pre-mortem on the pre-mortem process, what are the things that will commonly go wrong that makes a plan of a pre-mortem fail?

• 23:32Speaker 03

  Well, that's one of the reasons that I've been writing this piece with Paul Johnson and Paul Sonkin. We see people making mistakes. There are ways that the pre-mortem will fail. First of all, if the leader doesn't have buy-in, then the leader will come up with something trivial, and everybody knows they have to not surface their prime concerns and choose something that's safer and minor. So that's one part. We see pre-mortems fail when the person who's facilitating will say something like, okay, we've seen the plan, let's do a pre-mortem. What could go wrong with the plan? That's the wrong approach.

• 24:12Speaker 03

  What can go wrong with the plan? That's a typical critiquing approach. You can't say, what could go wrong? You say, I'm looking at a crystal ball. The plan has failed. There's no doubt. That changes the mindset that instead of just sort of musing about it, now you're seeing a disaster, and you're now in a mode of what caused that disaster? It changes the kinds of issues people identify.

• 24:36Speaker 00

  Why is it that it changes it?

• 24:38Speaker 03

  There is a study from 1989 about perspective hindsight, that giving people the perspective of a future situation where something that is counterfactual has occurred, all of a sudden opens up access to kinds of thinking and speculation that people ordinarily wouldn't engage in. And the metaphor I have is like an eclipse. Right now, if we stare at the sun for like quarter of a second or so, it's really bright, and we look away. But during the eclipse, and I had a chance to see the eclipse a year or so ago when it happened,

• 25:19Speaker 03

  when the eclipse occurs and the moon blots out the sun, and you look up, and you see the corona, you see all of these trails streaming out of the sun, and it's just gorgeous. And I had these issues I wanted to study during the eclipse, and I was just paralyzed by the beauty of it. But it was all invisible because the sun is so blinding under normal conditions. The pre-mortem is a sort of a way of blotting out what you normally think about and your enthusiasm for carrying out a plan. And so you need to cover that up so that these other issues can emerge and become visible.

• 25:56Speaker 02

  That was what we tried to do in the paper, was kind of tie some of these concepts back to research that's been done. But as Gary said, you're reframing the problem, and then it's easier to find a reason for why something has happened, why the project has failed, as opposed to developing different scenarios of things that might happen. So it just kind of shifts the mindset. It's very nuanced, and it's very subtle. And that's why we feel as though the pre-mortem is misused, because they'll just say, okay, what could go wrong? And they think they're doing a pre-mortem. But just that subtle framing kind of kills the effectiveness.

• 26:39Speaker 04

  I think there's one more thing that we sort of swept under the rug, is that Gary says at the end of the meeting, you pause, take a deep breath. We're talking 15 seconds. And then you say to people, you have two minutes to write down why the plan failed, and no one has a problem writing down two or three reasons. And that, to me, is so important because we've given them five seconds to think about it, and then two minutes to write, and everybody is scribbling things down. So that knowledge is in the group, at the tip of their tongue. They're thinking about it all through them planning meetings, and nobody brings it up. And the pre-mortem allows you to change the mindset, as Paul said, it failed. Why did it fail? Then you give them two minutes, and you get to take

• 27:23Speaker 04

  the number of people in the meeting, multiply that by three, that's how many reasons you're going to get why it failed, none of which were brought up at any point in the planning session. And certainly none of them were brought up in the approval process. For me, the power of the pre-mortem is we're now allowing people to express their concerns that they've had all along, that they've been afraid to bring up. And I think the thing for me, I think Gary, 100% believe, agree with you, that if the leader doesn't take it seriously, everybody hides. But the most important factor for me is psychological safety. So psychological safety is the notion that it is safe to say what you're most concerned about. This captain in the example said, this just isn't going to run. Would have been very easy for him to sit on his hands and not say anything. And yet, if you give

• 28:06Speaker 04

  him permission, he brings up the most critical issue, change the plan completely.

• 28:11Speaker 03

  And he hadn't said a word in the whole meeting up to that point.

• 28:14Speaker 04

  And that, to me, is the power.

• 28:17Speaker 00

  We're going to take a quick break in the action to tell you about RidgeLine, the investment management platform built for firms who are rethinking how work gets done. You're at your desk, looking at a line graph of your firm's growth. The line is moving sideways, linear growth, and you know exactly why. You're constrained by your operational bandwidth that's been the same for so long, it just feels like the cost of doing business. RidgeLine is built to change that line's trajectory, to be the inflection point that takes your growth from incremental to accelerated with the team you already have in place. RidgeLine is the only AI-native investment management platform that brings together the front, middle, and back office

• 29:00Speaker 00

  with one set of data and powerful AI agents that automate away operational drudgery, so your talent can shift from managing old technology to materializing new ideas. RidgeLine helps asset and wealth managers understand and improve their firm's growth trajectory. To see how your firm compares to 16,000 of your peers, visit ridgeline.ai/trajetory. If you file Form ADV, your firm is already there. Then talk to RidgeLine about how you can do more with the team you already have. How do you encourage that psychological safety in the context of running the pre-mortem?

• 29:41Speaker 03

  We describe what the pre-mortem is going to do, and if it makes the organization too uncomfortable, then we might simply not bother, not waste our time doing a pre-mortem. So it takes courage for an organization to say, okay, let's do it. But organizations have learned to trust us, and they've seen the results of what happens when we describe other pre-mortems that have occurred, like the story that I told you, and they said, we want some of that. But you're certainly challenging a culture, and a culture of intimidation, where people don't like to speak up because of the issue of psychological safety. One of the benefits of a pre-mortem is it starts to change a culture

• 30:26Speaker 03

  from a culture of hiding to a culture of candor, where we've been able to voice these concerns in a safe place. Because, you see, if I run a planning session and then I say, okay, does anybody have a problem with our plan as we're getting ready to implement it? We've just built a plan. It takes a lot of courage to say, well, here's why I think the plan isn't going to work. It essentially means we're all idiots, or we've been wasting our time. So that's really hard to do, and people don't want to do it. On the other hand, with a pre-mortem, the contest is to see who can come up with the most clever and worrisome issues that haven't been discussed at the time.

• 31:10Speaker 03

  So the whole incentive system changes, and now people are competing with each other to come up with really insightful problem areas that haven't been voiced before. It's amazing to see how it happens.

• 31:23Speaker 04

  And I think that subtlety, and I think what Gary just said is really important, is none of us really like counterfactuals. I've learned on your podcast many times that the counterfactuals are your best friend. As Annie Duke likes to say, we're trying to get to accuracy, not to being right. Well, if I'm in a meeting, and there's a lot of momentum, and everybody's excited, and I raise my hand with a counterfactual, I'm really putting myself at risk on many levels. I may be wrong on top of it. This is a concern I have, we should probably address. But if it ends up being a wall of goose chase, now all of a sudden I've exposed myself. The pre-mortem, you flip that. Now we're going to have a contest, who can be the most clever, the most insightful, who can come up with the most serious risks that we've not talked about? And what I found when I've run my pre-mortems, that little subtlety is shockingly powerful.

• 32:07Speaker 04

  Because people, all of a sudden now, it's no longer a career risk. It's sort of like, oh, how clever can I be? How insightful can I be?

• 32:13Speaker 03

  This is how you show off.

• 32:14Speaker 04

  Yeah.

• 32:14Speaker 00

  Whenever I read about, whether it's in your paper or Gary's story about this fighter pilots, it's always the introverted person who doesn't speak up, who seems to have the insight, that's the mind-blowing insight. So how do you get that person in the room to consistently be part of the discussion?

• 32:39Speaker 03

  Well, that's another part of the rhythm of a pre-mortem, is that we go around the table and everybody gets a chance.

• 32:46Speaker 04

  Not only gets a chance, has to.

• 32:48Speaker 03

  Everybody has to come up with something, and is expected to. And so that's a part of it. There's also a structure that, as I go around the table, I say, okay, Ted, what's the top of your list? And you tell me what you have, and then I'll say, Paul, what do you have on your list that Ted hasn't already covered? And we do it one at a time, one item at a time. So you don't get to give your entire list. And there is some dynamic in some groups where people who are extroverted rather than introverted, they want to take more than their allotted time. They want to be able to tell you their whole list, they want to sort of go on. And that's not going to work, because everybody else is waiting.

• 33:30Speaker 03

  And so this is a way of controlling them and giving everybody equal airtime.

• 33:34Speaker 04

  And you create democracy in the group. You force democracy. Everybody has to vote, which gets that introvert to actually say something, and often a very good insight.

• 33:43Speaker 00

  Is there significance in the process to writing down that first response as opposed to it being an open, but person-by-person verbal session?

• 33:54Speaker 03

  It's too easy to say, yeah, I thought of that, or just not to keep track of what you had considered. So this really forces each individual to go on record, at least on the paper in front of them, and write down what their ideas were.

• 34:08Speaker 02

  You also get that production blocking, and that people, if they don't write it down, they'll forget. So it's a lot in the cognitive science that there is importance to writing it down as well.

• 34:20Speaker 03

  There's one more issue here. By having people write it down, then they're each working independently. So I'm not influenced by what other people are saying. These are just my own ideas.

• 34:31Speaker 00

  Are there any other features of the pre-mortem that are kind of essential to making it work?

• 34:36Speaker 02

  What we've done a lot of work on is the wisdom of crowds and cognitive diversity. And we think that if you think about an investment committee meeting where they're discussing a stock idea, you have senior people and you have junior people. And the junior people, like, fear stifles ideation. So they're afraid to say anything, because they've only been there for six months, and the portfolio manager's been there for 20 years. And as Paul said, they could be wrong, and they don't want to look stupid, so they keep their mouths shut. When you think about all of the money that firms spend on that talent, it's wasted, because there's a lot of cognitive diversity

• 35:21Speaker 02

  that is lost if these people don't articulate what they're thinking. And that also happens when you get the conversation, which is dominated by one particular individual, is that that cognitive diversity is lost.

• 35:35Speaker 03

  And we once did a study where we had four-person teams going through an exercise, and it's called a garden path exercise, where we introduced something at the beginning that made it seem that certain thing was going to happen, and everybody was preparing for it. But that was wrong. We were trying to mislead them. And then we dribbled in cues to indicate that it was something else going on. And we wanted to see when the team would get off the garden path. We ran seven teams. None of them got off the garden path. The initial assumption held for the entire session. So that was a failure. But we had each person keep a diary of what they were thinking. And we found that in every team,

• 36:17Speaker 03

  at least one or two people had noticed the anomalies and had worried about it. So the knowledge was in the team. You might say the subconscious of the team. But nobody said anything. And so that's the kind of fear that we're trying to overcome.

• 36:33Speaker 04

  And I think if you get back to the diversity comment, is that the more people you have in the group that bring a different perspective, the more cognitive diversity you're going to get almost by nature. And so if you create psychological safety, you do the pre-mortem where you force everyone to contribute, and people are going to come up with their own. If you have this cognitive diversity, you're going to get a broader list of items. I ran a pre-mortem a year ago on a project we were going to launch. 27 people in the room, wrote 27 things down on the first pass. Five of them were path-critical. Of those five, I'd only thought of one. And so I was blown away. Because I was team leader, I took it very seriously. We wrote stuff on the board, I was like, I hadn't thought of that. That's a really good one. And it's that power of getting the cognitive diversity,

• 37:18Speaker 04

  forcing the interaction, forcing people to contribute. And the 27 things, five of them who cared, five we'd kind of thought about already. And then you start to get into some really good stuff. And there were five that were path-critical, like the computer in the army thing, that were going to kill the project. I hadn't thought of four as a team leader. And we hadn't in the process. We'd had a very long process. So it was just very powerful, the combination of the two.

• 37:42Speaker 02

  Ted, as you can see, as we're going through this process, there are like four or five really nuanced things that if you don't get it right, the pre-mortem doesn't work, and they're very subtle. So just something as simple as having each person say one idea, or that problem reframing, saying that the project has failed, or requiring each person to say one thing. If you don't get each of those steps right, it doesn't work.

• 38:15Speaker 00

  So let's say we do it the right way. We take our deep breath for 15 seconds, and we've got two minutes of writing it down, and the leader starts, and we go through this, and now we've got 27 ideas on the board. But Paul, as you said, four of the five important ones you hadn't thought of. Great. What if the real issues were in the other 22? So how do you know when you go through this? Or more importantly, when you go through and you've got all your ideas on the board, how do you then make sure that you're using those ideas to improve the process?

• 38:52Speaker 03

  So after you get everything out in the open, that can pretty much end the meeting. Then we do another phase. What ideas do people have for mitigating that? But then the team leader takes all of that material and goes away and thinks about it and sorts through and said, might say, okay, this one, hmm, I was ready to dismiss it. But now that I think about it, it might be more important than I imagined. So there's a subsequent phase of replanning and reassessment using all the material.

• 39:21Speaker 02

  The other thing is that there's no system that's foolproof. So going back to Rumsfeld, that he said that in terms of risk, there are the known knowns, there are the known unknowns, and then the unknown unknowns. What the pre-mortem does is you're trying to take those unknown unknowns and make them known unknowns and reduce that. But you're still going to have unknown unknowns.

• 39:46Speaker 04

  And the key is that you're never going to eliminate all of them. But you learn so much in this process, and it doesn't require much time or energy or investment, that it's borderline free, and you do pick up a bunch of things that you didn't have before. I guess if you could invite God to the pre-mortem, we could get everything. But short of that, the list is going to be so far expanded that if it's really a critical plan, you could do a second pre-mortem. Replanning, come back in a day, a month, or whatever it is, do another pre-mortem after time has changed. You're never going to be 100% foolproof. There's always going to be some unknown unknowns. But you have mitigated enormous amount of risks, and we think that's incredibly powerful.

• 40:23Speaker 00

  There are a bunch of other techniques that we see investment teams use to try to mitigate risks upfront. What are some of your favorite ones in addition to pre-mortems? Or what are some of the ones you don't like as much?

• 40:36Speaker 03

  Let me start with the ones I don't like that much. So one of them is the devil's advocate approach, which gets a lot of publicity. I'm only aware of one study that tried to evaluate the effect of a devil's advocate to see if it had an effect. And they got a significant result, was significantly worse than not using a devil's advocate. And the reason seems to be, if we're a team, and I appoint somebody to be the devil's advocate, then I don't know if the person is coming up with these ideas because they're of genuine concern or because that's the person's role. But I assume it's because it's their role, and they don't really necessarily believe in it. So it's easy to discount a devil's advocate. It also means I don't have to be critical.

• 41:19Speaker 03

  I've outsourced it to a devil's advocate. So that's not healthy for a group.

• 41:24Speaker 02

  Oh, so one thing, and I guess I call devil's advocate, Dr. No. And that's the role that I've played within organizations or have tried to create, and you're a skunk at a picnic. And it's just, you piss people off, and people don't like it. The other thing is that if you just have one person do it, or a small number of people, you're really losing that cognitive diversity of the organization.

• 41:53Speaker 03

  Right. And so what you want is for people to articulate genuine concerns, which a devil's advocate doesn't, and want to have the psychological safety to be able to articulate that. Another approach is to use red teams, and that's a very powerful method, and certainly a very impressive one. I like the method of red teaming, bringing in outsiders to review it. How does the red team work? There's a great book by Bryce Hoffman called Red Teaming about his observations of an army unit at Fort Leavenworth who was set up to do red teaming. You just bring in outside experts who haven't been involved in preparing the plan so that they can do the critique. And so they have different kinds of expertise. Now, we're talking about diversity. And they don't have the investment in the plan.

• 42:37Speaker 03

  They don't feel like they have to make the plan work.

• 42:40Speaker 04

  And the problem with red teams, particularly in the investment business, is since we're making decisions almost daily, or certainly fairly frequently, having a red team coming in and prepping them is probably prohibitive in terms of time and energy. Works really well on a big plan. Works really well where there is outside expertise you can bring in to influence it. But when we're making decisions more in process on a regular basis, investment decisions, you just couldn't scale it in the right way.

• 43:03Speaker 03

  That's an important issue, is what kind of effort is involved for a pre-mortem? If you took two hours to do a pre-mortem, that would be terrible because people don't have that time, so they wouldn't use the pre-mortem very often. And it would reduce the energy. And then people would be sitting back in their chair rather than leaning forward, ready for their turn in the barrel to be able to say, here's my best salvo against this plan. Pre-mortem should take no more than a half hour, maybe 20 minutes. If it goes for an hour, that's too long. So there's a virtue of keeping it really brief because that compresses the energy rather than dissipates it.

• 43:40Speaker 02

  And when you think about it, in the investment process, like if you buy a stock that goes wrong, it is so painful that if you could just eliminate a small fraction of the mistakes, it makes your life so much better. And the batting average with investments, it's like 55% for someone that's a hero. So if you just make one or two less errors over whatever time period, it's a total win, and it's not a lot of time.

• 44:11Speaker 03

  Another technique is risk assessment. And there are a variety of risk assessment methods, and a lot of research has gone into potential risk factors. And so having people systematically go through all of the risk factors. And that strikes me as fairly mechanical and fairly insensitive to context because it's the same factors all the time. Whereas for a pre-mortem, you're looking about what can go wrong with this plan in this context, in this situation. And so that's going to be always changing.

• 44:41Speaker 00

  So Gary, as you're going into any type of organization, and you're pitching them on doing a pre-mortem, how do you describe what the benefits of the pre-mortem are to those organizations?

• 44:53Speaker 03

  First of all, we have collected some evidence about what happens to confidence when you use a pre-mortem. And we compared it to other methods, like listing pros and cons, listing cons, just offering critiques. Offering critiques seem to have no impact on reducing confidence. Listing pros and cons are just cons, reduce confidence. But the pre-mortem reduced confidence more than any of the other techniques we measured. So that's one of the benefits.

• 45:21Speaker 04

  And I think, Gary, what you mean by that is overconfidence.

• 45:24Speaker 03

  Right, overconfidence.

• 45:25Speaker 04

  Right. We're trying to reduce the hubris in the group, that we figured everything out, the plan's perfect, let's go, nothing can go wrong. So any kind of critiquing method is about reducing overconfidence, increasing the humility, and recognizing that things can go wrong.

• 45:40Speaker 03

  Thank you for adjusting that.

• 45:41Speaker 02

  You don't have a lot of overconfidence on Wall Street.

• 45:46Speaker 03

  So we assume overconfidence is pretty much the norm in many settings. And so pre-mortem has that effect. A second effect is it changes the culture to some extent, to one that accepts candor. And so where people are feeling that they can speak out, and of course, they have spoken out with their bosses and supervisors present, and the world didn't change. They didn't get fired.

• 46:11Speaker 04

  I think one of the follow-ons is you do a couple of pre-mortems, which I have now done. You get to the point where you realize it's so incredibly powerful. You're sort of an idiot not to do it. It's not very much time. You're tapping into your group. You're building confidence within the group to be candor. You're getting more honesty, more transparency. The group dynamic improves. I've failed to find anything that hurts because it's really talking about a half an hour, 45 minutes of time commitment. So even that time commitment, that after you've done a couple of them, you'll get groups that refuse to not do that, right? We're not going to make it to say, I know you're this, Gary, I know you feel the same way. I'm at the point now where I won't, any plan I'm in, small or big, I won't go forward without a pre-mortem. It's just, once you've done two or three of them, it's so powerful.

• 46:54Speaker 04

  You're sort of like, why would we not do it? And then that increases the dynamic in the group. Everyone is now more candor during the planning process. We've gotten to the point where the pre-mortem is sort of not that important because we've been really critical in a very constructive way along the process. We still do it. But at the end, it's this sort of sense of confidence where we're being a much more thoughtful group and sort of figuring these things out. That, I think, is the real benefit, is start to do them. You do one that works well, I don't think you ever go back.

• 47:21Speaker 03

  And another advantage is you make the group smarter because now we're listening to each other and realizing, I came up with one, I'm looking at you, Paul, and there were four others that I didn't come up with. So at the end of that session, you were smarter, you were more sophisticated, your mental model was richer. And you also have a chance to see somebody who maybe didn't say a word, that they really, even though they were sitting there quietly, they were thinking, and that they have something to offer, and it would have been missed otherwise.

• 47:55Speaker 00

  Great. Well, we've mentioned this paper and we'll certainly put a link to it. Where is this paper sitting today?

• 48:01Speaker 04

  We published it in a fairly academic structure. It's called "Rendering a Powerful Tool Flaccid: The Misuse of Premortems on Wall Street." We put it out into the wild. We want people to read it. We want people to use it. Certainly, we will consult with them if they have any questions in any way. In more formal setting, Paul or I will certainly come into an organization. Gary loves doing these, so he's available. But he's the busiest man I know, and he lives in Washington, DC, so a little bit harder. I teach this at Columbia Business School, so if somebody's interested, they can come when we talk about premortems for that lecture. Yeah, our goal is to just give this away to everybody as much as we can.

• 48:36Speaker 02

  I guess we haven't officially released it into the wild yet, but we would have it on our website, pitchtheperfectinvestment.com. And then we'll also try and put it on a SSRN, so it'll be available to the masses. And we're looking at different academic journals to possibly get it published, but we're just not sure if jumping through all the hoops and actually getting it published helps.

• 49:02Speaker 00

  All right, speaking of which, what's happened to Pitch the Perfect Investment since you guys were last here talking on the show?

• 49:08Speaker 02

  It's actually sold really well.

• 49:11Speaker 04

  I'm going to brag more. I recently taught a class at Columbia Business School. It was an investment class for one week. One of the students, very experienced hedge fund manager in Australia, was taking the class, just heard it was a good class. And he thought, you know, I might as well read a little bit of the book on my flight to New York. He thought it'd be great. He's read a lot, written a lot, very experienced. And he said he started reading it, and he didn't stop until he read the very last page. So I've heard this from time and time again, people, particularly practitioners, that say, you know, "It's not really designed for me. It's really for a student, somebody coming out of business school." And they start reading it, and they're like, "Holy smokes, this is fantastic." I've heard that now enough that I will repeat it. Whereas when we were on last time, it was just being launched, and the thought was, "We hope it's good."

• 49:54Speaker 04

  You know, people love it, and I can't encourage... And I always joke with them, "The book is $33 on Amazon. Paul and I each get a buck and a half. We'll rebate the buck and a half if you buy the book." But I really encourage people to read it because I think it's great. And I think that they walk away and they're like, "Holy smokes." Either it just reminded me of some of the things I knew, maybe it gave them an insight. So I can't encourage people enough, and I'm happy to rebate my royalty.

• 50:19Speaker 00

  Great. Well, Gary, while we have you here, before we let you go, I want to ask you a couple of closing questions. What's your favorite hobby or activity outside of work and family?

• 50:29Speaker 03

  There's a number that I enjoy. I like movies. I like to read. I like to jog. I like to play the game of Go. That's pretty much it.

• 50:37Speaker 00

  All right. What's your biggest pet peeve?

• 50:40Speaker 03

  People thinking that complex tasks can be boiled down into checklists and procedures, and you just have to follow the steps, and you don't need expertise. And you just have to follow the steps and don't be an idiot. And that's just too shallow.

• 50:53Speaker 00

  Doesn't quite work that way.

• 50:54Speaker 03

  No.

• 50:55Speaker 00

  All right. What reading do you almost never miss?

• 50:59Speaker 03

  New York Times Sunday Sports section.

• 51:03Speaker 00

  How about what teaching from your parents has most stayed with you?

• 51:06Speaker 03

  The idea that when you hit what looks like a barrier, rather than saying, "Okay, that's as far as I'm going on this path," often there's a way around it. And not to give up so quickly, but to look around and see, "Are there any leverage points? Are there any opportunities that I haven't considered?"

• 51:26Speaker 00

  All right, last one. What life lesson have you learned that you wish you knew a lot earlier in your life?

• 51:33Speaker 03

  When I see people make mistakes, instead of immediately becoming critical and impatient, to become curious. Why would somebody smart do something that looks stupid? Maybe there's a reason. Maybe I need to try to investigate further.

• 51:50Speaker 00

  Great. Gary, Paul. Paul, thanks so much for joining me.

• 51:54Speaker 02

  Thanks, Ted.

• 51:55Speaker 03

  Thank you very much for having us.

• 51:57Speaker 00

  Thanks for listening to the show. If you like what you heard, hop on our website at capitalallocators.com, where you can access past shows, join our mailing list, and sign up for premium content. Have a good one, and see you next time.

• 52:13Speaker 05

  All opinions expressed by TED and podcast guests are solely their own opinions and do not reflect the opinion of Capital Allocators or their firms. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of Capital Allocators or podcast guests may maintain positions in securities discussed on this podcast.