E347: The $26B CIO Who Turned Superforecasting Into Alpha

How do you manage a $26 billion public fund while keeping every investment decision disciplined, every team member calibrated, and every partner accountable? In this episode, I sit down with Mark Steed, Chief Investment Officer of AZ Public Safety Personnel Retirement System, to explore how super forecasting and probabilistic thinking shape portfolio management. Mark shares how lessons from Dr. Phil Tetlock's the Good Judgment Project inform every investment decision, why intellectual humility and calibrated confidence drive better outcomes, and how simplifying portfolios into broad buckets creates flexibility and competition for capital. He also unpacks the role of co-investments, structural alpha, and first principles thinking in public markets.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00David Weisburd?

  How does super forecasting play into how you manage $26 billion today?

• 0:04Speaker 01

  This is a great question, and I love that we're starting with Dr. Phil Tetlock. So Phil Tetlock spent decades proving that like most experts are terrible forecasters, not because they're dumb like in any way. It's just that they're like most of us overconfident and like they don't keep score. So, and I know this firsthand, but back in 2010, Dr. Tetlock organized a team to compete in what was really a four year forecasting competition for the Department of Defense. And I was on that team, so I was a guinea pig. And over like the subsequent four years, which I think went from 2010 to about 2014, grinding through geopolitical questions with like real scoring. I learned what we call calibration and what calibration thinking actually feels like.

• 0:48Speaker 01

  So he eventually published his findings in the book Super Forecasting, which became a bestseller on Amazon, probably around 2016 or 2017. And I basically brought that methodology back to my investment team. So what does that mean for us right now? Every investment decision requires a very explicit probabilistic forecast. And let me explain what that means. It's not like I think this fund looks good, but it's, I believe there's a 55% chance, for example, this say manager will outperform the MSCI healthcare index by 400 basis points over the next five years or five years from the close of the fund. So it's specific. You have to have a probability forecast. You have to have a clear definition of success and a time horizon and a confidence level.

• 1:31Speaker 01

  So the discipline isn't just about, you know, being right. It's about being calibrated. So when you say you're 90% confident, are you 90% right? And so it's about knowing the difference between like what you know and what the information allows you to know and what you believe.

• 1:44David Weisburd?

  So how do you track your team's investments internally based on a super forecasting methodology?

• 1:50Speaker 01

  We use the Briar Scores. And the Briar scores are something that were championed by Phil Tetlock. And he has since commercialized his, his research into what's called the Good Judgment Project. So the Briar score is just a mathematical formula that ranges from zero to two. So lower score is better. So a zero Briar score would be that you're 100% confident and 100% right. So you're basically like, you know, an all-knowing kind of omniscient, omniscient being. And a two is that you're 100% confident all the time, but like 0% right. And just for reference sake, a Briar score of 0.5 would be that you're sort of a, you're sort of a coin flip. But it's not just about the Briar score. You also have to attach kind of a rationale, you know, how did you get here? And so we're tracking the quality of the decision-making as well.

• 2:35Speaker 01

  And we can measure, you know, hey, if somebody's overconfident or underconfident, what would have happened to specific investments? If you actually had that, if you're actually appropriately calibrated. So that's, that's one way that, that we do it to make sure that when, you know, first of all, there's an objective score, but also that people are right for the right reasons. And that, that level, I think of introspection is really important. And that's the main way that we track it.

• 2:57David Weisburd?

  So this is a fascinating thought experiment in and of itself, but you've actually implemented at Arizona PS, PRS. What are the second order effects of this?

• 3:05Speaker 01

  First of all, intellectual humility just becomes absolutely contagious. So when everyone's score is on a wall, we're all competitive people, right? Nobody walks into a meeting claiming some kind of, you know, certainty that they don't have. So that social vindication element really is looming large. Second, I think we make fewer big mistakes because the process forces you to articulate, you know, what has to be true for the investment to work. And step one starts with setting what we call the base rate, meaning you take this outside view and you ask like, what usually happens? What usually happens to say a private equity fund or whatever it is that you're looking at? And that keeps you from falling prey to sort of like, you know, this, this illusion of control.

• 3:47Speaker 01

  The third one is, and maybe, maybe the most important is that it changes the culture of disagreement. So it, in some, I think, investment committees, it's like this really combative sort of, you know, gladiatorial event where there's a clear winner and a clear loser. Or where you have people who are just really good at debate, who can take up a lot of air in the room, or maybe it's just the senior people in the room talking and the junior people don't, you know, don't speak up. All of that goes away when you have this objective scoring and this, and this objective, very quantitative approach. So it sounds more like, hey, I'm sort of thinking 60% confidence in this investment. But you're kind of at 80. Like, how do we bridge, how do we bridge the gap?

• 4:27David Weisburd?

  Expert calls have always been one of the most powerful ways to build conviction. But today, investors are asked to cover more companies, move faster, and do it with leaner teams. With AlphaSense AI-led expert calls, their TGS call service team sources experts based on your research criteria and lets the AI interviewer get to work. The magic is in the AI interviewer, purpose-built and knowledgeable-based information to conduct high-quality, context-rich conversations on your behalf, acting as a trusted extension of your team. Then they take it one step further. Your call transcripts flow natively into your AlphaSense experience and become queryable, searchable, and comparable. So your primary insights plug directly into earnings prep, digital work streams, and pitch books with zero tool switching.

• 5:11David Weisburd?

  And with AlphaSense expert call services, the AI-led expert calls are just one option because we know the importance of a hybrid expert research approach. AI for coverage and efficiency, humans for complexity and conviction. It's the institutional edge that scales research without scaling headcount. For hedge funds, that means validating thesis assumptions across dozens of experts before earnings instead of a handful. For private equity, it means faster pre-IOI scans and deeper commercial diligence. For investment banks and asset managers, it means pulling real operator perspectives straight into models and sector positioning without disconnected tools or manual handoffs. All of it lives inside the AlphaSense platform, trusted by 75% of the world's top hedge funds alongside filings, broker research, news,

• 5:56David Weisburd?

  and more than 240,000 expert call transcripts, turning raw conversations into comparable, auditable insight. Take advantage of AlphaSense AI-led expert calls now. The first to see wins, the rest follow. Learn more at alpha-sense.com/HowIInvest. So everybody wants to improve their score, so they're actively looking for feedback from the rest of the team in order to better calibrate their position versus this being the zero sum, this is my investment. And somebody being against investment and you've kind of butt heads in the sea or some way.

• 6:27Speaker 01

  Exactly. And it's very, it's really fascinating. Some of the best forecasters on the team are people who have come to us with absolutely no prior investment knowledge. Now, they're sort of probabilistic thinkers and or they're what we call numerate, meaning they just sort of think in terms of sort of quantitative, but it's not like they're running real complex models sometimes. It's just that that's, you know, they're thinking in terms of likely outcomes, likely ranges, you know, and again, they start with the base rate, what normally happens, and then they start looking for data that can kind of move them off of it. And it's not so robotic, you know, but you can sort of say, hey, look, on average, there's kind of a 30% chance of these things, but I'm looking at this other data and that kind of is moving me, you know, I moved up kind of five or 10% here, right? But then this kind of moving back down 5%. So you're putting guardrails around a conversation and it's not, it's not a

• 7:11Speaker 01

  subjective, and I find that that's just really important.

• 7:14David Weisburd?

  I've been going down this rabbit hole of figuring out how the most innovative people in the world become the most innovative. And prime example, of course, is Elon Musk. I've interviewed a lot of his friends and a lot of people around him. And one of the things that I've learned about how to tell whether somebody's crazy or extremely innovative, one way is, does it actually work? So you could tell post facto, in the beginning, Elon Musk was trying to create a private space company. Even his closest friends, his most innovative friends had an intervention, showed him all these videos of space rockets crashing. But he really had first principles. So he had these ideas, these building blocks of logic, and he had this thesis on why it would work.

• 7:56David Weisburd?

  And the thin line between whether somebody's crazy or extremely innovative is actually not based on them or even people's reactions. It's based on the first principles logic. So I know, A, I know that sounds crazy. Okay, fine. Now, B, now look at my logic, poke holes in it, either to improve it and to improve the thesis, or to kill the thesis. But if the first principles hold, even though it's completely crazy, then it's actually innovative, not crazy.

• 8:22Speaker 01

  In the context of our organization, I think that's what this forecasting application really does. Because what you're trying to build over time is calibration, right? Are people, when they say they're 90% confident, are they 90% right? If they say they're 60% confident, are they 60% right? So we write down, of course, all the training questions and also all the investment recommendations. And we're looking at those resolutions, right? And we, for any individual on the team, we can say, well, here are all your 60% confident forecasts, right? Here are all your 80% confident forecasts. And where you write eight out of 10 on those, 80%, six out of 10 on the 60. So when you get to innovation, that becomes really important. Because what you want as a CIO is to know that your team is calibrated. Somebody comes to you and they say, well, here's an idea we probably haven't thought of.

• 9:03Speaker 01

  We start thinking through, you know, okay, well, is this more like, is this a 50/50 bet, right? And that, we can get into, you know, how that works with the specific investment examples. But like, that is really important to help, like we said, help parse the ideas that are truly crazy from the ones that are actually innovative.

• 9:20David Weisburd?

  This reminds me of Bridgewater and how Ray Dalio created the principles where everybody kind of openly fights their positions in order to get better. But upstream of that is if you have this organization where people know that they could come to your organization to basically tune and calibrate their investment skills. That's a hell of a recruiting advantage over other organizations that might be seen as tribal or political.

• 9:42Speaker 01

  Now that we do it this way, I can't imagine doing it any other way. Because when you, we have that level of objectivity, it just levels the playing field. And people who wouldn't normally speak up, you know, because they're newer or for any other reason, actually have a voice. And it's not necessarily that they speak up more, but it's the people who tend to be overconfident, right? That will start to defer. And I think that's just a really healthy team dynamic, for sure.

• 10:06David Weisburd?

  There's this paradox of knowledge where the deeper you go into a topic, the more you realize how much you don't know. So said another way, if you don't know a lot, you're going to have a lot of times a very high confidence. So there is this negative correlation between somebody's confidence and sometimes their depth to their knowledge.

• 10:19Speaker 01

  That's well said, because there is like the paradox in decision making, which is the faster you admit what you don't know, actually the better you are at making decisions.

• 10:29Speaker 01

  We see this pattern in the forecasting results of our own team. So what will happen is they'll make a few forecasts early on and let's say the results are really good. So then they build up all this confidence, right? Without really getting any, any better at the calibration process. And then sort of like mid, you know, mid sort of forecasting career, right? Then they get a bunch of really bad scores. And then they figure out, okay, wait, wait, you know, maybe I was overconfident. So then they, then they start to pull back and they think, okay, I'm the first ones. Maybe I got lucky here and I, I confused that luck with skill. And then I got overconfident. Now you start to break them down and that's the process of calibration. And they start to, they start to pull the confidence down and your, and their accuracy will start to go up. And that's what you want to see.

• 11:10David Weisburd?

  Do you go through this super forecasting process as well?

• 11:15Speaker 01

  Let me give you an example of how this would work on the team, right? So you have high, you know, kind of have high and low confidence. And let's just take like a low, like a low confidence example, right? So the low confidence example, oh, I always get asked this, right? Can you, can you make a low confidence investment? And low confidence doesn't mean, you know, don't invest. It's just like poker, right? You don't fold a hand just because you're not holding the aces. So this is classic, the way we approach this. Classic expected value theory, right? You multiply the probability of being right times the upside alpha subtracted by the probability of being wrong times the downside, right? And then if the number's positive, you sort of have a case. So like, here's an, like, here's an example. I'll make the math easy on myself on the podcast here, right? So say you're like looking at a private equity fund and it's focused on a specific sector.

• 11:59Speaker 01

  We'll benchmark them, say on, on like a PME basis against the sectors like public, public index. And let's just say for the sake of argument, right? That history tells you in any vintage, say two out of 10, you know, sector specialists will add a thousand basis points of alpha over the sector benchmark. So your starting base rate is 20%. You can be 20% confident, you know, that's the outside view. What normally happens, 20% confident that a manager is going to add a thousand, a thousand basis points of alpha. And then you go hunting for information that moves the needle. So maybe their prior fund did it and that maybe moves you up from a 20% to a 30%. Maybe the team's intact, maybe there's a proprietary sourcing angle, they've got a playbook.

• 12:40Speaker 01

  But let's just say for sake of argument, that kind of moves you to 50%, like a coin flip. So 50/50, right? Meanwhile, let's say if you're wrong, you estimate that you lose about 100 basis points relative to that sector index. We take the average of the other eight managers and let's just say they're a minus, you know, they're minus 100. And so the math is you have 50% times 1,000 basis points, so that's 500, minus 50%, you know, times 100. So your expected value is 450 basis points, right? That's not certainty, but that's a solid investment and that's basically how we approach it.

• 13:13David Weisburd?

  Last time we chatted, you mentioned that you're being benchmarked against the S and P 500, whether you like it or not. Why is that?

• 13:19Speaker 01

  So here's the uncomfortable truth of institutional investing. Every dollar we put into private equity, real estate, hedge funds, you know, what have you, it's a dollar. We didn't put the S and P 500 index or a bond index, but nobody, nobody compares alternative investments to the bond index. So the S and P wins the benchmark conversation, like just by default. Probably because it's like the one number everybody already knows. It's familiar, it's on the news every night. So even if it's not your explicit stated benchmark, it's the one in the back of every journalist's mind, every legislator's mind, every board member's mind, every constituent's mind. And like the S and P has been, you know, extremely good for the last 15 years, which makes the conversation even harder.

• 14:03Speaker 01

  And my job is to justify the complexity, the illiquidity, the fees of a diversified portfolio. And the only honest defense is long-term risk-adjusted returns over full market cycles. And we're not trying to beat the S and P every year. We're trying to not be the fund that blows up when the cycle turns. But like, as I say all the time, diversification is a absolute punishment for sure in a bull market, but it's a miracle in a crisis.

• 14:29David Weisburd?

  And it gets even more complex, given that you manage for police and firemen. So it's not just one single decision maker where you could basically explain and educate them on the market. There's thousands of underlying members that are de facto going to use S and P 500 as its mental bedstone.

• 14:45Speaker 01

  Absolutely, and you have to keep your audience in mind. And that you just, and I think that I think it's good that you sort of keep, you know, keep that idea in the back of your head. Because it does keep you really honest when you're, when you're adding investments. And I think that's, you know, not a bad way to construct a portfolio. I do think you have to, you have to have kind of like, you know, what your constituents understand in the back of your mind, you know, at all times. So as annoying as it is, I think it is a healthy way to build a portfolio. And I think it's a, you know, a fair way from a public policy standpoint to interact with your constituents.

• 15:16David Weisburd?

  And you have to construct your portfolio in these two concentric circles. What, who your audience is and what is the ideal portfolio and find a way to kind of match both of those.

• 15:24Speaker 01

  By and large, you want to kind of keep the portfolio constructed in a way that like people will understand it. Right? I mean, that's the, I think that's, that's really critical. I mean, these are people, right, who didn't sign up to study asset allocation and their retirement is not in their own hands. And it can be kind of disconcerting to kind of like trust other people and, you know, with your retirement money. So I do think you have an obligation to construct something that they understand. Now, at times, there are things where you, you just know, look, this is not going to be well understood, you know, to a layperson. But we feel strongly that it's going to be helpful in the portfolio. And then, you know, you had that conversation with your board. But I do think you have to kind of keep what other, you know, what your constituents will understand in the back of your mind at all time.

• 16:02David Weisburd?

  Tell me about your portfolio construction today and maybe how that's evolved over the last few years.

• 16:07Speaker 01

  So when I became the C I R in 2019, we had something like 10, like we 10, 10 asset classes, I think is what we had, you know, maybe more. So, and I've changed that to simplify it remarkably. So we start with a really simple premise, right? Our liability. At the end of the day, there's a longstanding stream of payments to first responders, right? And so for me, the mission is in this order. One, have money when money is due. Manage portfolio volatility so it doesn't translate into contribution rate volatility that employers and members just can't absorb because it makes it hard. And do all of that, I think, spending as little money as possible. That's the whole job in that sequence. It's like I said, we do feel an obligation to make it understandable to the people whose

• 16:51Speaker 01

  retirement this is. Again, they're not signing up to study stocks and bonds. They signed up to serve. So we really break the portfolio at a really high level into three broad categories. And for us, that's called capital appreciation, contractual income and diversifying strategies. Because those labels, I think, describe how investments actually behave, not sectors or anything like that. And we tell members, within each of those, there's publicly traded investments and there's private ones. That's basically it. So the edge in my mind isn't having like all these exotic buckets. It's just having fewer of them and filling the buckets really well.

• 17:26David Weisburd?

  You could think of the buckets as essentially constraints. So if you have a specific public equity and private equity, and for some reason you see private equity as more risk adjusted, then you're arbitrarily constrained into this public equity where you don't want to be playing just because somebody came up with these buckets.

• 17:42Speaker 01

  I could ask, why have the loosely defined buckets? And I think to your point, I think the rigid buckets can, I mean, the glib answer is the rigid buckets can lead to just stupid decisions, right? So I think the problem with defining asset classes by like sectors, having an asset allocation, I mean, where you have sectors and sort of capital structure. So you have a real estate allocation, but then you also have a public equity allocation, right? So you get into these conversations about, well, what do you do with a real estate equity guy or a real estate credit guy, right? And I just think that that gets conceptually messy pretty fast. So sure. Real estate, you know, sometimes behaves differently than stocks. But, you know, how often does it really, right? And with like, what degree of confidence? And take, probably the most recent example would be commercial real estate, right?

• 18:27Speaker 01

  Commercial real estate has been like a long time darling of the inflation protection crowd. And then COVID hit, followed by like, actual inflation. And commercial real estate didn't help because other forces were swamping the inflation signal, right? So the theoretical benefit just got buried in the specific like facts on the ground. I think making the buckets wider increases competition among investment ideas. And for a pension fund with a longer horizon where we can stomach some of that volatility, the more competition for capital, I think just leads to better outcomes fundamentally. And so loosely defined ranges lets us be flexible and opportunistic, which is a huge advantage. The buckets are, in our minds, just kind of a map, but like, you know, any navigator knows the map's not the territory.

• 19:08David Weisburd?

  Are you essentially implementing TPA into your portfolio? And if not, why not?

• 19:13Speaker 01

  I'm gonna be honest, I can't describe TPA. And every time I hear somebody like define it, I just think like, that kind of sounds like what we're already doing. And, you know, maybe we all arrived at it independently and maybe that's a sign, right? Maybe the logic of it is pretty sound. But I think either way, the core question is, you know, within each sort of bucket or sort of risk category, what's the best complement of assets available to us, you know, right now? Given our size, it's not like, hey, is this a, you know, a good credit investment? Or to your point, like, is this a good private equity deal? And maybe, you know, private equity is just not all that attractive. But it's like, is this a better use of capital than the next best alternative with a similar risk and correlation profile? And so I think that reframing is just really, I think, really important.

• 19:56Speaker 01

  And I think it's generally hard to execute institutionally because it just requires overcoming this gravitational pull towards asset class silos, which has just run, you know, a huge, deep trail down all of our cerebral cortexes. People get comfortable defending their buckets, right? And I think, like, my understanding of TPA is that you're just, you know, you're trying to compete across all the buckets all the time, right? And pair things with common risk exposures to things, other things with common risk exposures.

• 20:19David Weisburd?

  One of the benefits of this bucketed approach or endowment approach is that you have your best athletes in specific asset classes. So you have your real estate credit, your real estate equity, you have your private equity, public equity. How do you build a team that's able to be so flexible? And how do you build incentives mechanism in order for your team to be flexible?

• 20:38Speaker 01

  That's a great question. So I think the answer at the end of the day is, you know, you just have to hire the right people. Public funds have like various constraints. You have to hire people that are intellectually honest. You have to hire people that just never stop asking questions. And the nice thing about having fewer buckets is you don't have anyone that's territorial, right? I can say to someone on my team, hey, listen, you've got to manage the private equity investments right now, but let's be clear, this could be 0%. There's nothing obligating us to have any money in this asset class, right? So I think the good thing about that, in fact, I love the PM that we have who's running real estate right now. He has spent probably the better part of three years just saying

• 21:21Speaker 01

  no to everything, but that hasn't been his only assignment. He's worked on a bunch of different asset classes. So I think you just have to, you know, going in, tell the people, look, you know, you're going to be athletes, you're not going to be married to a single asset class. And that just, that just makes people really honest. And it's not hard for people to say, look, I don't think there's anything to do here in fixed income right now. So I'm not, I'm not making any recommendations or real estate where we've just been saying no, not re-upping with any of our partners. And, you know, nobody's worried about their job. It's all about what's the best allocation, you know, if we're going to take this kind of risk, where's the best opportunity in that market right now? And, and I think that's just a better way to run the portfolio. And it's, I think it's resulted in improved performance for us for sure. When I started as the CIO, we were in the bottom decile pretty much across all the time horizons. And now we're in the top third across all

• 22:06Speaker 01

  the time horizons. And I think you're having the right people is part of that. And then also simplifying the asset allocation. So you have greater competition for deals because it just raises the bar.

• 22:16David Weisburd?

  How big of your strategy is co-investments and has that changed?

• 22:20Speaker 01

  So we haven't been real active co-investors in a lot of the credit-like instruments within what we call global private equity, which encompasses private real estate, private real assets, buyouts, venture capital, all of that stuff, probably about 10 to 15% of that allocation. So it's probably maybe 5% of our overall portfolio and increasing. We have a big focus on just trying to get closer to the assets. Co-investment is going to become a more important part of our program for sure.

• 22:54David Weisburd?

  When I interviewed Chris Aylman, who was CIO of CalSTRS for 23 years, he said one of the ways that they really generated alpha was structural alpha in co-investments. And one of the only things that really worked in co-investments was rules-based co-investment. So creating a diversified pool of co-investments. Lo and behold, the alpha there was the fee savings. So it's somewhat of an interesting view on private equity in general, but they were able to use their size and advantage and capture this co-investment structural alpha.

• 23:22Speaker 01

  It makes a lot of sense. If you focus on a handful of partners, we tend to like the sector specialists. And co-investments, I think, from sector specialists are just of a higher quality than, say, maybe a more generalist or diversified manager. Now, like, obviously, there's exceptions here and there. But I think that's not a bad way to do it. And what you're effectively doing is just creating your own private equity fund. But the deals have already been vetted by a GP that you've underwritten and trust. And so you can have effectively a private equity program with substantially discounted fees or no fee and no carry, even if they perform in line with all the other deals in a private equity fund. You've maybe increased your return on those by three or 400 basis points because you've eliminated that gross to net spread.

• 24:02David Weisburd?

  You're one of the best first principles thinkers that I've interviewed in terms of CIO. I want to get your opinion on something. So I've been thinking about this concept of the semantics on, call it investing. Some people call it allocation, other people call it investing. And then I had Sam Zell's long-term partner, Mark Soder, and he calls it own, owning. He said Sam Zell never said, I allocated or I invested. He said, I own this asset. Something about that, to me, feels more powerful when you're thinking about which assets do I want to own? Not where do I want to allocate or where do I want to invest? Do you think there's something there?

• 24:30Speaker 01

  I do. So in fair disclosure, I have not heard that. But I've always hated the, I've always, always hated the term allocator. Right? I mean, I don't see much of a difference whether you're buying a stock or investing in a fund. P S P R S, we do everything. We've got some, we've got some direct investments. We can buy and sell stocks and bonds. We've got ETFs that we buy and sell. We've obviously commit to private equity and private credit funds. We've done co-investments. But even if we didn't have all of that and we were just allocating to funds, the practice is the same. I mean, it is an investment for sure. And I think we, I think allocators becomes just more of a semantic. I mean, we're just trying to distinguish between the players in the market more than anything else. But I like ownership. I like that idea. I'm going to have to use that and I'll give full attribution to that because.

• 25:15Speaker 01

  Obviously, for me, accountability is a huge thing.

• 25:17David Weisburd?

  To be fair, this podcast is how I invest. If I could call it how I own, if that sounded better, we would also call it how I own.

• 25:24Speaker 01

  Yeah, that's true.

• 25:26David Weisburd?

  But I have to be intellectually honest, I do think own is stronger than invest. I think invest is still passive.

• 25:31Speaker 01

  You could say like maybe semantically like ownership is like maybe, maybe something was given to you or you inherited it or something like that. So it wasn't maybe necessarily a certain decision with an outcome that you have to realize. But yeah, I mean, I certainly prefer invest over allocate any day. But I like, I like the ownership idea just because, again, start to finish, you own the work and you own the outcome. And I think that that's your semantics, your words matter and it's important for enforcing behavior. So I do like that idea.

• 25:58David Weisburd?

  What's something you've changed your mind on in the last 12 to 24 months?

• 26:02Speaker 01

  I'm anchoring to, you know, the most recent geopolitical events and some of the conflict in the Middle East. But we've been on a rampage to really humble the confidence of our, of our investment partners. I probably would have thought three or four years ago that the confidence that our partners have in their specific outcomes were merited. I no longer think that as we're looking at our managers across the board, because we do sort of implicitly benchmark them, using sort of the same kind of forecasting process that, that we have internally. And a lot of them are, are overconfident. And I, and I think, you know, that's been a fundamental shift in my mentality is that, you know, we, it's not, it's not a 50 50 whether they can do what they say they do, but like we have to be a lot.

• 26:46Speaker 01

  I mean, the default has to be like, there's a, you know, maybe a 20 or 30% chance that they can do what they say they're going to do. There's a lot of marketing, but I think treating partners with a much higher level of scrutiny in this environment is really important. So I've gone from, from thinking that, you know, look, they probably have a pretty good handle on things to just thinking, certainly in some conversations that, you know, there's marketing, I understand it from a business building standpoint, right? That, you know, certainty is what sells, you know, or if you're on the other side of it, pessimism, if you're, if you're shorting things, pessimism just always sounds smart. But I think what everybody has to do right now is just reduce their certainty in whatever they're doing, whether they're long or they're short. And I think that's critical.

• 27:24David Weisburd?

  There's such a performative aspect to this confidence. We just invested in a company called Legora and LP was asking me about horizontal versus vertical AI. I had this itch to say, really, nobody knows. How could you know horizontal versus vertical AI? And if Marc Andreessen is on the record saying he doesn't know, Peter Thiel doesn't know, why would I know? But you have this pressure to justify this position and everybody expects it. You have to basically justify this unknowable variable without saying the more intellectually sound thing, which is, well, if vertical AI wins, or in the 50% of cases that wins, this has an expected value of 5X. But in half the cases, this might be a 1X or it could even lose 1.

• 28:06Speaker 01

  I would be in that camp. I think, I would think, look, I think vertical AI is probably like the way to go. It's probably still where I think things are going because, look, with vertical AI, it's a huge moat, right? It takes, it takes a lot of resources to just understand the sector because every sector is extremely complicated, right? So I, there can be a strong case made for that. But I think with AI generally right now, right? Everyone has to be careful. We, we don't know the full impact of AI and I would be real hesitant to take a view on whether horizontal or vertical is going to be the winner. And I, and I love the way that you position it, right? I mean, if these guys who are the experts in the field don't know, then I think it's fair for us to say we don't know.

• 28:51Speaker 01

  And there's a similar discussion going on with like home builders and interest rates. You know, if you listen to like the earnings calls of home builders, you know, they don't really have a clue where the housing market and interest rates are going, right? They can kind of speak for like, you know, the next year or so. But if they don't know, and you would think that they're probably as close as anyone to it, then you have to say, we don't know. And that's really important in terms of how we make decisions when we're trying to create those, those base rates. You know, I'd be real hesitant if anyone on my team said, hey, look, I'm 60 or 70% confident I know like, you know, the interest rates will go up by at least 100 basis points in the next year. Or I'm 60 or 70% confident that horizontal AI is going to win because you'd have to convince me that like the base rate right now, according to the experts is 50/50. It's a coin flip and that's me.

• 29:33Speaker 01

  Max uncertainty and you'd have to point to some pretty substantial evidence that would move you off of that. We all have to be very humble right now about what we know and what we don't know. And like I said at the beginning, the paradox of improving is just admitting what you don't know. And the more you do that, then you know, I think the better, the better you are about making decisions. Now that doesn't mean you're paralyzed, right? But you have to approach investments and say, look, I'm, I'm 50% confident. So if I'm looking at this investment on an expected value basis, meaning I'm 50% right. What do I think I gain if I'm right? You know, times like the negative alpha, if I'm wrong and at, you know, 50%. So you could still be making investments into that, but I think you have to say this is a coin flip. It's close to a coin flip. And there's another sort of behavioral component to this too, which is we're often really good cognitively. It's sort of the yes scenario. Will horizontal AI win or will vertical AI win?

• 30:19Speaker 01

  Because you can kind of point to various things and you can build a case in your mind because our, our human brains are reasonably pretty good at that. But what we're not good is generally the no case, right? So will horizontal AI fail to win or will vertical AI fail to win? Because that's really cognitively expensive. You have to like, look at all the reasons it possibly could and just blow them all up. And so it is a delicate balance right now, but I would certainly be closer to that 50 50 threshold.

• 30:43David Weisburd?

  Just a pressure test that, do you have GPs that come and pitch you in probabilistic manner? And if so, how do you react to that?

• 30:50Speaker 01

  No.

• 30:53Speaker 01

  I would say not usually, you know, the conversation with GPs, you know, usually goes something like this. We aim to perform over a market cycle, right? So then our team's job is to, is to pin them down and say, well, what's like a market cycle for you? Well, like five to 10 years. Well, which is it? Five or 10, right? And generally partners are, you want to keep that flexibility because right? You know, we all understand the, the future's uncertain and you know, there's compliance and things like, and things like that. But it's also a huge problem in the LP community. If you can't get your potential partner to define like what success looks like, then any result is going to do, right? This is the classic, just drawing the bullseye around wherever the arrow landed. So I think what you have to do is say, hey, listen, I need a minimum, a minimum success definition here, like what?

• 31:37Speaker 01

  And they'll say, well, we're looking for mid teens over market cycle. Okay. Well, like, what is that, like 15, 13, 18? So you pin them down on that. Well, okay, at least 13. All right. What's that market cycle look like? Well, it's not really a market cycle. We're trying to do the, beat the benchmarking. Oh, okay. So it's a relative performance. So our team, their job is to just sit there and pin down, what does success look like? Cause I maintain as an LP, you're never going to be the expert in any one area, right? You're talking to venture capitalists, you're talking to CTAs, right? You're talking global macro funds. And I maintain if they can't define success, that's problem number one, right? Two, if they're defining success and you finally get them to say, cause we do have GPs that'll do this. And that's sort of like the threshold for getting into our portfolio.

• 32:18Speaker 01

  They'll say, look, we're looking for a 12 to 15% return on a rolling 18 month basis, about 60% of the time. Okay. And then we're just watching them, you know, every 18 months, we roll it forward. Hey, are you hitting that, hitting that return 60, 60% of the time? And I maintain like you're since, because you're not an expert, you don't have to know everything that they're doing. But if they're miscalibrated and they say 60% of the time, we're going to hit that return threshold, but they're only hitting it 40% of the time. You don't have to know what they're doing. You just know that they don't know what they're doing because they wouldn't be that far off. And in fact, you see a lot of GPs who are overconfident. And that's a problem because if you go back to the expected value construction, if you think your success is 60% and it's actually 40, well, then your, your equation has a negative expected value

• 33:02Speaker 01

  and you end up with performance disappointment.

• 33:07David Weisburd?

  You're essentially forcing super forecasting on your GPs.

• 33:10Speaker 01

  Yep. Yep. Well, we will impute it and we will ask them directly. You know, so if it's a liquid fund, we'll certainly ask them that. If it's a private equity fund, obviously we're getting sort of like a whole, kind of walk through the example I gave earlier. Right. But then you can also go portfolio company by portfolio company. And you can say, hey, this portfolio company, where do you think revenue is going to end up 12 months from now? And they'll say, kind of here. And you say, well, how confident are you? Ah, we're 60%, right? Okay. Portfolio company B, what do you think? Right. And then fast forward a year, you know, you can, you can start to compile some data on them. And you can say, look, you guys are actually pretty well calibrated or you seem to be routinely overconfident. And I just think LPs really want to work with groups who are appropriately calibrated because then they're sizing bets appropriately. And that's a little bit how we do it.

• 33:54David Weisburd?

  If you could go back to 2007, you just left JP Morgan and joined PS, PRS. What is one piece of timeless advice you'd give a younger Mark that would have either accelerated your career or helped you avoid constant mistakes?

• 34:06Speaker 01

  The number one thing is to communicate to any young investor that most of what you look at will probably not do what you think. So when we first started doing, when I stepped into the CIO and we were forcing everyone on the team to assign confidence levels to investments, the confidence levels were really high. 75% confident that this fund is going to achieve the success definition, 60, 80%. And it was a bit of an awakening. Let's say you're looking for top quartile private equity funds. Okay, well, then that means by definition, most of the people across the table from you are not going to be there because only 25% of them can be and 75% of them will not be.

• 34:49Speaker 01

  So what I hear from other investment teams or individuals, they sort of kind of go into these meetings thinking like, okay, this sounds like a good idea. I'm going to listen to them and then I'm going to try to poke holes in what they're saying. But that's coming from a position like you're giving them, you're basically saying they're going to do what they're going to do and I have to find information that disproves it. Or they'll come in and say, maybe, which is a little bit better, like, I'm going to be on the fence, I'm kind of 50/50, let's see if they can convince me. But it's completely different to sit there and say, there's a 25% chance they're a top quartile fund. Most likely, they will not be, right? And they're really going to have to convince me and I've got to point to very, very specific criteria, right? That would move me from 25 to, you know, say, 50 or 60% confidence.

• 35:31Speaker 01

  And I just think that's a healthy frame, but I think it takes some time. But doing that, and again, starting with that outside view, that most funds actually do not perform, I think is a much healthier way to approach investments. And when I was, back in 2007, when I was starting, I didn't realize and understand that. I do now, and that's one thing I would definitely, you know, beat into my 2007 self from the get-go.

• 35:56David Weisburd?

  Set slightly a different way. The market, when people look at historic returns, they look at an asset class and maybe returns 8% or 11%. They intuitively think it's just like this linear growth. But especially in the most spikey assets like venture capital, some years you're getting 40%, some years you're losing 18%. It's really quite chaotic back and forth. So what do you do with that? You do two things. One is you have to discount any kind of projections. So if you're trying to get top quartile, you probably need to pick a top decile fund. If you're going to pick a top quartile, the odds of it staying top quartile, according to University of Chicago's Steve Kaplan, is 52%. But that also says 48% of the time you're not going to get that. So you have to be more ambitious in order to hit your target. And then secondly, is when you're right, you better make it count. You better make sure that the symmetry is there.

• 36:41David Weisburd?

  Because if you have just an asset class that is going all over the place and the median and the mean is basically the same amount, then it's not a good asset class.

• 36:48Speaker 01

  So part of this is just sort of when we start with this outside view and creating base rates, what you're really trying to do is kind of control for skill and luck, right? And so that's why it's really important to pin a potential partner down and ask them to define the success metrics. I'll use this analogy all the time. Like, you know, take the University of Michigan Stadium, right? Biggest stadium in college football. If you had everybody stand up in that stadium and if you flip heads, you stay standing. If you flip tails, you sit down. You do that 15, 15 times, 20 times. I don't know the math off the top of my head, but it'd be 0.5 times 0.5 times 0.5. You'd probably end up with a handful of people standing at the end of maybe 15 coin flips. But it's not like they did that intentionally, right? But you can imagine like what happens. Like imagine they were investment, you know, they were like venture capitalists or whatever.

• 37:33Speaker 01

  You know, they had some home runs. But what you don't know is whether that happened on accident or whether that was skill. Going back to my analogy, imagine that, say there's five people. Imagine that those five people or one of them said, "I'm going to flip 15 heads in a row ahead of time, right? Before it started." And they actually did it, right? That would matter a lot more. So. So the, I think defining that success and the outcome beforehand really helps you interpret the results because otherwise there's a lot of revisionist history, right? And they'll take credit for anything that turned out really well, right? And they'll sort of, they'll sort of just like, you know, uncritically accept that and they'll assign bad luck to anything that didn't turn out well. So I think from the, from the jump, having that definition of success is just really important.

• 38:17David Weisburd?

  Well, Mark, it's been two years since I've been trying to get you on the podcast. Ever since you won innovator of the year for institutional investor. Congratulations on, belated on that. And thanks so much for taking the time and for sitting down.

• 38:29Speaker 01

  Hey, no problem. This has been fun. Thanks for having me.

• 38:30David Weisburd?

  If you found this conversation valuable, please click follow how I invest so that you don't miss the next episode with the world's top investors.