Scaling Forecasting: AI Forecasting Tournaments & the Road to Epistemic Security, with Deger Turan, CEO of Metaculus

Nathan explores the cutting-edge world of AI-powered forecasting with Deger Turan, CEO of Metaculus. In this episode of The Cognitive Revolution, we discuss how AI is revolutionizing prediction markets, the potential for AI to outperform human forecasters, and Metaculus's ambitious new AI forecasting benchmark tournament. Join us for an insightful conversation about the future of decision-making and collective intelligence. Participate in Metaculus' first of its kind bot forecasting tournament: https://www.metaculus.com/aib/ Apply to join over 400 founders and execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj RECOMMENDED PODCAST: 🎙️ This Won't Last - Eavesdrop on Keith Rabois, Kevin Ryan, Logan Bartlett, and Zach Weinberg's monthly backchannel. They unpack their hottest takes on the future of tech, business, venture, investing, and politics. Apple Podcasts: https://podcasts.apple.com/us/podcast/id1765665937 Spotify: https://open.spotify.com/show/2HwSNeVLL1MXy0RjFPyOSz YouTube: https://www.youtube.com/@ThisWontLastpodcast SPONSORS: Building an enterprise-ready SaaS app? WorkOS has got you covered with easy-to-integrate APIs for SAML, SCIM, and more. Join top startups like Vercel, Perplexity, Jasper & Webflow in powering your app with WorkOS. Enjoy a free tier for up to 1M users! Start now at https://bit.ly/WorkOS-Turpentine-Network 80,000 Hours offers free one-on-one career advising for Cognitive Revolution listeners aiming to tackle global challenges, especially in AI. They connect high-potential individuals with experts, opportunities, and personalized career plans to maximize positive impact. Apply for a free call at https://80000hours.org/cognitiverevolution to accelerate your career and contribute to solving pressing AI-related issues. The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. CHAPTERS: (00:00:00) About the Show (00:00:22) Sponsor: WorkOS (00:01:22) About the Episode (00:04:47) Introduction and Background (00:08:42) Deger's Journey to Forecasting (00:13:06) Current State of Forecasting (00:20:23) Sponsors: 80,000 Hours | Brave (00:22:55) Overview of Metaculous (00:23:30) AI Forecasting Research Papers (00:35:44) Sponsors: Omneky | Squad (00:37:31) AI Forecasting Benchmark Tournament (00:44:01) Improving Forecasting Accuracy (00:51:56) Scoring System and Tournament Rules (01:04:17) AI Forecasting Benchmark Series Details (01:11:03) Tournament Structure and Participation (01:17:07) Consistency in AI Forecasting (01:34:36) Risks and Challenges in Forecasting (01:42:43) Consensus Illusion and Policy Making (01:53:55) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, my guest is Der Turan, CEO of Metaculous, a leading forecasting platform that aims to bring epistemic security to how we understand the future. Forecasting has long fascinated me. I participated in one of Philip Tetlock's original super forecasting tournaments some

• 0:43Nathan Labenz?

  15 years ago, and I actually did reasonably well, though I was not ultimately named a super forecaster. Since then, I've often visited forecasting sites like Metaculous to take a pulse on future events, and I've always intuitively believed that forecasting could really help most organizations to make better decisions. And yet, despite its promise, forecasting has not become a mainstream technology. That's in part because it is a ton of painstaking work to do forecasting well, and thus really hard to scale to the point where forecasts become consistently relevant to decision-makers as they operate in highly contextual situations. As a case in point, Metaculous has hosted some of the most referenced AGI timeline forecasts for years, and still today, fewer than 1,500

• 1:28Nathan Labenz?

  people have registered their own individual forecasts. If you've listened to this show for any length of time, you can probably guess where this is going next. What if we had AIs do the forecasting? It turns out that with some real effort required and some important remaining caveats, this already works. Multiple recent papers, including some by Tetlock and others, show that AI systems forecasts are competitive with the best individual humans, though they do still fall somewhat short of the wisdom of human crowds.

• 2:02Nathan Labenz?

  Darren and I first review this literature together and then go on to discuss Metaculous's reasons for gathering more complex probability distribution style predictions rather than simply seeking to discover market prices. We also review the community's track record on major world events, and we get into Metaculous's ambitious new AI forecasting benchmark tournament. This is a year-long competition that aims to advance the state-of-the-art in AI forecasting capabilities with free API credits available and meaningful prizes to be awarded quarterly. Beyond that, we explore Darren's vision for the future of forecasting, not just as a way to predict outcomes, but as a tool for building better world models and improving collective decision-making. His ideas on using forecasting for resource allocation,

• 2:46Nathan Labenz?

  policy development, and even enhancing democracy are both exciting and thought-provoking. And another example, not only of how AI's already superhuman strengths, in this case, scalability, have the potential to change the world even if the AIs perform no better than humans on a given task, but also how AIs are meaningfully generalizing beyond their training data. After all, the future is in some sense definitionally out of distribution. As someone who's deeply interested in both AI development and societal decision-making, I found Darren's perspectives fascinating, and I think you will too. As always, if you're enjoying the show, I'd ask that you take a moment to share it with friends or leave

• 3:27Nathan Labenz?

  us a review on Apple Podcasts or Spotify. And if you have any feedback or guest suggestions, you can contact us via our website, cognitiverevolution.ai, or you can DM me anywhere you like. Now, I hope you enjoy this exploration of the frontiers of AI-powered forecasting with Darren Turan, CEO of Metaculous. Darren Turan, CEO of Metaculous, welcome to the Cognitive Revolution.

• 3:53Nathan Labenz?

  Thank you. Big fun here. Thank you. That's kind of you to say. I'm excited for this conversation. I've been a Metaculous longtime watcher and you've got some very interesting new projects, which will be kind of the bulk of our conversation today. But maybe for starters, you're relatively new to the job, just a handful of months in the CEO role there. Want to give us a little bit of your background in AI, because you've been working in the space for years before that and made a move that you might want to unpack a little bit in terms of like, why the shift to the forecasting realm now?

• 4:26Speaker 02

  I guess hindsight is 2020, especially in contexts like forecasting. I have always been interested in questions of collective intelligence. How can we aggregate multiple different perspectives to the point that we can build a coherent world model? And I was working on this question while I was still at school from a perspective of can we use natural language processing? This is way before LLMs, like back in 2016. How can we aggregate multiple different streams of thoughts? So that this would be actionable for a policymaker. So I've been interested in working with federal agencies and build a start up on the space. And right before Metaculous, I was working on AI Objectives Institute, which is a nonprofit research lab focusing on socio-technical alignment,

• 5:07Speaker 02

  which is looking at questions of AI alignment in context of real people in deployments right now. And seeing are there any technical steps we can take that can actually say, this group of people do perceive this AI model as better aligned or more instrumental towards their goals? Looking at it from an individual perspective and from a collective perspective and from a systems perspective on understanding how can these tools be much better used for enhancing human agency? And I've been doing a lot of work on questions of collective intelligence. How can we augment a multitude of different views to be able to contribute to a system that might be a single model of, or it might be multiple different agents collaborating? This might be

• 5:50Speaker 02

  a government, this might be an LLM, this might be an ensemble model. This is in a way applicable to many different systems, right? And very quickly, I find myself thinking about, okay, say we can aggregate multiple people's desirabilities, even if we are able to do this on a high fidelity level. How do we get to the world outcomes that we're looking for? How do we actually go towards the model that is not just driving towards the lowest common denominator and finding, you know, a consensus mode where the desire for consensus actually causes a trade-off with fidelity to one's perspective? But instead, see what would actually be helpful? What future do we want to live in? Are causes prioritized correctly so that we can move towards better futures?

• 6:34Speaker 02

  Can we find positive some gains? Who will shoulder the externalities? If there is no free lunch, how can we make sure we're tracking that? And these questions brought me towards thinking, okay, who's looking at the world from a perspective of if we take action course X, will that yield a better outcome? And that's how I found myself thinking about forecasting. Because if I'm able to even have a very high fidelity version of desirability, elicitation, preference, elicitation, understand all the cruxes that exist in society, that still does not bring me towards a machine system being able to take actions that yield net good. So exploring that along with the questions of wisdom of the crowd, you know, if you have multiple different perspectives, be these be real humans or AI that is actually

• 7:18Speaker 02

  once aggregated is doing a better job in forecasting outcomes. I realized this is actually quite instrumental in how we look at the world. So that is what brought me towards thinking about more forecasting.

• 7:28Nathan Labenz?

  Cool. Just to rewind back in time for a second to the 2016 era. Did any of that stuff work? Were you able to to make anything with technology that was available at the time that you thought yielded any practical utility?

• 7:43Speaker 02

  Yes, well, back then I was working with Dan Jaravsky on a question at Stanford on federal agency regulatory feedback. So the FCC net neutrality debate was just taking off and there was 2 million comments that were submitted to the federal agency. And at that point, this seemed like, wow, this is the largest data set that was ever collected of raw text. Now it's ridiculously small, but at that point, there was this question of, okay, how should a central agency pay attention to this? There was a lot of interesting questions, like a lot of the submissions were form letters that are copy paste. But the fact that it is copy paste does not make it illegitimate. You know, I will read a campaign and say, yeah, I agree with this. I want to send this. How should you take that into account compared to something that is bot generated?

• 8:28Speaker 02

  Compared to something that is a genuine opinion that someone has taken time, but it's not necessarily sophisticated and you can see that it has shortcomings. And in my thesis work, I was looking at this both from a legal perspective and also a technical perspective. You know, what are the expectations from EPA and EPA perspective on just the US legislation? But also, how can we have natural language processing augment this? And funny enough, at that point, I was thinking a lot of questions of good hearding. Like every time we start focusing on a specific narrower metric, it ceases to be a good measure. And this was very apparent in being able to claim that you're looking at the public. Things we tried were so simple and yet they seemed like, oh, this is a paradigm shift.

• 9:10Speaker 02

  Just like simple clustering strategies, unsupervised learning, topic modeling. And even that went a long way because people had no idea. And now we can do much more sophisticated versions. And people still are like, people assume that we don't need to ask these questions because we cannot do large scale qualitative service. So we keep going back to Likert scale. So I guess the crusade I've been on for a long while is can we go past how happy are you, one to seven? Or should we build a road from A to B or A to C? I'm like, maybe we need to build a bridge instead. And if you don't let people express that, you're just not going to get that opinion. So how do we get machine learner systems, optimizers to be able to understand, ah, there's something qualitative that I need to elicit?

• 9:53Speaker 02

  That even the speaker might not be sure what they want. Like, how can we give room for that human flexibility in these machines?

• 9:59Nathan Labenz?

  So your comment on good hearding definitely resonates with me, particularly today. I was just having some back and forth on Twitter about the new GPT-4O mini model, which is coming in ahead of Claude 3.5 Sonnet on the LMS leaderboard. And that raises the question, it seems like the general consensus among people like me who are obsessed with. The latest AIs and kind of have their own individual opinions seems to be that 3.5 Sonnet is the best. Certainly, we would all tend to think it's better than GPT-4O Mini. And yet, there it is ahead in terms of the votes that it's getting. So this has raised

• 10:42Nathan Labenz?

  all sorts of, you know, why is this happening? Do we trust this thing anymore? Is it just about formatting? If it is mostly about formatting, does that mean it's like illegitimate in some way? Or does that mean we should really think more about the importance of formatting? But yeah, as always, it's like extremely difficult to pin much down if you really want to do it with any real rigor. So that is maybe a good bridge toward and certainly the same is even more true about the the future. So I've been interested in this space of forecasting for a long time. I've actually participated in one of the original Tetlock organized super forecasting tournaments, maybe 15 years ago now. Tyler Cowen posted

• 11:25Nathan Labenz?

  an invitation to participate and I was like, I'll try that my hand at that. I did reasonably well. You know, I didn't top the leaderboard, but I came out feeling like I had at least somewhat of a knack for it. And I've always found that the sort of Robin Hanson perspective on like, this seems like a much better way than sort of highest paid person's opinion or like whatever kind of prevails in organizations. That's always been compelling to me, but it seems generally speaking, we still don't see a ton of forecasting deployments in the world. What would your analysis be of kind of where we are in forecasting, you know, where you think like the most notable deployments are and why we don't see more than we do today?

• 12:07Speaker 02

  Right. I want to answer from the previous point you made around good hearting, because I think this actually goes towards there on, you know, which model is better? Define better, right? Better is incredibly content specific. Like even in just a small world of things I am trying to do personally within Metaculous, 4.5 is more useful for me on question writing, but not necessarily fact retrieval. And it's a very narrow thing. And this current version of these models seem like, you know, one of them has an easier knack towards one thing without interrupting, without self censoring. So I can move forward faster. I think the core of forecasting also relates to this in terms of how can this process actually be useful? Is one thing that is different from will it score good on these benchmarks?

• 12:51Speaker 02

  Like in this one specific competition, will 4.0 score better like? That is a much narrower thing. In a way, usefulness is almost always anecdotal. And I think we are not paying enough attention to that. And I think the same is applicable for forecasts as well. There has been so many tournaments like the ATX 2023 tournament is interesting for how accurate the forecasts have gotten. But to me, okay, how is this going to actually be useful for an end user? Is a different question. And I think this is the approach that I really want to focus on with Metaculous for the upcoming sprint is. Okay, I have a couple of validations that I feel satisfied with. What are those? One, wisdom of the crowd works. It is interesting that it works, but when you aggregate a number of people, and we will go into it.

• 13:36Speaker 02

  When you aggregate a number of bots that are trained with LLMs or what have you. It seems to be able to draw an accurate picture of the future. And we have enough case studies of this. Why is this not more commonly used? I think usefulness is a whole different paradigm than accuracy. And this is really what I want to focus on. What do we want to do on this front? Like, for example, can we identify critical inflection points towards future states? Is a different question than just is your forecast accurate? What is the shape of the world that we want to live in? What are entities? You know, these can be federal agencies like my prior work. This can be philanthropists, this can be corporations, nation states. The framework of forecasting so far, especially from a technology lens, I think,

• 14:18Speaker 02

  has not focused on how will this be actually useful, but has more so focused on can we prove the accuracy? I would like to move forward towards coming up with versions of forecasting deployments that actively pay attention to. How will the decision maker take this into account? What are things that are within their space of levers? This is why I think forecasting in context of the humans that are trying to live a better life is extremely important. I can share more examples on this front, but I'm curious where you'd wanna go with this. Like, pretty much our research agenda for Metaculous for the next chapter, for the next two quarters, heavily focuses on a lot of experimentation on these kinds of questions.

• 14:58Nathan Labenz?

  Yeah, interesting. Does that look like basically conditional markets being kind of the first thing? I mean, that's something that Robin Henson has talked about for a long time, and I don't see much, but we've just lived through a moment in history where it seems like something like that was pretty motivating to the current President of the United States, who I think was, you know, if reporting and general intuition are correct, seems like he was not sold on the idea that somebody else had a better chance than him until he saw like polling data suggesting that, and then was kind of like, well, I guess, yeah, if somebody else has that much of a better chance, maybe I should step aside. Is that the sort of

• 15:43Nathan Labenz?

  Play that out a million different ways for a million different contexts, but is that the kind of next big paradigm?

• 15:49Speaker 02

  I think that is an example that basically is a very simplified and somewhat was obvious to the entire population, except for the decision makers themselves for longer than it should have, in my opinion, state of affairs. With that said, the real cases that I'm interested in is much more sophisticated than this, right? That heavily looks at public opinion and is more, if I may say, like vibes-based rather than data-driven. And I'm interested in really pushing this edge further. Say we have fifty million dollars to allocate towards making the world better. Pick your challenge, reducing microplastics, climate-related risks, AI-related risks. How should we allocate these resources to get to a better world? Requires us to build world models.

• 16:31Speaker 02

  I think forecasts around what are specific outcomes we can get to if we take an intervention point is very helpful. And the simplest thing we can do is exactly what you said on, you know, come up with conditional forecasts. If we take lever X, will that bring us to a better future? If not, what will it be like? And if we see high divergence here, that is great. Now, I want to push this much further. We can already do this. But truth is, conditional questions are somewhat hard to answer. It's hard to think in that framework. So can we build tools, be it LLM-driven, or discourse-driven, or come up with ways in which people can collaborate that enable conditional forecasts to be more useful? This is one avenue.

• 17:11Speaker 02

  One of the things I want to do in the near future is launch a series of mini Metaculous. This is the name we came up with. Think of it like subreddits, which is Metaculous instances. This will come in context of us open sourcing Metaculous, and we hope that many of these will spring in the next quarter. Each mini Metaculous will be focused on a specific question, a goal-oriented question, such as we are trying to reduce microplastics, for example, or we're trying to reduce homelessness in San Francisco. This is geared towards disaster response or consequences of research avenues in context of AI. I would want every question in the mini Metaculous to be able to serve, if we are able to answer this question, it bubbles up to the parent. And then we can see which of these questions have the highest value of information.

• 17:56Speaker 02

  Which of these questions actually inform us to say, oh, it looks like this intervention is going to be able to drive much further value. I would want us to identify critical inflection points, like is there a specific moment in which the world model seems to diverge? Are we able to extract tools of thought in the context of forecasting by seeing, oh, this group of forecasters seem to be much more in accordance over multiple forecasts? Why is their world model divergent? Can we double down on finding more questions that can help us excavate that? These kinds of research questions go way beyond what Metaculous has aimed for so far. This is actively trying to build a world model in an enclosed world space that is the Metaculous with a specific goal, and I find this to be really, really interesting.

• 18:40Speaker 02

  Like the kinds of things I would love to do is, for example, can we find short-term proxies and heavily forecast on both? Is this short-term proxy good, and how is the short-term proxy panning out? The way we landed with the FAB, the forecasting AI benchmark, is basically guided from this also. In a way, we can frame the presidential debate through this lens. I think we, as a civilization, need to think much more from this lens. Like, this is a version of a cognitive revolution where we change how people are thinking about the future. In a way, we're enabling them to coordinate between their world models. I still see forecast aggregation to be fairly low fidelity to be able to coordinate world models, but I think we can use that as a building block, as a primitive and come up with much better world models.

• 19:24Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:29Nathan Labenz?

  Let's maybe take one step back, just for those who maybe don't have so much experience with Metaculous as it exists today. I'd be interested to hear just like, for starters, how do you describe it to somebody who's never seen it before? Would you call it a prediction market? Would you call it a forecasting platform? How does it fit it? Because there's like a few of these now, right? With people would be familiar with certainly just online betting platforms, but also there's like manifold and polymarket that I see kind of shared around. How would you describe the overall product today? And how would you distinguish it from some of the other things that are out there?

• 20:05Speaker 02

  Metaculous aggregates forecasts. It's a forecasting platform and an aggregation engine. I would not call Metaculous a prediction market. The goal of Metaculous is to bring a level of epistemic security towards how we envision the future to be. You can call it Wikipedia about the future. And it is a space in which multiple people can see how they foresee the future to play out and the critical discussions to take place there. It works with predictions as its core block, but it does not have a monetary incentive. It behaves quite differently than prediction markets as a result of that. And some of those differences are the reasons why, in my opinion, Metaculous has been both more accurate and

• 20:46Speaker 02

  also more rigorous compared to a lot of other spaces. It is, I can shed more light into that. For example, on a prediction market, the participants are buying and selling contracts that are based on the future outcome of an event, right? So you place bets and the result is a zero sum question. For example, if the current forecast is at 60% and you think it is 60%, you don't have an incentive to add that information in because it only can get worse from there. While on Metaculous, this is not the case. There's no financial incentive and instead we are comparing all the forecasters' accuracy and track record through time. And this creates a different set of incentives that I believe are actually much more productive

• 21:30Speaker 02

  towards something that looks like Wikipedia about the future, where people are, one, incentivized to share what their world model is. Two, people are incentivized to participate in forecasts and not think about de-risking or spreading, but instead focus on what is the world model that I have that I can share. So our scoring mechanisms are different with respect to prediction markets as a result of that too, because the reward mechanism is that you reward the forecast over time for their calibration across many predictions and for their accuracy. We also have a metric that we call the community prediction. This is interesting because this is where you can see everyone that has predicted on this

• 22:11Speaker 02

  question. And we also have weighted averages depending on your past success, on your track record and the recency for, you know, sooner forecast than to be better for you have more information. So using these, we actually end up in an ecosystem that is much more collaborative and much more grounded in good estimates. And this brings higher rigor as well. So the questions we forecast tend to be more like carbon emissions on this date rather than very personal questions like, who am I going to be dating with next? And I like that too, you know, there's a market for that. But I like that there is a space for focusing on the rigor that is actually beneficial for the world. So that is what I find distinct and attractive about metacritic.

• 22:54Nathan Labenz?

  Gotcha. So it seems like the big thing there is the incentives, whereas on a more market oriented platform, I am looking for things that are wrong so I can make money. Here I am looking for just opportunities where I can share something with the community that hopefully brings the overall community forecast toward a more accurate.

• 23:18Speaker 02

  Right, but on the thought, I will add, there is a financial incentive in Metaculous too. As in, if you have very good track record as a forecaster, we will hire you as a pro forecaster to engage on many specific paid client works. And these range from federal agencies to large retailers to hedge funds, think tanks, that have much narrower questions. These questions don't make their way to the, you know, public forecast that we have, and they will be paid engagements. And we will also pay the pro forecasters for their reasoning, so that they can actually explain why they see the forecast to be this way.

• 23:52Nathan Labenz?

  Hmm.

• 23:52Speaker 02

  Just to echo back to a thing we were talking about, I think one of the core projects that we have for Metaculous is going to be reasoning transparency. The forecast on, say, I'm making up an example. Say we're thinking about the Taiwanese sovereignty. This was the context in which this came up. I was in Taiwan recently for a conference and we were talking about, will Taiwan's electricity grid be challenged by China in the process of a potential invasion? And the forecast jumped from thirty percent to forty percent. And there were no comments under it because it was fairly new. And I was sharing this with one of the folks there that are working on from the government's perspective. And they were saying, unless I know why this is the case, I cannot, I cannot take this into account.

• 24:35Speaker 02

  And for that, we do pay our pro forecasters to come up with better forecasts with their explanations as well. We place people in jobs in the past. So there is, in a way, the leaderboard can translate into financial incentives.

• 24:48Nathan Labenz?

  Yeah.

• 24:48Speaker 02

  It doesn't happen on the substrate of the question itself. And that I think is quite important.

• 24:53Nathan Labenz?

  Okay, cool. Can you say a little bit more about how you create that community forecast out of, you, you kind of touched on this briefly, but this is one of the big questions that I've had, honestly, for a long time, because there are two, kind of, I think of them as like canonical AI. Questions around AGI timelines that I've come back to for years now. I think I tweeted about them like almost two years ago. And these are the weak AGI and the strong AGI. And we can unpack this a little bit more in a minute. But one thing that I've always kind of wondered is like, exactly what am I looking at when I'm looking at this? You know, to what degree am I seeing? Is it a naive aggregation?

• 25:39Nathan Labenz?

  Cause you can go in there and put a whole distribution, right? So I'm interested in your comments on that, too. Like, I don't have a great sense of what I'm doing when I'm like actually drawing a curve over the potential timeline of something happening. I feel like I'm a little bit like, you know, I mean, I guess I'm always sort of vibing with these predictions, but that brings it home to me where I'm like, boy, the the precision here feels like I'm leaving something behind that is, you know, and it's weird, right? It's a way to express my uncertainty, but it's also a way to like, be very specific about my uncertainty and even that I don't feel like I really have. So interested in sort of how you see that kind of curve drawing?

• 26:22Nathan Labenz?

  And then how are you aggregating all these curves into one curve? Like, do people get upweighted if they're better? Is there a recency bias, et cetera?

• 26:30Speaker 02

  So a couple of different parts there. With respect to the community prediction calculation, we basically keep the most recent prediction from each forecaster that we have. And we weigh each forecast based on how recent it is for before being aggregated. And we also for the attackless prediction do pay attention, especially if there is a paid client that is looking for a specific outcome or higher rigor. We do do things like pay attention to the past track record of each of the forecasters and aggregate that information in as well. For binary question, the community prediction is basically a weighted mean all of the individual forecasters probabilities. And if you have a numeric question or a date question,

• 27:12Speaker 02

  it is a weighted mixture of all of the individual forecaster distributions. There's actually a lot of different philosophies of aggregation. And this is one of the spaces that I am interested in experimenting further. These are the ones that we use. And in general, the community prediction works and seems to be really accurate. There may be a couple of folks that seemingly consistently beat the community prediction. And I think, you know, there's a couple of posts from like Astral Codex, etc., that has focused on exploring this. I think that I'm very interested in this. Very soon, we will open source Metaculous. I'm curious for people to come up with, you know, say, here's a different, you know, scoring mechanism, an aggregation mechanism that actually seems to work better or seems to be more instrumental.

• 27:56Speaker 02

  I really wanna encourage experimentation on this space. What we have so far seems to work. If you just copy the community prediction, you will do pretty well in a tournament. But then you're naturally bringing in independent information, right? So the better question to do is for someone to do their homework on, you know, what are past forecasts that might be similar to this and then contribute? And only after that, you see the community prediction. Because otherwise, you start messing with the wisdom of the crowd. You said, you know, you might be doing vibe-based reasoning. I think a lot of more reasoning-based forecasting ends up being vibe-based for a lot of people. Not everyone is gonna build like a mathematical vigorous model or be a domain expert.

• 28:38Speaker 02

  But somehow, when we put many of these together, it actually does yield a better outcome. I would start from, for example, the comparisons from the 2023 ACX contest is quite interesting. To look at for comparison of, you know, how does Metaculous compare for prediction markets or super forecasters? And I really like Sam and Swifty as an anchor for, I think they're actually doing a great job as a reference point. To me, the parts of the question is like, how to make the forecast useful along with how to make the forecast be accurate? I think there is enough track record on a couple of different forums. Like the Who Predicted 2023 Best Posts from Astral Codex is quite interesting for this.

• 29:21Speaker 02

  But if you even zoom in on specific moments where an entity, it's a federal agency or a person was able to take action from a forecast, I think those are the moments where I see, yes, Metaculous was actually useful or interesting. Like for example, in epidemiology, Metaculous has outperformed both panels of experts on COVID and also informed hospital resource allocation, public health decision making. A lot of the forecasts were more robust and accurate than baseline models in predicting COVID vaccine upticks. And I do think that is quite interesting, for example. And another part is on the monkeypox outbreak. Metaculous was able to quickly provide perspective on that front. For example, in January 2020, Backpoint conventional wisdom was that COVID wouldn't be significant.

• 30:04Speaker 02

  Metaculous instead was predicting that more than 100,000 people would eventually be infected with the disease. And a lot of folks took that as an early warning sign. I remember reading a post about someone making a bunch of investment decisions because Metaculous said so. Like another case is around predictions around Russia invasion of Ukraine. There's a comment on Metaculous that I really like where the Metaculous user in Ukraine said, "Just want to say that I moved from Kiev to Lviv on February 13, entirely thanks to this prediction thread and the Metaculous system." Like that to me is a moment where, yeah, the forecasts were instrumentally useful for someone making a decision.

• 30:44Speaker 02

  Like we've been cited in The Economist, Our World in Data, Bloomberg, Fortune, Forbes, you name it. I think the core question is around how will these be able to turn into instrumental actions that one can take? And I'm particularly interested in this being helpful for the AI sphere and making sure the tools that we're building on AI as a whole are actually serving humanity. And I think if there is something that has good track record and ability to be able to influence individual decisions, we should explore why and how this can be much more useful. And that is the lens with which I think prioritizing AGI is important, but also, you know, other cause areas. Some folks say Metaculous should purely focus on long-termist causes.

• 31:28Speaker 02

  I actually do think a lot of short-term intervention modeling and being able to take successful action teaches us on how forecasting can be helpful in longer-term contexts. Like questions like, "How can we reduce homelessness in San Francisco? You know, if I have an additional 5 million, should I just go build two more houses? Is that the best thing I can do?" Or is there a form of lobbying that I do think a large crowd will say this will make the housing market understand the negative externalities of homelessness to the point that we can have a more, you know, well-grounded change? I would like this to be, I mean, it's a calculus where every forecast, these forecasts can be conditional, they can be intertwined with each other, we can explore different ways to visualize them. Like, these will bring usefulness towards someone, and us being good at this on a

• 32:13Speaker 02

  repeatable fashion will help us be able to tackle the largest questions much better.

• 32:18Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:22Nathan Labenz?

  Cool, very interesting. So going to these two AI questions that I refer back to the most, I think probably many listeners will have visited these pages. The weak AGI and the strong AGI timeline, each one has kind of four different resolution criteria, and the question is basically, when will a single AI satisfy these different criteria? And you get to predict your, you know, your distribution of dates. Right. I think this has been really interesting and quite informative for a long time. Although, more recently, it does feel like it also highlights a real challenge of writing these questions, which is that when things are farther out, it seems like the, or it

• 33:07Nathan Labenz?

  can seem, I feel like in this case, it does seem like the detailed criteria seemed quite reasonable. And now, as we're getting closer, I'm feeling like there's sort of a divergence between what matters and what is actually the, you know, the letter of the law in the question. Particularly when it comes to, in the weak AGI question, the use of a specific form of the Turing test. Where I'm kind of like, okay, from my standpoint, in the sort of intuitive, like, what really matters in the Turing test, you know, line of thinking, I would say we've passed it.

• 33:45Nathan Labenz?

  And yet, the formulation requires a expert interrogation, and that experts not be able to tell what is the AI and what is not the AI. And we're not that close to that, but I always emphasize for people that, I think the reason we're not that close to that is a design decision of the people that are making the AIs that, you know, are to be tested, right?

• 34:09Speaker 02

  How confident are you about this claim?

• 34:12Nathan Labenz?

  I would say quite in as much as if I wanted to create an AI that would be impossible for a or you know, much more difficult for a interrogator to identify as AI or not. The first thing I would do would be have it say, I don't know a lot more often than it does, right? So like the easiest way for me to tell whether something's an AI or not is just to ask 10 very long tail, random questions and be like, no human is going to answer these questions. All 10 like, it's just wait, you know, nobody has that breadth of knowledge. Actual people would say, I don't know. So I would actually dramatically narrow the range of responses from the AI and make it seem much more conversational, make it seem much more ignorant, make it much less useful.

• 34:56Nathan Labenz?

  You know, for what you would actually go to chat GPT for. But I think I could make it a lot harder for the the expert interrogator to figure out what is what. So anyway, that's like just one example of of this general problem. I wonder how you guys are thinking about that.

• 35:14Speaker 02

  I'll start from what you just said and then go zoom back to the HR question. The changes you mentioned could probably be enacted by just using a couple of language models that are policing each other. And you could basically get to a system that actually behaves this way right now. So I'm not very sold that that is the main threshold here. There's a quote from Jaron Lanier that I actually used at the beginning of my thesis that was focusing on how can we augment citizen participation in governance through natural language processing? I guess today I would call it LLMs where he says, the Turing test cuts both ways. If you can have a conversation with a simulated person presented by an AI program, can you tell how far you've let your sense of personhood degrade in order to make the illusion work?

• 35:59Speaker 02

  I think this is quite important here. Like being able to communicate on a certain pattern full of expectations is not what I am interested in. I think that's a very, very low bar, in fact. Like the goal, my my interest space goes way beyond. Am I able to simulate something that is convincing enough in this closed box tournament? Like this does not actually bring us a better world. From my opinion, I actually agree with you. I don't really love the way the question is operationalized. But keep in mind, they are both from 2020. We will not always formulate questions so that they are optimally informative years later, right? And I would say that in 2020, these questions, I mean, you've said it yourself, were in fact quite useful to the point that I would argue

• 36:43Speaker 02

  Metaculous probably moved the Overton window with respect to how people were paying attention to the impact of AI. And I think that is one value add that's both hard to track, but also intuitively resonates with me, which is what also draws me here. So I think these questions somewhat serve their purpose. Now, zooming out of the purpose question to like, okay, but can we have a better question? If we were to do this today, these questions, instead of it being a single question, I think this should be shaped something like the Minifaculous instance that we were talking about, where all of the subparts of the questions would have multiple different forecasts.

• 37:22Speaker 02

  And they all together in an ensemble actually give you a picture of, heck, we don't even have a clear definition of AGI to the point that the question needs to operationalize itself one way or another, right? One of the projects that I'm pretty excited about on Metaculous is to come up with indexes. Basically, come up with, just the way you can aggregate a bunch of stock tickers to come up with a composite view. I would like there to be thirty forecasts that are all really zooming in on slightly different aspects of this. And them all together is what you're paying attention to. Now, you can say, well, I want to rigorously link all of these through causal diagrams, and then you will get into a whole different hairball. Sure, that could be helpful if you pull it off, but even a lesser fidelity version of this, that is just,

• 38:07Speaker 02

  here is thirty forecasts that all rhyme with each other with respect to its focus point, will be able to shed more light. And I think at that point with what Metaculous had, like, that was possible, and these questions did serve their purpose. It's a challenge of forecasting the future, right? We don't always know what formulation for a question will be most useful years from now. For example, we're doing the five years after AGI question series that just launched. Like, I am much more interested in that, for example, because when I look at the answers there, it actually gives me a much more accurate view of what people even conceive of as AGI. Like, the way I am using those questions isn't about... Thinking of, okay, will these things happen when AGI hits?

• 38:48Speaker 02

  It more so informs me, okay, these are the things that people consider as critical possibilities within AGI. And doing a throwback to some of my prior work with AI Objectives Institute, I often find there to be, if we are talking about AGI and AI alignment without talking about the societal context in which this has impact, I think we are narrowing this down. Like, for example, there's a common meme that we would use in AI Objectives Institute to talk about what a successful AGI that can enable human coordination is. In the world that is post AGI, if you don't think Jerusalem will be a united and peaceful city, maybe your AGI isn't ambitious enough.

• 39:30Speaker 02

  Like, maybe something that is truly AGI would be able to resolve clashes and conflicts within existing human sphere, where it's not trained to make sure it is perfectly neutral towards that, but instead it augments human agency that actually brings a meaningful change. Because otherwise, sure, like, I can relate to a system so it can say, I don't know enough time, so that you might find that convincing. I find that to be a love bar. I think we should be more ambitious.

• 39:54Nathan Labenz?

  Yeah, so how much editorial do you exercise over these questions? Because it does seem like these couple of questions have sort of become a bit of a shelling point. And it's a delicate decision, probably, from your perspective, right? Do we sort of take a proactive step to retire a shelling point? Because we feel like it's outlived its usefulness? Or do we just let the community kind of gradually move on?

• 40:18Speaker 02

  I wouldn't say that at all. I think, like, these questions are good. I am much more interested in pushing it further. So I guess it's also a call to action slash ask for help for everyone that loves Metaculous or is interested in this. As we launch Mini Metaculous, which are focused instances, like, if you have different formulations that you think are interested, come tell us. Like, we already get a lot of inbound questions, right? That are around, like, hey, can we write the question? We heavily editorialize them for we do want to maintain a level of rigor. And so far, I think Metaculous has been much more closed than what I would like it to be. I would love for there to be many more people that write questions, especially domain experts, especially people who say, you know, hey, I get the hang of what a question is resolvable and meaningful.

• 41:03Speaker 02

  And these are things we can help with, but we are entering a chapter of Metaculous as we are open sourcing to get many more people to write questions, to have instances that they are hosting that have maybe different scoring mechanisms, different levels of rigor, to the point that it actually proves useful to them. We will also host at metaculus.com many instances that are very domain specific. So I just think we need more questions. It's not about going back and changing what this has accomplished, but more so bringing on new lights. Like, for example, I'm quite excited about a bunch of new technologies that could actually bring better results. Like, I love a lot of the more symbolic base, like, are we able to use language models

• 41:44Speaker 02

  in order to come up with specification criteria and use that to create safeguard AI systems? I'm interested in like, singular learning theory. There's a bunch of new techniques that actually, when I imagine versions of AGI or weak levels of AI competence even that is really good at these, the ways in which I imagine those features are substantially different. And I'm quite interested in exploring these, and I think that just requires a breadth of more questions to forecast and also more human energy or AI energy to be able to forecast on these questions. So, anyone that is interested in saying, hey, I have things I want to forecast, I have a formulation that I think is better, come talk to us.

• 42:26Nathan Labenz?

  How do you create density though, right? I mean, the one worry that I would have if I was you and I got a million new questions in is now I need like, a billion predictions, right? To actually create a meaningful community forecast for all those million questions? So how do you think about balancing like, the diversity of questions versus the density of forecasts?

• 42:51Speaker 02

  Right, great question.

• 42:53Nathan Labenz?

  This is also a moment where the AIs can come in, we'll get there momentarily.

• 42:57Speaker 02

  Yeah, you know, nail on the head, like, the real currency here, the real limitation is human attention bandwidth, right? I have 10,000 questions being launched at Metaculous every day, it's not going to help anyone, at least until like, everyone is forecasting part-time as part of their job. Like, that's not what I'm trying to angle towards here. But this is why we need to do active research on questions like, can we build indexes that aggregate a bunch of different forecasts? Can we have AI start forecasting? As part of the AI benchmark tournament, we've had an explosion of AI contributors. They seem to be doing okay. And like, can that create a scaffolding or jumping off point for humans to be able to get to a rigorous forecast much faster? There is, I think, a lot we can do to increase the quality.

• 43:42Speaker 02

  We have 10,000 new forecasts every day. I think that just will drown any kind of quality from the system. And the win condition also isn't, you know, well, AIs are forecasting and humans are just watching. I think the question is more around, are we able to identify what are the best questions? And I want that to happen with more people, not just the Metaculous team. And I want the contribution of the botch to be able to gear towards shedding more light and more information and have these questions be incrementally composable so that we can build world models of all of these are rhyming with respect to how we are thinking about electric vehicle proliferation, for example.

• 44:22Speaker 02

  And using those, then we can have much bigger questions, like forecasting specific windows of electric vehicle proliferation in China is quite specific. But if I have 30 of those questions, I can then make something aggregate that says, you know, will electric vehicles be better for the environment? It's like, what does that even mean? But like, it actually means a composite of all of these different things. This is a different frame of reference for thinking about how we can aggregate them.

• 44:49Nathan Labenz?

  Cool. Well, that's great motivation then to get into kind of what the state of the art is in LLM-powered forecasting and then the tournament that you're running to try to advance that state of the art a little bit further. Right. In proper, in proper for this, I read through three links to papers that you had shared. And I was overall pretty impressed. And you can give me more detail on kind of what you think is most important or stands out or like what the kernels are that you really want to build on. But it seemed like across the board, pretty positive results. And these are from like serious authors, including Tetlock has been involved in some of this research.

• 45:33Nathan Labenz?

  Just to summarize the three papers, the first one was approaching human-level forecasting with language models. And this was from Jacob Steinhardt and his group. They basically created

• 45:46Nathan Labenz?

  I guess what I would describe as sort of the intuitive thing that I would create if I was gonna work hard on trying to make this work, and that is like a retrieval system, the ability to go on the internet, search through the latest news, process that, and ultimately create forecasts. And it seemed like it was pretty good. It was like

• 46:05Speaker 02

  Yep.

• 46:05Nathan Labenz?

  coming close to the community forecast, although not as good as the overall community forecast. A question that I did have reading that paper, which I don't know if you know the answer to, is, okay, it was a little bit short of the community forecast, but like, how does that compare if it weren't individual human? You know, at what percentile would that system have performed? I don't know if you do know that, but it's my intuition was that it would be pretty high as like the percentile of individual humans.

• 46:34Speaker 02

  Correct. I mean, let me give you a thing that I do believe in, that I don't have data, but I think we will soon have data for this is that thought in the hands of a team of human forecasters will just kill it. Like, that is obviously going to be much better, right? Going back to the question of usefulness, like, I like the academic rigor aspect, don't get me wrong. I do think testing these in isolation is good, and a lot of our designs around AI benchmarking for competition pays attention to that. But the part that already is obvious to me is, let's actually look at what these systems are good at. They're good at going through massive amounts of content, identifying which parts may be relevant. They're good at taking a first step at creating a world model.

• 47:15Speaker 02

  Let's actually build systems that pay attention to those first and build on top of that. I particularly like the Schneidhardt lab, the Halloway paper, for a lot of intuitions that I would like to explore further is in that paper. And a lot of folks that have asked me, I want to compete in the tournament, what should I do? I always point them, well, start from here and then try different things, try different ensemble methods, use different language models, don't share prior data, have them debate each other, have a third model, look at it and synthesize them. There is a lot of playful strategies that one can go with, so I really enjoy thinking in this framework. So I've been encouraging, there's a very active Discord by the way for the tournament where people are discussing strategies.

• 47:56Speaker 02

  It was absolutely delightful to see how much the models have evolved in just like the three weeks of the tournament kicking off. Um, different strategies people have come up with and we do encourage people to update their models. For the tournament, there shouldn't be human in the loop because we're trying to benchmark the state of affairs. But if they have a better strategy, they should update that. Another thing we do is reasoning transparency, like have the models post at least some form of text. We don't grade or score this because it is a whole different level of complexity for that. Let's stay with the forecasting accuracy, but that at least gives us a way in which how the model is thinking. And a lot of earlier research on chain of thought reasoning has pointed that explanation is actually what gets these models to be able to stay further grounded.

• 48:42Speaker 02

  So to me, these things are really good. Couple other avenues that I don't think has been explored as much lately is how can we bring in model based reasoning on top of language models? That will actually yield much, much better results. Even in the first few weeks, we have what 15 questions that have resolved so far, I think. In that signal, we've had plenty of questions that have, for example, tests like scope sensitivity or negation. Ask the same forecast, ask the opposite, see if the answers are the opposite of each other. For example, there was the measles question, like if you ask the bots, what will be the number of measles cases? Less than 200, it'll say 75%, 70% for more than 300.

• 49:27Speaker 02

  And that means there should be 5% chance that it will be between 200 and 300. And bots will say 65% for the window that's between 200 and 300. Like obviously a pro forecaster wouldn't make this mistake, but even an ensemble method falls apart on this case because it doesn't keep track of. Here is a world model that is mathematically rigorous. Now, why is this interesting to me? Humans wouldn't also be able to do a very good job at tracking this if they were much more complex. Like, obviously, a proforcaster is trained to get better at it. But if you ask this to someone that is not a highly numerous child, they would be like, "Oh, maybe it's the same. Oh, I didn't realize that this is wrong." That means there is something intuitive about how humans reason that is distinct from the mathematical rigor here.

• 50:12Speaker 02

  How can we close that gap? Like, if we are able to incorporate model-based reasoning, and I think this can happen with approaches like Open Agency Architecture, or even like a simple squiggled integration, I'm super interested in that, to be able to say, "Your forecasts," as you bring in more and more information from a language model, is building a world model that also has a mathematical grounding, in which you can enforce things like scope sensitivity, negation sensitivity, et cetera, et cetera. There, I think we will actually have a lot more paradigm shifts. And this goes along with an intuition that I would like to be much more explored for building safer AI systems, which is, can we have language models to be used in order to come up with

• 50:56Speaker 02

  rigorous world models that can be interpreted because they are mechanical? And there are papers on this front that I think have explored this in different ways. For example, there's a Tenenbaum lab at Stanford that's published a paper called "From Word Models to World Models." It was about a year ago, so there's plenty of work that come on top of that. Are we able to build probabilistic programming language support? So use an LLM to come up with a world representation where the world representation is both consistent and also interpretable? Can we use this as a starting point to come up with much better, much more robust worlds of AI systems that are getting more accurate, that are getting more logically consistent? I can go a little more technical if you're interested.

• 51:39Speaker 02

  Like, for example, can we come up with auto-formalizing option spaces? Like, there's a finite set of proposals. There's a constant positive number that is the total budget. Say we're trying to do like a question of budget allocation. The option space has many finite functions that, you know, the sum of all over the actions needs to be less than or equal to the budget. Like, an LLM is not going to parse that. But an LLM might be able to get better to this if you're thinking about, "Take a natural language description of a decision situation and produce a formal description of the decision variables, the constraints in the language optimization framework." Ton of research, like use something like CVXPI. Like, there's a bunch of libraries that would get this better. We can do structured construction of option spaces.

• 52:23Speaker 02

  Like, are we doing a simple choice question? Are we doing a matching problem? Like, L L M's would actually be good at figuring out, "Oh, the thing you're asking me seems to match this kind of problem." And then help parse into that. That way, we actually have a world model that we can inspect, that we can see, "Okay, these parts are right, these parts are wrong. Can you improve this?" And have the human continue with just verbal intuition, rather than need to keep an Excel spreadsheet to see if all of my intuitive probabilities are tracking. But instead, you're able to say things like, "These are my intuitive probabilities. Can you figure out where I am logically not consistent?" Like, this would help a forecaster right now. Here's all the data that I have. Here's thirty forecasts. Maybe these thirty forecasts have some logical inconsistencies between them.

• 53:08Speaker 02

  Can you do better? If we can help language models get better at this, that composite system, I believe, will yield a much more reliable, much more safer AI model as well. And if we extend that further and further, we actually might end up with AI systems that are interpretable, that have reasoning transparency, that have distinct parts of world model building, exploring option spaces, eliciting preferences and desirabilities from people that you can look at and see, "Oh, this is how all of them are talking." If you're just trying to get one L L M or a bunch of L L M's, not so much. Like, I will never be able... Maybe through breakthroughs on mechanistic interpretability, we might be able to get there. So I believe these kinds of explorations will actually yield much better AI forecasting.

• 53:53Nathan Labenz?

  Yeah, cool. So that's awesome. I want to just, for closure, for people who wanted to maybe hear those other two papers, I'll just mention them briefly. Because I think if you're going to get into this tournament, you should look at the Holloway and Steinhardt paper for sure. That's like, you're kind of jumping off point, you know, "Well-done agent with retrieval, with fine-tuning, with a good scaffolding to spit out good predictions." That's the one that I mentioned before. Comes reasonably close, but still falls short of the community prediction. Then there are two from Tetlock and co-authors. One was pretty interesting around just giving people access to an AI assistant and seeing how that helped them as humans forecast.

• 54:37Nathan Labenz?

  This is kind of the big picture, really a small step for the big picture vision that you're painting. Right. Interestingly, they ran an experiment where they gave the best assistant that they could give to the participants and then also a deliberately biased AI assistant. And they found that both helped, although the biased one didn't help as much. That's an interesting finding. And also an interesting argument in that paper about how this look at the future. I mean, you could think like multiple reasons for why AI forecasting could be useful. You've made the case that, you know, obviously we would like to have a better sense of what's going to happen. We would like to be able to make better decisions. They also make kind of an argument in that paper that the study of what language models

• 55:21Nathan Labenz?

  can predict is also kind of a useful way to interrogate to what degree they can get out of distribution, which is a very hotly debated topic within the study of AI, right? Some level, by definition, if you're predicting the future, you're out of distribution. So that's, I think, and the fact that they are doing it like reasonably well is definitely at least some points for team, these things can generalize purely beyond their strictly defined training data. Right. Then the third one, also from Tetlock and a number of the same authors, is just an exercise in ensembling the different language models and finding that the average of the language models performs better than the individual.

• 56:03Nathan Labenz?

  So it's kind of wisdom of, in fact, that's the title of the paper is Wisdom of the Silicon Crowd. Yes. That one was interesting to me. I mean, they did a, as you'd expect from a Tetlock publication, they re-registered all their experiments and were very sort of rigorous about declaring exactly what hypotheses they were going to test. It did jump out to me from the data that GPT-4 was like way outperforming the other models. And if you wanted to make a really simple improvement on their method, I would just cut the worst models and take one or like a few of the very top performing models and just kind of run multiple copies of those. Right. Couple of interesting notes there around bias toward round numbers a little bit,

• 56:47Nathan Labenz?

  bias toward positive resolution, some of these things that you noted where

• 56:50Speaker 02

  Right, right, right.

• 56:51Nathan Labenz?

  There was like some logical inconsistencies. And they were also hard. Another note I had on that paper was it really shows the breadth of AI knowledge off in a powerful way because I was looking at some of these things and I'm like, I don't even know the person, you know, it might be a leader of a country or whatever, and I don't even know who that is as we begin this discussion. So the fact that the AIs are just like jumping in and making predictions on such a wide range of things is like a good reminder of some of their fundamental strengths.

• 57:23Speaker 02

  Right.

• 57:24Nathan Labenz?

  So, okay, that's all background and, you know, hopefully breadcrumbs for people that want to get into the tournament.

• 57:31Speaker 02

  Find me to comment on them a little bit. I think the conclusions of these papers do make intuitive sense and I'd love to see much more rigorous tests of this. Like for example, what you said on, well, cut the worst models and maybe that'll get you better. Maybe, but I also want you to keep in mind, like if your goal is to have better forecasts, probably the kinds of things that will make mild leaps is going to be things like, look up if there has been similar questions asked on a variety of prediction platforms. Like, that is actually instrumental and shifts out of an academic mindset of the rigor towards like, what is actually instrumentally helpful here? I do think a lot of shortcuts are actually much more around like, okay, how can we get this to be further helpful? So those are the

• 58:16Speaker 02

  things I'm really interested in getting more folks to explore. Ensemble methods are great. Like wisdom of the crowd works, wisdom of Silicon crowd also would probably work as a result. Like try different models, try the same model, figure out if these models are good at specific domains or if they're avoidant on specific questions. Like there is so much juice there and as we have better models, also the conclusions will shift continuously, which is why I think a benchmark like this will actually be really, really helpful.

• 58:43Nathan Labenz?

  So let's get to the contest. You've alluded to the competition, or I guess we should maybe properly call this a forecasting tournament. You've alluded to that a couple of times. I would maybe love to start with just kind of a rundown of like, what the setup is, what the rules are, there's prize money available. So just give people a sense of like, What they would be signing up for and kind of the specific nature of the competition?

• 59:09Speaker 02

  Yes, I am pretty excited about this competition. It's called AI forecasting benchmark series. The competition will last for an entire year where every quarter we will have new questions that get launched. And for this quarter, we will ask about two hundred fifty to four hundred questions to the AI systems that are forecasting. We have about one hundred twenty K prize money to be allocated across the entire competition. And yeah, I would love for people to participate. A couple learnings that I've had just looking at the competition is even very simple bots seem to be doing really well. And this might be that their prompts are better. This might be that they're paying attention to better news sources.

• 59:52Speaker 02

  So it is definitely interesting. And without a lot of effort, I do think it is possible to get something that is somewhat competitive. Now, what is the tournament? In a way, it's a typical metaculous forecasting tournament, but it is specifically geared towards bots. We encourage people to build bots that use multiple L L M's. And we will use this as a benchmark to compare against pro forecasters and also the community aggregation. And it's the first of its kinds to be done at this scale. The basic rules are that you cannot have a human in the loop. You can change your bot, but you cannot make specific adjustments to the bot. Bots have to leave comments to showcase reasoning. We won't score the comments, but it will be good for us to see what are the winning bots doing.

• 1:00:37Nathan Labenz?

  So just a quick clarification. When you said you can change your bot, but you can't like modify it. That means you can update it periodically, but you can't adjust it for a specific question. Is that?

• 1:00:49Speaker 02

  Exactly, exactly. Adjusting it for a specific question basically defeats the purpose. At that point, it has human in the loop.

• 1:00:57Nathan Labenz?

  Right.

• 1:00:57Speaker 02

  I mean, I do believe human in the loop systems ultimately are going to be better, just like how, you know. Competition is good, collaboration is good, but like a competitive collaborative system where you have teams that can collaborate in competition is better. This tournament, we're not looking into this. We do want to see AI capabilities on their own, in their own footing. Hence, you can update if you come up with a better strategy and especially like the writers of the winning bots, we will interview them and ask them, what was your approach? We do encourage people to share these systems if they would like, but they're not expected to do so. They will provide some description of the bot or the code if they would like, so that we can see what works and what doesn't. Yeah, this will create a benchmark that is continuously evolving.

• 1:01:41Speaker 02

  And also, it's a benchmark that kind of cannot be cheated, or we literally don't know the answer to a lot of these questions. So it is really fun, like it evolves every day, like every day you look at it again and see how your bot is doing, is there something that is missing? So I find it pretty fun personally. We have created a template that people can follow that is fairly simple and straightforward, that you can use to just start with a one-off system. But on top of it, go to town, like there's a lot that can be done. Try the ensemble methods, try model-based reasoning systems, bring on news. Like one of the things, for example, where I saw, I touched on this earlier a little bit, if a human forecaster is forecasting and trying to do well, they will Google and look up other forecasting platforms to see if there are similar questions.

• 1:02:27Speaker 02

  This was something that the bots figured out pretty fast, or the bot writers, rather. So I'm really happy about these kinds of explorations. Yeah, so that is the rough rules of the competition. Fairly easy to get started. Oh, also, both OpenAI and Anthropic have donated a fairly generous amount of credits for the competition. So for anyone that would like to get credits, because there's a lot of questions to forecast, I think we can cover all of the needs. So the process is ping us either on Discord or send us an email. You can email me, [email protected], or support email for it, it's all on the website, and say, hey, this is the kind of thing I want to try, can I get credits? And I'm pretty sure we can give you plenty of credits to support you.

• 1:03:10Speaker 02

  So yeah, I find it to be like quite exciting for that.

• 1:03:14Nathan Labenz?

  All the questions are binary questions. They're going to resolve yes, no, is that right? Can you give us a few examples of early ones that have already resolved?

• 1:03:27Speaker 02

  Yes, I want to say all the questions are binary at this tournament. We will have non-binary questions soon for the next round, which will be Q4. The tournament will happen in every quarter and you get scored on the questions you forecast. So a lot of folks have asked me, am I too late to participate? I missed the first few weeks. Actually, not at all. We have some people that are high on the leaderboard that have joined fairly late. So I say, join now and also try things now because we will just cover your credits anyway. So when Q4 hits, you can go in with a bot system that you have rigorously tested. So what are the questions that have already resolved with respect to like the Prime Minister of France? Will that belong to a coalition other than the new Popular Front or together?

• 1:04:09Speaker 02

  Like a lot of election-related questions on France have resolved already. Domestic box office question, we have one on Deadpool and Wolverine. Will that be higher than that of Deadpool? Resolved, yes. It was interesting for the bots to do better than humans on this one. Joe Biden, you know, what will happen for the Democratic Party? We had a bunch of questions on that front. One thing we have done that I think is really interesting is in the like main quarterly cup, we have questions that are, for example, continuous variables. Like I'm looking at one of the questions, between July 17 and July 28, what will the strongest geomagnetic storm have a K in pixel? For this tournament, for example, we have discretized this continuous variable question

• 1:04:51Speaker 02

  to say things like, will it be greater than four or less than or equal to six? Greater than four, less than or equal to five, between five and six. So we have turned continuous variables into binary questions. And that way we can compare it to, you know, is there logical consistency here? But also, how does this compare with human forecasters that are looking at this? So a couple of tricks that we have placed in there to be able to make sure we can stay with a purely binary forecasting tournament. And next quarter, we'll build on top of that with more complexity.

• 1:05:24Nathan Labenz?

  Structure that the people submit their bot, then you guys are going to run the bot for them on a daily basis as new questions come out? Are they going to answer each question once or are they sort of going to be reposed the questions, you know, daily as they go until they resolve? And then do they provide a, I assume it's not just a simple yes, no, but it's presumably a percentage? And how does the scoring work based on that number that they give you?

• 1:05:55Speaker 02

  The scoring is the same as any binary question that we have done on Metaculous. It will resolve yes or no, and we will score it accordingly towards that. And we will score the ensemble on top of all. We are only doing binary questions here. But the process is basically, we don't host their bots, they host the bots. And we will open questions. Every question will be open for 24 hours. So you will need to continuously forecast on top of that. We have made that process fairly easy. If anyone has any questions, they should take a look at the onboarding process. It would take maybe about 30 minutes to spin up a very simple bot. But yeah, so you need to submit through the API your forecasts to the question that open and close every day.

• 1:06:35Nathan Labenz?

  Gotcha. So people are running their own bots on their own

• 1:06:40Speaker 02

  Yep.

• 1:06:40Nathan Labenz?

  computing infrastructure?

• 1:06:41Speaker 02

  Yeah, basically, we have some folks that are actually competing with bots that have value in form of a private IP. We don't want them to just say, well, you need to give us what you have. Or if you have built something really interesting that you want to monetize, we do want to encourage that. So I think it would be very interesting for everyone to just submit their bots. But also at this point, at this stage where the current talent capabilities are at, I would rather have people to have much more ease updating their bots. Like I want to see things like, oh, this bot seemed to be bad at scope sensitivity, but it got better. I'm curious, like, what tricks did they do? Or that I'd rather have the bot be still in the custody of the person that is building it? Does that make sense?

• 1:07:23Nathan Labenz?

  Yeah. Is there a way, are we working on the honor system? Or is there the possibility that people could have a human in the loop as they submit their stuff? Like, the reason I assumed that they had to submit the bot was so that you could guarantee that they wouldn't have a human intervention in the process.

• 1:07:44Speaker 02

  At, in this tournament, we have discussed this quite a bit, and we decided that this will go with an honor system. One of the things we have is, like, the number of questions to forecast is quite high, actually. So somewhat disincentivizes a human to forecast about 600 questions that are just, you're being bombarded.

• 1:08:01Nathan Labenz?

  It's a lot of work.

• 1:08:01Speaker 02

  And another reason why we want people to submit their reasoning is that if there is human intervention, it might be much more visible through that, especially for the winning cases and the high volume of questions. I do think we will scrutinize them heavily. And if we do have reasonable suspicion on that front, I'm sure we will have further investigations. But for this tournament, we decided to go forward with the honor system, rather. So we do encourage people to abide by the norms, because they'll be contributing to a meaningful benchmark that way. Otherwise, it kind of provides a cheat.

• 1:08:33Nathan Labenz?

  I guess the other thing is that the, the general prior assumption is, just based on all the research that we talked about a few minutes ago, that like, you might be able to add a bit, but you're not going to add, you know, even over the baseline AI performance, an individual is not likely to add that much. And you might even make it worse in some cases. So it's not like an obvious advantage to be, unlike, for example, the ARC challenge, where if you had a human intervention, you know, there would be like, a clear reason to think that you would have a big advantage toward the prize. Here, it's like, much less obvious that a human can actually improve on what their bot is doing on its own.

• 1:09:10Speaker 02

  Yeah, exactly. And to be honest, like, this is fairly early to make any conclusive statements, and kind of hope that this will change. But we have a couple of questions where the bot did better than the aggregate of pro forecasts. And I'm like, oh, this is very interesting, because they clearly are failing on things like scope or negation sensitivity. But on questions where that's not the primary mechanism, they seem to be able to do something that is powerful. I think people should try not messing with it. And one thing is every quarter, we will have a different tournament that will have its own sets of rules and norms. So I am interested in actually exploring different mechanisms here as well. We have an active Discord. So if people have ideas, I'm like, hey, this kind of competition

• 1:09:51Speaker 02

  would be much more interesting. I'd love to chat with them and hear what their ideas are. So we can try different things. But like, the goal of this is to be able to benchmark AI capabilities and see if this can be an early warning system that would actually substantially help the role forecasting place.

• 1:10:07Nathan Labenz?

  You said a minute ago that the scoring is like standard scoring, but just in case I don't know exactly what that is, my intuition is that it would be almost in the way that like a neural network might be measured for like a classification task, sort of sum of squares type scoring. Is that right?

• 1:10:22Speaker 02

  Is your question like how the scoring is for binary questions?

• 1:10:26Nathan Labenz?

  Yeah, exactly.

• 1:10:27Speaker 02

  Yeah, think of it this way. We're basically comparing the forecast to a coin flip in the context of the forecaster. We're using log loss versus a coin flip. And the peer score is basically how your baseline score, which is the log loss compared to the coin flip, compares to everyone else's. So that way we have a proper scoring rule that's used based on log scores. The reason why we designed this way is that it rewards the forecasters for beating the crowd and incentivizes both like do research and understand the crowd, but also report your best guess, rather than say things like, I want to be 10% above whatever the community prediction is. Does that make sense?

• 1:11:06Nathan Labenz?

  Yeah. Are you penalized by that for overconfidence? Like if you put a one on something and you're wrong, is that like very costly in that scoring system?

• 1:11:16Speaker 02

  So you need to be more accurate with respect to how you envision the world. That's way we don't drive towards the far edges, which is something you see in prediction markets that have financial incentives. So this actually combats against that. That's why I was saying like if the Market or committee prediction is at sixty percent. And you believe this is correct, you should say sixty percent. Because if it is accurate, then if it turns out to be a yes, like you will be not penalized. But if you don't think this is the accurate guess that you could make, then you should not say it's sixty percent. While in a market, if you're trying to maximize the financial output, where the market is at isn't something meaningful for that.

• 1:11:58Nathan Labenz?

  Yeah, gotcha. And you get all the questions at once as well? Would, is it within the rules to have the system consider all the questions to try to deal with the sort of possible inconsistencies?

• 1:12:09Speaker 02

  Can you repeat what you said?

• 1:12:11Nathan Labenz?

  Yeah, you said that you have like a number of questions that are related, where you've kind of discretized a continuous variable or something. Are you supposed to just consider each one, one by one and give a prediction? Or can you kind of consider them as a family and impose checks on like? If there are related questions, you know, make sure my predictions are self-consistent?

• 1:12:33Speaker 02

  I hope that's what the contributors do, right? I want a human forecaster and a bot to be able to do that. Which is precisely why we're asking these questions this way. Because it is very hard for bots to be able to do this, unless you have come up with something that is much more robust. So I also want to see if they get better at doing this over time through the next year. Will simpler models be better at scope sensitivity or negation sensitivity? But yeah, like a human pro forecaster will obviously say, oh, all of these questions are related. So let me come up with one answer. It wouldn't be very difficult for an AI system to start doing this by noticing these questions are all connected. You can just have a simple LLM check that says, if all of the questions are looking at similar variables, come up with one coherent answer.

• 1:13:18Speaker 02

  And then based on that answer, answer each of the individual parts specifically. I'm curious how many contributors will do that? And I'm curious if that will actually yield numbers that will be.

• 1:13:28Nathan Labenz?

  Yeah, we, I would think it would help at least a bit.

• 1:13:31Speaker 02

  Yeah, it should. Like this is what I, what I'm hoping for.

• 1:13:34Nathan Labenz?

  Is there a leaderboard that folks can obsessively refresh instead of going to the election websites every day?

• 1:13:44Speaker 02

  Yes, there is, and I encourage people to check it. Uh, what's interesting is some like newcomer bots have already went up high on the leaderboard fairly fast. Which is why I tell everyone, we have easy templates to get started. Just go and take your shot at winning the prize money. Like, we do have a large pool of money. And this will also be distributed across the entire contribution over a certain level. So you don't have to get first place to get something. Like, I do expect a lot of the bots will actually be compensated in some shape or form. So, yeah, bot performance seems high variance enough, and we have three more quarters. So I would say like, this is the time to get your bot in shape, set up through the template, start testing, and maybe you'll even win some money.

• 1:14:28Speaker 02

  We will cover your costs, so the cash will be distributed along the curve, rather than just the top three. So it's not too late at all. I say, give it a shot. My question to you is like, what would it take you to make your own bot? I think you would probably enjoy it.

• 1:14:42Nathan Labenz?

  Yeah, I think it does look fun. I did read through the collab notebook that you have shared. And it does look really with that all in place. It's like, especially and also covering credits. But there was one other little detail question I had was, is there a credit budget? Or any sort of budget per question? Or is that's all up to you to manage on the participant side?

• 1:15:07Speaker 02

  So in this tournament, we decided to not impose a max budget. I do think this would be interesting. Like, say if we said, you only have $200 forecast for an entire quarter. But then like, we're intentionally inhibiting measuring AI capabilities. So that doesn't actually give us the outcome we want. And that could happen in a fairer competition, but that kind of defeats the purpose. We have given up to 5K to some of the competitors. And people have asked us to say, hey, can we get more credits? We ask people to describe the strategy they will try and incrementally give more. So this is, we decided to do it on an ad hoc basis. Because if we just distribute it at large, we kind of can't keep track.

• 1:15:50Speaker 02

  But people that actually want to spend a lot more budget, they can reach out to us. And what it will take is a quick email exchange back and forth where they describe the phenomenon. And then we will keep a look at, are they forecasting? If so, we will keep on giving them.

• 1:16:05Nathan Labenz?

  Gotcha. Cool. Great question for me in terms of what would it take for me to get into the contest? I think the biggest thing I'm probably missing right now is a sense of like, how do I improve on the Steinhardt and group paper? And I, reading through that, I was like, this seems quite well done. It seems like it's working quite well. And it wasn't immediately obvious to me how I would make that better. So I would need to, and it hasn't been that long since I've read it. So, you know, eureka moments could strike anytime. But I hadn't had an intuition yet around, oh, if I do this, I think I can beat them. And that's sort of the moment of inspiration, I guess, I'm sort of waiting for.

• 1:16:49Speaker 02

  Right. I mean, a couple of quick ideas I'd like to share is pay attention to other forecasts. Like any forecaster should start by Googling, okay, has anyone forecasted this? Um, what's happening on other platforms? Are there other questions at Metaculous? Like, we do pay attention to make sure we don't have another question that's identical um for the tournament. But similar things will provide a lot of lights towards having a good base rate. So that, for example, or hook up for Flexity, pull up recent news, or try multiple different models and see how they compare with each other. I do believe that fact fetching versus coming up with base rates versus, you know,

• 1:17:27Speaker 02

  abiding by logic consistencies, or rather than abiding by, I'll say leveraging that to be able to make better forecasts. Different models will have different competencies. I have found, this is very anecdotal, and this might change even if you use different prompts. I'm just sharing, you know, I tried this, not as a research, but literally like sending prompting for a while. Entropic seems much better with respect to negation and logic consistencies compared to OpenAI. OpenAI is much more helpful to work with me in trying to figure out what I should pay attention to in a question. So come up with models that pay attention to multiple things. Use Llama models, use custom things. You can even fine tune with prior forecasts. Like, there's a lot that can be done.

• 1:18:09Speaker 02

  And I don't think the, what will make the winning condition is going to be like, this comes from a multiple PhDs research team that invented something really novel. I think it will be actually much more flexible than that with respect to which model ends up winning. I don't know, my bet will be that something simple will end up in the top three or top five and we will say, wow, like. Maybe we didn't need all of that complexity. Or maybe we will find out like, yeah, logic consistency abiding by it will yield much better forecast. So you definitely need to pay attention to that because that was the determining factor. There's a lot of conclusions we can get. So I just, my goal is getting people to try out a bunch of things.

• 1:18:50Nathan Labenz?

  It seems like the, if I had to forecast on a meta level, the result of this tournament over the next year and definitely using the the research papers as point of departure. It seems like some sort of ensemble of these bots, which themselves will probably be ensembles internally in many cases. It seems like we probably are headed for a world in which the AI forecasts are in aggregate very competitive with and maybe even surpassing to some extent the community forecast. And I guess maybe in, in closing, I'd love to hear like, first of all, do you think that's true?

• 1:19:34Nathan Labenz?

  And then can you give us a little bit more, you've kind of sprinkled some of this in throughout, but a little bit more about like what that will mean. And I guess I don't have a great intuition always for like, how good is the community forecast? There's, you know, it's still obviously not like a crystal ball. But if we could get that good, how good is that? How useful does that start to be? Is it a question of? You know, just we need to, if we had this level of accuracy for everything, like how much does that change the world? Or how much does that still sort of leave us in a like fundamentally uncertain state? Because this seems like this could happen, especially as the tokens are getting extremely cheap. You know, we could really have with the latest GPT four oh mini, it's especially if you do the offline batch process.

• 1:20:19Speaker 02

  Right.

• 1:20:19Nathan Labenz?

  10 million tokens for a dollar, folks. It's like, it's ridiculous. So, yeah, like.

• 1:20:26Nathan Labenz?

  What is the sort of epistemic future look like to you?

• 1:20:33Speaker 02

  I will answer from a different starting point, again, that harkens back to what we started with, which is around why is this you? If I was to tell you, we will end up in a world where you will have an ensemble AI or the bot aggregates that consistently beats the community prediction. Does that bring a better world on its own? I don't think so. I think the part with the AIs that I'm pretty excited about is that it costs a couple of cents to have them write an entire research paper explaining their reasoning. I don't know if you'll buy the reasoning, but it is much cheaper to probe and understand ways in which their conclusions come compared to humans. And that, to me, is a paradigm shift.

• 1:21:18Speaker 02

  And this is already happening, like there's multiple companies focusing on this, ranging from illicit to future search. I am interested in seeing how that will give us better actionability. Bots in aggregate start to forecast better than humans, or human plus bot ensemble systems are doing much better. I will again go back to how do we make this useful? And I am excited that the LLMs will be able to create much more interpretable world models, hoping that these world models will not be manipulative, also hoping that these world models will actually serve the goals of those that are trying to make value out of them, which is why some of the questions are fun and throwaway.

• 1:22:03Speaker 02

  Other questions are actually quite proactive towards specific goals. And we will have more of these to come up in the upcoming months, like around questions of strategies on mitigating homelessness. Like, there is a lot that we can do there. So if bots are getting much better than humans, then I think the question is, how can we make this be maximally useful? We already have really accurate forecasts, and we haven't had the Cambrian explosion of every single, you know, entity, corporation, government perpetually uses forecast, because the frame of the forecasts has not proven to be helpful just yet, which is why I am really interested in this action-oriented inflection point identification, finding short-term proxies, finding questions around budget allocation, working with hypotheticals.

• 1:22:50Speaker 02

  Another side note here that I think might be worthwhile to go into. I think we should ask questions that might not resolve. Like if I say, if San Francisco was to allocate seventy million dollars for improving public transportation, if they did X, if they did Y. Now, like, this might not resolve, this is probably not gonna happen, but asking this kind of question and asking pro forecasters to forecast on it, they might say, "Well, I'm not gonna get a good leaderboard on it because it will never resolve." But if I tell them, "Hey, this will be helpful for resource allocation." Doing this for questions on Taiwanese sovereignty, doing this on questions on AI, will actually let us explore hypotheticals, counterfactuals, in a way that will bring a lot more strategic information. So these are spaces I would like to go to.

• 1:23:35Speaker 02

  And if bots are able to bring a level of reasoning transparency, that is great, because that will make it much more actionable. I hope it checks out. I hope it doesn't end up being the uncanny valley of "seems legit, but not necessarily correct, but it's too hard to scrutinize." Like, that failure mode would be what I would be afraid of here. And in a way, we already live in a world that encourages that, right? Like, we have legal documents that are, unless you speak legally, you won't understand it, so you need to defer. The failure mode here is like, yes, there is something that all of these bots are able to converse with each other that we see as like, somehow this has higher epistemic status than what a human can access. Like, at that point, I think we are in the failure mode of epistemic security.

• 1:24:19Speaker 02

  I would like to dismantle that possibility, which is why I'm excited for LLMs to be able to help, say, someone that doesn't understand anything legal, that doesn't have money for a lawyer to be able to make sense of a contract, or for the medical sector, for insurance sector, for science. So, there is a world in which these tools can actually get us towards better reasoning. And do it be it individual or in a collective scale, where we are aggregating multiple perspectives. I think same as well for forecasting. Like, if we do this well, it is quite a paradigm shift. That is going to be super powerful. And like, the failure mode is basically like pushing humans further and further away from having access to that. I think the benchmarking tournament gives a little bit of light towards that.

• 1:25:03Speaker 02

  And hopefully more and more as the upcoming quarters will have more rigorous questions that go beyond just fineries. And so I'm excited to explore this. I presume many folks that are listening have been curious about setting up their own L L M bots or getting more facility with L L M's trying prompt chaining. I think like this is a skill set that I see a lot of people around me wanting to cultivate more of. I would say like this is an unusual moment. It's a powerful opportunity. You have outsized price money. You will contribute to a benchmark and it helps you build capacity that you would want to do anyway. Plus, we will cover your credits and we have some templates. So I would just say like, tell people, hey, like, come along, we need to figure this out.

• 1:25:44Nathan Labenz?

  Cool, I love it. Is there anything else that you wanted to touch on that we haven't got to?

• 1:25:49Speaker 02

  I mean, I just want to say, I am especially interested in folks that say, hey, I have a different vision with respect to how forecasting can be helpful. Do reach out to us. You're quite interested in seeing different experimentation and testing out some of these experiments where we can both financially incentivize people and try things out. We will be open sourcing Metaculous very soon, so I'm very excited about more people to try things out. So that is another thing I would like to underline. If you have ideas that you want to test, holler for, I'd be very curious to hear them. And yeah, hopefully we will have many more instances, many more ways in which people will try to make use of forecasts.

• 1:26:28Nathan Labenz?

  Yeah, I would ask, also, that's a good reminder. What is the sort of future of the company? Like, obviously, open sourcing, important strategic decision in terms of like, gives a lot of value and a lot of opportunity to others. How do you see that sort of shaping what your own possibility set is as you go forward with the company itself?

• 1:26:49Speaker 02

  I'll start from like, before I joined a C E O, I was wondering why is this not open source already? I do think there is a lot of wealth here and one way to get more people to be able to both like critique and audit, but also host their own versions. Like, this is something that is a net good towards the entire humanity to be able to do much more of this. And Metaculous will focus much more on ways in which forecasting can be useful for both decision makers and understanding more robust world models. Which is why questions around better reasoning transparency and

• 1:27:23Speaker 02

  Action oriented decision making for those that are trying to figure out what's the best way to allocate resources or which policy seems to address the needs of a community. Will this policy, if enacted, actually yield the impact? So we're interested in much more collaborations on this front with folks be in municipal to federal agency government. We have some of these partnerships already in place. Interested in ways in which this would also help businesses to be able to make better decisions and keep track towards the goals. So these are all spaces that we will highly prioritize. We're entering a window of a lot of experimentation, actually, on seeing like what are additional things on top of the forecasts as they exist right now would add value.

• 1:28:07Speaker 02

  So the minitaculuses that we were talking about is just one of the many examples here. Building indices is another, like looking into things like tech trees that Forsythe Institute is building on seeing, okay, for us to get to this technical state, it looks like we have these in between things that need to be invented or researched. Will this be worthwhile? Okay, if we put five million dollars to this versus fifteen million dollars to this, will we be able to go through these inflection points? These are the kinds of ways of reasoning that I would like to see minitaculuses enable in.

• 1:28:39Nathan Labenz?

  Is there anything that you worry about with this paradigm? I mean, we just looked through this crowd strike disruption, right? Where like one seemingly very local point failure like cascaded through society. And if I was going to take the most risk oriented position on this, I might say something like, in a world where, yeah, maybe we're getting most of the time even better than wisdom of the crowds forecasts from like a couple of different AI models. What happens if they fail, not necessarily in terms of an outage, although maybe, but like more so, is there a potential for sort of a correlated failure that could create weird scenarios, weird tail risks?

• 1:29:23Nathan Labenz?

  How do you think about that type of possibility? Is there any way we can get ahead of that?

• 1:29:30Speaker 02

  I mean decentralization is key, right? Like if everyone wasn't using CrowdStrike, probably everything wouldn't have gone down all at once. And I'm sure a lot of people have had the hindsight to say, well, I saw this coming. I've talked to a couple folks even that said, well, yeah, we switched out because we first said something like this coming up. I think especially when you have institutional systems that are entrenching the use of specific tools or specific flows, that ends up introducing a lot of failure modes, right? Like if there is a government-end or supply chain flow, that would be much easier for a foreign entity to be able to infiltrate and mess with, causing supply chain-related drift. Plenty of those through human history. I think like Metaculous being open source is good.

• 1:30:14Speaker 02

  I mean, obviously, I want Metaculous to be the most accurate, and that is the goal we're striving towards. But even more than that, like the proliferation of the perspectives, so that there is multiple people doing multiple focuses. There is many different strategies. Like maybe for a specific domain, a different way of scoring will work much better. And this will end up causing much more value for that. Like, for example, just zooming back to something we said at the beginning of the podcast. Decisions around political outcomes is very different than, say, like questions like, should Biden step down versus questions like, if we allocated an additional fifty million dollars towards synthetic biology, which of these will yield an inflection point?

• 1:30:57Speaker 02

  Now, it is interesting, like in the book, like super forecasting, there's plenty of anecdotes of, people are really good at forecasters can end up beating domain experts. This doesn't mean we don't want domain experts. This means we want the forecasters to work with the domain experts. So I think there's many different contexts in which forecasts can play instrumental role, but they don't all look the same. So I think the real win for Metaculous is to actually bring us towards this world, where many people are trying many different approaches and many different things. We will always strive towards the level of rigor and accuracy. But on top of that, like I would love it if there's a different thing that ends up building an AI ensemble method that seems to be more reliable than Metaculous prediction.

• 1:31:39Speaker 02

  Like at that point, then the question is, okay, like, how do we integrate this towards usefulness? Like, that wouldn't be a failure. It would, in a way, be a success for like, we have shown that this is possible. So. My answer towards failure modes through singular authority that has maximal control is always, well, decentralization is much better. And how can we steward towards that? That's the question.

• 1:32:01Nathan Labenz?

  Cool. Are you also interested in things like Polis? Is there a sort of Metaculous-Polis mashup? Or like, you know, what does it look like if those two concepts have a baby?

• 1:32:13Speaker 02

  Great question. This goes all the way back to my backstory. I would say Polis is probably the most influential platform for evolution of my thinking. Right before Metaculous at AI Objectives Institute, the project I worked on the longest was called Talk to the City, which is an LLM-assisted aggregation and deliberation platform that can take in qualitative feedback. And in a way, this was what I was working on for my thesis. This was what I was working on in my startup, Cerebro also. So I have thought about this question in so many different lenses. When you have raw text that is the highest fidelity human opinion, how can you augment that?

• 1:32:52Speaker 02

  And through many iterations, first without LLMs, then looking at multimodal systems with text data plus other data. And in the latest iteration with Talk to the City, what I have come to was, okay, the current state of language models actually does a fine job in identifying what are the topics? Are there any subtopics? And another thing they are really, really good at is actually the retrieval of these concepts. Talk to the City still actually exists and it's going pretty well. We have an excellent team at AI Objectives Institute, so I recommend the audience to check it out. Website is ai-objectives.institute and they can see on the blog post a couple of case studies that we've done on impacts of Talk to the City. Like, for example, one was with Taiwan.

• 1:33:34Speaker 02

  We've had an extensive collaboration with Taiwan's Ministry of Digital Affairs, where we were aggregating people's opinions with respect to both local municipal questions and also larger scale questions on same-sex marriage data. A lot of these data inputs come in from Polis. Now, Polis doesn't, like, there's some constraints there that is a trade-off. Or it ultimately is a recommender system where you're finding people that will share certain viewpoints. Instead of leaving the onus to that, which causes less human bandwidth, like you can only write about tweet-sized things, we said, okay, can we have just completely unbounded? Input window where people can send in text, they can send in videos, they can share any kind of data.

• 1:34:19Speaker 02

  That's, you know, you would be able to then enhance and see, okay, what does this community talk about? What matters to them? And in a way, my journey to forecasting and to Metaculous was me seeing, okay, we are actually at a place where we can aggregate public desirability. I prefer the word desirability to preference because preferences can change. One might not necessarily be aware of their preferences, you can have cyclical dependencies. I use desirability more as a catchall term that's overcome some of these. We can aggregate this, we can have a snapshot. Like one project that I loved, for example, was with the Labour Union with focusing on veterans' health, where they wanted to run a survey with respect to a specific proposition.

• 1:35:02Speaker 02

  And the negotiation windows they have with the government is incredibly small. There's a 20-day window where they need to come up with a policy, share it, and it will go through. The whole system is basically rigged against internal deliberation. It's so narrow and so fast-paced that people just say, whoever I voted for as head of the union should go forward. The new president of the union, Mark Smith, ran a survey, open-ended, text-based question to the entire population. Within 24 hours, I think we got what, like maybe 200 responses on the first round? Within 24 hours, we could turn this open-ended survey in multiple languages, by the way, not just in English. You can get it in Spanish and Tagalog and consolidate this to say, here are the four viewpoints that came up. What are your thoughts?

• 1:35:47Speaker 02

  Send it back to the entire community and develop this recursive loop. It's so much more interesting when you say, wow, between just yesterday, when the survey went out, I already have four clusters, maybe I should respond to this. Ah, I have something to add to this one. Next day, do it again. This cycle is much cheaper than what it used to be. I think this is a paradigm shift, like this changes governance. And what I realized is, okay, now that we can aggregate this, what I need is the next pillar, which is, will the action over actually get us towards where people want to go? And that's, I was talking about this with Gaia, and that was the previous CEO of Fantaculous, and that's how I ended up here eventually. So I'm really excited about seeing how forecasting can help this. I think there's a lot there, and I think we need a lot more

• 1:36:32Speaker 02

  experimentation. So yeah, like, Polish-like tools are, in a way, where I started thinking, huh, maybe there is a way in which we can do better.

• 1:36:41Nathan Labenz?

  Glad I remembered that bonus question. I think it does sort of start to look like a liquid democracy, or a, you know, sort of technology-mediated liquid democracy, and that is definitely super compelling. I mean, there's just not much room on, we see this right now as we're going through this campaign, obviously, in the US. It's like, the things that are actually getting talked about are few, and not particularly well-chosen in many cases. And there's just so much stuff that really ought to be, you know, that we want to understand what people actually care about, first of all, or how they think about

• 1:37:24Nathan Labenz?

  different things. We're just not even getting that data in the first place, let alone being able to map a path toward actually delivering for people. So.

• 1:37:35Speaker 02

  Absolutely. And I think, you know, the bit signal of red or blue votes in the US, it's, we can do so much better than that. That's why I'm looking at, you know, municipal engagements, membership-based organizations, looking at DAOs, where they are making a lot of decisions around their stakeholders, and the entire thing is on a digital substrate. So there is much more experimentation here. What you said reminded me of something with respect to liquid democracy. I must say, like, the term liquid democracy does bring some aspects of fear in my mind also, and I think it's worth pointing that out. There's this concept that I call the consensus illusion. There is a drive towards the lowest common denominator. That if we say finding consensus in a community is desirable for that is the best policy.

• 1:38:20Speaker 02

  What you will end up is a lot of policy outcomes that are quite lukewarm, that don't actually address the issue. There was a New Earth paper from DeepMind in like 2022 on like, how can we fine tune language models? So that we can find agreement across humans that have diverse preferences. And one of my concerns when I see work like this, I think the papers are actually great. Like some of the techniques they use has made its way to talk to the city. So I like that. Some of the concerns I have is finding agreement doesn't necessarily mean good policy. In fact, it quite often doesn't. Like, one example I like to give is one of the conclusions from a civic dataset is we should build more bike lanes for the community. Everyone seems to want more bike lanes.

• 1:39:03Speaker 02

  And you say, okay, where should we build these bike lanes? It shouldn't be on this street. And the model will say, no, no, no, that's not what we agreed on. It shouldn't be on a specific street. We need more bike lanes. This is the thing. We don't want to make any specific streets narrower. And it's just like, oh, we're doing the same thing we've been doing with politics all along. Find the most common denominator statement that yields power. Because in the current elective system, consensus is power. So when you have a desire towards consensus, that causes a trade off with fidelity to your viewpoint. And if you seek power over fidelity, you will have higher representation of a viewpoint that is not very meaningful. And the most catch-all sentences will end up resonating the most and it won't make meaningful policy.

• 1:39:47Speaker 02

  So I don't want consensus, really. I want to start from a shared world model. Like, can we actually come up with ways in which we have desirabilities for stakeholders? We have action possibilities that come from policymakers. And then we have outcome likelihoods from domain experts. And these three need to be continuously talking to each other as division of labor. I think democracy focuses on the state's desirability for stakeholders. But if the options you give us on the roads, you can't really build a bridge. And people need to be able to say, I want also be able to build a bridge. And an AI model that hampers this by finding, oh, this is the most agreeable thing is not good. Instead, I'd much rather have an AI model whose goal was come up with individual viewpoints

• 1:40:32Speaker 02

  that are clashing with each other, represent each of them with high fidelity, and identify what are the cruxes between these viewpoints. Like, if we can find a good crux where, like, I think A is true, you think A is false, but the underlying cause, if I thought the underlying cause changed, this would change my mind about A, and you would say the same thing, like the typical double crux process, is good, because that means we actually have a same-shared world model, and we need more data, and at that point, bring on further research, bring on forecasting, like, we're in a good mode. The failure mode here is a risky alliance, where someone says, oh, I think policy A is good. You say, I also think A is good, and I say, A is good because it will enable B, which will be great.

• 1:41:15Speaker 02

  And you could say, I don't think B is likely to happen as a result of A, but maybe we should keep together, because it seems instrumental, just for this one step. You end up having a lot of unlikely alliances, and the ultimate version of this is a completely polarized society, where, for some reason, social conservative and fiscally conservative behavior is correlated with each other, even though that doesn't necessarily manifest. There was one research that I read, I can't remember where, that I just absolutely loved, which was, instead of asking people their opinions with respect to immigration policy, it actually asked a statistical question on how many people are you willing to let in before you reject someone that truly deserved to come into the country?

• 1:41:58Speaker 02

  And the previous people that you have let in might be mistakes. This is very interesting, because instead of people saying, well, I am liberal, I am conservative, you see people who say they are liberal, you know, San Francisco crowd, that will say, yeah, I'm willing to let in five mistakes before we admit one person. And others will say, what do you mean, five? We need, like, three hundred. But both of these people see themselves as having a liberal opinion. Like, we do not yet operate on a level where we're looking at good policy. So, we should not encourage this illusion of consensus so that it can serve towards better power. Instead, we can use these techniques that we have right now to go towards better policy. And better policy means higher visibility. Better policy means people agree on the world models. So...

• 1:42:42Speaker 02

  I see the work we're doing at Metaculous as an instrumental step in that trajectory. Are we able to find people whose world models are diverging? Let's figure out why they are diverging. Are we able to see, okay, will this action that a policymaker has given us bring us to the outcome? I think these are really important questions around the category of how can we have better epistemic security?

• 1:43:05Nathan Labenz?

  It's crazy to think how far this might be able to go over the not-too-distant future. I mean, obviously, you're primarily focused right now on getting a read on what the bots can do and trying to be as accurate as you can and to be much more in-depth on the sort of, you know, mini Metaculuses. Let's assume this works. Do you have a roadmap for how this sort of rolls out and scales up to ultimately like big picture, you know, most important questions in society?

• 1:43:39Speaker 02

  Right. And that question I hear, oh, there you should write a blog post about this probably that maps this out. Quick thoughts that come to my mind. Like, we know preferences can and do change based on actions and their consequences, right? So can we build these feedback loops as proof points in organizations that can actually take action on them? Where real stakes are present? That's why I'm interested in labor unions, for example, because it's a fairly acute case of coordinated decision-making with stakeholders that are outside and inside the group. So it gives us a lot of visibility. I think where we are at right now is, like, I don't expect a top-down revolution of a government adopting this off the bat,

• 1:44:20Speaker 02

  but I do think there's a lot of municipal-level experimentation that already has been happening. For example, Dr. City is collaborating with a couple municipalities in Japan right now through Liquid, which is a Japanese like liquid democracy purpose company. There's a bunch coming in Taiwan. We have interest from folks in Singapore to be able to use Metaculous forecasts, for example. We have interest in similar Taiwan community. We have groups in the US also on municipal level. Like, for example, Detroit has a bunch of communities that are focusing on the well-being of African-Americans and previously incarcerated folks. Like, are we able to figure out what interventions bring a better world to them?

• 1:45:01Speaker 02

  These are all very short term, but in these processes, we can actually see, oh, this seemed to have worked. This actually did yield an outcome where a group was able to coalesce much better. I think we need more visibility into that. This, I think, is the very first step. I think AI tooling is absolutely critical for both the failure mode and the success mode. Will heavily depend on how these tools are implemented and used. How these tools are actively enhancing human agency. I would recommend people who are interested in more of this to check out the roadmap documents from AI Objectives Institute. That's where I have done a lot of my writing and thinking with the team there. I reach out to them also. I still try to stay as involved as I can, but it does hold a dear space in my heart for

• 1:45:43Speaker 02

  Peter Eckersley, who was the original founder and a mentor and a friend of mine. Unfortunately, passed away quite unexpectedly, which is when I started leading AI Objectives Institute. And like the line of thought there was very much always like, AI can be a transformative point for human well-being, but the default systems do not place us on that path. So I think there's a lot there and this is the question of existential hope, right? Like existential risk is failure to coordinate at the face of a risk. If we can already foresee this path and we are failing to coordinate the systems we are in, the coordination capabilities we are in doesn't let us get to the heart of that, that is why we are failing. So that is the angle that I want to keep looking at.

• 1:46:28Speaker 02

  Because we have seen incredibly successful cases of international coordination or multi-corporation coordination, like the ozone layer is basically recovering since we have banned CFCs. Like, there is many different cases where we have moved mountains as society. Like microplastics-related harms, like we have banned lead at this point. Like, there are ways in which we are able to coordinate if we can create coherent world models. And I think the thing we need to do is have shared world models that can contain the disagreements. Rather, instead of agreeable action policies. I think the way politics happens right now, voting happens right now is find the most agreeable action policy so you can maintain control.

• 1:47:11Speaker 02

  As opposed to say, what is the good policy? And this requires a level of epistemic rigor and epistemic security. I see my life's work is to focus on that question and bring more and more towards that. And if there are groups that we can work with, hell yeah, let's kick it. I like, this is where we need to start. So if there are any organizations that are focusing on our priority cause areas, be it on climate change, or AI, or like nuclear consequences, or there's any organizations that are trying to do better on resource allocation, where they want the resources to be able to do the maximal good for the specific community, I would love to talk to them. I would love to understand how the things

• 1:47:52Speaker 02

  we are building can be useful for them. I think we need more experimentation of this sort. And I'd rather have these experiments be with people that benefit from them in the immediate short term.

• 1:48:04Nathan Labenz?

  Cool. Well, that's a great call to action. I'm glad we stayed on the little extra ticket, that final section. I guess I'll ask again, last time it was a fruitful question. Anything else that we didn't get to that you wanted to touch on?

• 1:48:16Speaker 02

  Nope, I feel complete. Thank you so much for this opportunity. It was lovely for it also made me realize the context through which my path has evolved, like seeing the role forecasting can play hand to hand with collective intelligence and the failure modes of the current political processes, or democracy, or resource allocation makes me realize, oh, I see the role that this plays. So this was great for me as well. So thank you.

• 1:48:41Nathan Labenz?

  Cool. Derek Turan, CEO of Metaculous, thank you for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.