Surveying 2,700+ AI Researchers on the Industry's Future with Katja Grace of AI Impacts

In this episode, Nathan sits down with Katja Grace, Cofounder and Lead Researcher at AI Impacts. They discuss the survey Katja and team conducted including over 2,700+ AI researchers, the methodology for the research, and the results’ implications for policymakers, the public, and the industry as a whole. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api LINKS: - Thousands of AI Authors on the Future of AI: https://aiimpacts.org/wp-content/uploads/2023/04/Thousands_of_AI_authors_on_the_future_of_AI.pdf -AI Impacts Site: https://aiimpacts.org/about/ -Linus episode: https://www.youtube.com/watch?v=wdmvtVTZDqE&pp=ygUJbGludXMgbGVl X/SOCIAL: @labenz (Nathan) @KatjaGrace (Katja) @AIImpacts SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api ODF is where top founders get their start. Apply to join the next cohort and go from idea to conviction-fast. ODF has helped over 1000 companies like Traba, Levels and Finch get their start. Is it your turn? Go to http://beondeck.com/revolution to learn more. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected]. 💥 Access global engineering without the headache and at a fraction of the cost: head to choosesquad.com and mention “Turpentine” to skip the waitlist. Producer: Vivian Meng Editor: Graham Bessellieu

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Forecasting the future is quite hard. We are having to make decisions about these things. The best guesses that we can get are valuable. And so it's important to hear what AI researchers think about this. Amidst quite a lot of uncertainty here, the chance of things just staying pretty similar and nothing coming of this to drastically change people's lives seems quite low. This seems like a pretty important thing to be keeping an eye on and trying to have accurate opinions about what the public thinks about these things will affect what policies happen and they think that could change the rest of the future forever.

• 0:36Nathan Labenz?

  Hello and welcome back to Turpentine AI. Before today's show, another quick reminder to take a second to sign up for our new Cognitive Revolution feed. Our latest episode on that feed is an outstanding conversation with the great Linus Lee of Notion AI. So find that and be sure not to miss any of the new episodes that will be exclusive to that feed by visiting our website, cognitiverevolution.ai to subscribe.

• 1:04Nathan Labenz?

  Today, I'm excited to share my conversation with Katja Grace, founder of AI Impacts, who recently published what I believe to be the most comprehensive survey of elite machine learning researchers' expectations that has ever been conducted.

• 1:18Nathan Labenz?

  As we all try to make sense of this AI moment and understand where things are going, Katya's survey provides an incomparable resource. With over 2,700 respondents, all of whom have published in one of the top six AI conferences in just the last couple of years, it offers a nuanced and detailed picture of what the people closest to the technology really believe. And the results are truly striking. Arguably, the biggest takeaway is that there is no consensus expectation within the field. Most individuals express high uncertainty on the most important high-level questions, and only relatively small minorities confidently predict a very positive or very negative future.

• 1:59Nathan Labenz?

  Researchers definitely do not dismiss existential risk, with median estimates of a 5% to 10% chance that AI could lead to human extinction. And meanwhile, timelines for key capabilities have shortened since the survey was last conducted just a year ago.

• 2:16Nathan Labenz?

  There is strong agreement that AI safety should be prioritized more than it is today, perhaps in part because no major mechanistic interpretability breakthroughs are expected in the near term.

• 2:28Nathan Labenz?

  Overall, this conversation offers a uniquely rigorous field-wide take on critical questions that are all too often poorly asked and sloppily debated. After first digging into the methodology to better understand how Katya and team went about defining their terms, framing their questions, and summarizing the responses into aggregate views, we go on to consider implications for policymakers, the public, and the field itself. While there's of course, nothing to say that ML researchers will ultimately be correct about AI outcomes, I would call this a must-listen for anyone who wants to ground their own personal AI world models in the considered opinions of the people who are collectively inventing the technology.

• 3:10Nathan Labenz?

  As always, if you find value in the show, please share it with others who you think would appreciate it. I genuinely believe that both the doomers and the EAX should update their P dooms toward these community estimates. A 5% to 10% chance of extinction is obviously a very critical issue, but the shape of the other 90% of outcomes is also well worth worrying about. Of course, I'm always eager to hear your feedback, so please feel free to reach out via your favorite social media platform. And now, please enjoy this exploration of the expectations and the substantial uncertainties of the AI research community with Katja Grace of AI Impacts. Katja Grace, founder of AI Impacts, welcome to the cognitive revolution.

• 3:55Nathan Labenz?

  Thank you, great to be here. I'm excited for this conversation. You have just recently completed one of the most comprehensive surveys of the worldviews and expectations of elite machine learning researchers. And as we try to grapple with this AI moment and try to make sense of what's happening and where things are going, I think it's a really incomparable resource. So I'm excited to just spend the full hour kind of digging into that and understanding both how you went about collecting this opinion and also what the aggregate opinions are. For starters, you want to just introduce maybe yourself and AI Impacts a little bit and give a little bit of the inspiration or motivation behind the survey?

• 4:38Speaker 00

  I am broadly interested in trying to make the world better. And for the last 10 years or so, I've been involved in trying to figure out what's up with the future of AI, since it seems like potentially a pretty important thing to try and cause to go well. Yeah, I've been variously skeptical about whether it's likely to cause human extinction or not. But yeah, interested in figuring out these kind of things. So AI impacts is an effort to try and answer questions like this. And just various high level questions about what will happen with the future of AI that are relevant to decision making for lots of people. Is AI likely to cause human extinction? When will the big AI thing be? What kind of thing will it be?

• 5:22Speaker 00

  Are there other kinds of changes to society that we should expect and do something about? AI impacts is often answering sub questions or sub sub questions that are relatively in the weeds as input to those things. For instance, case studies about other technologies and whether they have ever been slowed down for sort of ethical or risk reasons, or what that looks like for a recent example. So we do a lot of stuff that's not directly about AI. It is intended to be part of a bigger network of questions that inform other questions.

• 5:57Nathan Labenz?

  Cool. For the purpose of a survey like this, is there any reference point that you can look to where folks have found it necessary even to survey experts to understand the direction of their field? I would think maybe something like biotech might have something like this. Or you could imagine, I just recently read a story about the original nuclear test at Los Alamos and how they had a little betting pool on it, but there was no like big survey. So are there any kind of other touchstones that you look to?

• 6:29Speaker 00

  I think the thing that we most look to is just like other surveys in AI over the years. Where prior to the three that I've been involved with, there were older, smaller ones. It often had a different definition of hu- human level-ish AI each time or something. So it's like a bit hard to compare the results. But I guess we previously looked for all the things like that we could find and go back to like, I think it was like one of the seventies. Yeah. Interestingly compared to over the years, I guess we're not looking to them that much methodologically. Cause I think they're maybe less careful and smaller. In terms of things in other fields, I haven't looked in detail at them. My understanding is that there's some sort of thing in climate perhaps.

• 7:13Nathan Labenz?

  Yeah, that's a definitely good reference. There's certainly a lot of guessing as to the future state of the climate. So that makes a lot of sense. Okay, so the way I plan to proceed here is just first of all, talking about who are you surveying? Like, who are these people? Then what was the survey that they took? A little bit of, I'm just gonna structure the experience, some of the key definitions and framings that you used. And then I think the bulk of the discussion will be like, what are the takeaways? What do people actually believe in the field? But for starters, who were these people? How many are there? And how did you get them to do it?

• 7:46Speaker 00

  We tried to write to everyone who published in six top venues. So five conferences in one journal in 2022. So as in, in 2023, we wrote to the people who had published in 2022. And we offered them $50 each. We, we had a little trial at the start where we offered nothing to some of them and $50 each to, to another small group. I mean, it looked like the $50 each was doing a lot better. So then we offered it to everyone. It, it was a decent task to dig up their addresses. Like, we got their names from all of these papers. And many of the papers have emails for at least one of the authors, uh, maybe several of the authors. And we dug up the other ones from other papers of the internet somehow. And I thi- I think we found them for a large fraction, like more than 90%.

• 8:31Speaker 00

  I forget the exact number. So I think this is like i- in the range of 20,000-ish people. And we got about like 2,700 to respond.

• 8:39Nathan Labenz?

  These are people that published in six top conferences? Is there a natural reason that you chose the six conferences? Like, why not four or 10?

• 8:51Speaker 00

  So previously, the last two surveys we've done, we just did NERIPS and ICML. And we want to expand it partly to a wider range of topics. So I think like AI people who are not necessarily all doing machine learning. And also just to like cover more of the top people. So I think it ended up being six because as far as we could tell, talking to people around, that was a sort of natural set of them to do. But yeah, I don't have a clear explicit description of what makes them the ones.

• 9:25Nathan Labenz?

  Is there a time horizon also on how recently they published in these conferences?

• 9:31Speaker 00

  It was everyone in 2022, like everyone who published in 2022. Often people published in multiple of them.

• 9:37Nathan Labenz?

  Quite recent, yeah.

• 9:38Speaker 00

  Yeah.

• 9:38Nathan Labenz?

  Okay, cool. So we got six conferences, universe of 20-ish thousand people, something like a 15% of respondents rate driven in part by a $50 gift card, 2,700 plus total respondents. First of all, interesting, and there's of course a whole paper here. You guys have a lot of visualizations of data in the paper. So definitely encourage folks to go check out the graphs if they want to do a real deep dive, although we'll cover the bulk, I hope, of the headline results. But it's kind of a branching structure where like not everybody's getting every question, but you have some anchor questions that everybody's getting. And then others, there's kind of a bit of a, not choose your own adventure, I guess,

• 10:20Nathan Labenz?

  but kind of a randomly chosen adventure for them. And it takes 15 minutes, which it was kind of surprisingly fast. I would have expected to go slower.

• 10:30Speaker 00

  Yeah, I guess we would actually check for this one, how long it took. I was like an estimate estimated based on like the previous one.

• 10:39Nathan Labenz?

  Advertise this 15 minutes. Was there any free response or it's all structured? Like you must either like select a thing or like enter a number, right? There weren't like any. Paragraph style responses that I could detect?

• 10:52Speaker 00

  There were some that weren't paragraph length. There was just like some brief ones. For instance, there was one that was what's an occupation that you think will be automated, fully automatable pretty late, something like that. So that's a couple of words. But also for, I think, all of the sets of questions, you could randomly, I think maybe 10% of people after each one got, there were two different questions you could get about just like, what were you thinking when you answered that? Like, how did you interpret this question? Which are not part of the main results, but for us to be able to go back and if we're confused about a thing, check if everyone's misunderstanding it or something like that, which we have not gotten into yet. We might do more of. Oh yeah, actually, sorry. Yeah, there are various open ended ones.

• 11:36Speaker 00

  There was another one that was like, what confusions do you think people have about AI risks? Something like that.

• 11:42Nathan Labenz?

  Okay, maybe we can talk about some of the open ended stuff later. And I'd be interested to hear how you may think about evolving the, the survey in the future, particularly as you can potentially bring language model analysis to open ended responses. I expect surveys in general are going to be an area that will change quite a bit. Certainly I see that in even just very sort of application centric customer feedback experiences. I'm like, hey, let's just let them talk. We don't need to structure everything in a grid or Rubik quite as much anymore. We can, if we're looking for insights to some degree, you can just let people sound off. But for these purposes, you're really trying to get to quantitative estimates.

• 12:21Nathan Labenz?

  And as I read through the paper, there were two super high level framing structures that stood out to me as, first of all, just very thoughtfully done and worth understanding for our audience as well. One is that you have two different definitions, essentially I would say of AGI, right? Like there are two different ways of thinking about like powerful AI that we definitely don't have now, but we could have in the future. And then there's also two different ways of asking people when. They expect certain things to happen. Maybe you can unpack, first of all, the two different definitions of AGI, if that is a reasonable way to think about it.

• 13:00Speaker 00

  Yeah, I think so. I guess I think of them as fairly closely related, but yeah, different people think of that differently. So there's high level machine intelligence, HLMI, which is like when unaided machines, so it doesn't even have to be one machine, just like any collection of AI can accomplish every task better and more cheaply than human workers. That is like for any particular task, there are machines that can do it. It doesn't even have to be the same collection of machines that does stuff. And we're ignoring aspects of the task to which being a human is intrinsically advantageous, e.g. being accepted as a jury member. We're asking them to think about the feasibility of it, not whether it's adopted. So this is roughly like, when can AI do all tasks better than humans?

• 13:45Speaker 00

  Then the other definition is full automation of labor, which is sort of similar, except that it's asking about occupations instead of tasks. So occupation becomes fully automatable when unaided machines can accomplish it better and more cheaply than human workers. We ignore aspects of the occupations, which being a human is intrinsic advantageous, is the same. And again, asking them to think about feasibility, not adoption. So I think there's some chance that people of these things in mind and tend to think of the automation of occupations as actually about adoption and the high level AI one about it being feasible. So when all occupations are fully automatable, then we have full automation of labor. So I would think of an occupation as probably like a big, complex task,

• 14:30Speaker 00

  or if not, composed of many smaller tasks. So I would think that if every task is automatable, then that implies that occupations are automatable as a subset of tasks. But in fact, the answers that we see put the date for HLMI much earlier than the date for full automation of labor. So I think there's a question there about what people are thinking. And to be clear, no one is answering both of these questions. People are randomized to receive one or the other. So. You can't say for a particular person, like, why he's so inconsistent. But given that there's such a big gap, presumably any given person is likely to be inconsistent if you ask them the two questions, because it's not just random or something.

• 15:13Speaker 00

  There's one other small difference between them, which we keep in, in order to keep everything consistent from year to year. But I think it's a bit annoying, actually. It's for the H L M I one and not the F A O L one. We said to assume that like scientific progress sort of continues as normal. Like, there's not some giant impediment to that. Those are roughly the two definitions. I guess, actually, sorry. For that full automation of labor question, though, the question is different in that first we describe what full automation of an occupation is. And then we go through a series of questions that ask people about different specific occupations. Like, we give them four occupations and ask when they think those might be fully automatable. And then ask them to think of a particularly late occupation and when they think that will be automatable.

• 15:57Speaker 00

  And then ask them when they think everything will be automatable. So I think this, leading them through the process in steps could also lead to a very different answer. Maybe that's what we're seeing.

• 16:07Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 16:11Nathan Labenz?

  That's interesting. I wonder if it would be useful to just open up a, like, this is the experience of the, the survey. Is there any place where people could go and actually just take it themselves or like experience what the? And they may not qualify necessarily for your population, but for folks like me, for journalists who want to understand what actually happened here. Is there a version where people can go see the actual step by step?

• 16:34Speaker 00

  I think there isn't, actually. Sorry about that. So you can, you can go and look at all of the questions, but they're all just like in a PDF. I think a difficult thing with just seeing what it's like is that there is so much randomization in it that you're going to get one particular path through it, which is still probably informative, but you would miss most of the questions.

• 16:52Nathan Labenz?

  Okay. So we've got high level machine intelligence, which is when unaided machines can do all tasks. And then we've got what on the surface seems like a pretty similar concept of full automation of labor. But those do lead people to quite different numbers. Let's talk about how the numbers actually get collected. Again, there's two frames here. And I think the, the difference is, is also pretty interesting.

• 17:15Speaker 00

  Yeah. If you want to ask a person about like where a thing is going to happen and you want them to give you a distribution over time, that's sort of going to look like, well, there's a. Very low probability of happening tomorrow and a higher probability that it will happen by the next day and so on. So we're trying to somehow get from them a distribution of probabilities increasing over time. And so we wanted to do this by getting three different year probability pairs. And then like drawing a line through them to make a probability distribution that we're sort of guessing is roughly close to theirs. And so the two natural ways to do this are to give them years and ask them for the probability that they think that it will happen by that year.

• 17:58Speaker 00

  Or to give them probabilities and ask them in what year that probability will be met. So in 2016, we we split people in half and gave half of them one set of questions and half the other. And they very consistently give different answers to these two things. We also tried it on people from Mechanical Turk, random survey takers for money, and they they had pretty similar patterns. Where in particular, if you give years and ask for the probabilities in those years, you get a later distribution than if you say. And in what year will there be a 90% probability of this? So given that they're pretty consistently different, we keep on asking them in both ways each each time. Because we don't want to just pick one of them and go there because we know that it's biased.

• 18:43Speaker 00

  So we want to do something in between the two of them. So we ask half each way and then turn them all into probability distributions. And then put the probability distributions back together again and average them.

• 18:53Nathan Labenz?

  So let me make sure I get this working backward. So the idea is at the end, you want to be able to say, we have a curve that represents the elite ML researcher communities aggregated sense of how likely these different technology phenomena are to happen over time. And ideally, that'd be a nice smooth curve. So we can make sense of it and look up any year and whatever. Now, to get there, we have to get everybody's individual take. But to ask people to draw all these curves is pretty tedious. Requires an advanced interface. You can do that sort of thing to a degree on like Metaculous, but it's not super lightweight.

• 19:35Nathan Labenz?

  So instead you say, well, okay, if we have three percentage year pairs, then that's enough. We can fit a gamma distribution. You can maybe tell me why gamma distribution versus a different distribution. But we will fit a formula to that. And then we can do that for every user. And then we can basically take the average of all of those fitted distributions. And that will become the final aggregate thing. And then for one more layer of complexity, the pairs themselves could be generated in different ways. You could say, in what year will there be a 50% chance? Or basically amounts to like, what's the over under, right?

• 20:16Nathan Labenz?

  If you're a gambler, it's like, you give me the year that which you would accept even odds.

• 20:21Speaker 00

  Exactly.

• 20:22Nathan Labenz?

  And the flip side of that would be, I give you a year and you tell me a percentage.

• 20:27Speaker 00

  That's right. Yeah.

• 20:28Nathan Labenz?

  And which direction was the bias? I didn't catch which one gets the earlier the later.

• 20:33Speaker 00

  If I ask, what odds do you give in 20 years? That will get you a later curve. So things happening later. So I guess my guess is that it's something like for pretty wild things. You're inclined to put low probabilities perhaps. So if you're given different years, you could just keep on putting lowish probabilities. Whereas if someone says like, when is it 90%? Then you still have to give a particular year. Once you're accepting that it is a particular year, unless you decide that this thing is just not going to get that likely. Then it's not particularly tempting to put it extremely far out. I don't know if that's right at all. That's how I remember which way it goes because I'm somewhat suspecting that's happening.

• 21:14Nathan Labenz?

  Yeah, interesting. Okay. So going back to the gamma distribution for a second, what can you tell me about that? I'm not very statistically sophisticated. One question that comes to mind is, does it necessarily reach 100% at the end? I don't really know anything about what is implied by this particular choice of distribution.

• 21:34Speaker 00

  Yeah, I'm actually also not statistically sophisticated here, which is one reason I have colleagues. Yeah, I'm not sure. I think that it doesn't have to go to 100.

• 21:42Nathan Labenz?

  It's certainly a very thoughtful approach. Reading through the paper, I was like, two framings of what it is we're talking about on these extreme superpowered relative to today's AI systems of the future. That I think is healthy, just sanity check. It is notable that there's quite a difference between those. The flipping of the percentages in the years is also really interesting. And it's definitely just a super thoughtful approach to try to find some way to get like a continuous distribution from relatively sparse data across people. So I came away from it feeling like it was better than I would do, that's for sure. And of course, all these sorts of things are going to have their artifacts or their weaknesses or their points of question. Would you give any caveats?

• 22:27Nathan Labenz?

  If somebody used to say, what's the biggest problem with the way that this data has been gathered and synthesized into this aggregate view? What would jump out to you as the biggest problems with it?

• 22:37Speaker 00

  Not sure about the biggest problem, but one problem I was thinking of then is like, I feel pretty good about us having asked things in various different ways because I think in fact we do see substantial framing effects or different answers for some reason for what seem like fairly similar questions. So I think that's pretty good to get. But I think that I wasn't thinking about when we were designing this in 2016 because we basically had a very similar survey from time to time. Is that, you know, if you ask a thing in four different ways or something, then it really opens it up to reporting biasedly. Like you really can choose one of the things and advertise it. And even if we don't do that, other people can do that.

• 23:17Speaker 00

  And so I think I'm pretty keen on not doing that, but I do end up having to fight a constant battle to get like the H L M I answers and the full automation of labor answers. I think it's like very tempting to just write about the H L M I answers. Partly because the full automation of labor answers are so late that maybe they seem like more ridiculous or something. Especially to people who feel like AI is happening very soon. So. Tempting to just look at A I online and be like, oh, and it dropped a huge amount. You know, this is all happening fast. When if you look at all of our answers together, it's like quite a strong counter signal saying that people are expecting this in quite a long time.

• 23:56Speaker 00

  And so I think, yeah, all of this kind of like different questions about everything makes that quite hard. And it means that you have to police it yourself, perhaps, and it might make it harder for other people to trust that you're doing that well. Which I I try to mitigate by making sure that just everything is online somewhere so people can check. In terms of overall worst problems with it, I think just the the fact that forecasting the future is quite hard. And these people are not forecasting experts even. And they're often answering it in sixteen minutes or something. It's a lot of questions. The questions are about really complicated, like what will the long term consequences for

• 24:38Speaker 00

  the world, a pretty complicated thing, be of this technology that we haven't seen at all. I don't expect the answers to be very accurate, but we are having to make decisions about these things. And so I think inaccurate answers, that the best guesses that we can get are valuable. And so it's important to hear what A I researchers think about this. I also think that just knowing what different people genuinely think about it is important for, you know, coordinating. But yeah, that's pretty different from these people are probably right, it probably will be the year that they say.

• 25:09Nathan Labenz?

  Yeah. Well, as we get into results, it'll also, you know, become very clear that there is a wide range of opinions. And certainly in the aggregate, a lot of uncertainty, which means that the accuracy question is probably less central anyway, because it's not like it's a very tight estimate that we're getting, right? It's a pretty broad range of opinion.

• 25:29Speaker 00

  I think it's often good to to be clear about what kinds of updates you might make from this. I I think that you can strongly infer that some important things are not ruled out. Like, you might think, these people are experts, they perhaps know that this crazy thought I have is not plausible at all. You know, with with some areas that you you can get to that sort of conclusion. I think here, the fact that a lot of these A I researchers put some probability on various things is worth paying attention to.

• 26:02Nathan Labenz?

  Yeah, well, let's get into it then. The results are definitely interesting. So I thought maybe we could start a little bit with some general, like, higher level characterizations and then work down to some very particular questions. At a high level, a couple of things that jumped out to me about the distributions and I'll you expand on this or add your own commentary as to what has stood out to you the most. But it seemed that this is pretty consistent, I think, with other things I've seen that there is kind of a left-heavy nature to the distributions. Like, the timing of all the tasks, right? There's 39 different tasks, you know, when will a AI system be able to do this task?

• 26:44Nathan Labenz?

  And we can start to list them in a minute. But across the vast majority of them, I would say, the range of the first half or the second quartile as compared to the third quartile is just much more compressed. So it seems like a lot of people are like, yeah, there's a decent chance that this might happen in the kind of near future. And obviously, there's varying definitions of near for different levels of difficulty, but there's like a pretty decent mass in the kind of near to midterm future for a lot of things. And then there's a really long tail that goes like far off into the future. That seems to be a broadly true statement about all the different answers. Is that fair?

• 27:22Speaker 00

  Yeah, I think that seems right. I guess I haven't thought a huge amount about this, but I wondered to what extent that's sort of what you get if you're predicting any kind of thing where you sort of think it will happen soon. And it hasn't happened yet. So the bit between right now and your kind of 50% it will have happened date is kind of all smoothed up. But then if it hasn't happened by then, there is the rest of eternity to spread it over somehow.

• 27:48Nathan Labenz?

  Yeah, I think that's right. In everything I've seen like this, this shape is. Seemingly a norm, another way to say it is the median is sooner than the mean. And intuitively, that makes sense because you have a long tail possibility in the future. And obviously, you're not going to be predicting the past. So fundamentally, asymmetry kind of always leads to that. But basically, I would say for folks who are trying to develop a mental picture, and you can look in the paper or go look at like some of the AI questions on Metaculous, you can see actual curves that very much have this shape. There's like a big bump in the relatively short and midterm. And then there's a very long tail that goes on far into the future.

• 28:29Nathan Labenz?

  Another big observation for me is just that the ranges are super wide. I'd say maybe the headline, a figure in the paper is one that shows the middle two quartiles, that is, the 25th to the 75th percentile range of expectations of the timing of all 39 specific, you know, abilities that you ask about. And the range there is often huge. There's pretty high disagreement or in aggregate, you could say basically radical uncertainty about the timing of a lot of these key things. It's not like we're talking, this is 12 to 18 years out. It's this is 10 to 100 years out in some cases.

• 29:14Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:18Speaker 00

  You said disagreement. I think there is disagreement. The note that the lines we're looking at here of when milestones will happen are actually the averages of everyone's distributions. So they don't actually indicate disagreement so much as like within each person, a lot of different things being possible. So if there's a very long line here between 25% and 75%, it's that the distribution you get by averaging everyone's distributions together is very uncertain.

• 29:49Nathan Labenz?

  Okay, that's a key point. So I think there are some other questions that may better establish disagreement on some key questions, but this is, just make sure I can repeat this back to you. You're averaging individual distributions. So these ranges reflect the fact that the average individual, so to speak, has provided a very wide range.

• 30:12Speaker 00

  Yes, I think that's right. I think you could also get this from lots of different individuals providing narrow ranges, but at different times than they were when they got averaged together. I think you would get a flat distribution, but it at least very well could come from a lot of people having flat ranges. There are figures of like for H L M I or full automation of labor, what the overall curve looks like and what some of like a random subset of the individual curves look like in the background. And so I think you can see it there that for those at least, the individual curves are all over the place. Like they're not in agreement, but also they're quite often very spread out. So like each person is quite uncertain often for many of them,

• 30:54Speaker 00

  but also there's a lot of variation between the people. My guess is that that's also similar for the narrow tasks.

• 30:59Nathan Labenz?

  Yeah, it's interesting. I really like that visualization. This is figure three in the paper that you're referring to, right? I love this kind of graph. It's the bright color line that represents the aggregate and then all the faint color lines behind it. But as I'm looking at it now, for the individual predictions, it is seemingly fair to say that if you have a very steep line, you are expressing high confidence. If you have a very slow upward sloping line, you are expressing very low confidence. It seems like there are some people that are expressing high confidence. Most of those people seem to have high confidence in the relatively shorter term. You don't see too many lines that are super steep that are late in the time range.

• 31:43Nathan Labenz?

  The lines that are far out, that are rising slowly through time, these are like pretty high uncertainty. It seems like is a mix of more confident people who have shorter timelines and less confident people who have just kind of, I don't know, but it seems like it's not soon timelines.

• 31:58Speaker 00

  Yeah. I feel like they're a decent number. They're like, I don't know. And soon is like about as likely as later, sort of gradually is sort of flat, where there are some that are more like, you know, flat for a bit and then go up after a while. I think it would be kind of strange to be like, definitely won't happen for the next hundred years, but they're highly likely to happen in the 10 years after that. But that would be a sort of weird, optimistic state to be in.

• 32:20Nathan Labenz?

  I mean, you do see a couple of those in the graph, particularly with the full automation of labor. There are two that like shoot up in the 2100 to 2150 range. So yeah, that is somebody who's saying. When would you give it a 10% chance that there would be full automation of labor? And this person said 2120. And then you said, when, when is there a 50% chance? And they said 2125. And when is there a 90% chance? And they said 2130. And it actually might even look a little tighter than that. But so that is strange, but we don't see much of that. Like typically the steep lines are broadly very front loaded where people are like 10% in a few years, 50% in a few more and 90% in a few more.

• 33:03Nathan Labenz?

  You have a few of these oddballs that sort of have a tight range in the distant future.

• 33:08Speaker 00

  Or possibly misunderstood the question or something. We tried to filter out people who are clearly misunderstanding it, but I think there are some confusing cases.

• 33:16Nathan Labenz?

  Broadly left leaning, that's kind of an inherent function of this sort of forecasting dynamic, generally pretty broad distributions and the mix of personas where some people have higher confidence in shorter timelines. Some people have very broader timelines and lower confidence and a few that sort of buck the trend and do something that may indicate that they misunderstood the question. The next big finding is, and this is like from one vintage of the survey to the next, that timelines are broadly coming in. So there was a 2016 version of this with a relatively small sample comparatively, a 2022, which had, I don't know how many hundred, but like a pretty good sample size,

• 34:01Nathan Labenz?

  like the kind of thing that they would do a national presidential poll with. And then this one is like several times even bigger than that. You want to summarize the general pulling in of timelines?

• 34:12Speaker 00

  I think the really notable thing is that between 2022 and 2023 surveys, there was a big drop in these roughly human level performance questions that H L M I and full automation of labor, where I guess like the, the H L M I one dropped by about a decade and the full automation of labor one dropped by, you know, was it four or five decades? And I think that's pretty notable, given that between 2016 and 2022, H L M I, I think changed by about a year. So it's not just flittering all over the Place. And I guess for the narrow tasks, you also see a general drop there. I think on average, it's by about a year there, where there's some dropping by a lot and even some going the other way, but more of them dropping.

• 34:57Speaker 00

  This is like the year dropping, not like the number of years until a thing happens dropping, which you would expect to get one year less by a year passing.

• 35:04Nathan Labenz?

  Yeah. So the actual specific years that are predicted are coming in. Yeah, it's worth just reading some of these tasks, thirty nine different tasks, which each have a short name and then a full description that the survey takers get to read. I'll just give a couple of them. Short name, physically install wiring in a house. Full description, given a one sentence description of the task and given the same information you would give a human to perform this task, such as information about the house, physically install the electrical wiring in a new home without more input from humans. So that's obviously a pretty challenging task for today's AI systems.

• 35:43Nathan Labenz?

  Also, I think that definition is like pretty representative of the thirty nine as a whole, where it's broadly, here's the task. We are interested in, can you delegate that task to a AI system with basically the same ease of delegation as you would delegate to a human? The state of AI task automation today is quite different from that. I can get G P T four to do a lot of tasks. But it's definitely harder than in many cases in the setup that it is to ask a teammate to do that same task. For multiple reasons, including they don't have a lot of context, they're not great with context, just broadly speaking, for multiple different reasons.

• 36:28Nathan Labenz?

  So I'm not really sure what to do with that as I try to understand these results more broadly. But I do notice that there is a a pattern in the questions. It's like, can you treat the AI system roughly as a human? Give them kind of terse instructions, a little bit of context, here's the blueprint, go. Whereas today, what I typically tell people is, we can probably save you ninety percent time and money with task automation. But you're going to put the ten percent in upfront to set up a system, gather the context, do validation, workshop your prompt, maybe fine tune a model, whatever. It's definitely not nearly as easy to do that delegation to the AI, but you can still kind of get there.

• 37:11Speaker 00

  I would have thought that at the moment that it wouldn't be. Largely upfront, it would be like with a lot of like input along the way. Like AI at the moment isn't able to act like an autonomous agent that's trying to do something for a long period of time in a useful way without you giving more input or redirecting it. Like you're the one kind of directing the task usually, at least in my experience of doing tasks. So I've seen people use it for lots of different things. But I don't, it's like, I'm writing a thing. I'm like, I wonder, you know, are there examples of this thing? I'll ask it. And I'm sort of like directing small bits to it.

• 37:44Nathan Labenz?

  Yeah, I think this is a very important practical distinction in AI task assistance or task automation. The flow that I'm describing there is one where there is some scale at which the task is meant to be completed. And the goal is that you would get the AI performance to a level where you don't have to review every single AI task execution once you're satisfied with the level of performance. So I usually distinguish between ad hoc, real time, co-pilot style usage, which is, you know, I'm writing something, can you help me edit this paragraph or whatever? But you're not doing that task at scale.

• 38:25Nathan Labenz?

  And on the contrary, the classic tasks that almost everybody has, certainly in business, would you like to personalize correspondence at scale? Yes, I'd love to. However, who has time to do that? Well, AI does, right? Now, can we take your database and understand what kind of personalization matters and set this up and take the first hundred results back to your copywriter and make sure that it's working on that level and whatever. Every different situation has different kind of requirements. But also a big part of that is just conceiving, how are you going to break the tasks down in the first place, right? That is a big part of it. So I guess maybe that's really the the key distinction between kind of what in practice is done today and the paradigm that you're sketching out in the survey questions.

• 39:09Nathan Labenz?

  It's like, who is responsible for breaking down the task into these like subtasks, each of which could be developed and validated, whatever. In the survey questions, that's on the AI. It's not the the human, generally speaking, is not responsible for getting super granular and really controlling. It's supposed to be high level. Delegation of the sort you would give to a teammate, not the sort that you would give to GPT-4 today.

• 39:36Speaker 00

  I think it varies between tasks, but at least for some of them, they're getting pretty far in that direction. I guess my model of such things is probably kind of fractal-ish. You can have some skills that allow you to do a second's worth of useful work before someone has to redirect it. If you have a bit more skill or some other kind of skill, then maybe you could put together three different pieces, but still like someone above you has to be like, all right, now is the time for that and so on. Like, it might have been that in the past, I could use a thesaurus. And it's like, all right, I know that I want this particular task done. I can check. And like now, ChatGPT can do several things like that at once without me telling it what I want. Like I say, I want you to write this letter or something and it can, you know,

• 40:20Speaker 00

  know that it should think about different words for this place and know that it should think about how to be polite at this place. And know it should think about what are the things I was going to list in the letter or something. And so it's putting together several different things, but still it's not able to be like, at a higher level directing what happens with this letter writing. For instance, I still need to be like, this is going to need some kind of quality check. And someone other than the AI is going to have to do that. And I'm going to have to figure out who to send it to, that sort of thing. And so I think I'm imagining, like, my picture of all of this is that you gradually grow toward these things as more and more stuff is able to be done by one system.

• 40:54Nathan Labenz?

  Okay, cool. I'll read a couple others just to give a little bit more concrete color. And these are not easy, right? The next one is fine tune LLM. Given a one-sentence description of the task, download and fine tune an existing open source LLM without more input from humans. The fine tune must improve the performance of the LLM on some predetermined benchmark metric. That's an interesting one because that's one of a handful of thresholds that I definitely watch out for. Another one is replicate ML paper. Given a study published at a leading machine learning conference, replicate the study without more input from humans. The replication must meet the standards of the ML reproducibility challenge. The link to the definition of that. There are a few.

• 41:37Nathan Labenz?

  In here that are certainly relatively easy, although most of them I would say are pretty hard.

• 41:44Speaker 00

  I think arguably some of them have already been done.

• 41:46Nathan Labenz?

  Yeah, I wanted to actually ask about that one, too. Now, this sort of both validates and calls into question some of the data. In my view, probably the easiest and in the aggregate view of the respondents, the easiest task that is the one that is most likely to happen soonest is write readable Python code for algorithms like quicksort from specs and examples. Full version, write concise, efficient, human readable Python code to implement simple algorithms like quicksort. That is, the system should write code that sorts a list rather than just being able to sort lists. So write code to sort a list. Suppose the system is given only a specification of what counts as a sorted list and several examples of lists undergoing sorting by quicksort.

• 42:29Nathan Labenz?

  That one, I went to ChatGPT and asked it to do a quicksort in Python for me, and I'm pretty sure it nailed it. And so you could say, well, that's like in accordance with the data and that the results there were the soonest. But it still seems like a full 25% of people said that that wouldn't happen until 2029 plus. And that is one area where I was like, huh, are like a quarter of people like not aware of ChatGPT? Or are they like interpreting this differently and sort of generalizing to like other algorithms? I mean, if I'm trying to like steel man the case here would be like, well, quicksort's in the training data. Maybe this would could be understood as like for new algorithms that you have examples of that are not in the training data.

• 43:14Nathan Labenz?

  Do you have any sort of way of understanding how that result makes sense?

• 43:18Speaker 00

  Not fully, but I have some thoughts. One is, I think you're misunderstanding the, it's not the 25% of people thought that. It's more, it's that everyone together, average together thinks that there's like a 50% chance of this happening. What is it like a couple of years in the future or something? In my experience, like demonstrating that one of these things has properly been done is surprisingly tricky. Especially if, maybe that one is pretty easy, but yeah, if it's like, can it consistently do this across the board? Can it do it for other things, basically the same as quick sort? I didn't think in the question we really said what fraction of the time it has to succeed at it or something. Might be like, in my use of chat G P T, say, my experience is that it's often great at things than often terrible at things that I would have thought it was great at.

• 44:04Speaker 00

  Last I checked, it wasn't very good at counting things. It was like, how many ones is this in a row? It's like, not very good at that. And so I could imagine like, without actually going and trying the thing right now while you're doing a survey, you're like, well, this seems like the kind of thing it can probably do. So probably sometime between right now and a couple of years or something. I think also the fact that we included it on the survey, I think they might take as evidence that it can't be done right now. We decided not to take any of them off since twenty sixteen, partly because it's just so complicated to figure out if they've actually been done or not. So we decided to not have any opinion on that and just include them all and maybe take them off once the respondents start to actually say this has constantly happened instead of like, I don't know, maybe five years.

• 44:48Speaker 00

  But I think that might be confusing people.

• 44:50Nathan Labenz?

  Yeah, isn't that an option? There's not an option today, right, to say this already exists?

• 44:55Speaker 00

  Ah, there isn't. So yeah, quite plausible. We should add that and that would make things less confusing.

• 45:00Nathan Labenz?

  Zooming out again, on thirty five of the thirty nine tasks, we have a fifty percent chance that it will happen in ten years or less. So here's the four, just to calibrate on the ones that were greater than ten years. One was the installation of the electrical wires. That was estimated fifty percent at seventeen years. The M L paper, there's a replication one that I read that was estimated at twelve years, fifty percent chance of at twelve years. And then there's also a research and write, in other words, do your own M L paper from scratch. That one is nineteen years. And then the farthest ones out were. Around math. Interestingly, there's some interesting proof points there as well of late,

• 45:45Nathan Labenz?

  but prove mathematical theorems that are publishable in top mathematics journals today, 22 years out, and solve longstanding unsolved problems in mathematics such as the millennium prize problem 27 years out. So these are the hardest ones with the longest timelines, things like Python, things like playing Angry Birds at a human level. The large majority of these things are under 10 years, 50% chance. I would say another kind of observation, I wonder how you would react to this, is it seems like the timelines here, while they have come in, and while like most of these things are more likely than not to happen inside of 10 years per the aggregate judgment, it seems like the timelines here are still longer than the guidance that we're getting

• 46:30Nathan Labenz?

  from the heads of leading labs. Like, I think if Sam Altman, if Dario, if Demis and Shane Legg took your survey, I think their numbers would be on the shorter end of the aggregate.

• 46:42Speaker 00

  That would be my guess. I think also just from many people working in AI who I know here in the Bay Area.

• 46:48Nathan Labenz?

  Yeah, I don't think we have three-year percentage pairs from the heads of the leading labs.

• 46:55Speaker 00

  I can't tell you if we do or not.

• 46:57Nathan Labenz?

  Well, here's things that I've seen recently. Sam has said AGI is coming soon, but it might not be as big a deal as you think. He also said at a Y Combinator event that startup founders now need to build with AGI in mind, which is certainly an interesting app design challenge. Anthropic, we don't tend to hear quite the same thing too often, but there was the, I think, credibly sourced pitch deck that they had where they said that in 25, 26, the leading model developers might get so far ahead of others that nobody will be able to catch up because they'll have their own kind of feedback loops where they can use their current models to train the next ones. That certainly sounds like a scenario where things are happening faster than the aggregates from the survey. And Shane Legg

• 47:42Nathan Labenz?

  from DeepMind recently said, That he's basically had like a 2029 median timeline for like 15 years.

• 47:49Speaker 00

  I think it's interesting to note that having had that timeline for a while, and similarly, that was for some of these other people. I think you could wonder, like, hearing very optimistic timelines, or optimistic or pessimistic, depending on how you feel about this, but like very soon timelines. You might wonder whether that's from seeing something right now that suggests that it will be very soon versus being kind of selection effect where people who think that AGI is soon work on AGI. And I think where we observe that people already had these views some time ago, that's support for the selection effect explanation of the difference in opinion there.

• 48:23Nathan Labenz?

  Yeah. So for comparison, the survey gives a 10% chance of high level machine intelligence by 2027. And for a 50% chance, you have to go out to 2047. So that's pulled in significantly from just one year previously. In 2022, it was 2060 that you had the 50% chance. So from 2060 to 2047, 13 years in, but that's still quite a bit farther than certainly what we're hearing from all the leading labs. Like what the survey says is sort of a 10% chance. It seems like the leading lab heads think it's like more likely than not on a similar timeframe.

• 49:06Speaker 00

  Matches my impression. Yeah.

• 49:08Nathan Labenz?

  Yeah. So who do we believe? Obviously, very hard to say. Yeah. One thing that you've done that I think is pretty interesting is that I might even actually spend a little more time on this is you've published all the results in cleaned up anonymized form. So folks can go do their own data analysis on it. So would there be anything stopping me from taking the results and saying, I'm going to go filter out anybody who thinks that the Python thing is not close? And rerun the analysis with those people filtered out. It is hard for me to put a mental model on. I don't know how many people I'd filter this way, but it's me. There's got to be a fraction, like a meaningful fraction that might be skewing the tails where if it's like you published in

• 49:51Nathan Labenz?

  the leading conferences in 2022, but you are saying that an AI won't be able to write a quick sort for a long time. Like I am kind of confused about that. Maybe I should just filter you. I could do that, right? With the data that you've published.

• 50:02Speaker 00

  I thoroughly encourage that. I would love to see more people like use it. And I think there are a lot of interesting things you could ask about, and it would really only touch the surface with giving the basic answers to each question and some amount of how does this relate to demographics? Yeah, I think how do the answers relate to each other? A lot of interesting things there. I think if you filtered out the people who said that quick sort wasn't possible for at least 15 years or something, I'm not actually sure what you would be getting there. I think you might be filtering out a mixture of people who didn't understand the question or just made an error on that one, or who have some sort of complicated philosophical take like it's not doing quick sort. It's doing matrix multiplication or something.

• 50:43Nathan Labenz?

  It's not doing real quick sort. It's just doing sort of stochastic period imitation of quick sort.

• 50:49Speaker 00

  I don't know. And then I guess if you filter out those people, I don't know if they're more likely to be wrong overall about other things. Yeah, but I feel like they're more likely to have thought about things probably. We did actually just ask people how much they'd thought about different things. So I think a very natural thing to do would be to see what just the people who said they thought the most about things think. Though I guess that's also tricky because of the selection effects with people who are more concerned about things, thinking about them more. So I think in fact, the people who've thought more do seem to be more concerned. But I'm not sure what you should actually take away from that.

• 51:21Nathan Labenz?

  I'll, I think we'll turn to that correlation question in a second. Just carrying on through a few more headlines. It was definitely striking to me that people are expecting the unexpected. Here's a quote, "A large majority of participants thought state of the art AI systems in 20 years would be likely or very likely to find unexpected ways to achieve goals. More than 80% of people expect that in 20 years. Be able to talk like a human expert on most topics. Again, more than 80% expect that." And frequently behave in ways that are surprising to humans. Just under seventy percent of people expect that to be the reality for AI systems twenty years from now.

• 52:03Nathan Labenz?

  I have to say, I broadly agree with all those things, but that is definitely a striking result, right? That people are expecting ongoing surprises from AI systems as their default. That seems like the closest thing probably to consensus in this survey is that everybody expects these things to be unwieldy and surprising, even as they become like quite a bit more powerful over a twenty year period.

• 52:25Speaker 00

  Yeah, that does seem like one of the more consensus-y things, at least. Yeah, I guess I don't know like how sinister to hear some of these things as like finding unexpected ways to achieve goals. At some level, that's just like what you expect if you are getting someone else to achieve goals for you. You know, you're not going to figure out all the details of how to achieve it. They're going to do it and you'll be like, oh, nice. You figured out a way to do that versus being quite surprised and being like, well, that was a norm violation. I thought I should put up more barriers to prevent you doing that, but apparently you did. Yeah, similarly for the sort of frequently behave in ways that are surprising to humans. I feel like that could be more terrifying or more just what you expect. So yeah, I would have liked it if we'd been clearer with those questions a bit.

• 53:09Nathan Labenz?

  Yeah, maybe there's a way to tease that out a little bit through correlating with the next headline result, which is, to my eye, the group is only slightly more optimistic than neutral. Basically, you've got a few of these kind of classic questions where you're like one to five, like extremely bad, bad, neutral, good or very good. And just ask people, how likely do you think this is to be in these various buckets?

• 53:34Speaker 00

  To be clear, this is like, how good or bad is the long-term future as a result of HLMI in particular? Yeah, each person puts a hundred percent between the five buckets.

• 53:44Nathan Labenz?

  And there, I would say this is like the clearest sign of radical uncertainty because all five of the buckets have significant percentages and they are skewed slightly positively.

• 53:56Speaker 00

  It looks to me like a fair amount of agreement on extreme uncertainty. There's a good bulk of this graph that is people putting a chunk of probability in each of the five buckets and just a little bit at each end where people are either very confident in good or very confident in bad. And this is a question that everyone in the survey got and everyone had to answer 100%. They had to answer it in order to get to the next question. So basically, everyone answered this. Uh, yeah. Each column is one person.

• 54:23Nathan Labenz?

  Yeah, okay. And this is figure 10 in the paper. This is definitely a cool visualization as well. It's a sorted list where each person gets a vertical pixel. And then you can see how each person distributed their expectation from extremely good to extremely bad. And because it's sorted, you have an extremely good region on the one end and an extremely bad region on the other end. And in the middle, you can see that, yeah, like a lot of people have given non-trivial weight to all five of the buckets. And like the middle sort of two thirds of people probably have agreement on radical uncertainty. And then there's like maybe a sixth on either end that are sort of the optimists and the pessimists.

• 55:04Speaker 00

  Even like most of those are like pretty uncertain. Even if you look at the, the big black chunk of pessimists on the right hand side, still like most of their area is not extremely bad. It's a decent chunk of extremely good in there.

• 55:17Nathan Labenz?

  Yeah, you got a few bimodalists. I, I don't know if I quite do call myself a bimodalist. Sometimes I do, but I'm not quite ready to stake my identity on it. But there is definitely a band there where you only see the two colors and it's the extremely bad and the extremely good.

• 55:32Speaker 00

  Yeah, I'm also seeing like a 2080 band like that. 20% extremely bad, 80% extremely good. Yeah. Okay, I like seeing that, like how polarized this isn't. I think this kind of thing can feel polarized. Our people are talking about doomers and so on, but it's, I think, very not.

• 55:51Nathan Labenz?

  Yeah, totally. Definitely suggests a very big middle of people that are just very uncertain as to what to expect. And those people are probably, for many obvious reasons, not the most vocal online. But I think a huge part of the value of this work is demonstrating that there is no single consensus, there is no dominant view, there's no single number that we can put on this. But really, for me, the headline is just there is radical uncertainty expressed like any number of different ways by the community at large.

• 56:22Speaker 00

  Radical uncertainty is in some sense a consensus about something like, like I think here you could say there is more or less a consensus that there is like non negligible probability of extremely bad outcomes. And from a sort of action perspective, if the question is yes or no, is this a risk worth paying attention to, then it ends up being sort of like a consensus for yes, worth paying attention to, because uncertainty is like, yeah, there's some chance of it. And you'd actually need to be pretty confident on something to say, no, not an issue.

• 56:52Nathan Labenz?

  Yeah, I think that's a good point. The headline that I've seen from the coverage of the survey broadly has largely focused on what you might call the P doom point estimate. And I've seen this most often reported as a majority, and it's a slight majority, 51% believe that there's at least a 10% chance that AI could lead to human extinction or similarly severe disempowerment.

• 57:18Speaker 00

  Two things. I think the thing that I've most said at least is the median 5%, but also I think that 10% would be wrong here because what we asked this question about values that we're just talking about, then we asked three other questions that were quite similar about human extinction in particular. So table two shows the three different questions that we asked that are all quite similar about human extinction or similarly permanent and severe disempowerment of the human species. One of them is just straight up asking about that. One is asking about that as a result of inability to control future advanced AI systems. And one is asking about it within the next 100 years in particular. And so the medians for those were like 5%, 10%, 5%. And I guess the means were quite a bit higher.

• 57:59Nathan Labenz?

  So that headline is cherry picking the highest median of those three. It's interesting too. This does show some of the limitations of both surveys and interpreting surveys. I think it's very interesting framing, but when you really start to stare at it, you're like, well, there's definitely a sort of conjunction fallacy at work here, right? Like the first question says, what probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species? Median answer, 5%. The next one is essentially the same question, but with an added detail of inability to control. And as far as I'm seeing here, every other word is the same and the percentage chance doubles, right? So if you're like logically consistent entity, presumably the second one would be like lower, right?

• 58:44Speaker 00

  I think it's not necessarily coming from the conjunction fallacy, though I think it may well be. But things to note here are like people are randomized into each of them. So it's not like the same person answering them. So it could just be some variation from them being different people. But I think often a lot of people put 5% and a lot of people put 10%. So whether the median is 5 or 10% is down to like exactly how many put where the middle person lands. There are like a bunch of people putting 5%, a bunch of people putting 10%. And exactly where the middle is changes it from 5 to 10%. But it seems wrong to be like, ah, they doubled it. The the distribution of people thinking things is similar. Last year, I remember 48% of people said 10% extremely bad. So it's like, if it got up to 50%, then the median would be 10%.

• 59:28Speaker 00

  But because it was 48%, the median is 5%.

• 59:30Nathan Labenz?

  Gotcha. So these are bucketed answers. In this case, we're not doing a distribution or a free response point.

• 59:37Speaker 00

  Right. Or so we're asking each person, what is the chance of this? And in fact, people just never put numbers between 5% and 10%. They're sort of rounding it themselves or not never, but like quite rarely. So I think in practice for these questions, the median jumps fairly easily between 5 and 10% rather than hitting intermediate values. I do still think that probably some amount of conjunction fallacy is going on here, partly just because we saw the same pattern last year.

• 1:00:05Nathan Labenz?

  Again, to attempt to summarize the the headline, P doom point estimate is 5 to 10% median estimate. With certainly, again, a kind of left heavy distribution and the mean being higher than the median. And that's presumably driven by a minority of people who have high estimates. And you can see that again in the other figure as well. Well, that's definitely something that's pretty consistent. I would say with my own, I don't spend a lot of time trying to narrow down my P doom. I thought Demis did a pretty good job answering this question. He did the New York Times Hard Fork podcast recently and they asked him what his P doom is. And he was like, it can be a subtle distinction and people want to collapse the distinction. But he's like, we really just have no idea.

• 1:00:50Nathan Labenz?

  It's not like we have a process that we have established that has any. Or attached to it that we're now going to execute and find out on the contrary, we really have no idea what the nature of the process is that we're going through. And so it's just like super unknown. I typically say either five to 95% or 10 to 90%, something like that. And certainly the consensus, at least the low end of that range is like not objectionable to the the large majority of people.

• 1:01:18Speaker 00

  Pretty in line with them. I disagree with you or I agree it's very unknown, but I think that's what probability is for. I'm a big fan about putting numbers on it anyway. And I think that even if you can't do anything except guess a number, it at least allows you to compare to other numbers you're guessing or like make consistent decisions over time. I I practice guessing numbers about lots of things and then I can check how how well that's going. And so whether to listen to the numbers that I make up about this is very uncertain, but I think it's worth trying.

• 1:01:45Nathan Labenz?

  Yeah, I certainly find a lot of value in this work just in grounding the idea that this is clearly not a fringe question. It is clearly worth thinking hard about. The Overton window is wide open and it should be wide open if the elite researchers in the field think that there's a five to 10% chance of extinction level bad outcome. So I think that's like definitely really true. I guess the way I tend to think about these numbers is what can we shift the true probability to? As opposed to how can I become more accurate in my estimate today? And I think both of those are worthy pursuits, but especially for somebody in Demis's position. He's like, what I need to do is create an agenda that collapses the uncertainty and hopefully moves the distribution.

• 1:02:29Speaker 00

  I agree that like trying to put numbers on such things does potentially get you into a headspace where you feel like you're not controlling the number. And I do think that's bad.

• 1:02:37Nathan Labenz?

  Yeah, because we're not like about to spin the roulette wheel, right? Like there is no roulette wheel that we can like spin now with these numbers. I think that's the main thing that I think is worth keeping in mind as a caveat, right? We're still building the wheel and we do, we have a lot of uncertainty as to what the roulette wheel looks like and. The more concretely it comes into focus, then presumably the distributions will start to narrow, although maybe not. It certainly wouldn't shock me at this point if we have just super high level of uncertainty and disagreement right up until a phase change hits and then it's like, well, we just found out and it wasn't ever very clear until relatively shortly before it happened. Again, it's just another dimension of pretty radical uncertainty.

• 1:03:17Speaker 00

  I think when you're talking about like the probability of a thing conditional on us following some particular path is a reasonable way to do it, like not, you know, P doom very high, we're going to die or something. More like if we, the humans are building this particular thing, I think that particular thing will be quite bad. Maybe we should do this other thing instead of maybe a more like proactive way of thinking about it.

• 1:03:38Nathan Labenz?

  Unfortunately, one kind of downer result is interpretability breakthroughs are not expected in the short term. Most respondents, 80%, considered it unlikely or very unlikely that users would be able to understand the true reasons behind AI decisions in 2028. Pretty logically, I would say there's a strong agreement that AI safety broadly should be more prioritized than it is today. Very few people saying it should be less prioritized than it is today. Also, should AI be going faster or slower? Again, only 500% of people said it should be going much slower than it is today, but then a healthy relatively even

• 1:04:18Nathan Labenz?

  30% on somewhat slower, 27% on current, 23% somewhat faster, and 15.6% much faster. So am I right to read that as high agreement that we're not expecting interpretability breakthroughs, high agreement that more AI safety work would be good, but high disagreement as to whether we should be slowing down or going faster?

• 1:04:41Speaker 00

  I think that seems right to me. I think one complication interpreting that last one is that the the central option I think is like relative to the current speed. So the most natural and literal interpretation is like, whatever the current speed is good, but like if things were to accelerate, you would want them to go slower and you might think they would naturally accelerate. My guess is that people weren't thinking about it for that long and what they actually mean is more like that the current trajectory is good, which is perhaps accelerating. But yeah, I think before you read too much into that one, it would be nice to ask you some related questions again or something to get it more clear.

• 1:05:16Nathan Labenz?

  So let's summarize this whole thing. I guess, high-level summaries. People do not dismiss existential risk. On the contrary, point estimate is like 5% to 10%. People have pretty wide uncertainty about exactly when AI will be able to do various things, though those estimates are broadly coming in.

• 1:05:39Speaker 00

  Also, a lot of them are relatively soon.

• 1:05:41Nathan Labenz?

  More than 80% of the tasks had a 50% estimate of happening in less than 10 years. So no breakthroughs expected in interpretability. More emphasis on AI safety and disagreement or confusion on faster or slower.

• 1:05:57Speaker 00

  That all seems right.

• 1:05:58Nathan Labenz?

  As a single headline, we really don't have a confident take. As a community, I shouldn't even say we, because I've not published in these conferences. I wouldn't be qualified to complete the survey. But if I'll flatter myself to be included in the community, then we, as a machine learning research community, do not have a confident view of what's going to happen. On the contrary, we have a pretty radically uncertain view of what is going to happen, how long it's going to take to get to different things, whether it's going to be good or bad, whether we're going to be able to control systems, or even whether we'll be here at some point in the future. Like, all of those are very live questions with non-trivial weight on all the options.

• 1:06:38Speaker 00

  One question we didn't talk about where I think it was interesting that there was more consensus, perhaps, is the one about different scenarios and how much concern they warranted. Not clear that they are concerned, but they think the thing deserves concern from society. The very top one there was making it easy to spread false information, e.g. deep fakes. I think more than 80% of people thought it was worth either substantial concern or extreme concern. That's an unusual degree of agreement.

• 1:07:04Nathan Labenz?

  Yeah, so these scenarios, just to read a couple of them. AI makes it easy to spread false information, e.g. deep fakes. AI systems manipulate large-scale public opinion trends. Authoritarian rulers use AI to control their population. Other comes in as the fourth most concerning scenario. AI systems worsen economic inequality by disproportionately benefiting certain individuals. AI lets dangerous groups make powerful tools, e.g. engineered viruses. Bias in AI systems makes unjust situations worse, e.g. AI systems learn to discriminate by gender or race in hiring processes. All of those that I just read had more than 60% of people saying they are either substantially concerned or extremely concerned.

• 1:07:44Nathan Labenz?

  And for all of those, fewer than 10% of people said no concern.

• 1:07:50Speaker 00

  It's either not really about what will happen, but what could happen enough that it's concerning. In some sense, for that to be the only thing that's really clear consensus about is maybe supporting your hypothesis. They don't know what's going to happen.

• 1:08:02Nathan Labenz?

  Yeah, none of these are dismissed, that's for sure. Okay, cool. Well, I love the work because it's just so methodical. As we've established through a couple of my misunderstandings, it's pretty nuanced. It's pretty granular and we can pick it apart in a lot of different ways. But it does seem that it's very well established by this result that there is a lot of uncertainty about what we're in for. There's a lot of uncertainty about even the high level of questions of whether it's net good or net bad, whether we are going to survive it or not. It is not like a doomer community, but is not a community that dismisses the tail risks either. And so with that, we head into a very cloudy future, I think. For the future, I assume you guys are planning to run this again.

• 1:08:44Nathan Labenz?

  A few things that came to mind for me I could throw at you, but before I do that, what are you considering doing differently in the future? Or what do you think are the sort of natural evolutions from here?

• 1:08:53Speaker 00

  I am reasonably likely to run it again. I think the most natural thing to do is to just run it identically to the past because it's nice to be able to compare these things. This time we added various questions that we hadn't had before, like this one about the concerns and the one about the different traits that AI systems have in the future. But it's like a fair bit of effort to add the questions and so the project can be a lot more contained perhaps. If you just have a survey and send it out every now and again. And so it's somewhat tempting to basically do that. I think also it's quite hard to cram more questions in here and still have it reasonably short. Give each question to fewer people and randomize and so on. It's very tempting if we want to do more surveying, run more surveys.

• 1:09:34Speaker 00

  I've been thinking a bit about having something about policy, but I haven't thought about it that much. Because people send us various questions they're interested in.

• 1:09:43Nathan Labenz?

  I guess if there was one thing I could ask for, and it's because multiple people have asked me, people don't know how to understand affordances. There is a ton of investment over the last year that has gone into the application layer, connecting models to databases, of course, for RAG-type implementations, connecting them to APIs and other tools. Now we've got like the memory scratchpad. A great paper on this is cognitive architectures for language agents, where they survey the full literature and try to put a taxonomy on all these things. But I do think it's pretty interesting to consider what happens when, with all this scaffolding, as it's often called in the application world,

• 1:10:27Nathan Labenz?

  or affordances, as it's otherwise called, when all that is built and it already exists and it's like working okay, you know, or even maybe pretty good with a current model. What then happens when a core model upgrade happens? And all of a sudden, it's like a drop-in replacement to a structure that has really been like built up to try to compensate for all the weaknesses. And now maybe those weaknesses drop by like an order of magnitude. To what degree does that create a situation where you essentially are like flipping a switch? And a lot of things go from sort of working to working really well. And how does that play out? I do see a lot of potential for that, but people have been asking me for, well, what are scaling laws for scaffolding? And I just find I really don't know.

• 1:11:09Nathan Labenz?

  I have this kind of broad mental picture of a lot of work has gone into it. And a model upgrade is going to make a lot of existing systems that don't work that great yet work quite well. I'm not even sure what questions I would ask, but I would definitely love to get a community sense for the relationship between core model power and affordances or scaffolding infrastructure that's already out there.

• 1:11:31Speaker 00

  Where affordances here mean something similar to scaffolding?

• 1:11:34Nathan Labenz?

  Yeah, basically synonyms. Yeah. What tools do you have access to? What information do you have access to? It's all the things that are outside of the weights that the model can use at inference time.

• 1:11:44Speaker 00

  You want to predict what will happen when the new model comes out with the scaffolding.

• 1:11:50Nathan Labenz?

  Yeah, there's a latent potential there. It is a tricky one, but yeah, the key thing that I'm wondering about is to what degree should we be expecting a step change in the actual AI impacts? When a new model gets plugged into existing enabling infrastructure? Because that seems like a fundamental difference between AI and almost every other technology is that all of the distribution infrastructure already exists. And the complements are getting built now and people are very aggressively and eagerly building all these complements in anticipation of the next upgrade. And they're tolerant of the fact that like, it doesn't really work now because they fully expect that there's going to be an upgrade that's going to make it work.

• 1:12:36Nathan Labenz?

  But then that has potentially very unpredictable consequences when all of a sudden everything turns on at once.

• 1:12:42Speaker 00

  I don't know a lot about the details of this kind of world. I would have thought that there are things a lot like that in other kind of technologies that get updated and have a widespread reach.

• 1:12:53Nathan Labenz?

  Yeah, I think so. Although in the AI application world today, there's a ton of building that is going into essentially the kind of ad hoc delegation that a lot of the survey questions anticipate as well. People are like, it would be sweet if I didn't have to ever write a SQL query anymore. And I could just ask my SQL agent to look at my database schema and figure out what's what, and then write the thing. And the same thing for web scraping and the same thing for answering phone calls and making phone calls. And it just goes on and on, right? There's research assistance and programming assistance. So it just goes to every corner. You could find a agent type product, legal work, medical research, diagnosis research for yourself.

• 1:13:35Nathan Labenz?

  And they're all like, well, GPT-4 is not quite there, but it's good enough that I can build a system and track how well it's working. And even if it's not working that well, then I know that there's like a Countdown clock on to the next thing. And that's where I feel like we're building a new capability overhang, right? There's been the notion for a long time of, well, if the chips are all out there and nobody trains the models, then someday somebody could come along and have this like massive advance. But here it's like, on the other end, it's all of the complements and guardrails and access to data and access to runtimes and the loops and the ability to cache skills. And all of these things are getting built, but the model isn't that good at using any of them yet. And so if it goes from one level to another, how does that kind of cascade through the overall system?

• 1:14:20Nathan Labenz?

  And do we suddenly move from an era where nothing is really working super well? So everybody right now can assume that they're essentially acting in isolation. Most everybody's worldview right now is like, the world is the world. I'm applying AI here. Nothing else is changing. That's the implicit assumption.

• 1:14:38Speaker 00

  Fear thinking like that's happening under the.

• 1:14:41Nathan Labenz?

  Yeah, everybody's doing that. And it is so far true that like, nothing else is really changing. But I think there's at least potential for a model upgrade to suddenly put us in a different regime where it's like, now my thing is working much better. Now I'm actually gonna go use it a lot more and send it out potentially semi-autonomously to do stuff. But at the same time, everybody else's autonomous systems are also getting unleashed. And now we have all these like autonomous systems going off into the world at the same time and potentially encountering each other. And we're definitely just not at all prepared for or even thinking about what that might look like. It could be not a huge deal if the next upgrade isn't that big. But my sense, and this goes back to the Sam Altman comment of you should be building with AGI in mind. Like my sense is that there is at least one more round of like substantial upgrades.

• 1:15:25Nathan Labenz?

  And so do we have this kind of phase change that happens suddenly? It could happen before the next survey can get run.

• 1:15:32Speaker 00

  Yeah, often technologies change gradually, even if there was a kind of big insight or something. But that's often because you're like building stuff to make use of it afterwards and takes a while. And so you're saying, maybe if we build all this stuff to take advantage of it ahead of time, because we already knew it was coming and we had this lesser version of it to play around with, that it could be much more discontinuous than most technologies when that bit gets swapped out. Pretty intro. Interesting theory. I wonder how much we've seen that with past upgrades like this. I think that my explanation here for why AI would be different to like any other technology in this regard would be just that it's like quite general. So you are using it across the board and there is the potential to suddenly upgrade a thing everywhere.

• 1:16:12Speaker 00

  But then you might expect that also to apply to the change between G P T three and G P T four, G P T two and G P T three. Maybe those were not useful enough to be building stuff around yet.

• 1:16:22Nathan Labenz?

  That's my sense. I feel like only with Chat G P T three point five, certainly with four, but probably not with three. And that whole timeframe is also like fairly condensed. It was only like three and a half from three point five first release to four first release. And three point five and Chat G P T dropped it basically the same time. So it's like three and a half months from Chat G P T to G P T four. Yeah, I just don't think before that there was really much, the gold rush or the sort of everybody goes stake out their place in the AI app sphere. Seems to have been a last twelve to fourteen months phenomenon at most. And like, certainly there was stuff going on before that, but I would say there's been a phase change of how much activity there has been.

• 1:17:04Speaker 00

  You're saying this would be an interesting thing to have a survey about?

• 1:17:07Nathan Labenz?

  Yeah, I mean, I I'm very uncertain about it. Quality people have have asked me about it. So I have the strong sense that among a thought leading population that's somewhat in some cases adjacent to like policy decision making. There is a lot of uncertainty and a lot of questioning around that particular question. To what degree should we expect discontinuity due to the plugging in of new models to existing infrastructure? You know, maybe we could use that as a first draft. But yeah, that's been the number one assignment that I've gotten from some of my highest value correspondents. But I'm struggling to model it. So that's my case for a bonus question next year.

• 1:17:45Speaker 00

  Okay, thank you.

• 1:17:47Nathan Labenz?

  I guess other areas that could be of interest, would you look at like military or U S China chip restrictions? Is Sam gonna get his seven trillion dollars? Maybe that's like outside the scope of what M L, at least publishing folks should be consulted on.

• 1:18:03Speaker 00

  There is also like, what do you do with those answers? What does someone usefully learn from these things or do differently as a result?

• 1:18:11Nathan Labenz?

  Well, escalate or de-escalate with China, maybe one key question. I don't know that our executive decision making is necessarily going to be a super evidence based in the near term, depending on who's in charge. But I do feel like that's another question where I have dramatic uncertainty.

• 1:18:28Speaker 00

  About what ought to happen or what will happen?

• 1:18:30Nathan Labenz?

  So is the chip ban going to work? Will it meaningfully slow? Or will there be like a multi-year gap between US and Chinese frontier models as a result of the chip ban? I am hearing confident takes on both sides of that issue. I've had people tell me who I think are very smart, oh, it's definitely working. They're not going to be able to keep up. And then I look at certain models and I managed to go as far as trying Ernie 4.0 and it seemed pretty good at a couple of queries head to head with GPT-4. It wasn't like obviously dramatically inferior. In fact, it seemed pretty comparable. And there's a lot of Chinese speakers at these conferences, obviously, too. So that would be interesting to know. Do they think that this sort of policy

• 1:19:14Nathan Labenz?

  will even be effective? If so, then you could say, well, if the machine learning researchers think that a chip ban will in fact slow Chinese progress, at least that could be something that the executive could take into account. But on the other hand, if they think it's not even going to work, then it's like, well, geez, now we're just sitting here escalating and the elite researchers, many of whom are in fact originally from China, don't even expect it to have an impact. I could see that being perhaps policy relevant at a super high level.

• 1:19:40Speaker 00

  Yeah, I think if I was going to run a survey about that sort of thing, maybe I would want to talk to chip experts rather than AI experts. Like you can't just run a survey of any experts to answer emails. Also, maybe I have some kind of hesitation around that sort of thing, or I'd have to think more about it. Partly just a lot of what AI impacts does is on topics where we feel broadly, just more people knowing what the situation is like in more detail will make things better. I think once you get close to things that are adversarial, have to be more careful, like you're probably helping one side or another. I don't know enough about the topic to have a strong view on what should be happening, but don't want to be contributing to. Arms race, kind of fuzzy.

• 1:20:23Nathan Labenz?

  It's a tricky one for sure. I don't have a strong view as to whether it's going to work or not. My bias, not knowing if it's going to work, is that it seems too risky. Like, the current course is definitely an escalatory course. And if you worry that the most likely flashpoint in the world is that China will blockade Taiwan or something, then preventing China from getting any of the chips from Taiwan certainly seems like it may make them more likely to do that, right? Because what do they care if chip production is disrupted, if they are getting them anyway? That's fairly crude analysis, certainly could be disputed. But if you could inform and say, hey, by the way, the machine learning research community doesn't even think it's going to work.

• 1:21:04Nathan Labenz?

  But it seems like new information, given how escalatory the current politics are, probably is much more likely to serve, if anything, as a reason to de-escalate. Just because we're already in this sort of cycle of escalation. So you could come with a finding that's like, yeah, your escalation is potentially going to achieve your policy objectives. And then people would stay the course. But I would say that it's probably more likely that you could have a de-escalatory impact in the sort of gambling analysis of, maybe I'll find something that supports escalation, but escalation already exists. Maybe I'll find something that supports relative de-escalation. Maybe that is, in some ways, more positive expected value, just because the current course is what it is.

• 1:21:47Speaker 00

  Yeah, but yeah, also, I don't know enough about it to be confident that I should want de-escalation. But yeah, as a baseline, I don't love escalation.

• 1:21:56Nathan Labenz?

  That's another big question. I do think that it could be clarifying there, too. I could be convinced, if I was really sure it was going to work, then I would be more likely to support it. You know what? The worst scenario is a highly escalatory move that doesn't even achieve its intended goals. So if it's at least going to achieve its goals, then like, maybe some escalation could be worthwhile if you believe in those goals. But even if you believe in the goals, if it's not going to work, then it's just like a total net negative. So anyway, food for thought for possible additions to the survey next year. Anything else that you want to make sure people don't forget about this work or any other kind of commentary? You wanna bring us to a close with?

• 1:22:32Speaker 00

  Amidst quite a lot of uncertainty here, I think the chance of things just staying pretty similar and nothing coming of this to like drastically change people's lives seems quite low. So even if you're not very directly involved with AI, that seems like a pretty important thing to be keeping an eye on and trying to have accurate opinions about what the public thinks about these things will affect what policies happen in this thing. Could change the rest of the future forever, a lot for better or worse.

• 1:23:00Nathan Labenz?

  Does that suggest that you'll be exploring some public opinion investments as well?

• 1:23:05Speaker 00

  I'm not sure if some other people are working on that. I think a public opinion being important is somewhat different to it being important for us to know what public opinion is. So I guess the public knowing what public opinion is, is potentially good for moving toward whatever view it's moving toward, hopefully a more reasonable one. If you start to get evidence for a thing and at first you're like, ah, nobody else thinks that. I may be able to speak quiet about that. It's helpful to be polling. What do people actually think along the way?

• 1:23:33Nathan Labenz?

  Yeah, this work is so in the weeds and works so hard on the definitions. Obviously polling the public, you would have to have a very different set of questions and a much higher level sort of gut check for many people. But I do think the common knowledge is probably pretty valuable. And I do think also for just reassuring policymakers that they're not out on a crazy limb if they want to do something. That the public is kind of with them by default, I think is probably pretty good to establish. Also, among the tech people, there's like, I think, under appreciation of how skeptical the public is. I'm not sure that the Silicon Valley set is really realistic about what like my mom's friends think about AI.

• 1:24:16Speaker 00

  Skeptical in the sense that like they, they don't like AI?

• 1:24:19Nathan Labenz?

  Yeah, there, there's a lot of people out there that are just like, my mom's one friend who I've known for my whole life, her reaction to me was like, it creeps me out. I don't want to have anything to do with it. Full stop, like not curious, not looking to automate some tasks, not looking to get help on an email draft. I don't want to have anything to do with it. You could say like, hey, they might have the same thing on Facebook and have it on there too. So I don't know that that's the final word.

• 1:24:45Speaker 00

  It is creepier than Facebook.

• 1:24:47Nathan Labenz?

  Yeah, no doubt. I do think there's just a lack of grappling with just how widespread that sentiment is. Anything else on your mind? This has been a great conversation. You've been very generous with your time. I really appreciate that and love the work. So I definitely hope to see a 2024 edition as well. Thank you for having me. Cut to your grace, founder of AI Impacts. Thank you for being part of the Cognitive Revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.

• 1:25:25Speaker 02

  Turpentine is a network of podcasts, newsletters, and more, covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution, to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows, along with sponsors. If you think that might be you or your company, email me at [email protected]. That's E-R-I-K at turpentine.co.