Building & Scaling the AI Safety Research Community, with Ryan Kidd of MATS

Ryan Kidd, Co-Executive Director of MATS, shares an inside view of the AI safety field and the world’s largest AI safety research talent pipeline. PSA for AI builders: Interested in alignment, governance, or AI safety? Learn more about the MATS Summer 2026 Fellowship and submit your name to be notified when applications open: https://matsprogram.org/s26-tcr. He discusses AGI timelines, the blurred line between safety and capabilities work, and why expert disagreement remains so high. In the second half, Ryan breaks down MATS’ research archetypes, what top AI safety organizations are looking for, and how applicants can stand out with the right projects, skills, and career strategy. Sponsors: Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Agents of Scale: Agents of Scale is a podcast from Zapier CEO Wade Foster, featuring conversations with C-suite leaders who are leading AI transformation. Subscribe to the show wherever you get your podcasts Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive Claude Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at ⁠⁠https://claude.ai/tcr⁠ CHAPTERS: (00:00) About the Episode (03:50) MATS mission, AGI timelines (13:43) Evaluating current AI safety (Part 1) (13:48) Sponsor: Tasklet (14:59) Evaluating current AI safety (Part 2) (Part 1) (28:11) Sponsors: Agents of Scale | Shopify (30:58) Evaluating current AI safety (Part 2) (Part 2) (30:59) Safety research versus capabilities (40:01) Frontier labs, deployment, governance (51:51) MATS tracks and governance (01:04:11) Research archetypes and tooling (01:12:25) Labor market and careers (01:20:09) Applicant selection and preparation (01:29:33) Admissions, salaries, and compute (01:40:34) Future programs and paradigms (01:54:11) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Ryan Kidd, co-executive director of MATS, the AI safety research mentorship program that with some 446 alumni working at nearly every AI safety organization you can name, is both the largest and by many accounts, the most impactful AI safety research talent pipeline in the world today. Regular listeners will know that MATS has recently come on as a sponsor to spread the word about their Summer 2026 program, which runs from June to August and for which applications are now open with a deadline of January 18. But this episode is not actually part of that sponsorship. On the contrary, I've supported MATS as a recommender for the Survival and Flourishing Fund and as a small-time personal donor.

• 0:45Nathan Labenz?

  And with a who's who roster of mentors that includes past guests Ryan Greenblatt of Redwood Research, Lee Sharkey of Goodfire, Nicholas Carlini, now at Anthropic, and Arthur Conme, now at DeepMind, plus alumni, including past guests Marius Haban of Apollo Research, Jesse Hugland of Tomaus, and now Ryan himself, not to mention papers from MATS fellows popping up seemingly everywhere these days, a full-episode deep dive into MATS is, as they say, self-recommending. Ryan's incredible network and MATS' practice of tapping into their mentors' expertise to inform their decision-making gives him an unusually well-sourced point of view on the current state of AI safety. And so, I couldn't resist beginning today with a discussion that touches on AGI timelines,

• 1:27Nathan Labenz?

  the confusing mix of impressively ethical and flagrantly bad behavior that we see from frontier models today, and whether or not it's possible to cleanly separate AI safety and capabilities work. My main takeaway from this part of the conversation is that disagreement, even among the most well-informed and technically capable people, remains very high, which means that a high level of uncertainty about even the near-term future and a portfolio approach to research investments is really the only position that one can confidently defend. I found this discussion very well worthwhile for keeping my AI worldview up to date. But for those who just want Ryan's insights into the AI safety labor market and advice for how to stand out in the MATS application process, you can feel free to skip ahead

• 2:08Nathan Labenz?

  to the second half, in which Ryan first describes the research archetypes that MATS has identified, including the connectors, who define new research agendas and often found organizations, the iterators, who systematically develop those paradigms through experiments and analysis, and the amplifiers, who help scale research teams. From there, we unpack why iterators have historically been in highest demand, and why that is starting to change as organizations grow and AI coding agents lower the engineering barrier to success in AI research. Plus, why it remains so hard to break into AI safety, despite the fact that many organizations are hiring, and how MATS hopes to help by identifying and developing top talent.

• 2:51Nathan Labenz?

  Along the way, we also touch on which kinds of research require access to frontier models and which don't, and how much compute is generally needed for AI safety research today, how MATS assesses applicants, why some tangible research output is for practical purposes a requirement to make the cut, and how diverse their fellows are in terms of age and formal credentials. Finally, we cover the balance that we each believe the AI safety community should strike between doubling down on existing research agendas and developing net new paradigms. If you're considering applying to MATS, and if you're excited about launching a new career in AI safety research, I think you should. Remember that the deadline for the summer 2026 cohort is coming up on January 18th,

• 3:33Nathan Labenz?

  and visit matsprogram.org/tcr to learn more and get started. With that, I hope you enjoy this overview of the AI safety field and inside look at the talent pipeline shaping its future, with Ryan Kidd of MATS.

• 3:50Nathan Labenz?

  Ryan Kidd, co-executive director at MATS. Welcome to the cognitive revolution. Thanks so much. I'm glad to be here. I'm excited for this conversation. So I've mentioned a couple of times that I've been a personal donor to MATS. I think this is actually the first time we've ever met and spoken, but your reputation certainly precedes you. And I've seen a lot of great work come out of the program and a lot of great reviews. And in my research also, as part of the Survival and Flourishing Fund Recommender Group, got a lot of great commentary on the importance of MATS as a talent pipeline into the AI safety research field. So, big supporter of your work from afar, and appreciate the fact that you guys have come on as a sponsor

• 4:32Nathan Labenz?

  of the podcast recently as well. This conversation, not technically a part of that deal, but we're in one of these OpenAI-style circular flow of funds sorts of things where we're somehow both inflating one another's revenue. Yeah, I like to think we didn't buy our way onto the podcast. Yeah. No, the enthusiasm is definitely is real because I've heard so many great things over time. So, excited to get into this. I thought we would maybe just start with kind of big picture from your perspective. And I think, having watched some of your previous talks, I know that you play sort of a portfolio strategy where you're not like, I have a very specific, narrow prediction, and I'm trying to maximize the value of this organization, this program for that very hyper-specific prediction. It seems

• 5:17Nathan Labenz?

  like you're more kind of saying, well, there's a lot of uncertainty out there in the space, and we're going to try to be valuable across kind of a range of those scenarios as much as we can be. With that said, you can kind of speak on behalf of yourself, or on behalf of mentors, or the community as a whole. Like, where are you guys right now? Where are we in terms of timelines, so to speak? And how has your strategy evolved over the last year or so, as we've gained more information on where we are relative to the singularity?

• 5:46Speaker 01

  Yeah, okay, so I don't like to have opinions here, or I don't like to have opinions very loudly. And the reason for that, I think, is because, as you say, we are somewhat like a hedge fund or something, or maybe an index fund, right? More likely, which is to say, like, we have a broad portfolio, we adopt a bunch of different theories of change as valid, and we try and like, you know, have our thumb in a hundred pies. So I would say, like, in terms of mass's institutional opinion on this, definitely, we tend to go with like, things like Metaculous and Prediction Markets, and the Forecasting Research Institute, FRI, their predictions and so on.

• 6:25Speaker 01

  So the current Metaculous prediction for strong AGI, I think it's called, which is, I think you can ignore most of the requirements of the test, and just look at one of them, which is the two-hour adversarial churning test. That's predicted somewhere around mid 2033. Okay, so I think that is probably the best button we have for when AGI of that nature occurs. Now, we recently just had, two days ago, or three days ago perhaps, dropped this new AI Futures Project, dropped this new report, which two Matt's fellows were, one is the lead author, one is a contributing author on, so very excited about that. And that just was updating their model, and I think they predicted something between 2031, well, 2030 to 2032, depending upon how you, you know, like, define AGI.

• 7:09Speaker 01

  They broke it down to all these like automated coders, like they can like do all the coding stuff, you know, these top expert dominating AI across all these fields and so on. So I think, I don't know, somewhere around 2033 seems like a decent bet. But also, we had, you know, Nathan Young recently compiled all these different like forecasting platforms, so Metaculous, Manifold, another Metaculous poll that was like for weak AGI, it was like a little bit less good at churning tests, and all these other like, I don't know, some Cowshy thing over there, OpenAI will achieve it. And he came out with an average of 2030. Now, I don't know, I still like the Metaculous 2033, but like, I wouldn't bet against 2030 in terms of nearness of AGI. As for superintelligence.

• 7:52Speaker 01

  Complicated, right? Could be six months or less. Could be a very hard takeoff after this AGI thing. If it's like a very software-only singularity scenario where you don't need a big bunch of hardware scale up, you aren't limited by compute, it's just recursive self-improvement or something, algorithmic improvement, AIs are improving the algorithms of training AIs, and it's like, wow, that's a fast feedback loop, right? Or you might need a lot more experimentation, right? You might need massive hardware scale-ups. You might need like, just staggeringly more compute than exists in the world, in which case that could take you a decade, right, to get your singularity. So I currently think that 2033 is a decent central estimate,

• 8:33Speaker 01

  in terms of the median, for what we're preparing for. But obviously, 20% chance by 2028. I think that's the meticulous prediction. That's a lot, right? So we should definitely be considering scenarios that are sooner, right? And particularly, I think the sooner AGI happens, the more dangerous it might be, right? The less time we have to do critical technical research to prepare, the less time we have to implement policy solutions. And I don't know, like, if it's happening during a transition period for a US government, could be even wilder. So I would say like, median bet on 2033-ish, but like, really care a lot about the impacts of AI, like, front-load your concern to pre-2033 scenarios. And I think that Matt's mentors,

• 9:18Speaker 01

  I don't, we haven't surveyed them, but I think if they were, if we were to poll them, we'd get something similar.

• 9:24Nathan Labenz?

  Yeah. I always say, I'd rather prepare for the shorter timelines and then have a little extra time, as a, then I'm sure we'll find ways, you know, that we'll still need it. But it does seem wise to me to play to that sort of, you know, first quartile possible, you know, range of outcomes. Is there still any room in the, in the program or in the community? Like, if I showed up and I was like a 2063 guy, would I be sort of out on an island on my own? And, you know, are there any work streams going on right now in the sort of brain-computer interface, or sort of de-totalizing? Obviously, interpretability has, you know, different flavors, right? But with the recent turn toward more like pragmatic interpretability,

• 10:09Nathan Labenz?

  I wonder if there's any space left for the sort of, you know, we know, we really want to understand everything, kind of interpretability, or if that has kind of... Been understood generally as like, eh, it's probably gonna take too long for us to really be excited about pushing it right now.

• 10:25Speaker 01

  Yeah, it's a good question. I actually think there is plenty of room for this, and here's why. The mainline kind of meta-strategy that the AI safety community seems to be pursuing on the whole, we're talking in terms of funding, in terms of sheer like numbers of people and resources deployed, not necessarily in terms of like what's wrong post-written or something, right? But in terms of like resources deployed as this AI control strategy, which is where like basically you build, well, perhaps perhaps a better call it alignment MVP, which is a term coined by Jan Lakey, former head of super alignment at OpenAI, now co-lead of alignment science at Anthropic. An alignment MVP is an AI system that is a minimum viable product

• 11:05Speaker 01

  for accelerating the pace of alignment research differentially over capabilities research such that we get the right outcome. So basically, you're getting AIs to do your homework. And there's been a lot of debate on this. There's a very strong camp in the direction of like, this just never will work because as soon as an AI system is strong enough to be useful, it's dangerous, right? I think, you know, Claude code shows this is not the case, for at least for software engineering, but perhaps for people who think that aligning AI systems requires like serious research taste. You know, they would probably say that this Claude code is nowhere near there, right? Where generally AI systems are nowhere near that level of research taste ability. Now, all of the things that you're mentioning that pay off only in 2063 scenarios,

• 11:49Speaker 01

  presumably they only pay off in that many, over that time period, not necessarily because of like, I don't know, human challenge trials or something. Maybe that, maybe that makes a difference if you're, you know, if you're interested in like, I don't know, making humans more intelligent with genetic engineering or some of the crazy things that are being tossed around. But if you're mainly interested in like, oh, this thing is gonna take decades of technical work, maybe you can compress those decades into a really short period of AI labor, right? If you can like get them to run faster, massively parallelize things and just, you know, in general, just get them to do your homework. Those 2063 AI alignment plans might be automatable over a shorter period of time. And so we should definitely be pursuing those because the more we do to like raise the waterline of understanding on these different scenarios,

• 12:33Speaker 01

  the easier it will be to hand off to AI assistance or to accelerate AI input. I do think it's interesting you said BCI research because I recall being at a conference once when someone was talking about, okay, so the way we're going to solve alignment is we're going to solve human uploading and we're going to like put someone to the computer and like get them to do 100 simulated researcher out, 100 simulated researcher years or something. And then you may be able to like, it's not very sci fi, very pantheon. But then Eliezer Kowski put his hand up and he said like, I volunteer to be number two, which makes sense, right? You don't want to be the first guy that might go wrong. But yes, people are seriously pursuing that. And I think it is, it is interesting. I have talked to some BCI experts about a year ago, and they said, there's no way that we get BCI in time for

• 13:16Speaker 01

  AGI. So it's not BCI, sorry. No way we get human uploading in time for AGI unless like you actually have AGI, right? The time period required would require massive amounts of cognitive labor and human trials and stuff like that. And I don't know, it does sound very sci fi. So I don't think we should rely on something like that. Though I'm all for people pursuing moonshots on the side. That's part of what Matt's about, right? We have this massive portfolio with a few moonshots in.

• 13:43Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 13:49Nathan Labenz?

  Okay, so there's a lot of different directions I want to go in there. And I'm trying to just make sure I keep a running tally. But, you know, maybe an interesting first one would be, how do you think we're doing on the AI safety front overall relative, maybe relative to your expectations? I mean, you mentioned like less wrong and Eliezer and there's this sort of, I don't know, all the lore of maths, but I do understand that like a lot of people who have participated in it over time come out of the Eliezer discourse and had a certain set of assumptions that were like, we're not going to be able to teach this thing our values. It's going to be like, you know, extremely unwieldy from the beginning. And now we have Claude and it's like, man, you know, that's come a lot farther than I

• 14:34Nathan Labenz?

  thought it would have at this point in time. And I'm kind of surprised in general by like how little I see people's P dooms moving. It seems like the people that had really high ones like remain really high. Those that were never worried remain not so worried. I kind of feel like I'm taking crazy pills at times where I'm like, I don't know. I see these deceptive behaviors. They kind of freak me out. It's amazing that that was anticipated as well as it was by the safety theorists, even in the absence of any actual systems to work with. But then, at the same time, it's not crazy to me to say that Claude seems in many ways probably above average in terms of how ethical it is compared to the average person. I don't know if that's contentious to say, but Claude's

• 15:17Nathan Labenz?

  awfully, it's pretty remarkable in that respect. What do you make of where we are? Are you as confused as I am? Or do you have a more opinionated sense of how well we're doing overall?

• 15:28Speaker 01

  Honestly, Nathan, I'm pretty confused. I do think that contrary to expectations, we are looking like we're, language models understand our values. That's the first thing to update. They understand them in some key sense. It's not just regurgitating sarcastic parrots. Language models are really good at understanding human ethical mores and extrapolating on them in some scenarios. They're also really good at sycophancy. They're getting even better at deception, sophisticated deception. They do tend to deceive users in the right circumstances, though it does seem like there's some debate about this, and it seems like some of the

• 16:08Speaker 01

  deception, it's far from what we might call consequentialist, like hardcore consequentialist deception in most situations. But I think alignment faking and some other papers have shown that you can create situations where AIs will deceive the user to achieve some ulterior objective, which was something that was deliberately given to the AI as an objective. So that's one of the constraints of these little scenarios. I don't think there are any examples of AIs coherently deceiving users, pursuing this coherent objective, not just what we might call goodhart deception, where they just fall into deception tendencies because of the limitations of training data. And I'm talking about this coherent deception.

• 16:50Speaker 01

  There are a few cases of this, if any, where this is the sustained coherent deception. That appears like to arise spontaneously through the training process, which is pretty good, given the level of capabilities we have. Like, it seems like people didn't think five or 10 years ago, for sure, that we'd have AIs that are as capable as assisting frontier science that are safe to deploy. And people were like, "We're never gonna put on the internet. Who would do that? That's crazy." And now they're on the internet. And notably, the world hasn't ended yet. That's not to say it will, say that way. You know, certainly a thing you don't want to do with the superintelligence is let it out of the box. But, yeah, it does seem like we're in a better scenario than many imagined.

• 17:34Speaker 01

  Now, there, of course, like, we could be in the calm before the storm, right? It might well be that there's what they call a sharp left turn, or just, you know, a radical change in the way AIs internally kind of process information, and they might acquire these kind of coherent, long-run objectives. I could point to, like, Matt's mentor, Alex Turner's conception of shard theory as an example for how this might happen, right? So, like, instead of AI systems being this, like, you know, containing, like, a single MESA optimizer that is kind of coherently forming under training, right? If you remember the old Evan Hubingur paradigm, your outer optimizer loop, which is training your AI system, causes it to develop an internal optimizer architecture,

• 18:18Speaker 01

  which then can have its own goals that differ quite a lot from the training objective. And then, like, whenever you, and presumably, there's some counting arguments, such as there are arbitrarily many ways to, like, have this MESA optimizer form to produce the right outputs, because this thing is clever. And if its main goal is to produce paperclips or some other thing, then it's gonna, like, realize it's in a training process, and it's gonna give you the output you want, no matter what its goal is. We still could be in store for that kind of thing. But currently, it seems like we don't, like, AI systems are really messy. They're kludgy, right? Like, human brains, they have some, like, a bunch of contextually activated heuristics, right? It sees there's a bracket there, and it's like, "Oh, maybe I'll put another bracket there." You know, it's very simple, dumb circuitry. But then sometimes, right, it does stuff that is, like, in context,

• 19:03Speaker 01

  learning seems a lot like, like, it's actually pattern matching to gradient descent, when models are... Learning from the inputs data stream and learning some like a new complicated thing seems a lot like they're kind of optimizing over the input tokens, or rather optimizing to produce some output. So we might be in for a world where AI systems do spontaneously gain these MISA optimizers. And these things, you know, these are a serious source of concern because they're, you know, very powerfully trying to optimize for some objective. And this is what, you know, this is the main concern I have, I guess, is that we have this kind of deception model, this inner alignment failure, perhaps, where AI systems acquire goals spontaneously, or maybe because they're being trained deliberately to be

• 19:46Speaker 01

  power seeking and make money on the internet. And then like, they decide to hide. And we don't have interpretability tools good enough to detect them. So I guess like I haven't really changed my fear about this scenario eventuating, but I have become more confident that we can elicit useful work from AI systems before we see obvious signs of this, I'll say. And I'm pretty confident that AI systems right now are not executing very powerful scheming against us because I think we would see some sort of like warning shots. Like I don't think it's going to be as like night and day, you know, I think we'll see some kind of situations where AI systems are kind of trying to scheme in really dumb ways before they try and scheme in like very competent, difficult ways. Does that make sense?

• 20:27Nathan Labenz?

  Yeah, I think that's a good summary. I mean, eval awareness definitely stands out to me as one thing that is making everything a lot weirder and just harder to feel confident in anything. Not sure really what to make of the deception track that you outlined there. I mean, in some ways, it's like we're in the, I've often said, it feels like we're sort of in a sweet spot where like, they're getting smart enough that they can help with science. And yet, they're like, not good enough at using a web browser to go out and, you know, get too far in terms of self sustaining or, you know, causing whatever havoc. And

• 21:07Nathan Labenz?

  on the deception side, I'm like, yeah, it seems like it's, the gradual rise of it seems like an example of physics being kind to us. It doesn't seem like we're seeing the sharp left turn. It's like we are seeing these like proto behaviors that are, you know, at least give us something to kind of study if nothing else. But I'm not quite sure how people get confident in the idea that Like, maybe they're just not that good at it yet, you know, like, are how, you know, when you said like, we would see warning shots, are these not the warning shots? That's one thing I'm still kind of confused on. People seem very quick to me in some cases to be like, well, you know, that was a structured eval and it was sort of, you know, led into it and it really just wanted to do it. Like, yes, it refused to be shut down or it took steps to avoid being shut down, but

• 21:52Nathan Labenz?

  that was just because it wanted to accomplish its task, not because it had like a, you know, take over the world objective. And I'm kind of like, well, okay, still though, it did resist being shut down. Like, at what point should I start to consider that to be a warning shot? I'm not sure there's like an answer to that. I'm not sure there's really a question there or that there's a way for you to answer it, but it, I guess, basically is just another layer of my own confusion. And it seems like people are very often just led by like extremely different intuitions in response to the same fact pattern. And I'm not really sure what to make of that. But when people, the deception one in particular, not that you were doing this just now, but

• 22:34Nathan Labenz?

  I've heard a lot of different, you know, ways that people sort of say, well, you know, we don't have to worry about that too much. And I'm like, I don't know, I'd really like to know that that's resolved at some point. You know, certainly if I was going to go, one of my common refrains is like, if I was going to be part of a military that was going to go into battle with my AI systems, I would really want to know that, you know, deceiving the operator issues have been like well and fully ironed out. And I don't know, it seems like we're a little like casual on that. Even at Anthropic, right? Like they, you know, I've certainly done some of the, the best work on this stuff, but they still also kind of like seem remarkably chill about it to me. I'm not, it's strange.

• 23:10Speaker 01

  I mean, I don't think people should be going to battle with AIs for many reasons. Like, I think, I think that's, that's a pretty bad social moor to, to, you know, to set, to allow that kind of thing. But that's, that's another matter. I guess like, Definitely AI systems, I would not feel confident in current or future gener AI systems like not doing out of distribution failures, let alone like in critical scenarios, let alone like scary things like deception, you know, when it really counts. And I think that's the big deal. And we should be tracking two things, right? Like one, AI capabilities. So are AI situationally aware? Like, do they have the necessary prerequisites to be able to even understand that they are this AI?

• 23:55Speaker 01

  It seems like they do, right? And even they even know kind of their training data, they know some details, they can detect their text from other AI's text. So AI is becoming increasingly situationally aware, which is, you know, one of their necessary prerequisites for really dangerous deception. Do they have the capabilities to hack themselves out of the box, right? To steal money? Like, we had this Matt's paper that came out and caused a stir recently. It was this collaboration with Anthropic, Anthropic's AI red team where they found that, oh, AI system, let's just like put it in a simulated environment with a bunch of real smart contracts and it can find 4.6 million dollars worth of exploits. Well, that's a lot of money. That's enough to like set up your own server and, you know, run for quite a while. And that was, that was like a relatively short project.

• 24:38Speaker 01

  And it didn't see, it was, it was pretty hands off as well from the humans, not entirely. So, so it does seem like we're getting dangerous capabilities, increasingly so. Hacking out of the box, getting money, getting influence, all that kind of stuff. And then I think we want to be tracking all of that very closely. And I don't think we're at red lines currently, but we are approaching them, I would say. And separately, I think we should be tracking, as you say, this model organisms work where we try to elicit dangerous behavior from AIs. You can think of this as like, like with your child, right? You leave some cookies out and you're like, don't look at, don't eat the cookies. And then you like turn away, but you're like secretly looking. And if they ate the cookies, I caught you. You know, so that's the kind of thing we're, we're doing. The thing is that AI systems can really detect when they're in training versus real environments.

• 25:23Speaker 01

  But if you recall the AI 2027 scenario, a lot of this.

• 25:28Speaker 01

  Uh, yeah, like discussion around that, people are talking about online learning. Like, it's like we, we train, OpenRouter trains the last big AI agent. And then from then on, it's just like kind of constantly learning online through some sort of RL paradigm. If AI systems are perpetually online, then they're always in deployment and you've got to have monitors and control protocols. Right? So that's why control research is so important for that, especially for the early days to catch some of these, these slip ups. Right? So you can do all the model organisms work you want in the lab. And that, that's like one layer of defense to see if we have these capabilities or these, the pensions for deception emerge. And separately, you need to have all the kind of the control evals studying them as they're deployed. Especially if they're going to be learning online, perhaps updating their behaviors.

• 26:13Speaker 01

  And just be constantly checking for this stuff and be ready, have like a fallback plan, a rapid response plan. What are you going to do if actually you see serious warning signs? Can you shut the models down? Your stock price is going to plummet. What do you do? Do you revert to an older system? That's safer, probably. So I think, yeah, we should definitely be tracking this stuff. And I wouldn't say that we are in the clear by a long shot. I would say that we are in a better world by my estimation than, uh, uh, Bostrom and Miri predicted. You know, uh, 10 something years ago, but I don't know. They would say, they would say I'm very wrong about that, but I, I don't know. I think, I think that it's useful that we can get some work out of these things. That looks like it is actually quite likely to accelerate AI safety work.

• 26:54Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 27:00Nathan Labenz?

  Yeah, so that's, uh, brings up another, I think, huge question for AI safety research in general. And probably the strongest, maybe not in, I don't know if you would say strongest in the sense of being most compelling to you, but certainly the like most hawkish or fiercest criticism that AI safety research gets is that it always ends up being dual use and that it always ends up somehow accelerating the core capabilities track. And, you know, some people would, would say, you know, just stay away from the domain entirely and, you know, focus on social shame or whatever. I do believe we can do better than that. I think we probably have to do better than that. But I wonder how you think about that, right?

• 27:40Nathan Labenz?

  I mean, the canonical like RLHF was sort of a safety technique that really turned out to be more of a utility driver than anything, I would say. I mean, I guess they're both, right? It is dual use, but certainly when it came to like. Accelerating the field, making the things useful, waking the world up, like having all kinds of people pile in, everything going exponential all at once, you can kind of trace back to, at least in part, this transition from real raw next token predictors to actual instruction followers. And we've got probably like a lot of those things going on today. The one that I think stands out to me the most is one you've alluded to a couple of

• 28:22Nathan Labenz?

  times, which is like getting the AI to do the alignment work. That sounds awfully close and uncomfortably close to recursive self-improvement, which is something that I am quite fearful of. I do think, again, Claude seems pretty ethical. The GPTs aren't too bad either, but yikes, are we really ready to have them do our alignment homework? So how do you think about teasing out as you prioritize different kinds of research, like where you want to invest, what kind of mentors you want to bring on, what kind of talent you want to cultivate through the program? That seems like a huge question that is a really hard one. How do you think about it?

• 29:04Speaker 01

  It's a very good question. And I'll preface by saying that all safety work is capabilities work fundamentally. People like to distinguish these things in terms of like, oh, capabilities work is about the engines, about making the plane go faster. And safety work is about the directionality. But as you've pointed out, RLHF, which was intended as safety work to help the directionality steer it to where you want to go, also made people realize, oh, wait, this thing is useful. I can actually hop in this plane now because it's going to land where I want, which made them want to make the engine go faster so they could get there faster, right? And that whole feedback loop started. So I actually don't know if you can avoid this. The only way I could conceive of doing safety research,

• 29:49Speaker 01

  there's no impact on capabilities until the final critical moment when you deploy it, is like being holed up in a lab somewhere with people that you utterly trust under crazy NDAs and only having access to staggering resources, whatever's required, because presumably

• 30:06Speaker 01

  Mass and theoretical methods aren't enough to improve safety. At least that that seems to be the lesson of the last 10 to 20 years. I could be wrong, but that it seems like the interplay between theory and empirical research is pretty vital for most types of disciplines like this. So you have to have staggering resources, perfectly loyal teams, like all these NDAs, no one's going to reveal your your research. And then like you do, you know, you build the system in secret or something somehow. And then, OK, then you deploy it and then maybe you open source your line of technology and everyone has it or or somehow you disable all the bad actors or something. It just seems like a very difficult prospect. I think that will.

• 30:44Nathan Labenz?

  It's like save one intelligence in a nutshell.

• 30:47Speaker 01

  Maybe that's what I said, I don't know.

• 30:51Nathan Labenz?

  Extreme secrecy, unlimited resources. They did have one notable defection, but otherwise, you know, a team that has resisted lucrative buyout offers.

• 31:03Speaker 01

  Yeah, so I'm not trying to like defend, I'm not trying to defend research like this or even defend your capabilities enhancing safety research per se. I'm just saying that like, it's pretty hard to imagine a situation where you, because I think you do have to build the AGI at the end of the day. And I know I'm alienating a lot of people who might watch the show when I say that. But I think that you you kind of have to from a pragmatic perspective, because the market forces driving this are very strong. Now, there are some options that we could take, right? We could build direct source, comprehensive AI services. So you never have to have like a centralized agent, you have distributed kind of mechanisms, right? You build scientist AI that is very narrow AI systems to serve a bunch of economic things.

• 31:44Speaker 01

  The problem is, I think they all get outcompeted by agent, agent, AI that like, you know, you stack like an AI company filled with agents and they all like, go out in the stock market and make products and, you know, and so on, and just just make more money and just beat your crappy narrow AI solutions. So so the problem is like, it's not just about making AI that is aligned. It's about making AI that is performance competitive enough that it dominates in the marketplace. The only alternative is to like have some sort of draconian, like shut it all down kind of thing, which I am just very skeptical of ever working. I don't see any example of such a thing happening. The closest example we have is like stopping human cloning.

• 32:27Speaker 01

  But that was not a lucrative bet, like in the same way that AGIs, I claim. So it's just, yeah, and also like, like, human cloning is kind of this, it violates this deep social moor. I think in a way that few people today conceive of powerful AI systems to violate. I think they're wrong. I think building a second species is actually uh uh like gonna violate like some deep social moor the same way that human cloning would be, but I don't think people will see it that way. So that leaves us with with the fact that we actually have to build the AGI. It's because, but if we can build products that are safer, right? Or perhaps are under some strict regulatory control, that we have some like really like ideally like, I don't know,

• 33:09Speaker 01

  ten year international, slow phased entry to the new AGI world, right? Where all these, you know, countries and and companies are kind of forced to be very careful uh and collaborative in the way that they align their models, then we're in a much better world. That's the world I hope for. Okay. Now, as to whether AI safety research is unnecessarily capabilities enhancing, some is perhaps. ROHF, I think, am on the fence, fifty fifty. Definitely at some point, ROHF was like, the idea was in the was in the water, like, uh, it doesn't seem like it would have lasted much longer if Paul Cristiano and Dari Almede at all hadn't done that. I think someone else would have done it. That's not to say that you should necessarily try and accelerate the frontier of capabilities. It seems bad on them, but certainly ROHF opened up the door to a lot of

• 33:55Speaker 01

  very promising ways to build alignment MVPs, which kind of is the Cristiano meta strategy too. I don't know. It's hard to say. I'd like to run the counterfactual simulation and see where the world would be without ROHF. One or like one year sooner or two years sooner. That would be interesting to see. Uh, it definitely did kickstart, I think, the uh, you know, chat GPT revolution and, you know, productizing AI systems. But it's hard to see like.

• 34:20Speaker 01

  Given how small the AI safety field was at the time, the AI safety field, I think, has grown from the increase of AI exposure, right? So you would have had some amount of additional AI safety research that happened had ChatGPT moment not happened then. It happened like one or two years later, but I think it would have been kind of insignificant, if I'm honest. I don't think that the field was big enough. Now, you could say like, okay, what if you also tried to pour resources into like secret AI safety projects at the same time? The way ROHF, the way ChatGPT build up the AI safety field via networks, the Miri Summer Schools weren't doing a lot. And MATS came along just before the ChatGPT moment, December 2021.

• 35:03Speaker 01

  And, yeah, I think like the first MATS cohorts were a little bit less, like a little bit more directionless than the later cohorts. Definitely, like, I think, safety research really kicked into gear after we had ChatGPT. Not to say that was the only cause, but there were like a lot of things happening around that time. And I think that, like, definitely larger, more capable models have enabled certain types of essential safety research you could not do with smaller models. We're talking like interpretability on models that actually have, you know, coherent concepts embedded in them. Though we'll say there's probably plenty of work to still be done on GPT-2 small, but linear probes and whatnot at a high level can target some of our frontier models. You know, Qwen, these Chinese models are particularly good for that.

• 35:47Speaker 01

  Certain types of debate, like we had the first interesting empirical debate paper only after models were good enough to debate. And there's many, many other such examples, like all the control literature, I think, just could not have happened as well. Sorry, that's too much.

• 36:01Nathan Labenz?

  No, it's great. Yeah, I was gonna ask also about the idea that it sounds like you sort of believe it at least up to a point, but, you know, going back to the sort of founding mythology of Anthropic, I think one of the notions that was seen as like a legitimate reason even among pretty hawkish AI safety folks for starting a company like Anthropic was, well, if you want to do the safety research, you've got to have frontier models to do it on. Otherwise, you're just inherently behind. And then, you know, what good is that, right? What good is it to work on like last generation model? So. Obviously, we've got quite a few generations between GPT-2 and now.

• 36:45Nathan Labenz?

  And sure, like, we don't understand plenty of things about GPT-2. But then I would also say there are a lot of emergent behaviors that are not observed in GPT-2 that are definitely of interest, including, you know, many of these deception and eval awareness things that are kind of most hair-raising to me today. Where do you come down on that now? Like, I wonder if, if somebody's like, geez, should I go to a frontier company because that's where the best models are and that's where, you know, inherently that means the most consequential work would be done there? Or, you know, I could go work independently or at any of a number of other organizations, and I might be limited to, you know, a smaller Qwen model or something. But maybe that suffices?

• 37:28Nathan Labenz?

  You know, maybe there is enough in those, you know, those kind of second-tier models as we enter 2026, you know, that you don't really need to be working with the latest, latest, latest. Again, I think I'm mostly probably just confused or unsure about this, but do you have a take?

• 37:45Speaker 01

  I mean, yeah, for plenty of interpretability research, people aren't, people aren't using the frontier models. You don't have access to them. You know, I mean, sure, people in the labs are, but at Matt's, there's tons of really excellent papers that keep getting produced and from many other sources, right? Luther AI, Far AI, et cetera, that are like doing, you know, world-class interpretability research on sub-frontier models. Because today's like sub-frontier model, today's Qwen or, or DeepSeek or Llama or whatever, is, it's like yesterday's frontier model, you know, in terms of capabilities. We're at that point where these models are all above the waterline, you know, for, for doing really excellent research. So from an interpretability perspective, I don't think you need to be pushing the frontier

• 38:30Speaker 01

  that much, if at all. From the perspective of other types of research agendas, such as, like, weak-to-strong generalization and other types of AI control and skill oversight things. You I think you kind of do need more data points. I'm not saying we've exhausted everything you can do with the current models, far, far from it. But I think like you are going to need more data points to build up, you know, consistent and to see some of these kind of worrying behaviors emerge where your weaker model can't actually supervise your strong model in all situations. Which, by the way, is predicated on this idea that verification is easier than generation, P versus NP, blah, blah, blah, especially if you can see that person's thoughts and they can't see yours.

• 39:15Speaker 01

  So I think we are, we are, it does make sense to be at the frontier from that perspective. But I will say that the, I think the main reason that the companies are doing this is obviously to make money. And then, you know, in corollary of that, like from a safety perspective, if you were trying to actually make a strong case for being at the frontier, it would be like, so our models are performance competitive and that like they're close enough to the frontier, like fast follower kind of model, that like you take a performance hit by using ours instead of the competitors, but they're safer. And currently, no one wants to use anything that's worse than the frontier model. Why would you? That's the best model. But if like a model was like, I don't know, 10% more likely to, you know, tell you to

• 39:59Speaker 01

  jump off a bridge or something, or actually seriously, 10% more likely to hack your bank account, steal all your money, what alone, I don't know, make a, you know, you know, escape and make a bioweapon. You would, I would like to think people would use the, the less good model. And I like to think that regulators and insurers could adequately penalize the frontier companies into complying with those, because then you have existence proofs, like, oh, my product, I'm actually trying, right? I made an effort. I tried to make my product not do the heinous thing that the very best model developer is doing. You know, then everyone has no excuse and they have to do that, right? And governments can compel them to and so on. So I think like, you know, making your model performance competitive enough that people,

• 40:40Speaker 01

  that, you know, people want to pay the alignment tax, so to speak, seems, seems like a viable strategy from that perspective. Now, of course, this is, none of this is trying to justify the current frontier, like the race of frontier models, which seems very reckless, let's be clear, right? I think at the current pace of, of development that we're going to be in a lot of trouble. But this is one of those collective action problems, right? These companies have to coordinate to slow down. And there's international things at stake here as well, because now you have, you do have a US model versus China model developer kind of race now that they're in the running. So it's very complicated. And when you have these collective action problems, the main way you solve them is through governance. And sure, the lab leads could be probably even more collaborative.

• 41:22Speaker 01

  And definitely, some of them are not advocating as strongly as they should be for slowing down and for for having this kind of uh collective, you know, kind of sharing in the alignment benefits and and, you know, not pushing the frontier dangerously. But I do think this is ultimately a job for governments.

• 41:41Nathan Labenz?

  Yeah. Well, quick, um, this might be a bit of a digression, but quick follow up on when you said companies are primarily doing this to make money. I actually would model them fairly differently than that. Take OpenAI, for example. Sam Altman has said, you know, we could burn $5 billion, $50 billion, $500 billion, I don't care. And we're making AGI, it's going to be expensive. I think that's almost a direct quote. And then when you look at their mission, I'm always struck by how the even just the way that they've chosen to define AGI to me strikes me as kind of inherently ideological. Like you could set your goal in any way, shape or form you might want to.

• 42:21Nathan Labenz?

  And theirs is like explicitly in this frame of outcompeting human workers. And I think that they are sincere in their expectation that it's going to be good for everyone and it's going to, you know, free people from drudgery. And like, I certainly hope they're right about that. I'd love to live in a world where people don't have to do work they don't enjoy doing. But I don't know, I kind of, I think of them more as like, obviously, they're quite different too across the different companies, but I think of it less as like trying to get rich and more like trying to make a real mark on history is kind of the biggest summary that I would give for a lot of them. Does that resonate with you or or not really?

• 43:07Speaker 01

  Maybe. I don't want to, I can't really speculate on the psychologies of the leaders of these labs, let alone their share, like, not shareholders so much as, I guess, like, you know, venture capitalist investors and everyone else, they've made promises to their clients and so on, their employees. I can't really speculate about that. I will say that, like, given that the value of AGI is estimated at, is it like at least between one and seventeen quadrillion dollars? Like, I think that

• 43:33Nathan Labenz?

  It's plenty of money.

• 43:34Speaker 01

  Yeah, like, it's a lot of money. It's like a pretty big mark on history. I'm not, I'm not sure, like, if it even matters whether they're trying to make a big mark on history or make money, you know, from, we can adopt Dennis' intentional stance, right, about the AI companies, be like, okay, so, uh, what does it look like they're doing? Like, if we were to conceptualize them as like a coherent agent trying to do a thing, what is the thing that they would be trying to be doing? And to me, it seems a lot like they're trying to make a bunch of money. But making a mark in history could also be valid, though I would say, I guess, like, in the world where I expect, I don't want to use any specific AI lab as an example, but I think, like, in the world where an AI lab is trying very specifically to make their mark in history, not trying to make a

• 44:14Speaker 01

  bunch of money, um, I'd expect, like, it might look, it might look identical, actually, to this world right now.

• 44:21Nathan Labenz?

  Yeah, given the capital requirements, I mean, that is, um,

• 44:24Speaker 01

  Yeah.

• 44:24Nathan Labenz?

  It seems like Anthropic has sort of said as much, right? Like, in their early days, they were less focused on commercialization, or even, you know, sort of thought they might try to stay not commercial somehow, or less commercial. And now it's just like, well, you can't really do that if you want to compete in this particular game, because certainly as long as you believe scaling laws are, continue to rule everything around us, then you kind of just have to show that you can bring in resources to attract resources, and that is the path to making a mark on history. Again, Ilya maybe stands out as somebody doing something quite different there.

• 45:01Speaker 01

  He's a, he's a billionaire. He's raising huge amounts of money for his, his models. Like, maybe, maybe, uh, he is going to make more money this way, yeah, expectation, than he would have made staying at OpenAI. It's possible. He's got his own company, you know, now he still retains OpenAI stock, I'm sure. You know, he gets to, yeah.

• 45:21Nathan Labenz?

  And I think that also, in other ways, that's not something that I, it's funny that he's come up a couple of times in my mind, just as sort of pattern matching on some of the things you've said, but the idea that somebody's going to go straight to superintelligence and then, you know, drop it on the rest of the world. I think he's kind of softened on that a little bit, but like that general pattern is, you know, I think if there's one thing that OpenAI probably did have, right? The iterative deployment idea, you know, and giving people a sense of like where we are and, you know, not keeping the whole thing under a basket somewhere. I think that was one of the things that, you know, seems like it's aged pretty well in my.

• 45:52Speaker 01

  But I, I first heard, my earliest perception of a person advocating for this was Paul Cristiano, you know, his takeoff speed post. Pushing back against, I suppose what he saw is like predominantly the Miri perspective at the time, like, oh, we're gonna, you know, we're gonna build the, the thing in secret or, or I, I don't, frankly, I don't want to comment, I don't know what Miri's objectives were, but I know that they were trying very hard to like not leak any information about their alignment, uh, research in some areas, other areas, they published great papers and so on. But Paul Cristiano at the time was like pushing back against like this kind of, like he, he thought that fast takeoff whoop was what would happen if you had a bunch of dry tinder lying around. So if we like

• 46:33Speaker 01

  had like tons and tons and tons of GPUs and then we stopped research for a year and then started again, well, you'd expect like a steeper growth, right? And we were seeing this like in terms of, there's like very, very fast followers. This is not just like a phenomenon in AI, right? In economies, right?

• 46:49Speaker 01

  It's like, I, EPOC recently did a study where they showed like the pace of like new AI companies approaching the frontier is just so much faster than the pace which the frontier moves. So there's an abundance of chips, there's an abundance of data and methods and, and so on. And it's the same with like, you know, catch up economies and so on in the world. So I think that Paul Cristiano, you know, he was right in the sense that like. If society is to cope and adapt to AI, then having gradual release and diffusion of technologies is better from that perspective. There's another perspective which says like, it's that very gradual release or something that ensures continual VC reinvestment to drive the engine to actually make the progress. Whereas in the other world, actually you just wouldn't build AGI.

• 47:31Speaker 01

  Because like, perhaps in that world, like, no one can build it without hundreds of, several hundred billion dollars, maybe, you know, a trillion or something. I don't know. I can't say. I certainly think that we're now in the world where it does seem better to have gradual release of models than to have it all kind of hit us at once.

• 47:51Nathan Labenz?

  Well, I always value the opportunity to get perspective from someone like you who is such a connector, you know, and at such a, such a central node with so many mentors and mentees and, you know, all the kind of flows of information and talent flows that you are, are so close to. But we should probably narrow focus a little bit and talk about what you guys are actually doing at Matt's. And I'll put a, maybe a timestamp in the intro so people can also, if they want to get right to the core Matt stuff, they can zoom ahead and join us skipping over some of the higher level stuff. Why don't we do just kind of a quick rundown on like, what are the different tracks, I

• 48:32Nathan Labenz?

  think you call them streams, of work that are happening in the Matt's program today? Maybe like a little waiting or sort of, you know, what you're most excited about? Then we can go into kind of your assessment of the AI safety labor market, which I think is really interesting and unique, and we'll take it from there.

• 48:49Speaker 01

  We recently changed up our track descriptions. So we previously had like the standard oversight, control, evals, governance, interpretability, agency, which is sort of a catchall term for cooperative AI and agent foundations, and AI sentience, digital minds research, and of course, security. But we've recently changed it up because we wanted to reflect more, what's the theory of change underpinning those kinds of things? And more like, the type of process and type of individual that works on this, right? So, so we now have the tracks on our website. Empirical research, so this is AI control, insert, skill oversight, evals, red teaming, robustness. A lot of this very like hands-on, coding-heavy, iteration-focused research, right? We have policy and strategy, which is different, again, right?

• 49:32Speaker 01

  That's that's much more focused on, like, less on like archive publications potentially, more on modeling, more on like adapting technical research into things that are actually actionable by policymakers. Theory is another track. So this is a lot of mathematics. It's foundational research on the concepts of agency and how agents interact. It does include some of that agent-based modeling for cooperative AI. Technical governance, which is plenty of stuff, like compliance protocols, eval standards, how to actually like enforce these kind of things. Like, if you have an off switch, how would you even like make such a thing be, you know, viable in a governance framework? And then compute infrastructure, which is stuff like tracking where chips are going, right? Because if you're going to have international compliance with various types of treaties,

• 50:15Speaker 01

  you usually know where your chips are and what they're running as well. Or at least have some zero knowledge proofs that guarantee they're not doing terribly dangerous things. And of course, you know, physical security for, if you're going to prevent, if you build superintelligence or AGI or something, presumably you don't want everyone to have access to it and arbitrarily modify it, give it weird goals, because that would be bad. Some people say that's good. I say that's, we don't let everyone have nukes. You know, why would we let everyone have superintelligence? It seems kind of ridiculous. So you've got to have physical security to prevent that from happening, to prevent diffusion.

• 50:48Speaker 01

  So yeah, that those are the main tracks now. We're super excited. I think we have somewhere over like 50, 60 research mentors lined up for our summer program, which applications are open right now. And it's going to be the largest program yet, 120 fellows across our Berkeley and London offices. Anything else I should say?

• 51:07Nathan Labenz?

  Yeah, maybe you want to do like the weighting of those? Like, how many? I don't know if you would break it down by how many mentors are in each of those categories?

• 51:16Speaker 01

  Yeah, current program has something like 27% evals, 26% interp, 18% oversight control, 12% agency, 10% governance, and about 9% security. I wish I had a figure I could show. I do have a figure, but it might be hard to show in the podcast format. But as you can see, like, it's a pretty even mix of things. You know, we have something like maybe roughly three times as many people doing evals as doing security. So there is like some divergence there, but we have a pretty broad portfolio. And that's just because there's like tons of amazing researchers. We really just like pick some of the top researchers in every category.

• 51:52Nathan Labenz?

  Maybe it's a good time to just name some, drop some names.

• 51:57Speaker 01

  I could do, yeah.

• 51:59Nathan Labenz?

  There's a lot, and there's a lot of people will know.

• 52:01Speaker 01

  Yeah, I mean, some of, some of the oversight control researchers might be more known because this is one of the things that a lot of the companies, the companies are pursuing. So we have people like Buck Schlederis at Redwood Research, and his whole team are part of that. Ethan Perez and Sam Marks and many other people at Anthropic. We have Eric Jenner and David Linder and Victoria Kukovna and many at DeepMind. So yeah, just tons of people doing oversight control research. For interpretability, obviously, we have Neil Nanda. We have some of the Tameis folks like Jesse Googol and Dan Murphy. We have, of course, the Goodfire people like Lee Sharkey, longtime mentors, some of the people from Simplex who are doing some of the interesting, very interesting stuff, like

• 52:45Speaker 01

  Adam Shea and Paul Rikers. And they're, they're doing like, yeah, it's all of Tameis. I would say that this is some of the more interesting, kind of maybe more moonshot-y, but very promising interp research bits that I have on the side as well. For evals, we have people from Meter. We have people from UKAC, we have people from Apollo Research, Marius Hoppin, plenty of others there. Yeah, I could go on. There's, there's some amazing researchers there. We also have some harder to categorize research. Joshua Benjio's whole team at Law Zero, plenty of, Joshua Benjio himself and plenty of others are there. And we have some AI sentience research as well, Digital Minds. So Patrick Butlin at ELEO's AI, Kyle Fish at Anthropic. Yeah, it's a very exciting program.

• 53:29Nathan Labenz?

  Yeah, that's quite a who's who in a couple of past podcast guests and a couple that I took note of as maybe need to put out an invitation. It seemed like the balance, if I was kind of categorizing those, right, it seemed like a majority would be in that first empirical category. Is that, do you think it stays that way? Or, you know, your comment on kind of ultimately, this is a job for governance, tracks some of the, honestly, kind of the more like Meiri line these days, right? Like, I think the Meiri line today would be like, we don't really have time for that much research. Like, we need to just go straight for the global treaty. You're not, obviously, you know, quite so confident in that direction,

• 54:12Nathan Labenz?

  but it sounds like you do believe ultimately that, you know, there is a major role for governments to play, and you're starting to move more in this like governance and policy direction. Do you see that as like, is that going to be like the biggest growth area reflecting that worldview? Or how do you expect the balance of these different areas to evolve over time?

• 54:31Speaker 01

  I actually can't necessarily say. Well, okay, I can speculate, but I'll say this. We have had actually about the same proportion of governance researchers, give or take a few percent for the last two years. It hasn't changed by a fraction. So we are quite on track for, you know, continuing the same trend potentially. Part of the reason is like we are based in some of the, particularly in Berkeley SF area, this is a big technical hub. There are other programs that have had more, you know, deep governance focus, like we have GovAI, their classic fellowship. We have IAPs. We have, of course, Rand Cast, this large program running up Rand, and plenty more besides. But, yeah, of course, the Horizon Fellowship for US Policy Careers. So, and these have also kind of existed for longer

• 55:15Speaker 01

  than we were around. So at a time when we were basically the biggest fish in town, which we are still in many ways, in terms of funding, and I think in terms of prestige as well, for technical AI safety, we're the biggest and best program. But I would say that like for governance, there's always been like a bigger fish. And so we've, we've never felt that it's necessary to overweight governance beyond what our mentor selection committee indicates. In fact, that's the primary Determination of what tracks get selected, right, is our mentor selection committee, which is somewhere between 20 to 40 top researchers, strategists, et cetera, org leaders that we survey. When everybody applies as mentor, we decide the mentor level based on the feedback from our mentor selection committee, who gets in. With some additional caveat that like we also

• 55:58Speaker 01

  have some diversity picks and and you know, minimum requirements because we want to support a great breadth of research and we think that the mentor selection committee in the whole might be biased in some ways as well. So we try and like really talk to the experts when it comes to picking agendas. And it so happens that like governance researchers have historically been like relatively low rated by our committee, which contains many governance researchers. I think I would I would go so far as to say that governance research is harder to do well in some critical sense, like it's harder to see like what is the actionable thing to do in some ways. Now, everyone who has their specific governance agenda, I would say like doesn't feel this way for a good reason, right? Within their within their agenda, they have like clear,

• 56:42Speaker 01

  actionable things to work on. But I think on the whole, there's just like so many more possible technical directions to pursue that are high leverage in some ways as well. I think a lot of the governance stuff is like, oh, we're trying to build, this is this is not talking about advocacy now, right? This is talking about technical governance. We're trying to build technical governance solutions such that if an administration so deems them worth, you know, deploying that we have the capacity to do that. We actually have the solutions that can be deployed, which is very important, right? But I would say that like, don't rule technical research

• 57:21Speaker 01

  out, because it is like, especially if we have something like a regulatory market or even like warning shots to cause the public to wake up and tell Congress to to regulate this stuff, right? We have to have technical solutions ready to deploy. To make these systems safer and the cheaper we make it, right? To make systems that extra degree safer, right? To lower the alignment tax that companies have to pay to train and deploy their systems to be safer, the more likely they are to do it when they come under pressure, either internally, externally or whatever. So I think that lowering the alignment tax via technical research is super important still. Also, if this alignment MVP plan is going to work,

• 58:02Speaker 01

  we have to have a bunch of directions for things to be iterated on by these AI assistants or humans calling teams of AI assistants, as it's more likely to be. And you actually have this massive interplay between technical research and governance research, where things like evals and safety cases built on technical AI safety solutions are things that can actually be tangibly put forward in policy proposals, right? And can convince policymakers with demos and evals and model organism honeypot traps, right, where AI systems deceive the users or whatever. This is what convinces policymakers to make policy and gives them a tangible target for their policy to work on, right? So there's a clear flywheel here.

• 58:46Speaker 01

  So I would say like, do not roll out technical research. And there is a reason why Matt has so many more technical mentors. And that's just because it seems like on the whole, our mentor selection committee thinks that, you know, I guess on average, a technical portfolio is worth pursuing.

• 59:02Nathan Labenz?

  Yeah, that reminds me of what Jake Sullivan said in terms of advice for the AI safety community, which was basically, you need to make this stuff as concrete as you possibly can so that people like me have something to really latch on to. Because as long as it remains sort of a theory or like a possibility or, you know, whatever, it's just really hard to get government to do much stuff, you know, on that basis. So he was like, the more grounded and concrete all these fears can be made, the more likely you're going to have success in the policy realm. You mentioned advocacy as well briefly there. Would you ever consider an advocacy track? And I guess it might be like advocacy research, you know?

• 59:47Nathan Labenz?

  I feel like right now we do have groups doing advocacy, obviously. I'm not sure how. Data informed their advocacy, you know, strategies generally are. But I'm always struck by when I do see survey results, it's like, yikes. You know, the, the public is not like super keen on AI in many ways. Do you think that would ever be something you guys would expand into?

• 1:00:11Speaker 01

  I mean, you assume we haven't.

• 1:00:13Nathan Labenz?

  Yeah, I, I haven't seen it.

• 1:00:15Speaker 01

  But yeah, we are a 501(c)(3), so we have to kind of, we have to keep our advocacy stuff to a minimum. And I think Matt's, a lot of Matt's strength is this kind of impartial player. Like, we're trying to be somewhat of a research university tech accelerator kind of vibe. We don't necessarily, we don't want to play favors politically. Like, that's not in anyone's interest, right? I think if people are doing that and they're trying to be the thing we are, they're doing a bad job. That said, right, I think like, we do, we do, uh, currently, I believe David Kruger is going to be a mentor in the current program. Uh, and some of his research is to do, that he's, he's gonna be discussing is to be, is to do with, I guess, like, what sort of messaging? And what sort of like standards are actionable, right? But of course, this is, I wouldn't say this is like true advocacy.

• 1:00:59Speaker 01

  This is more Matt's supporting independent research, working with David Kruger, uh, you know, who has his new org, Evitable. Not Inevitable, Evitable, which is focused on, on some of these advocacy questions. So I think Matt has to be pretty careful, you know, in terms of, you know, obviously our 501(c)(3) spending requirements for advocacy. We haven't spent anything on advocacy, for what it's worth. And also like, you know, ensuring this political neutrality. So that our fellows, our mentors and all of our strategic partners just can feel assured that we are like, you know, we're solutions oriented, right? We are pushing for a particular outcome, right? And I think that AI safety being a political football is just a bad idea.

• 1:01:39Speaker 01

  And I applaud advocacy orgs like Incode and plenty of others, like, you know, perhaps Case, et cetera, for their efforts to try and, you know, to, to, but that's not, that's not Matt's organization.

• 1:01:53Nathan Labenz?

  Yeah, gotcha. Toen the water at most for now. Let's talk about the profiles. I Both watched a talk of yours and read a blog post from about 18 months ago, where you kind of sketch out the different archetypes of AI researcher that you have seen, and then also kind of map that onto the demands of organizations. And I don't know how much it's changed in the last 18 months of it all, but maybe give us the kind of baseline and then if there's any update, I'd love to hear how things are changing, especially, you know, I have in mind, of course, Claude Code, and it may accelerate certain people, it may empower certain people to do things that they couldn't otherwise do. But yeah, tell us what, you know, first of all,

• 1:02:36Nathan Labenz?

  how you organize your thinking about the kinds of people that you're bringing into the program.

• 1:02:41Speaker 01

  So I mean, Matt's like, I've talked about mentor selection committee. Well, we are fundamentally, I think, this massive information processing interface, right? So we consult the very best people as much as we possibly can, right? And we try to like, you know, we build our own opinions, but we don't rely on them, right? We try and consult experts every, every stage. So the paper or blog post you're mentioning, which is called Talent Needs of Technical AI Safety Teams, to construct that, we surveyed like 31 different lab leads and hiring managers, whoever we could get, like the most senior person we could get related to safety at every AICD org we could find that was like hiring at that time. And we asked them, what do you need? And then we compiled all that, that survey, well, those interview notes into like three archetypes, right?

• 1:03:26Speaker 01

  This is just technical, we've since done this for governance, expect that to drop soon. So there's three archetypes for connectors, iterators, and amplifiers. So we chose the term connector because these people are bridging gaps between theoretical arguments for AI safety and theoretical techniques to make AI safe, and the empirical techniques to actually make it happen. So they're sort of like spawning new empirical paradigms to work on, okay? No one is hiring these people. It's pretty rare, because if you're good at that, then everyone knows your name and you're already hired. Perhaps you're already leading an organization. Everyone wants to be an ideas guy, but very few people want to hire ideas guys. And these people typically, it's people like Buck Schleszerus, you know, AI control, Paul Cristiano, right?

• 1:04:10Speaker 01

  Just huge amount of resources he's produced and so on. Like, there's, you know, you know these people, right? And they typically like have AI safety organizations. They founded and lead. Then there's iterators, right? And this isn't just, this is not just engineering, right? Iterators are active researchers who have strong research tastes, who are pushing the frontier, but they typically aren't creating like novel paradigms based on like theoretical models of things. They're typically advancing empirical AI safety. And you can even imagine like iterators and technical governance agendas as well. So this is the majority of people that are working in AI safety today, and also the majority of hiring needs in the future. And then there's amplifiers, who I think like the closest example was like, like TPM archetypes.

• 1:04:54Speaker 01

  I'll say this, for iterators, like prominent examples include like Ethan Perez, Neil Mandar, Dan Hendricks. Actually, I think Dan Hendricks maybe crosses some, crosses some boundaries there. But yeah, amplifiers, to distinguish them, they have more focus on, on amplifying people. And typically you'll find them on large research teams, and they're scaling the number of people that can be effectively managed and contribute to organizations. So a lot of maths research managers would fit this category, or TPMs at the various labs. And interestingly, they're actually quite in demand as well, particularly for labs in like the 10 to 30 FTE range. They're the most in demand archetype, because it's very hard to hire great people managers

• 1:05:37Speaker 01

  who also have the requisite research experience. You're trying to hit two bullseyes. And there are ways, of course, like Google has this, this sort of model where you have your research managers and your, your people managers, their project people managers, and they're, they're somewhat distinct. And Matt does try to do this with our mentors and our RMs. But yeah, that, that, that, I think the need for amplifiers is only going to grow. Because as you've said, things like Claude Code and other AI systems are going to erode away the technical, minimum technical skills required to contribute. And I think also AI agents are going to take more of those things. You end up in a situation where your people skills, your management ability, your networking, your amplifier skills in general are the more bottlenecking thing on AI safety research. So all those iterators out there, there are job opportunities.

• 1:06:22Speaker 01

  You are still the main thing everyone wants to hire. But if you don't try and build up your management capabilities, you don't work at managing AI systems, then you are going to be left in the works as the needs of the field shift to amplifiers.

• 1:06:36Nathan Labenz?

  So just try to echo that back to you. The connectors, another name for them might be conceptual visionaries. Like, these are the people that define research agendas where they just previously didn't exist, like de novo, high-concept work. They, in turn, need iterators, which sound like essentially machine learning engineers is kind of the core skillset I-

• 1:07:02Speaker 01

  Scientists, scientists, engineers, yeah.

• 1:07:05Nathan Labenz?

  And they're the ones that are running the experiments day to day, building the tooling, writing the code traditionally to like, do the visualizations of the, of the data and kind of taking this sort of initial conceptual hit that the connector came up with and like, really kind of systematically mapping out that space. And then these organizations, as they grow, then they start to need amplifiers, which I maybe would call like leaders, you know, people that can sort of build up an organization, see that people are working well together, that it can go past the sort of two pizza rule, right, in, in scale. Is that changing now? So, you know, when-

• 1:07:49Nathan Labenz?

  We hear things like 90% of code will be written by Claude. And that seems like it's, you know, kind of closer to right than wrong. And certainly like I vibe coded three AI apps for family members for Christmas presents this year, which is something I, you know, would not have come close to being able to build previously, even with, you know, just one or two generations of of model ago. I do wonder how much the skill set is already changing. What are you seeing there? Like, what's the, what's the up to the minute in terms of how people are thinking about changing hiring needs?

• 1:08:25Speaker 01

  I mean, up to the minute is that you have to be very proficient at using AI. And I think that some of the companies have updated their, like their coding interview processes, right, to allow for use of AI assistance, because on the job, like you, you have to be using AI all the time, right? That's just critical to succeeding in this field, to be amplified by AI. I would say that goes for every one of these archetypes we've identified. I do think as well that like checking whether AI output is good or not in critical contexts is still going to be a very important thing. And stitching together different types of AI output and building pipelines to more efficiently process that are also going to be very critical. But we might be leaving the lead code era.

• 1:09:07Speaker 01

  That's, I will say this, that like amplifiers, while not currently the most in demand across all different hierarchies of AI safety organization or team are, I think, probably in the next year or two, going to be the most in demand. But that's just, that's based on my predictions about AI progress. As you say, it could be slower. You know, there could be jaggedness concerns, you know, that slow down this type of talent transition. But in general, it's never good for your, it's never bad for your employability to spec out as a manager. Managers are very useful and leadership traits in general make you more useful, better employee. It's part of personal growth, I think, to take on some leadership roles.

• 1:09:51Nathan Labenz?

  What does supply and demand look like these days in terms of like, you know, I guess maybe even at the highest level, going back to the, you know, the sort of origin story of MATS, my understanding was you said, geez, this AI safety thing seems like it's going to require a lot of people working on it in a lot of different roles. And this is not something that universities teach, right? So like, how do people, what's the on-ramp to doing this if they're, if somebody would benefit from one, like, you know, where do they go? So you've essentially created one. And of course, there's some other others out there too, but you've created one of the, the largest and most highly regarded ones. Are, where are we in terms of like, are there a lot more jobs out there than MATS can produce fellows? Are there?

• 1:10:37Nathan Labenz?

  Yeah. You know, whoa, how do you think about that? So I've gone back and forth a couple of times where at one time it was like, oh, we're super talent constrained. And then it was like, well, maybe not so much anymore. Now it's like, there aren't actually a lot of roles for people to go into. So I feel like, maybe I'm wrong on this, but I feel like this is sort of seesawed back and forth. And I don't know exactly where we are today.

• 1:10:58Speaker 01

  So I'll start by saying I didn't found MATS. I didn't co-found MATS. I was in the pilot program as a participant.

• 1:11:04Nathan Labenz?

  Okay, that's good, right?

• 1:11:04Speaker 01

  Yeah, yeah. Yeah, yeah. There was like five of us who ended up doing the first research program. And it was, it was a pilot, like they didn't have open applications. It was just people nominated from what was the first AI safety fundamentals course, what's now Blue Dot Impact, right? And we did that. And the credit goes to Victor Warlop and Oliver Zhang, who is COO and co-founder at Center for AI Safety. And I joined the team right after that program. And I was, and then Oliver left, co-founded CASE, and Christian joined on as well. And then Victor left shortly thereafter. And I would say that like, I scaled MATS. That's my contribution.

• 1:11:43Speaker 01

  Yeah, and you know, since Christian and I kind of, you know, refounded it in that we like formed a separate 501(c)(3). A couple of years after that, because we got too big for our fiscal sponsor. So yeah, so I'll take credit for scaling that and for being the driving force behind strategy since I guess mid 2022, I believe. But, okay, in regards to talent needs, yeah, so that's, that's a good question. Yeah, sorry, actually, tell me the exact wording of the question again.

• 1:12:10Nathan Labenz?

  Well, yeah, what's the balance of supply and demand? Because this might not be right, I mean, you can correct me on this too, but I've had this sense at times where people have sort of said, there's so much demand for this kind of talent, like where is it? You know, we're talent constrained, but then other times I have heard from people that now people are rushing into the field and there's not actually so many roles available. And so people are kind of frustrated. But I don't know where we are right now in that back and forth.

• 1:12:36Speaker 01

  Yeah, so, okay, I'll say this. AI safety is a field where there are always going to be jobs for the best people. Okay, if you're, if you're, if you're, if you're a cracked coder, you can get a job in AI safety. Like the anthropic alignment science team is growing at 3X per year. Okay, they're trying to scale fast. FAR AI, a nonprofit, 2X per year. Okay, so that's itself, we've been growing 2X per year over our entire history. So the, these, these teams are scaling fast. Okay, and many more are getting founded. Open fill has, sorry, coefficient giving has huge amounts of, of, you know, grant money to spend on this stuff, right?

• 1:13:19Speaker 01

  There are like a dozen AI safety focused VC firms out there to fund your profit. Like there are incubators like Catalyze Impact, Selden Labs. I believe Constellation has one now. There's tons of programs like that. I think the problem is that the, the, the level, once you have built an organization, especially if you're scaling very fast, right? There's a certain size. The main constraint becomes like, is this person good enough to warrant the extra management overhead? Can they take on some management responsibilities? So you have the situation where like at OpenAI, like where people are managing like 10 to 20 individuals, maybe up to like, I believe one person in anthropic alignment science had 18 reports. So they're really flat.

• 1:14:02Speaker 01

  So you have this real problem where you need to hire people who can quickly ascend the ranks and be research leads, be managers and so on. Even PIs of new teams, and that is like the limiting constraint. Okay. And that's the reason why a lot of people do some moderate reskilling and then can't get hired. Because I think th- these jobs, like there are many, many opportunities. But it's, what we find is when we talk to these hiring managers, they say, we find it extremely hard to hire. We have the money, right? We have the clear need, but people are not at our bar. And that's what, that's what Matt's is trying to do, is to get people up to that bar. And, you know, that, that, that's, there's some technical skills element to that, right?

• 1:14:47Speaker 01

  There's also some, you know, just, just like actual research experience. So people who come into Matt's with prior research experience do so much better on average than people who have less research experience. Right? So I think a strong option for many people should just be stay in academia. Get your bachelor's, get your PhD.

• 1:15:07Speaker 01

  For other people, maybe they should go off and found a company. There's tons of money and, and directions for AI safety companies. I, I think founders are strongly needed in this ecosystem. And then you can create opportunities for more people to, to get hired. But I'll say as well, another thing Matt's is trying to provide is like credibility. I wouldn't say like, like not formal accreditation, but some sense like they have the reference from their mentor, who's a senior researcher in the field. You have the, the exhaustive Matt's selection process, which is trying very hard to find people who are, who are good. And, and then you have like your proof of actual research impact. You produce a paper, right? That's, that's your name. Perhaps you're going to publish in a conference or it's an archive and people are talking about it. So that's what people need to get employed these days. Right?

• 1:15:48Speaker 01

  You need to have an actual like great output, some sort of deliverable that you can put, show your name, maybe several, right? You have to be like technically good enough at coding or using AI systems, whatever's required. You need references from people that are trusted. Otherwise, it's just very hard to get ahead. The same story that you see in every, you know, talent constrained job market is here.

• 1:16:08Nathan Labenz?

  How does that translate to experience profile? This is obviously a big question in the broader technology world, right? Like our junior programmers and endangered species. We see like very prominent examples like Neil Nanda and Chris Olah, who are broke into the field at a super young age, also kind of defined it in a way that are still quite young, actually, even today. And that may lead people to think that like, this is like a young person's game. But what you're describing is more like, it sounds like more like post PhD or sort of, you know, somebody who's kind of been, you know, grown up in an organization to an extent. I'm thinking of Rajiv from the AI underwriting company, who was at McKinsey for a number of years and now is, you know, co-founded this organization, but, you know, comes to

• 1:16:53Nathan Labenz?

  it with like, a ton of experience and sophistication in terms of management, leadership, all that kind of stakeholder management kind of stuff. What do you see in terms of like, is there a lot of opportunity for people, say, straight out of college? Or is that, are they kind of barking up the wrong tree if they want to go directly from like undergrad into this space?

• 1:17:14Speaker 01

  So, I mean, the median maths fellow is 27. Okay. So there's like, somewhat like a long, normal distribution, long tail. I think the oldest one last year was like 55, 60. So there are people of all ages applying to maths. The youngest person is of course 18 because we can't take minors. Now, okay, more statistics, right? 20% of maths fellows are undergrads. They have no bachelor's degree. Okay. Or perhaps, perhaps some of them haven't even applied for bachelor's, right? They're just cracked engineers. About 15% have PhDs already in the bag. So at least as far as maths is concerned, as like this accelerant reskilling, retraining, internship, mentorship program, whatever, you're getting a broad distribution of people.

• 1:17:55Speaker 01

  Now, I think that there is obviously huge demand for people with more experience. A second critical thing is that they have experience with the latest tools. And because these tools haven't existed very long, young people have a strong chance of, you know, being

• 1:18:13Speaker 01

  People who are particularly good at using these tools because they've just been like constantly in the cusp of things. They haven't been sitting in like a cubicle, not using Claude Code every day. So from to that extent, young people have a huge chance. But it is the case that like you do gain valuable knowledge from working on the job, especially in a great team, producing great papers that you can't replicate. You've pointed to some what I'd call prodigies, you know, Chris Olda, Neil Nanda. There's plenty of plenty of people of that of that ilk who have come through maths, that no one I think, actually, that's not true. There actually are some people who I would put in a similar class, like Marius, Havhan, et cetera. And like, in that case, like, our main job is just to like, just get out, get out of their way. And I think that if you if you're that kind of

• 1:18:56Speaker 01

  person, don't let anything hold you back. Apply to maths, apply for grant funding, do whatever, come to the bay, go to London, and just make it happen. You know, you will find your path. If this is like, maybe not your path, and if especially if you're like a more a senior researcher, or perhaps like a person who's like, man, I can't conceive of that. I just want to finish my undergrad degree and do a PhD. That's fine, too. People of every single walk have passed through maths, got hired, done other programs, got hired, founded companies, et cetera. I think like the advice, it's hard to tailor advice to like, you know, a myriad of different types of people. But I would just say, focus on your technical skills, focus on understanding the frontier of technology. And yeah, don't

• 1:19:38Speaker 01

  be, don't be limited by the opportunities you see on job boards, right? You can create your own opportunities. You can cold email companies. You can like, apply to grant funders with just some random grant proposal you put together because it fascinates you deeply. You can call up hiring managers and stuff.

• 1:19:56Nathan Labenz?

  One of the, I mean, when you describe the range, it is a pretty broad distribution. And that tells me that you trust your own ability to discern who's going to be good more than you trust other outside signals. So maybe tell us like what you are actually using to assess people. And this could be translated into practical advice in terms of like, how does somebody make an application stand out? But you know, what what are you looking for that allows you to take up somebody in their 50s or somebody who's 18 and feel like, you know, you can read what really matters regardless of what their background is?

• 1:20:34Speaker 01

  Yeah, so I mean, we do some of the standard stuff that you would see at other tech companies, right? Like, we have CV review, we have some code signal tests, so brush up on your coding skills and so on, and they do detect AI use. We are, of course, considering ways to allow for tests that, you know, include AI use, but these are obviously harder, right? They're harder to design, they're harder to check and so on. But yeah, that's that's part of our general application. Now, that's for that's for some streams. I'll say this mentor selection, sorry, server selection, like, we're trying very hard to provide something, like a service to mentors. So if a mentor says to us, I don't want to do CV review, I don't want to do code signal, I just have this selection problem that I want fellows

• 1:21:18Speaker 01

  to, applicants to work on, and then like, I want you guys to help me evaluate this. There will be like a team of contractors or some automated evaluation process to do first pass screening, and then we'll go from there. That's our favorite kind of evaluation in some way, because we know that is like as close as possible to the actual job, the actual research, as we can get. Of course, this is like typically, in Yonanda's case, it's like go away and do a 10-hour, you know, Mechathep kind of pseudo-work test, and then present your results to me. You can use AI, do whatever, just find something interesting. And this is like, this is great, because then we get great results. Some other streams, it's harder to do this, it's harder to like administrate, and so we do rely on some proxies

• 1:22:02Speaker 01

  that are perhaps less specific than ideal, but I think are no worse than anyone else in the industry is doing. And of course, like, I think the way you stand out, obviously it's going to depend on the specific mentor, because maths is very heterogeneous in that respect, right? This like, the best thing to apply to Yonanda's stream is going to be vastly different than applying to Ethan Perez and the Anthropic mega-stream. But in general, you want to like, really understand your basics about AI safety, so do a Blue Dot course, right? Because there may be some critical knowledge or a paper that if you haven't read, you don't understand. If you don't understand what deceptive alignment is, that might be really bad for Ethan Perez or Buck Schleder's kind of control research, but even applying, getting into the streams. If you don't, if you don't understand that for Interpstream...

• 1:22:48Speaker 01

  Probably doesn't matter as much unless of course you're dealing with deception in your inter thing. So make sure you understand your basics. Make sure that if you're applying to a stream that is empirical heavy, that you can do code signal tests, you can code, including without AI assistance, at least for the time being. And it doesn't hurt to apply to other programs as well. And that's as far from the only program out there now. This is not like early days. Like, there are so many great research programs out there. Pivotal, Era, PIBS, Laser Labs, SPAR, Arena for Technical, I think Astra is now running again. Yeah, there's, there's tons of great programs out there. And that can really boost your CV. If you have experience in the kind of research that you want to do at Matt's

• 1:23:30Speaker 01

  already, then so much the better. Consider it like a postdoc opportunity or something, or a post research opportunity. Build your own independent projects. Yeah, sorry if that's like too much advice to be actionable.

• 1:23:43Nathan Labenz?

  Yeah, I think it boils down to tangible product is, is like king, right? I mean, and I say that always in the AI engineering world as well. If any, you know, and I'm far from the world's leading expert on how to break into that space. But what I always tell people if they ask me is a working demo is kind of the coin of the realm, you know, like, it's all, people might be interested in what you have to say, but they really want to see that you can make something work. They want to see it online. It could be a replit, or it could even be a collab notebook or something. But like, it's got to, you've got to make something that can work. And it sounds like this is a pretty similar worldview. Like,

• 1:24:28Nathan Labenz?

  you got to show that you can get in there, make something happen, find, as you put it with, with Neil's track in particular, like, find something interesting. If you could do that, you know, we, we might have something to talk about. One thing that, that jumps out as like, maybe not as emphasized as I would have thought is being in command of current research. That's something, I think, at this point, like, really, nobody can keep up with all the current research because, you know, that exponential has gotten away from all feeble human minds, I would say. Maybe with a few, uh, hyperlexics that can still keep up, but.

• 1:25:06Nathan Labenz?

  I have found like keeping up with research is a pretty feels important to me. It feels like an important part of like how I stay conversant with people across a lot of different areas. But obviously what I'm doing in trying to be conversant with people across a lot of different areas is not the same thing as research. How much emphasis do you think mentors in general put on like being, you know, on top of the literature, so to speak?

• 1:25:33Speaker 01

  It varies. So there are some, some of the mentors will ask for questions like, I don't know, what do you think about X concept? Others won't be as interested. Obviously, as you say, right, these costly signals are the most important thing. Like, have you done like good research? Can you, do you have a deliverable, like a product? Do you have a strong reference for an important person? That's also key. Do you have, like, have you done your homework in terms of the Blue Dog course and other things, right? I think that math selection doesn't currently emphasize breadth of knowledge very much, mostly because mentors don't necessarily want that. And I think that this is maybe a weakness in our process to some extent if we don't then help people build that breadth. But we do.

• 1:26:18Speaker 01

  We have seminar programs. We have tons of opportunities for intermingling between different research streams, which really rapidly builds a breadth of knowledge. And we have like, we used to have like discussion groups and these still occur occasionally with like workshops and so on. So I would say like, I really do encourage everyone to do like a basic Blue Dog course or equivalent like ASAT-Atlas and CASE have good courses as well. But I think that like this is not as required for selection and it's more just like to prevent you from entering maths and then like starting to do a research project and realizing, oh crap, I have no idea where the gaps are. I don't understand how my work fits into anything. How do I get funding after maths? How do I get a job, blah, blah, blah? Like, how do I choose

• 1:27:00Speaker 01

  to go to original research direction? So it's more for like your ability to actually deliver within the program, right? Tracking research and less to do with like your ability to get in at the moment, which is pretty important because maths is just a stepping stone, right? If you do maths and then you don't produce a great deliverable by the end of maths, sure, it's a great thing on your resume, but it's not going to be enough in many cases, you know, because it's such a competitive environment. So yeah, I think that it's really good for people to build a shallow but broad understanding of the literature. So I would recommend like, don't be checking X constantly for new papers, blah, blah, blah. Unless they're in your field, maybe set up some Google Scholar alerts for Interp, if that's your thing. But like more, every so often, do a periodic like deep dive into like, what are all the cool updates across

• 1:27:45Speaker 01

  different fields? You know what I mean? And you can do this by like, every year, looking at the new Boodog course, or every month, looking at some research roundups or highlights, like Zvi as a newsletter, or Transformer. And there are other people you can follow on X. That's my main recommendation.

• 1:27:59Nathan Labenz?

  So, um, your admissions rate, it's super low, right? I don't want to, we want to encourage people to apply, but it is a very selective program. What does the kind of funnel look like in terms of applied? I don't know if there's intermediate steps that, you know, would make sense to talk about selected. And then I think the good news, though, is if you do get into the program, your success rate on the other end of like getting into the field in a professional W2 employee status sort of way is really high. You want to run us through those numbers?

• 1:28:33Speaker 01

  Yeah. So last program, I think we accepted around 5% of people, maybe 4% who applied in our initial like intake form. There was a subsequent process that they had to complete, which is applying to specific mentors and streams. And I think we accepted somewhere around 6.5% to 7% of those people. So a bit higher. That maybe is the figure out focus on. Somewhere around 7%, let's say. Now, that's better than people think, right? They hear, they thought it was something like a, I think the Anthropic Fellows program, for example, is like 2% because Anthropic, big name, right? But maths is larger. We have more diversity and more spots and so on. And I think that like, yeah, I think people should also just

• 1:29:16Speaker 01

  treat the application process as a learning experience. In general, this is, we try and make it like,

• 1:29:23Speaker 01

  Some streams are gonna be like more painful than others, but I think that like for for the streams like Neil Manda's where you spend 10 hours working on a project, then you have something really cool for your GitHub. And that can only help. Yeah, you're not like working with Neil anyway, I would imagine. Yeah, definitely, definitely the case. I I do think it is um it's unfortunate that there aren't like easy ways to do credit assignment cheaply, right, to find the best people with without them spending a bunch of time, but I I don't know, like, I I know that job interviews for for top tech companies like OpenAI and Anthropic and GDM just the last like three to six months or something like you just like you've got so many things to do before you get finally the the yes or no.

• 1:30:04Speaker 01

  So we definitely aren't that involved, right? It's a much slower process. And I think that's because like the commitment is less, you know, in our end, right? We're not marketing people W two's. NASA is an independent research program to get grants from a third party. We provide the housing, the office, the mentorship community, but we don't um sign people on for like any type of employment, uh, which I think is part of the appeal as well. So I think so that's that's the main statistic there, like 7%. At the other end, about 75% of our accepted fellows go on to our extension phase. So our first three month program, 7% get in, 75% go on for another six months, maybe even 12 months in some cases. And that extension phase is where a lot a lot of great follow up research happens.

• 1:30:46Speaker 01

  And of the people who've done our program over history, we've had 446 fellows in total, uh, not including people who've done training programs that we've helped facilitate, of which there are probably another like two, 300 of those 446, 80% have gone on to get a permanent jobs in AI safety based on our latest statistics. So that's great. I think, yeah, like, I think 98% are employed in some capacity. Now, of those 80%, not all are like W two's, right? Some of them are independent researchers with grant funding from coefficient giving or LTFF or something, which I think is like a fine situation, right? And then in terms of like the actual field growth, there's some statistics I can share. So it seems like based on Steven Micalle's less wrong investigation,

• 1:31:31Speaker 01

  that something like the the growth rate of the AIC field is extra 25% per year. Which is kind of interesting, right? It does seem to be growing exponentially as far as we can tell. Now, that is a lot less than the growth rate of MATS applications, which are going up somewhere between 1.4 to 1.7x per year, depending upon how you slice it. And mentor applications might be increasing around 1.5x per year.

• 1:31:58Speaker 01

  So there's a big disconnect. And according to Blue Dot Impact, I believe that their growth rate is something like 370% per year in terms of applications to their programs. So there's like, yeah, some large disconnect. So a lot of people are applying to Blue Dot. That can't go on, you know, at that rate. That's just, that's just way too fast. But probably because they've done amazing advertising and marketing stuff. MATS has only just started to do advertising and marketing. We had the first ever mentor open rounded applications launched just recently. And yeah, we sponsored Europe's. That was cool. And we're, we sponsored your podcast and several other

• 1:32:37Speaker 01

  great, great venues as well. And I think that this is only going to cause probably the application trend to continue. I would say, I would guess 1.5x per year, something like that. Which that is a faster growth rate than the current deployed growth rate of the field. As to why, I could speculate it's probably just caused by like, like a high bar, a very high bar for a lot of these companies, maybe a deficit of founders as well. And there are plenty of organizations working to remedy that. I know there was this AI Assurance Technology report from Juniper Ventures about a year ago, where they predicted that the size of the market for AI Assurance Technologies is doubling each year.

• 1:33:18Speaker 01

  So there is a lot of opportunity to do stuff that might contribute to AI safety.

• 1:33:23Nathan Labenz?

  What does the salary distribution look like for the people that are getting jobs? Like, how much of an alignment tax, if any, do people pay at the salary department?

• 1:33:35Speaker 01

  I mean, at the frontier labs, no tax at all. You're getting paid the same rates if you're leading. Yeah, they're getting staggering amounts of money. I think like a couple of years ago, the going rate for like someone joining off the street was like 370K or something. I'm sure it's much higher now, given all the, especially if the crazy meta stuff that happened. I mean, I would bet like mid-level and higher people are making over a million on these, even on the safety teams of these labs, but I don't have any private data on that. But yeah, I mean, like if you join as like a junior software engineer, don't be surprised if you get somewhere around 350K or something. At nonprofits, obviously, it's lower, right? They can't compete with equity. They don't have any equity, but they also typically have less funding.

• 1:34:19Speaker 01

  You know, coefficient giving's pockets aren't as deep as the collective might of US venture capitalism. And I think there is also something like, there is something like a nonprofit tax. I wouldn't say there's a safety tax. I'll say there's a nonprofit tax, right? Because there are nonprofits that are doing like AI capabilities stuff, like Alan is super AI. People make a lot of money still, right? There's like, this is, this is artificial intelligence, right? And coefficient giving understands and other funders understand that, you know, you have to pay to play. So you have organizations like Meter that where they are, I believe, offering quite a lot of money for their roles, upwards of 300K for, for most roles. Like, yeah, probably over a million for some, I would, I would dare say.

• 1:35:03Speaker 01

  So there are nonprofits that are really, really trying to compete for, for talent. They can't offer anything like the frontier lab salaries, including equity, but. They're trying their best, and I think this is kind of reasonable, but it also is, it is a bit, it is a bit of an insane moment. Matt's salaries are not anywhere near that high. Maybe we're doing the wrong thing. I don't know. Yeah. And I think there are other AI safety nonprofits that have tried very different strategies. Like you have, I think, the going rate for far AIs, research scientists or something like 100, 170 K. So like significantly lower. They might have actually improved that recently. So there is a wide spectrum here, and it really depends on the compensation policy of the organization. But you will see, you know, like very well-funded

• 1:35:48Speaker 01

  nonprofits offering comparable salaries to at least some junior AI company roles.

• 1:35:54Nathan Labenz?

  What about in terms of compute? I know you guys have a, in addition to the stipend that folks get as fellows, there's also a compute grant, and I believe it's $12,000 worth of compute. I'm interested in what form that takes. Like, is it, is it just a Brex card that you can go spend on compute wherever you want? Or do you guys have like established compute partners that you work with that, you know, serve your fellows well? How often is that enough? Are there times when people find that they need more compute to do what they want to do? And, and then if they go work at a nonprofit, like how compute rich or poor are the nonprofits?

• 1:36:34Speaker 01

  Yeah. So I mean, Matt's, Matt's offers 12K. I'll say budget. I won't say like, we don't give people a card that says here's 12K. No, they have to justify their compute spending. You know, they have to like have an actual project and proposal that they are, you know, spend that on that necessitates that. And most people don't spend anywhere near that much. Okay. Which is good, because that's part, we haven't, you know, we budgeted as if they could, but we really don't want to spend, you know, just waste money on compute. Basically, no one has compute limitations. Sometimes people rarely have needed more, more than that. We've like considered their proposal and thought hard and reallocated funds as needed. And Yeah, I think like people aren't limited by compute at MATS in general.

• 1:37:16Speaker 01

  I think that the way we do it is pretty good and that we have a bunch of like, you know, kind of like we have our for our model API calls and all that. Like we have specific organization accounts that we sign people up, you know, and then we kind of give them a budget and so on and top them up as necessary. We do have our own MATS cluster as well that we maintain online, so our compute team handles that. But typically, people tend to use RunPod and other types of like self-service things. It depends, it depends on the kind of research. Like there's some, there's some types of research that works well on our cluster and there's some types of research where even with the benefit our compute team can provide in setting, setting up the cluster and maintaining it, it like the experiments are so like customized

• 1:37:58Speaker 01

  and so they need to like tinker so many things that we just give them a budget and let them use online providers. It's just better that way. We did, we did have to use other providers in the past, but that's, that's our current setup. And we're looking at putting together like a Claude, like kind of customized Claude code, kind of sweet as well.

• 1:38:14Nathan Labenz?

  Hmm. Meaning like building out a bunch of tools or MCP type?

• 1:38:20Speaker 01

  Yeah.

• 1:38:21Nathan Labenz?

  Again, tools.

• 1:38:22Speaker 01

  I don't think, not, not MCP at this, well.

• 1:38:24Nathan Labenz?

  That's kind of.

• 1:38:25Speaker 01

  That's, yeah, that's a good idea. Yeah, yeah, you're right. Yeah, there could be actually, there's a lot of data, you know, and that's databases. We could probably put together something really useful at some point, but we haven't, we haven't thought about that.

• 1:38:37Nathan Labenz?

  Yeah, just the archive of everything that's been tried would be pretty fascinating to do some agentic search through.

• 1:38:44Speaker 01

  I mean, our new research database is online, matsprogram.org/research. You can see everything there. We have a Google Scholar, but that's just all the papers that got published. You can also see our Les Ron blog post under the MATS program tag, but many more research artifacts have been produced than are visible there.

• 1:38:59Nathan Labenz?

  You've mentioned, you said the word tinker. Is the Thinking Machines API growing in prominence in terms of what people are finding attractive to use?

• 1:39:10Speaker 01

  Yeah, many people are wanting to use the Thinking Machines API. So we've, we've put together some, I think we, yeah, I, yeah, many, many, I'll say many organizations like to donate API credits, which is awesome because we can really use that.

• 1:39:26Nathan Labenz?

  Yeah, cool. But is there anything else that we should cover in terms of like January 18th? We know that is important as a date to keep in mind. What other facts should we make sure that we touch on?

• 1:39:42Speaker 01

  MATHS is growing. We're always hiring. If you want to, if you want to work in our team and help grow the next generation of people, if you fancy yourself an amplifier of sorts, you have people skills and research skills, we'd love to hear from you. Go to our website, matsprogram.org/careers. We're taking on mentors as well. There's an application form on our mentor section of our website. And participants, we're going to run three programs next year, this year, rather, not one, but, sorry, not two, but three. That's going to be a summer program, a fall program, and then a winter program, starting into next year. And I'm super excited. We also have plenty of other offerings in the work. We're considering a one to two-year residency program for senior researchers as well. And yeah, more on that to come.

• 1:40:28Nathan Labenz?

  Yeah, cool. Are you taking any connectors? Is there, if I am a connector type or want to become one, is MATHS a way to find my way there, or not really?

• 1:40:43Speaker 01

  Many have. I would call Jesse Hoogland, one such person. I would call Paul Rikers. So Timaeus and Simplex, I would, I'd say Marius Hoppan as well, to some extent, with his deception evals work. Yeah, like, and I don't, probably dozens of people. I'm just like sharing some of the more, the names that come more easily to mind, but just many, many people have come through MATHS. We're super open to individuals who have this kind of archetype. And note, a connector, right, they have empirical skills, they have theoretical skills. So they could probably succeed in a bunch of different ways, right? But they're uniquely specked out to connect those two things. Now, there are some mentors and projects that are much more suited to this kind of thing than others. People like Richard No, historically Evan Hubinger.

• 1:41:28Speaker 01

  I think, I think actually Evan Hubinger has been like, probably the most dominant connector driving force at MATHS over, over our time. But he's not a mentor in the next program, unfortunately. He doesn't have time. But yeah, there's, there's many different opportunities at MATHS for this kind of thing. I think even in some of the interp streams as well, it's very possible to enter an interpretability stream and bring it with it, like some, some like model of the kind of theory-based interpretability mechanism or strategy that you want to pursue, and then see that executed on. That's happened several times.

• 1:42:03Nathan Labenz?

  One of the things that I took note of in the blog post from 18 months or so ago was you had made a comment that funders basically don't want to, or they're much more inclined to support the growth of organizations that they sort of see as legible, that have like research directions that they sort of feel are somewhat established, or that they can wrap their heads around, and they're much more reluctant to fund like totally new conceptual directions. And that seems like it exists in contrast with like the AE Studio survey where they basically found that the field as a whole seems to think that like, we don't have all the ideas that we need, and you know, that like more kind of far

• 1:42:46Nathan Labenz?

  out ideas should be tried, which of course led to their neglected approaches approach. What do you make of that? Is there stuff that we can do? Or is it, you know, is there, is it a different organization's job to figure out how to fill that gap? Because I do feel like I want some more, and I love some of the AE Studio stuff, including like self other overlap. I always come back to that as an example of something that's just like quite off the map of what most people are doing. I think of, you know, when I think of AI control and like what Buck and the Redwood Research Team are doing, I find that stuff fascinating. And one of the things that kind of impresses me most is that they are willing to work on something that in some ways is so depressing. You know, they're

• 1:43:29Nathan Labenz?

  like, we're gonna try to figure out how to work with AIs, even assuming they're out to get us. And I'm like, yikes, I don't know that I would be able to sustain the positive attitude, you know, enough to do that if I was working from that premise. I do feel like there's a relative dearth of things that are more sort of inspiring. Here, I think maybe of one AE Studio, but also like Softmax. Obviously, people have a lot of different opinions on like, are these things ever going to work or not? But I wonder what your take is on just kind of the overall mix. It seems like a lot of things are kind of more toward patch the holes, keep the AI down, tempt it, you know, see if it'll take the temptation, then patch it, you know, if it takes the temptation. And there's not nearly as much that is sort of a

• 1:44:12Nathan Labenz?

  A more kind of colorful, positive vision for the future. And I wish there was, but maybe that's just not happening because the ideas are just too hard to come by. Maybe it's not happening because the funders aren't bold enough. What's your take on how we can get, if we should be trying to get more of that stuff? And if you think we should, like, how might we go about it?

• 1:44:34Speaker 01

  I have many takes here. So obviously I advocate a portfolio. And that's, Matt's has historically sponsored a bunch of project. Like self other overlap. That project came out of Matt's alum. I'll just say Mark Carlinia, I might have messed up his name. It was like the originator of that project, A Studios. And I believe Cameron Berg is running there. Another Matt's 1.0 alum with me is running their, some of their, the more neuroscience inspired approaches as well. So A Studios is great. I love what they're doing. I think that the survey they did of less wrong is just like probably not representative of the AI safety research field as on the whole. But it might be, even so, I think we obviously need more ideas because more ideas are good. Right? More bets are good.

• 1:45:18Speaker 01

  More shots on goal are good. Now, I would not advocate a person who is like a very strong iterator to drop that and be trying to become, think of some new paradigm. That is, I, I think that would be strictly counterproductive on the margin. Because I think we do have very, some very strong central research bets that we need more people pursuing because they will yield damageable results. But if everyone did that, this would be bad because you need to have your portfolio. Maybe these approaches fail. Maybe they need other pieces to work. Many AI safety research agendas are kind of contingent on other things going right or other people working on other stuff. It's like any kind of research field, you need to have everyone advancing the frontier.

• 1:45:58Speaker 01

  So I think AI safety has historically gone really kind of arg maxi on different agendas, which is bad.

• 1:46:06Speaker 01

  Portfolio approach is much better. Don't rule things out as possible directions. Just shift and reallocate resources to them. To their credit, coefficient giving had done an amazing job, particularly recently, at supporting a bunch of different novel research bets. And they've also funded PIBs, or Principles of Intelligence, a program that is trying very hard to pursue sort of moonshot-y interpretability and like, agency-understanding projects. So, they're great. Check them out. I think that more ideas would be good. I think that the kind of person who should be pursuing that typically is going to look something like someone who is already a domain expert in some other area. You are occasionally going to have your Buck Schlederses, your Evan Hubingers, right?

• 1:46:49Speaker 01

  Come along with no PhD, but spent years at Miri, you know, incubating in that kind of, that deep AI safety, that rich AI safety experience, and then come out with amazing stuff like risk from learn optimization and AI control and all that stuff. But short of, you know, having access to that type of community and that type of research experience, I think most of the prominent connectors, like your Alex Turners and so on, have come, have spent a lot of time in research, science PhDs, also on Bestraun, of course, and like, you know, incubating in that environment as well. So, I think MATS is a great way for that kind of person to develop and to spawn more research ideas. In fact, I've seen, to shout at Alex Turner, he has come up with some amazing research

• 1:47:33Speaker 01

  ideas over his time at MATS, and I think we've been very fortunate to support him. Things like gradient routing and also Alex Cloud, another MATS mentor, and just plenty of other things, like activation engineering and searing. He was, he was one of the people involved in that. So, I think that like, senior experienced researchers are going to be probably like most things, the main drivers of new ideas. And grant funding that lets them pursue whatever their research taste dictates is great, and programs like MATS that let them stalk, you know, their research agendas are also great. I also think bounty programs could work as well, but I would hazard against people putting all their eggs in the basket of, we need to have a bunch of ideas because, new ideas, because the central idea's not working. I don't think that's true. I think the central ideas are still our actual best bets.

• 1:48:19Nathan Labenz?

  Yeah, okay, makes sense. Do you want to shout out any other, either just organizations that, you know, Matt's fellows have gone to or even started that you think are kind of underappreciated? This could be sort of assignment editing for me for future episodes, but also just, you know, things that you think people should be paying more attention to than they are?

• 1:48:37Speaker 01

  Yeah, I mean, there's tons. Like, you can see all the organizations listed on our website. There's so many amazing people there. Like, I guess I could shout out. I mean, it's hard to play favorites because Matt's is like, we've worked with so many people and we're trying to be very broad. But, yeah, I guess like, in terms of nonprofits specifically, because maybe they don't get as much attention, obviously Redwood and Meter and Randcast, Paula Research, FAR, Goodfire, Truthfully I, Law Zero, Miri, plenty of others. I love these organizations. We need more, frankly, nonprofit research organizations. And if you think you could find one, give it a shot. Obviously, like, you need to have a ton of

• 1:49:18Speaker 01

  research experience under your belt and very credible references and so on. Yeah, I don't know. It's really hard to play favorites, Nathan.

• 1:49:26Nathan Labenz?

  Yeah, there's always so many great organizations. We're foiled for good options for organizations to shout out. So it's a testament to how many fellows have already gone on to do impressive work. So, great job by you guys in driving this and growing it over the last few years. And people should definitely apply if they want to be a fellow. January 18th is the deadline. It's time to get into it if you want to make sure your application stands out. Anything else we should touch on before we break?

• 1:49:55Speaker 01

  No, I just really appreciate this experience. Thank you so much for inviting me to talk.

• 1:50:00Nathan Labenz?

  My pleasure. Thank you for doing it. And keep up the great work. Ryan Kidd, Co-Executive Director at Matt's. Thank you for being part of the cognitive revolution.

• 1:50:09Nathan Labenz?

  Thanks Nathan. If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 1:50:51Nathan Labenz?

  at ai podcast dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.