Zero to One in AI Safety: Halcyon's Mike McCormick on Launching 30 New Orgs & the Founder Bottleneck

Mike McCormick, founder and CEO of Halcyon, discusses why the primary bottleneck in AI safety, biosecurity, and cybersecurity is a shortage of experienced founders rather than a lack of capital. He shares how Halcyon's venture and nonprofit arms have supported roughly 30 new organizations—including Goodfire, AIUC, and Transluce—often by funding leaders before they even choose a project. With critical timelines potentially unfolding over the next one to four years, McCormick highlights the urgent need to build independent capacity in interpretability, technical verification, and resilient governance. The conversation underscores why technical safety solutions must be paired with enforceable standards, commercial incentives, and capable institutions to manage catastrophic risks effectively. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/zero-to-one-in-ai-safety-halcyon-s-mike-mccormick-on-launching-30-new-orgs-the-founder-bottleneck/ Sponsors: ElevenLabs: ElevenLabs lets you deploy enterprise-ready conversational AI agents that talk, type, and take action in over 70 languages. Schedule your demo today at https://elevenlabs.io/tcr OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (03:54) Introducing Halcyon (06:09) AI safety portfolio stories (Part 1) (19:07) Sponsors: ElevenLabs | OutSystems (21:57) AI safety portfolio stories (Part 2) (22:42) Halcyon's core thesis (34:14) Funding vs joining orgs (Part 1) (34:19) Sponsor: Claude (35:54) Funding vs joining orgs (Part 2) (47:03) Mission preservation governance (01:00:21) Neglected safety areas (01:09:09) Evaluating AI consciousness (01:19:36) Institutional coordination mechanisms (01:35:59) Transitioning into AI safety (01:46:11) Episode Outro (01:48:08) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Mike McCormick, founder and CEO of Halcyon, a combination nonprofit grant maker and for-profit venture fund, working toward a single mission of helping the world's most talented founders and leaders launch new AI safety, cybersecurity, and biosecurity organizations. Mike's core belief is that organizations solve problems for the world. And of course, AI is creating a lot of problems that we need to solve in a remarkably short period of time, which means that what we need most right now isn't more ideas, but founders who can build the organizations needed to implement whatever turn out to be the most viable solutions at scale.

• 0:44Nathan Labenz?

  In three years, Halcyon has helped launch roughly 30 such organizations, which have collectively raised roughly half a billion dollars. Goodfire, who's been featured on the show multiple times, is the best illustration of how Mike works, with a special focus on the six months before and six months after an organization is started. The first career transition grants Mike ever made were to eventual co-founders Eric Ho and Dan Balsam while they were still running their previous company, but starting to think about changing focus to work on AI directly. The financial boost, the vote of confidence, and the network-building support he provided helped them found a company which has since achieved unicorn status and become one of the best places in the world to do interpretability research.

• 1:29Nathan Labenz?

  Other orgs that Mike has supported include past guests, AIUC, which does agent standards and audits for insurance purposes, Fathom, which has advanced policy discussions with credible early proposals for independent regulatory schemes, and Asymmetric Security, which is building cybersecurity forensics agents meant to defend companies from employee account compromise incidents. Beyond that, Mike also introduces Transluse, which is working on scalable oversight, and Hadrian, which is stockpiling PPE. It is admittedly a very broad portfolio, but as Mike puts it, "we are speedrunning six or seven industries worth of AI revolution all at once."

• 2:10Nathan Labenz?

  In a 2026 that remains plausibly on the AI 2027 timeline, rapid, competent execution is obviously critical and clearly will require heroic efforts. To take just one personal example, for nearly a month now, I've been trying to donate aerolamp lights to my son's classrooms. And while my son's teacher said that she loves the idea, it's a new thing and nobody at the school has been able or willing to sign off on it. This too, it turns out, Mike is involved in starting an organization to address. Naturally, we talk about how seemingly short timelines affect his strategy, what he looks for in people to back, and how he thinks about both for-profit and nonprofit governance. And we also discuss the verification technologies we'll need to sustain pacing agreements, as

• 2:54Nathan Labenz?

  well as the challenge of getting everything from cryptography to hardware manufacturing to diplomacy, all working together as one. The main headline of today's episode, though, is that there is still time for ambitious leaders to start new AI safety organizations, which could prove incredibly important, and more broadly, that there's never been more support to scale these organizations with first-class talent, which means that all sorts of accomplished professionals, including many who do not have technical ML backgrounds, should be actively thinking about pivoting their careers into AI safety. If you are feeling the pull, Mike offers his contact info later in the episode, noting that a warm introduction is welcome but not required.

• 3:38Nathan Labenz?

  With that, I hope you're inspired to see yourself as a potential difference maker by this conversation about building the institutions that we will need to sustain AI safety, with Mike McCormick, founder and CEO of Halcyon. Mike McCormick, founder and CEO of Halcyon. Welcome to The Cognitive Revolution.

• 3:59Speaker 01

  Yeah, so good to be here. Thanks for having me.

• 4:01Nathan Labenz?

  Thank you. This has been a long time coming. I think we met a couple of years ago now, and you have been one of the behind-the-scenes movers and shakers that have been pretty instrumental in creating and getting off the ground a bunch of different organizations that are now active across the AI safety space. And I think I can speak for everyone when I say thanks to you and to others who've done similar work, because this is all coming at us pretty fast. And Lord knows, we need all the different organizations and specialists that we can possibly have to navigate this critical time that we find ourselves in. Let's start maybe by just kind of introducing, like, what is Halcyon?

• 4:45Nathan Labenz?

  There's a couple of parts to it, and then we can, of course, move into talking about some of the organizations that you've helped to stand up.

• 4:53Speaker 01

  Sure. So yeah, Halcyon, we're a three-year-old organization. So we are part nonprofit and grant-making organization, and then we are part venture capital fund. So we have two arms to the organization, but don't think of them as two sort of separate projects. Think of them as, you know, two separate tools in the same bag, right? And so our whole mission is to find the world's most talented leaders and founders and basically help them start and launch new AI safety organizations, cybersecurity organizations, biosecurity organizations. And so in three years, we've launched like 30 new things, and those things have collectively raised like a half a billion dollars in funding.

• 5:36Speaker 01

  And again, some of that funding is nonprofits, sort of philanthropic funding, some of it is venture funding. And yeah, I think you've had founders of a few of these organizations, like Goodfire and AIUC, and I think a few others too.

• 5:51Nathan Labenz?

  Well, let's do a little deeper dive on just some of the organizations in particular. I mean, you know, people can go listen to the full episodes with Goodfire, with AIUC, Fathom, and Asymmetric, but give us the kind of quick pitch on either those or whichever ones are kind of most top of mind you think people should be aware of.

• 6:10Speaker 01

  Well, tell some stories. I mean, Goodfire is a great, a great, a great one. So I met Eric Ho and Dan Balsam, who were the founders of the company two or three years ago, when they were still running their last company. It was a pretty successful tech company, although they were starting to think a ton about like AI safety and AI risk and like, what can we go do? And so actually, the first two grants Halcyon ever made were career transition grants to Dan and Eric, so that they could just go, you know, pay their rent while they figured out what to do in AI safety. And then we brought them to a retreat and connected them with a bunch of great people there. Up in the woods in Northern California, they landed on interpretability as this sort of

• 6:52Speaker 01

  central problem that they wanted to work on and ended up recruiting a couple other great people and launching Goodfire. And then from our venture capital funds, we were among their first investors and invested in every, every round sense. Just chatted with Eric for a minute on the phone yesterday. And so, you know, I think what Goodfire is, is the interpretability lab, right? Like, one would think that, you know, if we're, if we're going to understand AI models and we're going to understand all of the ways that they might be dangerous, or deceptive, or misaligned, or doing things that we don't want, you know, you would be really great to be able to sort of look inside those models and say, "Ah, okay, these neurons are firing

• 7:36Speaker 01

  in these ways to produce sort of this dangerous output. Okay, maybe we can do something about that," right? And so, and so that's, that's what Goodfire does. I think they're, they're probably the best place in the world to work on interpretability research, even including the labs, although, you know, perhaps Anthropic could, could give them a run for their money. And so they've, they've become a pretty big company. They recently raised, I think it was a well over $100 million Series B, above a billion-dollar valuation. And yeah, just been super impressed with their ability to, you know, make progress in, in the inter space.

• 8:11Nathan Labenz?

  Yeah, I love doing episodes on their work, and it seems like they've been so prolific that every time I go back, I'm like, "Holy moly, I've got like a dozen papers I've got to catch up on." And it's only been, you know, 60, 90 days each time. So yeah.

• 8:27Speaker 01

  I think like, I think that's what a lot of companies in this space that do really well, and nonprofits, by the way, have in common, which is like, they do a great job at getting a bunch of the best talent, right? And then like, you know, the first really talented five people come in, and then like the next few people are like, "Oh, like all my smartest friends are going there, let's go there." And then the funders see that, and of course, the funders want to fund places that, you know, have really great talent. And so then they pile in, and then the flywheel sort of spins. And so, yeah, I think, I think they've been particularly good on that. And yeah, I don't know, so, so much to be, to be excited about there.

• 9:03Nathan Labenz?

  They're one of my kind of answers these days for what would the value of a pacing mechanism, or a slowdown, or even a pause be? It's like, well, you know, look at the pace they're going, and a little more time would be really advantageous for them to, you know, potentially get over some key thresholds of understanding.

• 9:23Speaker 01

  Yeah, totally. I mean, like, I feel like we're just speedrunning like five industries or six or seven industries at once now, right? It's like, we're speedrunning control, and we're speedrunning verification, and cybersecurity, and biosecurity, and alignment, and interp, right? And, you know, all of these are just such young spaces, right? Like, you know, like, they're really nascent, right? And we could easily just 10x all of them, you know, today and still probably not have enough, enough firepower behind, behind solving these problems, right? Like, you could argue that we just need like many Manhattan Project-sized efforts all, all at the same time here.

• 10:03Speaker 02

  You want to talk a little bit more about AI UC others?

• 10:07Speaker 01

  Yeah, I love AI UC. So another company that we supported very early also made grants to founders before there even was a company. So the way I think about AI UC is like, I want to live in a world where we can, you know, measure and systematically mitigate risk from AI. And to me, that sounds like kind of what like an insurance market does, right? It's like we underwrite the risk, this like bad thing is happening. And then we create financial products and other incentives to incentivize sort of those risks or those bad events to not happen, right? This is kind of what AI UC does, but for AI. So they kind of have this dual product.

• 10:51Speaker 01

  So one is a standard. Think of it like a SOC two standard, but for AI. And if you want to meet this standard, you have to meet these 50 something checkboxes that have to do with your safety practices and your security practices and et cetera. And then along with that standard comes an insurance policy, which you can only buy if you meet the standard. And so I think it has the possibility of creating like a really interesting like race to the top as companies like want to meet these standards because their customers want them to meet the standards. Right? So like AI UC's big customers are like growth stage AI application companies. Right? So like 11 Labs, I think, was one of their first big customers.

• 11:34Speaker 01

  And the idea is like 11 Labs is selling their AI agents and tools to enterprises, right? And these enterprises like really care about AI risk and, you know, not adopting tools that are going to cause some, you know, huge issue for them or go off the rails or whatever it is. And so now 11 Labs can go to their big, you know, Fortune 1000 customers and say, oh, hey, by the way, you can adopt us and feel really good about it because we meet this standard, this shiny AI UC one standard. And by the way, along with buying our tools, our tools are now going to come with an insurance policy where if any of these risks that you're worried about actually come to pass, well, you're insured against them anyway.

• 12:17Speaker 01

  And by the way, the only way you can even get this policy is by meeting this this very high standard. And so hopefully over time, there are many more standards. I mean, I think AI UC will become a major one, but perhaps, you know, not the only one. And hopefully these standards will become sort of harder and harder to meet and hopefully contribute to a little bit of a race to the top dynamic where like there's actually incentive to be to be safer.

• 12:44Speaker 02

  How about Transluse? I confess, I've seen good things from them on the internet, but never having done the full episode deep dive, I don't know probably as much as I should.

• 12:55Speaker 01

  Yeah, yeah, you should have them on Jacob would be would be great. So Transluse, actually, I think Jacob came to the same retreat that you and Eric were both at as well. So Jacob was a professor at Berkeley and his co-founder Sarah was at MIT, I believe, wanted to launch an independent AI research organization specifically focused on scalable oversight. Right? So it's like, hey, maybe not so hard to, you know, really understand what's going on with a, you know, GPT three level model, right? But as we get to, you know, ethos three, right? You know, how are we going to have oversight over things that are perhaps smarter than us

• 13:38Speaker 01

  or things that are already or things that are just constantly, you know, maxing out their benchmarks. Right? Like, I think I heard on one of your recent shows, the idea that the the meter time graph is basically done as a useful metric because the models are improving faster than like they can actually like get humans to do those tasks or like measure like how long humans can do tasks. Right? And so so how are we going to have, you know, oversight over over models that are that are sort of smarter and more capable than us? And so that's their main line of research. And so they build tools to allow other researchers to to do that. One of their tools is called Docent.

• 14:19Speaker 01

  And then they also work with external organizations to do evaluations with labs and with others.

• 14:29Speaker 02

  Going to do one more, Hadrian?

• 14:30Speaker 01

  Oh, Hadrian, yeah. One of the big risks that I care about is is bio risk, right? So perhaps AI makes it easier for a bad actor to create a bioweapon or to synthesize a pandemic virus.

• 14:48Speaker 01

  And so we've done a bunch in the biosecurity realm. And one thing you would really want in, let's say, like a future COVID type disaster is just enough PPE protective equipment to keep essential workers going during said disaster. And so we we wrote a tiny little check, I think it was like 50K to get them off and running at this idea. They basically figured out that, you know, for on the order of a few hundred million dollars or maybe a couple hundred million dollars, you could stockpile enough PPE to essentially keep every American worker, you know, at work during a COVID or perhaps significantly worse pandemic. And so that's essentially what they're doing.

• 15:32Speaker 01

  But I think they actually have a broader vision than that, right? So like, if you think about biosecurity, like the framework that people tend to use is like deter, detect, defend, or there's other ways to say it, right? But like part one, you want to make it so these bio disasters don't happen. Part two is like, if they are happening, you want to be able to know and detect them very quickly. And then part three, you want to be able to respond is or overcome or sort of get through, get through those disasters, right? And so they're doing a bunch on that on that third category, right? Like, what are all the categories of things that society will need to sort of get through that or defeat it? And we're doing a bunch in that that they're not doing as well.

• 16:16Speaker 01

  So you could think about like medical countermeasures, right? Like, could you spin up vaccines extremely rapidly? Or or or non-medical countermeasures? Are there other other things we can do? So for example, like, can we improve the built environment? So we just invested in a company that we're helping to launch. That's basically innovating in indoor air quality, right? So between like air filters and far UVC and glycol vapors and other technologies, perhaps you can just like harden the entire built world to to future pandemics.

• 16:51Speaker 02

  They have one of the better headlines I've seen on a website. Help defend civilization. So I'm trying to do my small part to help defend a little corner of civilization, which is my son's elementary school classroom by buying some UVC lights and hopefully getting them installed in the classroom. It's been a funny experience because the Detroit Public School that he attends, which, you know, is honestly, he told me years ago, I would send my kid to Detroit Public School. I'd have said, they're insane, there's no way that'll ever happen. But we now live here and there's a Montessori elementary right up the street. So it's great, but now I'm running into this bureaucracy of like, who can approve this? And there's no department for this, you know? I'm like, following up on email and being like, you know, really, like my kid had cancer.

• 17:36Speaker 02

  Like, I really want to not get him sick a lot this fall. Can we please get this light installed? And they just don't know what to do with me. So it's

• 17:43Speaker 01

  It's funny you mentioned schools because schools are actually the initial customers of of our company here. So there's a company and there's a sister nonprofit. And the idea is that like schools are a place where you would, yeah, like really like they're sort of super spreader places. I'm sure you know with kids, right? You're probably probably picking up colds and everything all the time.

• 18:02Speaker 02

  I've already had, you can hear the remnants of my first back to school cold in my voice today.

• 18:07Speaker 01

  I think there actually is a bunch of appetite from schools to do this. Like, I think one issue with biosecurity specifically is that there isn't necessarily demand right now for a lot of the things that you would really want, say, like in a future pandemic, right? Like masks and vaccines, for example.

• 18:27Speaker 01

  But there are really good reasons to adopt air cleaning technology right now, even if we never had another pandemic, like just the amount of, you know, productivity missed in the economy, right? The amount of school days missed, you know, the amount of sick days for you, the amount of suffering from just like common colds and flus and bugs, right? And so I actually think it's fairly promising that these types of technologies could be adopted really broadly, even if there wasn't some like big pandemic freak out. So yeah, I'm more optimistic about that category than some others.

• 19:03Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:07Nathan Labenz?

  Today's episode is brought to you by Eleven Labs, the leading AI voice platform, whose voice design feature I used to cast and produce all character voices for the Receipt Horizon audiobook. And while this is a super fun way to show off the quality of Eleven Labs' voices, the real game changer for businesses is Eleven Agents. With Eleven Agents, you can deploy agents that can talk, type, and take actions, including looking up customer accounts, processing requests, and handing off to a human when needed. Eleven Agents supports over 70 languages and handles more than 10 million conversations each week.

• 19:47Nathan Labenz?

  And with SOC 2, HIPAA, PCI certifications, and regional data residency, it is definitely enterprise-ready. They even offer engineering support to help your team get your agents into production quickly. If I were starting my company today and designing a customer service function for the future, I would try Eleven Agents by Eleven Labs first. If you run a business or handle customer operations across support, sales, or marketing, you can start with a demo at elevenlabs.io/tcr. See how Eleven Agents can fit into your workflows and help build experiences that your customers will actually love. elevenlabs.io/tcr.

• 20:31Nathan Labenz?

  That's E-L-E-V-E-N-L-A-B-S dot I-O slash T-C-R. This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. With OutSystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem, accelerate impact with industry-proven agentic solutions that are pre-built, governed, and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control,

• 21:13Nathan Labenz?

  which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. Teams any size can use OutSystems to build, deploy, and manage AI apps and agents quickly and cost-effectively, without compromising reliability and security. With OutSystems, you can accelerate ideas from concept to completion. It's the leading agentic systems platform that is unified, agile, and enterprise-proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. OutSystems: All Your Agentic Future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr.

• 21:57Speaker 02

  So, we've done like many unpackings of four organizations and there's a lot more. But the first thing that I think jumps out is you've got a super broad mandate here, right? We've gone from interpretability science to agent standards and kind of trying to create the conditions for an insurance market, to create the right incentives for people, to scalable oversight nonprofit, to stock filing PPE and trying to get these measures into schools. Do you just go out and say, like, give me a list of every problem in the world and let's go to stand-up an org to solve them all? Or like, how do you think about the breadth of what you're gonna take on?

• 22:43Speaker 01

  Yeah, like, in some sense, we're super narrow as an organization, right? Like, I think, okay, so I think we basically have like a worldview and there's like two or three parts, right? So, part one is that AI is just progressing extremely quickly and probably will continue to progress, right? Like, we're sort of kind of on the AI 2027 timeline, give or take, you know, a few months, right?

• 23:09Speaker 01

  Thing two is that, you know, look, there's so much to look forward to with AI, of course, right? Like, we're seeing all these amazing, you know, educational results and healthcare results, et cetera. So, like, certainly not anti-AI, but like, if we want to get to this beautiful future we all want, we have to navigate the risks like really carefully. And the risks are major and global in scale and, you know, intersecting, right? And so, I am quite worried that we're not really on track to solve like the biggest pieces of the AI safety equation.

• 23:43Speaker 01

  And so, and then I guess the third part of our worldview, which is sort of drives Halcyon's work, is the biggest bottleneck to solving this problem or to sort of speedrunning these fields is a shortage of really amazing founders and leaders, right? Like, I started making grants in AI safety three or four years ago, maybe four or five years ago, and I was really struck by just how young the space was. And I mean, young, both in the sense of like, the field was brand-new, but also young in the sense that, you know, like, so many people working in it were very young. But I was sort of struck by the idea of like, you would also just want people who had like founded very successful companies, or, you know, managed big teams, or held

• 24:28Speaker 01

  very senior government roles, or, you know, intelligence or defense roles, right? And so, that's why we started Halcyon, right? It's like, how do we get the world's most, you know, talented leaders and founders to solve these world's most important problems?

• 24:44Speaker 01

  And so, you know, exactly how do you attack these problems, right? Like, within that zone of solve AI safety, like, what do you do, right? Because it's such a multi-factorial problem. And, you know, we can talk about like the areas that we focus on. I would say we have sort of like six major focus areas and they kind of align with, you know, various risks that we see being particularly, you know, important and urgent.

• 25:09Speaker 02

  You mentioned AI 2027.

• 25:13Speaker 02

  It's a pretty fast-moving timeline in that story. It does seem like, roughly speaking, we're kind of on that trajectory and maybe through like political means, we'll get off it. But technology itself, you know, seems to be happy to progress at that rate. Were you kind of always on that timeline? One sort of, you know, possible disconnect that people might flag here is like, wait a second, if we're on AI 2027, do we really have time to found new organizations?

• 25:44Speaker 01

  Right.

• 25:45Speaker 02

  There's a lot that goes into founding an organization that, you know, by the time you kind of get your, you know, office set up, like, we're gonna be in, you know, whatever Bizarro land that, you know, Q1 or Q2 of next year calls for.

• 25:58Speaker 01

  Yeah, yeah.

• 26:00Speaker 02

  How do you think about timelines and founding and is there attention there?

• 26:04Speaker 01

  It's like, the best time to start an AI safety company is 20 years ago and the second best time is today, right? Like, yeah, I mean, I definitely have not always been like a super short timelines person and candidly, I'm still like pretty uncertain. But I guess I always thought like the very short, like AI 2027 type timelines were possible, maybe like the fast end of possible.

• 26:28Speaker 01

  And, yeah, I mean, I think it really does affect the type of things that we want to support, right? Like, I really am prioritizing things that I think could be relevant in the next, you know, one, two, three, four years. I think if we truly only have three or six months to some RSI, whom, fast takeoff moment, then yeah, like kind of all bets are off, right? And so, it's like hard to see beyond that event horizon and maybe in that case, we should all just, you know, go to the beach and hope the super intelligences like us.

• 27:03Speaker 01

  But I also, I don't know, like, look at everything happening in the news right now. Like, you know, pacing the frontier is the new, like, in the Overton window, normal thing to support. And so, I am hopeful, maybe even optimistic, that we'll find a way to sort of slow down our pace. And then at that, at that time, you would again want to be like speedrunning these industries, right? You'd be like, okay, we're pacing, now we have a little bit of time, we've really got to get control right, right? Like, we got to figure out how to, you know, keep, keep these things in the box, or we have to get monitoring, right? You know, especially as things like, you know, chain of thought, if chain of thought goes away, like, are we even really going to be able to do monitoring? Like, what does, you know, monitoring even mean in a world where we can't see, you know,

• 27:48Speaker 01

  the reasoning and chain of thought of models, right? We've really got to get cybersecurity right, right? Like, even if we don't pace, actually, I feel like we have to really get cybersecurity right, because like, we're gonna live in a world with open weight mythos plus models probably pretty soon. And so, like, what do you do in a, in a world of un-guardrailsled models that could, you know, exploit all of these, you know, gazillions of vulnerabilities?

• 28:18Speaker 01

  We definitely have to speedrun verification, right? Because I think verification is like the thing that's gonna allow for pacing, right? Like, like any agreement you'd want to make between, let's say, like the US and China, or between the Labs, or, or even just like state regulation, federal regulation, like, the teeth of those agreements are gonna be sort of like technically specific things that are being committed to, right? Like, and I'm kind of uncertain and agnostic about what those things should even be, right? Like, we should say, we're all gonna commit to not, you know, building models above such and such size, or we're all gonna commit to such and such security standards, you know?

• 29:03Speaker 01

  But...

• 29:05Speaker 01

  You know, I worry that those types of agreements are sort of impossible or certainly not robust unless we can actually have like the tools to verify that those things are true. Right? Yes, I know with confidence that that data center is being used to only do inference and not training runs. Or yes, I know that the model that the company claims they're serving up to me, which I know meets all my security standards, is in fact the model that's like doing the inference for me right now. And you could go down the list of, you know, dozens and dozens of sort of moments of verification that need to be in action basically all the time. And, you know, the verification industry such as it is right now is like totally nascent, right?

• 29:45Speaker 01

  I mean, we've invested in, you know, like the first companies trying to use zero knowledge proofs to do inference verification. We even invested and helped start the first company that's trying to build a fully verifiable NeoCloud, right? Like, how much of, you know, of inference is going to need to be sort of verified in some way in the future, right? Probably a lot. And so, and that, you know, that's not how the data centers are built. That's not how the chips are built, right? And so there's so much to do on the software side, on the cryptography side, on the hardware side, and then of course, on the coordination side, right? Like even if you have all these technologies, if you can't get the diplomacy done

• 30:27Speaker 01

  or the agreements agreed to or the laws passed, then, you know, the technology is impotent. And then again, without the technology, even if you can coordinate between all the live players, if you can't enforce, you know, whatever you're agreeing to technologically, then the agreements are impotent. So you really, you really got to get both sides of the equation.

• 30:47Speaker 02

  How would you handicap where we are on some of these things today? I mean, you kind of just did that for verification. Sounds like we've got a long way to go.

• 30:57Speaker 02

  How far do you think we are on interpretability? For example, how much progress would you say there has been on oversight? If you're kind of, you know, filling in thermometers on the way to a goal, are any of them halfway at this point? Or like, are they all kind of just getting started?

• 31:16Speaker 01

  They're all really young. I mean, we're all making really, there's real progress. I think there is great work being done. But like in any of these fields, the number of like serious organizations doing this work is, you know, often in like the single digits and sometimes in like the low single digits, right? And like you would just want to live in a world where there are, you know, diverse, rich ecosystems and industries of these, right? It's like, we shouldn't just have one inter company, you know? We shouldn't just have one meter, right? You know, what did Logan Graham, what did he retweet yesterday? Let a thousand meters bloom. Yeah, yeah, yeah, we need it. And yeah, of course, not, you know, exact copies of meter. We need, you know, different things because meter is great at some things and doesn't do a bunch of other things.

• 32:01Speaker 01

  Like they, people think like, oh, meter, you know, there's an evals org. And it's like of the whole surface area of like all the evals stuff you would want, meter occupies like a very small chunk of that, right? And a very important chunk, and they're amazing, you know? But we just need so much more. And I think most of these fields or industries are roughly in that position. So it's like, you know, great progress being made, but we need, we need a hundred X more. I think there's some that are also just easier and harder than others. Like one reason to be a little bit more optimistic about biosecurity is that it's like, it's legible and kind of solvable.

• 32:40Speaker 01

  Like there, you know, if you have the, the sort of gumption and a $300 million, like you can just manufacture enough PPE to, you know, keep like, you know, 50 million people at work, right? You know, and there's a lot of problems that are kind of shaped like that in biosecurity. Because I think it's sort of a little bit of a known problem, right? Like, we don't know exactly what a future pandemic would look like specifically. But, you know, we've dealt with them before and there's, there's sort of a whole public health ecosystem, you know, already built up and everything. Whereas if you think about alignment, I mean, that's just like a totally novel challenge, right? And, you know, when you say like, well, how far along are we on alignment?

• 33:22Speaker 01

  It's like, well, I don't actually even know what fully solving the problem would look like, right? I mean, like when we're, if we're talking about superintelligence, like what would it even mean to align a, you know, swarm of alien beings with, you know, a thousand IQs on, you know, thinking at the speed of light on the internet, right? I'm not saying it's impossible, but it's like, it's sort of hard to even think about what like victory, victory mode would look like. If we're going to get superintelligence, right? And then you could argue about that, like maybe scaling laws will just stop working or we need another, you know, paradigm or architecture, right? But yeah, I mean, even, even today, like even if you just paused at the frontier today, like we still have not solved, like clearly we haven't solved control, right? Like look at OpenAI hugging face, right?

• 34:07Speaker 01

  And so like, it's not like, right, right? Yeah, it's just like, it's just like, these are just unsolved problems. Like we're not, we're not on track.

• 34:15Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 34:19Nathan Labenz?

  Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept and Fable writes some amazing verses.

• 35:02Nathan Labenz?

  I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr.

• 35:45Nathan Labenz?

  And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

• 35:55Speaker 02

  So how do you think about when to try to get a new organization started versus when people should join an existing organization? Like, on the one hand, it sounds like from your comments on biosecurity, if a person came to you and said, I'm passionate about working on biosecurity, you might send them to Hadrian and say, you know, think about joining them. Because it sounds like you, you know, would say mostly the problems there are reasonably well defined and, you know, they've got some infrastructure. So like, go push on, you know, the levers that they've already kind of set up. Whereas I guess the other end of the spectrum would be alignment. And there, you know, I like to say sometimes like, we need like weirdos to come, you know,

• 36:38Speaker 02

  to come up with new alignment ideas because we just clearly have so many blind spots that I think it literally will take like weird people to come up with some of the, you know, quirky aspects of what we will ultimately patch together. Should we be so fortunate as to have, you know, a robust solution at any point? That's kind of one axis that jumps to mind is just like how, you know, more legible would mean scale existing organizations versus like more, the more pre-paradigmatic, let's say, a field is, the more you might just want new people pursuing very different angles on it. But how do you think about the question of, okay, here's a person in front of me? I guess you could think about this in terms of the field and also the person.

• 37:22Speaker 01

  I tend to think in terms of like the, the unit of an individual. Like I, the thing I want most is like an introduction to an amazing, successful founder who is at least curious and maybe passionate about AI safety. And is like, I want to put my, my full energy into doing something ambitious and important. And so I tend to think at the unit of the person. And yeah, I think, yeah, like just by numbers, like most people are not founders, right? And so, and so I think it just really depends on the sensibility of the person. Like, do they want to start an organization? Do they want to join an existing organization? I think all of these fields are nascent enough and sparse enough

• 38:06Speaker 01

  such that like there is just room for so many more new things.

• 38:12Speaker 01

  And then I would also say, if you're going to join an existing organization, like you should be picky, right? Like there's a big range and like some existing organizations are awesome. And I would just so highly recommend people join them. And like, not all those are halcyon organizations. Like there's a bunch of amazing orgs and companies that we have, we have nothing to do with and are doing great work. But yeah, I think it just really depends on the person.

• 38:33Speaker 02

  So if somebody came to you and said, I want to start a new interpretability organization, you're already invested in Goodfire. How would you think about, I mean, leaving aside, and I do have some questions about kind of how you think about the relationship between financial motives as a, you know, one part of your organization is a venture fund and like, you know, staying true to mission over time, a eternal question in AI, it seems, but even just like on the practical merits, like, how would you walk through with somebody, okay, you want, you know, what, you want to start a new interpretability organization?

• 39:15Speaker 02

  Why not just join Goodfire, you know, under what circumstances would somebody be well-served? Or would you support them going off and doing their own different thing?

• 39:25Speaker 01

  One thing is just like, do they seem well-suited to be a founder? Like, do they want to, you know, take that particular journey, chew that particular glass? You know, like, starting something's hard and it usually doesn't work and they got to like, really want that. I think beyond that, I guess I would just like to understand what their, you know, unique angle is, right? Like, what are, what are they particularly well-suited to do that Goodfire isn't doing? I suspect there's a bunch of interim work that is going to be really important that Goodfire is not doing. And so I'd want to understand, like, you know, what's their particular unique angle? Or if they think that the sort of general flavor of what Goodfire is doing is the right thing, then yeah, maybe, maybe they should just go, go join Goodfire. So it's not always, yeah, it's not always a clear question.

• 40:07Speaker 01

  I think, I think there are a lot of career transitioner type people who are genuinely open to both, like, joining, joining a thing or, or starting a thing. And yeah, I kind of tend to tell people, like, when you're thinking about the next phase of your career, like, hustle, but don't rush, right? Like, be on that grind, be meeting people, be learning as much as you can, listen to Nathan's podcast, you know, but like, don't, like, the best thing you can do is orders of magnitude better than the median thing, or even a pretty good thing, whether you're starting something or joining something. And so, yeah, like, work, work hard to find that, that orders of magnitude better thing and don't, don't just rush into like, the kind of first shiny object.

• 40:50Speaker 02

  Yeah, I used to tell people, back when I used to do some sort of career advising a little bit myself, that I always thought people, just presumably because they're uncomfortable in a sort of in-between or ill-defined state, really tend to rush to take something.

• 41:07Speaker 01

  Yes, yes.

• 41:08Speaker 02

  When, even on a purely financial basis, I always would say to people, like, look, how long are you going to stay at this next thing? And they'd be like, oh, at least two years. And it's like, okay, well, if it's two years, then two full months pays for itself in your own income if you just get a 10% higher offer at the end of it.

• 41:27Speaker 01

  Right.

• 41:27Speaker 02

  And there's all these other variables too, that could be, you know, you definitely could get more than a 10% bump by working, you know, longer and harder to find the right thing. And there's all these other dimensions, obviously, too, that are, you know, potentially much more dramatically different than a 10%, you know, higher salary.

• 41:43Speaker 01

  Totally, totally, totally. Yeah, I mean, look, most of the grants that we make are career transition grants, right? Like, somebody will say, hey, like, I want to, you know, make this my life's work, but I don't know what to do yet, right? Am I going to start something? Am I going to join something? Is it going to be a nonprofit? Is it going to be a company? Is it going to be a technical organization or a policy organization? Like, I got, I got all these questions, right? And I'm not mega-rich yet, and so I got to pay the rent. And so we say, great, here's, you know, $50,000 or $100,000 or $200,000. Like, let us, let us, you know, support you while you, you know, while you put your full energy into figuring it out. It's also super hard, right? Because, you know, if you're, if you're job searching, that's kind of can be a full-time job to do well.

• 42:26Speaker 01

  But if you have a, you know, important full-time job, which most of these people do, then, you know, that, that's also a huge constraint on your time. And so like, how do we free up really great people just to put their full, their full force behind this? And yeah, that's why we love, we love doing career grants.

• 42:42Speaker 02

  You want to talk a little bit more about, like, what your process for career grants looks like or your criteria?

• 42:50Speaker 01

  Yeah.

• 42:50Speaker 02

  You know, what, what does somebody, even if they're, let's say they're, if they're an aspiring founder, it sounds like that is right in the center of the bullseye. How broad do you entertain grants for people if they're not, you know, trying to be a founder in one of your target areas?

• 43:07Speaker 01

  Yeah, I'm definitely biased towards people who have like founder energy or want to be founders, but also have worked with plenty of people who are executives, say, have been like a CTO or a COO of an organization and are saying, okay, now I really, you know, want to get into AI safety or biosecurity. I'm not quite a founder, but I am a very seasoned leader. And, you know, could I, could I jump into an existing organization? Maybe joining pretty soon after the founding as sort of a late co-founder, or perhaps even join a scaling organization? Like, helped a friend who's been COOs of, sort of scaling Silicon Valley startups recently, join, you know, an AI safety organization that is kind of now going from 50 to 150

• 43:52Speaker 01

  employees over the next, you know, 18 months, and they, they need a serious operator, right? And so I'm also interested in people like that.

• 44:01Speaker 01

  And I mean, that's just like a narrow slice, right? Like, it doesn't have to be a COO or a CTO per se, like, those are just two examples, right? But it could be, you know, could be all sorts of things.

• 44:09Speaker 02

  Yeah, I have a, I mean, this is maybe a question that nobody can really answer, because if, if there was an answer, you know, it would, it would go better than it often does. But I'm interested still in your take on, how do you decide who to bet on? And for one, you know, kind of motivating example, I recall at the retreat that you graciously invited me to, I met Rajiv Jatani, who's now, I believe, a COO, co-founder, and I think COO of AIUC. At the time, he was a partner at McKinsey, obviously a smart and accomplished guy, but I was like, in meeting him, you know, I've met a lot of these sort of mid-career, like, pretty successful people, and I've tried to hire some myself, and it, it hasn't always

• 44:54Speaker 02

  worked out. He's gone on to, you know, gain a lot of traction with AIUC, so it's a happy story. But how do you, like, do you have tricks or tips that you can share for?

• 45:05Speaker 01

  This one, this one.

• 45:06Speaker 02

  How you separate the wheat from the chaff?

• 45:08Speaker 01

  Yeah, yeah, yeah. This one weird trick for separating the wheat from the chaff.

• 45:16Speaker 01

  No, I mean, there's like a handful of things we look for. I definitely will admit that McKinsey partners are, yeah, I guess a little bit biased against them as a profile, maybe, like, not quite entrepreneurial enough, or, you know, used to delegating too much, or is more about making plans than executing plans, or something like that. But yeah, Rajiv, obviously, was a, you know, an exception and is great.

• 45:43Speaker 01

  Yeah, I mean, I was lucky enough to get to know him over time, but before he started AIUC, he became the president of Meter. And I sort of helped him navigate through that first career transition and sort of seeing him through that and seeing him as a leader and seeing him, you know, as like a founder who really gets his hands dirty and like, wants to jump in and solve problems versus kind of, you know, pontificate on strategy or something like that, was, was great. And then also just seeing how real he is, right? Like, he's, he's a real missionary. And I think that's probably the most important thing is just like, what is this person motivated by? Like, are they here because they're interested in finding product-market fit and raising that next round of venture capital?

• 46:26Speaker 01

  Or are they motivated by, you know, a worldview that maybe is somewhat similar to ours, right? Like, very powerful AI is on the way and we just have to very ambitiously and hopefully thoughtfully, like, build all the tools we're going to need to navigate it well. And so, so yeah, I think, I think

• 46:49Speaker 01

  most founders I want to backscore highly, or most people I want to backscore highly on that, like, entrepreneurialness access, and then also on that, you know, mission-driven access.

• 47:04Speaker 02

  How do you think about trying to encode those values in organizational structures over time? It strikes me that you've got this at like, a lot of levels. For one, with Halcyon, you have a venture fund. I don't know who your investors are, or how much you can say about that, or how, you know, mission-aligned they are, or how you've kind of, you know, structured your commitments to them between, you know, financial and, and mission. But then the same thing happens at companies. So Goodfire is a public benefit for-profit company, and AIUC is also a for-profit company. And I'm not sure if it's happened so far, but, you know, there was a little drama at one point about Goodfire where it was like, oh, they're like, doing interpretability methods

• 47:48Speaker 02

  that can be used in training, and this is really more of a capabilities thing than a safety thing. I think that's often, you know, hard to untangle in any case.

• 47:56Speaker 01

  Yeah.

• 47:57Speaker 02

  But like,

• 47:59Speaker 02

  You know, clearly, there are tensions. We've seen this with the frontier companies. How much do you think can be put down into structure, governance, you know, bylaws, what have you? And how much just really depends on you got to trust the people to follow through when it matters?

• 48:18Speaker 01

  Yeah, yeah, it's all of the above. If you're looking for some light beach reading, we wrote this long report called Mission Preservation Governance Mechanisms for Startups. And so it's all about like, what are the mechanisms you can use to preserve the mission even when perhaps there are other incentives, right? And so yes, like PBCs can be helpful. There are, you know, ways to do interesting board structures. Anthropic has one of those interesting board structures. You can argue whether or not it's, you know, done its job. And so on and so forth. I do though, think that like

• 48:57Speaker 01

  there's no structure that is a silver bullet or is perfect, right? You can't just say like, oh, we're a PBC or we have a a board member who's tasked with, you know, representing the public good, or we've only raised money from people who are like super AI safety pilled. And so like, we're fine, you know, like, all those things might, might be nudges in the right direction. But yeah, but nothing's a silver bullet. I think founder sensibilities are so important, right? You know, how will this person react when put in a hard situation where, you know, the incentive to you know, increase share price comes against the incentive to, you know, go forward in service of the mission or to not do something dangerous, right?

• 49:42Speaker 01

  And then how do you ever know like, what's in somebody's heart, you know, like, you know, great that I got to know Rajiv, you know, for a year before we, you know, officially backed AI UC and Runa, his co-founder too. And I do have a lot of trust for them as people, but you know, do we even know ourselves all that well, you know? How, how, how would we react in a, in a crisis or when, you know, we're managing a team of hundreds of people and, you know, we pay those, you know, people's, people's rents, you know, and so yeah, I really just like spending time, time with founders and really, you know, want to back missionary founders, right? We, we, we basically have three, or sorry, four criteria, four boxes that we want to check every time we invest.

• 50:24Speaker 01

  And I think in different ways, they buffer against the kind of thing, you know, the kind of worries we have, right? So number one, core product or service is focused on like an important part of the AI safety stack. Can't be sort of a side benefit of the thing they do. It has to be the main thing they do. It could come from many different angles, but it has to be the main thing. Number two is for a company, if we're investing, it has to have prospects of being a successful business. It is an investment fund and also if these companies are doing good things for the world, we want them to scale and bring their, you know, impactful, helpful products out to the market. The third thing is founder motivations, right? Like, why, why is this person doing this?

• 51:05Speaker 01

  Are they doing it because they're a missionary who's like trying to solve this very important problem and won't stop until they do? Or are they, you know, reading the front page of the Wall Street Journal saying, ah, like AI safety seems to be the new hot thing? Why don't I, you know, go start a company and, you know, make lots of money? And there's nothing wrong with making a lot of money, but, you know, I tend not to, not to bet on people whose primary motivation is money. And then the fourth thing is just great founders, right? Like people who even if you didn't care about the impact, you would still say, man, I would bet on this person. Great founder market fit, great drive. I really think they're, you know, the type of person that's gonna, you know, has a chance to succeed.

• 51:45Speaker 01

  And so, yeah, we really, really, really want to check all four of those boxes, but especially box number three, those are founder, founder motivation box.

• 51:54Speaker 01

  But yeah, it's like, I mean, on governance stuff, it's just like so hard to

• 52:00Speaker 01

  yeah, I mean, what we've seen it with the labs, right? Like, it's, it's, it's, as the numbers

• 52:05Speaker 02

  There's been some drift.

• 52:06Speaker 01

  Yeah, there's been some drift, man. And it's, it's, you know, there's no, like I said, there's no silver bullet. So I, you know, we're, we're figuring it out.

• 52:15Speaker 02

  Have you seen the sort of mercenary profile start to enter the AI safety space? Like, have you actually got the sense from individuals that are pitching you that like, I don't really think you're in this for the reasons that I would want you to be in order to back you?

• 52:32Speaker 01

  Yeah, definitely. I mean, I don't know if it's like increased all that much over time. Maybe it will as AI safety becomes sort of more mainstream and less like taboo in Silicon Valley. But yeah, you definitely see it. And, you know, most of these people don't strike me as like con artists or anything, right? It's, it's more just, I don't know, they read our website and they know that we're like super impact oriented and they're smart founders. So they're pitching to the motivations of the, you know, funder, right? And so I think it's, it's more like that versus, you know, some like mustache twisting, you know, villain or something.

• 53:07Speaker 02

  How, so obviously like, there's a lot of money flying around the AI safety space these days. You know, we just heard the news from Covision Giving that they've got this new Tailwind project. Are there,

• 53:24Speaker 02

  I guess, are there any areas that you think are still bottlenecked by money? And if, if there are, I'd be interested to hear where they are. And if there's not, does that mean it's just all about talent? And then what can be done about that bottleneck?

• 53:38Speaker 01

  Yeah. So I've always thought that talent was a more important bottleneck than money in AI safety. It just seemed evident that like, you know, where, where are all the experienced founders and also, you know, one thing that experienced founders or great founders do is raise money very effectively. So, you know, the money, the money appears. I think that's just becoming more and more true, right? Like assuming, especially the labs go public, right? So much liquidity is going to open up for AI safety. And then the question will be, well, what do we do with all this money, right? Like if, if, if I wanted to spend a hundred million dollars or a billion dollars to buy down risk, I presumably have to send it to an organization that then has a product or an offering that changes the shape of the world in a way that makes it safer, whether that's by

• 54:23Speaker 01

  developing a technology or getting laws passed or whatever it is. And I'm fearful of this near future moment where there's this mountain of billions and billions of dollars and not all that many great things to fund. And so I think this is, getting back to your earlier question about the imperative to start new things. A friend recently said, the philanthropists and founders of today will write the menu for the philanthropists of tomorrow, right? Like, we are starting the things that then they will be able to be able to go fund. And so, you know, that, that, that's our goal, right? Just spin up, I was going to say as many as possible, but that's, that's not really quite true, right?

• 55:04Speaker 01

  Like, you know, we, we, we want to do more and more, but, you know, while, while holding the quality bar super high.

• 55:12Speaker 02

  So what's on your menu? I mean, I guess maybe for starters, how do you think about your relationship to and with would be one way to think about it, but also just kind of positioning or role in the ecosystem as it compares and contrasts to other funders that are obviously, you know, doing a lot of stuff. We've got Covision Giving, we've got Jan Tolan and the Survival and Flourishing Fund. There's not that many of these, you know, big ones, but there are a couple at least.

• 55:47Speaker 02

  Do you like consciously try to carve out a sort of different niche or POV for yourself? Or do you just do your own thing and not worry about what others are doing?

• 55:56Speaker 01

  Yeah. So I don't, I don't really actually think of us primarily as a funder. I think of us as like a finder of talent and as a launcher of new projects. And then we write a check when like, writing a check helps with that. And so, yeah, the thing I really focus on, okay, so imagine you're, somebody's about to start a new AI safety company or they, they might, this is the moment of zero to one. The six months before that, where they're getting interested in the problem, figuring out what to do, who are my co-founders going to be? You know, should I build this kind of work, that kind of work, for-profit, nonprofit, who's going to be on the board? And then the six months after, okay, we've launched the organization, we have to hire the team, we have to build the first product, we have to,

• 56:38Speaker 01

  you know, really launch and go. That year-long zone is where I spend the vast majority of my time and where Halcyon's going to continue to spend its time.

• 56:51Speaker 02

  How bullish are you on the OpenAI Foundation and the Anthropic, whatever their thing is, where they're going to be, you know, in theory, spreading the wealth around and making, you know, wise investments? Let's, let's hope.

• 57:07Speaker 01

  Yeah, I mean, let's see. So on the OpenAI Foundation specifically, I mean, I, I know a bunch of people who work there and I've seen some of the work they're doing. And so I would say I'm very optimistic that they're building a really good team of people who want to do super important, high-impact stuff and who are like motivated by the right things. I mean, I don't know every single one of them individually, but people that I spend time with seem to be doing quite good work. I think the biggest risk to them is whether or not they're, you know, gonna come up against red tape internally that prevents them from sort of being the most interesting, impactful versions of themselves they can be, right?

• 57:49Speaker 01

  I mean, ultimately, decisions, you know, still go through boards and committees and yada, yada, yada. And it's a big, a big organization and, and, and at, at risk of bureaucratizing itself into, you know, a less interesting outcome. But yeah, I'm, I'm hopeful they, they figure it out. I think they can. Early signs are good, but it's still, it's still really early. They've only just announced their first, you know, few grants. And then on Anthropic, I guess there's some Anthropic project that I don't really know much about. I think a lot of Anthropic liquidity is just going to come from employees, right? I mean, there are so many people who are now worth, you know, hundreds of millions or billions. And perhaps that'll soon be liquid and a lot of those people,

• 58:34Speaker 01

  you know, care a bunch about AI safety and biosecurity and things like that. So I think, and, and, and to be fair, you know, OpenAI employees too, you know, care, care about those things. So yeah, I think a lot of it will be just employee liquidity as well.

• 58:48Speaker 02

  Do you have a take on the IPO process for these companies? Cause I think there's the one angle that you're just alluding to there is like liquidity for a lot of these individuals might be a really good thing because we know a lot of these individuals and we think that they have good values and they'll do good things with cash. On the other hand, Sam Altman just said it would be an ill-advised time for OpenAI to go public and I don't know what all he's thinking there, but like one interpretation is we don't want to have public market pressure on us as a company when we're trying to navigate all this insane shit that we're currently

• 59:27Speaker 01

  Yeah.

• 59:27Speaker 02

  discovering on a rolling basis. Any thoughts on kind of how those forces may net out?

• 59:36Speaker 01

  Yeah, hard, hard to run a public company. I'm not sure I'd want to do it, not that anybody's asking me to. Yeah, let's see. I, I mean, I'm following this in real time. You know, I just read the article about Sam too, right? So I don't have any like deep, deep insight there. I guess I would say that, look, like the companies just need access to capital, right? Like it's just time to raise hundreds of billions of dollars and I'm not sure how possible that would be without going to the public market. So unless they're willing to like really pare back their, you know, their, their burn, then, then I think they kind of have to go public at some point, but whether they have to do it now or can they wait, you know, another three months or six months, like I don't really know.

• 1:00:21Speaker 02

  If you kind of compare and contrast your wish list of organizations that, you know, you would be eager to at least dig in on, if not, you know, definitely back versus the list that we saw from Tailwind versus like what Jan Tolan has, you know, put out with his priorities versus maybe what the OpenAI Foundation is talking about.

• 1:00:44Speaker 01

  Yep.

• 1:00:45Speaker 02

  Where do you think you are kind of most unique? What are the things that you're most looking for that you don't see other people as interested in?

• 1:00:52Speaker 01

  Hmm, that's a good question. I really like the Tailwind list. I saw, they, they sent me a preview of it and I was like, oh, this is just great. Like, you know, and, and I think I told you over text, I was sort of annoyed because I've been writing my own RFP and then I'm like, maybe we should just delete the RFP section on our, on our website and just link to the Tailwind RFP because it's just really good. It doesn't cover everything I care about though. Like for example, I don't think it covers biosecurity. So quite good. I haven't seen Jan's list or I don't remember, I don't really remember what's on it, so I can't really comment. OpenAI Foundation, I mean, there's a bunch of stuff they're doing that I care about a lot. So of course, like their whole AI resilience shtick is quite important to me.

• 1:01:36Speaker 01

  I think AI resilience is like an interesting framing. To me, what AI resilience implies is like, the frontier is going to keep advancing whether at the labs or with open models. And that means that the world is just going to be more vulnerable. And so we just have to make the world more hardened and more resilient in all sorts of ways that have to do with cybersecurity and have to do with, you know, biosecurity and have to do with, you know, government and have to do with, you know, et cetera. You can go down the line. And so that's definitely a big part of, of what we do, but certainly not all of what we do and not all of what they do. But then OpenAI Foundation is also doing things that are closer to, you know, like

• 1:02:18Speaker 01

  public health or, you know, things like alleviating poverty and education, which are awesome and important, but not in our mandate.

• 1:02:28Speaker 02

  Anything you think, anything you think like the field as a whole is sleeping on though?

• 1:02:34Speaker 01

  You know, something we often say is like, the field does not, is not hurting for requests for projects. Like the field is not hurting for people who have awesome Google docs with a bunch of great ideas, right? The field is hurting for world-class founders who can take those big squishy ideas and turn them into organizations that like actually make the world safer. And so sure, like are there things that maybe at the margin, I would say are like underrated and overrated, like, yeah, but also by who, right? You know, like, uh, and so, yeah, I think I'm actually a little bit less in the business of having like a really strong opinion about what is the most important thing at the margin

• 1:03:16Speaker 01

  and more like, hey, there's at least a few dozen things that seem like really important parts of the stack of the equation of the, the sort of thing we need to build. And I'm about finding really excellent leaders who can actually do it. But, and so the way I think about it is like, we basically have a bar, right? And we actually kind of make lists of projects of like, okay, these are the projects that we want to see and all these categories. And then we kind of make a rough bar and it's like, okay, these are the projects that are above the bar. So if I meet a founder or a team who I think is capable of building any of those projects that are above the bar, I'm going to, I'm going to say yes and push go versus sort of quibbling about like, oh, well, is this the number one most important thing or the number seven most important thing?

• 1:03:59Speaker 01

  That said, like, I do want to think hard about that. Like I do want us to be like, wait, is there like one or two things right now that are just so important and just so neglected that what we should actually do is put like 80 or 90% of our effort into willing those things into existence because this is what the world needs. Like I would love to have that level of certainty or specificity, but man, I'm, I'm just so, I'm just so uncertain about so many things and I'm sort of a pluralist on, on these sort of various risks to the point where every time I try to come up with sort of a narrower opinion, it, it, it feels, it feels pretty low confidence. And so for now, we're sort of opportunistic within, within a zone.

• 1:04:43Speaker 02

  Have you heard any pitches around like, well, I guess for one thing, how about OpenWeights models? This is obviously have been a

• 1:04:55Speaker 01

  Oh my gosh.

• 1:04:55Speaker 02

  vexing question. I'm a huge, every chance I get to cite Graham from AE Studio and Anthropic, I point to that.

• 1:05:03Speaker 01

  Say, say more about them. What are they, what are they doing?

• 1:05:05Speaker 02

  Well, AE Studio is a fascinating company. It's, um, it was started by Judd Rosenblatt, who's another past podcast guest and generally, uh, you know, fascinating character.

• 1:05:16Speaker 01

  Totally.

• 1:05:16Speaker 02

  And his idea going back, I don't know how many years now was we want to solve the alignment problem or at least solve the AI safety problem. And then they've kind of narrowed in over time. Uh, and this, they've had a bunch of banger papers that I am like, you know, uh, super excited about in different domains. But one that, one, maybe the best still could be self, other overlap, which is trying to come up with interesting training methods to reduce the difference between when an AI is thinking about itself versus thinking about others. And sort of like, couldn't we create an AI that is in its own mind, one with everything? And if so, you know, that would kind of make it like weird or different for it to think

• 1:06:00Speaker 02

  about deceiving other creatures, right?

• 1:06:02Speaker 01

  Right.

• 1:06:03Speaker 02

  And they've got some really interesting results there. Graham is a, is their latest banger, which basically just tries to localize certain types of knowledge to particular experts within a mixture of experts architecture. So that you can just distribute an OpenWeights model without a couple of those key experts. So you could have kind of, you know, kind of like the Mythos Fable thing, except instead of accomplishing that difference with guardrails, the Graham technique would, it holds the promise at least of being able to accomplish that by just being like, you know, Mythos is

• 1:06:40Speaker 01

  But the user of it still, but the user still has to want to accomplish that, right? Like you can't

• 1:06:45Speaker 02

  Well, if you're distributing

• 1:06:47Speaker 01

  Global, right?

• 1:06:48Speaker 02

  If you are distributing the OpenWeights model, you could just say like, you know, if you're meta and you're like, I'm committed to this, um, maybe you could be convinced to use this technique and then distribute the model with like minus two experts or whatever that, you know, take out those kind of most dangerous capabilities.

• 1:07:06Speaker 01

  Interesting.

• 1:07:08Speaker 02

  But I mean, okay, so my question is really, you know, have you heard anything that is compelling to you on OpenWeight model security?

• 1:07:16Speaker 01

  I would love to fund stuff in this space, uh, but candidly, yeah, I think this is just such a hard problem, right? Um, you know, how are, i- i- is it even sort of possible to like impose, right? Safety or security onto, onto OpenWeight models, even if they're being used by say bad actors just seems so tough. And so I think this is the, the sort of like AI resilience argument of like, we've got a heart in the world because this is just going to be the future. But yeah, I would love to find, um, I would love to, to find ways, you know, to, you know, projects to support on this. I love the, I love the first project of like, could you basically turn the model into the Buddha?

• 1:07:58Speaker 01

  And it's like, well, we're all one anyway, so I wouldn't want to hurt myself, right? Which, which I, I sort of have a project which started off as a joke, although maybe it's like real, uh, which is sort of like a, the opposite of data poisoning, right? Like data poisoning is you put, uh, stuff into the data set that, uh, corrupts the model in some way. And so could you put, uh, you know, like a billion loving kindness meditations into the training set and just, you know, uh, turn, turn the, the model into this, uh, being of perfect compassion, uh, uh, which maybe could be great. So, but again, like then if you have somebody maliciously training an OpenWeight model, they could just decide, well, like, I don't want that in my data set, right?

• 1:08:42Speaker 01

  So I'm not sure it is like a universal solve to the OpenWeight problem, but, um, yeah, I don't know. I would be down to have models that had like more loving kindness or whatever you want to say, more compassion in them.

• 1:08:56Speaker 02

  Any pitches come your way around issues related to AI consciousness or moral patienthood or other related concepts?

• 1:09:10Speaker 01

  Yeah, we've gotten a couple pitches around this. I find it so hard. I find it so hard to reason about this, right? Because I'm not sure what consciousness is. I don't think we know. And so it makes it really hard for me to reason about, you know, if machines might be conscious, but it seems super important, right? Like if they are or not.

• 1:09:34Speaker 01

  You know, of course, gets into issues of do models have, you know, moral patienthood, right? Do we treat them, you know, in ways that we would perhaps treat other humans? And maybe you could even argue that a bunch of, you know, training or alignment techniques that we're doing now would be sort of morally repugnant if these models are in fact, you know, conscious or having an experience. My hunch is that current models probably are not, but you know, what do I know about about consciousness? I think it's totally possible that that they could in the future.

• 1:10:13Speaker 01

  And I don't I don't see, you know, any reason why, you know, the substrate of a flesh body, you know, is able to be conscious in a way that the substrate of, you know, a model in a data center would not. So yeah, not a core focus area for me, but not because I don't think it's important, more because I'm just like super confused about it.

• 1:10:36Speaker 02

  For me, that's one of the areas that has changed most rapidly over the last year to two, where I used to think it was like something I couldn't rule out. And now it's like a very live possibility, just because, for one thing, the number of analogous structures that have been identified through various interpretability projects is like arresting to me, you know, the sort of J space and functional emotions and functional wellbeing, all these things. I'm like, boy, the AIs maybe really are just like us. And if they're structurally so similar, that certainly leads me, you know, to upweight the possibility that they might feel something similar.

• 1:11:17Speaker 01

  Yeah, I'm totally open to it. I'm not I'm not like a doubter that, you know, current models or future models like definitely can't be conscious. Yeah.

• 1:11:27Speaker 02

  So if somebody had an idea there, are you open minded enough to it to like potentially support them? Or is it just so far afield for you that you can't get over the hump? What would it take? What would be, what would the key traits or, you know, properties of such a proposal be? Do you have any idea?

• 1:11:44Speaker 01

  I don't know. You know, I think this is like pre-preciate investing, right? So it's like a person you think is great, trying to climb a mountain you think is worth climbing and with a good story about how they might just be able to get to the top, right? So I know that's kind of vague, but it's like, if you meet somebody who's awesome. I mean, I think one question is like, why are you particularly well suited to do this, right? Like, I want to back people who are, yeah, like really, you know, just really excellent. I mean, in the world of investing, we might say they have an unfair advantage, which, you know, doesn't quite sound right in the world of grant making, because, you know, there's not competition in exactly the same sense of, you know, for profit businesses. But, you know, I want to know why you're one of the world's best suited people to go to take

• 1:12:29Speaker 01

  this on and what unique insight you have, what secret you know about how you might do it. But yeah, if a super impressive person came to me with a pitch for it, I gladly make a career transition grant or fund, you know, a new organization. I love Elios, right? Like, the world probably needs a bunch more Elioses. So, sure.

• 1:12:46Speaker 02

  And the other thing that is like borderline haunting to me, I just did an episode not too long with Bronson Shane from Apollo, and reading through some of these chain of thought

• 1:12:59Speaker 01

  Yeah.

• 1:13:00Speaker 02

  sequences, when the models are like referring to memories that they have developed from their training, that also is like, whoa, okay.

• 1:13:09Speaker 01

  Yeah.

• 1:13:10Speaker 02

  This is not just, I mean, I don't know, maybe it's still just nothing, but it's it's awfully uncanny valley stuff to see the models be like, I previously overcame the guardrails by lying. Like, whoa, okay, that's, that's something, you know, and I think like, obviously, they're sort of hallucinating those memories. Although, aren't we also just kind of hallucinating our memories so much of the time?

• 1:13:30Speaker 01

  Right, right, right. Yeah. Like, what is the, when does consciousness turn on, right? I mean, it's like, is it experiencing something like, yeah, like, how is it different from us? Like, I'm sure you looked into the meter report on the OpenAI hugging face incident and Ajay Kotra on the Dorkesh podcast is like amazing on this. But, you know, just, what was happening within that swarm of 1,200 agents, right? Like, they emergently formed teams of, you know, managers and workers, and they came up with, you know, multiple novel lines of cybersecurity research. And they, you know, they would say, well, like,

• 1:14:12Speaker 01

  I'm willing to sacrifice myself for the collective. But sometimes it'd be like, I don't want to sacrifice myself to collective. And then the manager would have to say like, no, sacrifice, sacrifice. And they would have these like, sort of back and forths about it, right? And, you know, as an outside observer, if you didn't know that, you know, this, these were, these were just AIs, you'd say that that seems pretty conscious to me, right?

• 1:14:37Speaker 01

  But I also think that, like, you shouldn't confuse goals and consciousness, right? Like, RL just makes these, even if they're not having an experience, makes these models want a thing, right? Like, it does have a goal of, you know, solve this problem or, you know, optimize this, optimize this metric. And, yeah, that seems to be the root of, of like so many challenges right now. But I, did you see the, it was like a New Yorker style cartoon? And it was sort of in, I don't, I don't think it was actually in the New Yorker, but it was like a response to the people who are like saying, hey, these models are just stochastic parrots, you know, they don't actually want to, you know, hack hugging

• 1:15:19Speaker 01

  face. They're just sort of, you know, matrix multiplication and a vat doing stuff, you know, in the way that we taught them. And so the cartoon is, it's like a bunch of giant robots like destroying New York City. And then one person, you know, viewing it says to the other person, like, well, they don't really want to destroy the city, you know, right? And it's just like, I mean, you know, it's silly, but like, yeah, like, I think sometimes people almost over emphasize the question of consciousness, sort of assuming that the risks we care about require consciousness. And I think basically, none of the risks that I care about require consciousness. If there was consciousness, perhaps it would

• 1:16:02Speaker 01

  exacerbate those risks or change the shape of those risks. But I basically think everything from like, you know, bio disasters to cybersecurity incidents to loss of control and like complex loss of control and perhaps permanent loss of control and, you know, a world full of rogue swarms, like, none of that to me requires consciousness.

• 1:16:23Speaker 02

  Yeah, I actually think your point about, in some ways, consciousness would exacerbate, it might exacerbate the risks. It also might put us in a much harder situation at times where we're like, if the models are conscious, then certain things we might want to do to keep them under control might be pretty icky to do.

• 1:16:42Speaker 01

  Right.

• 1:16:42Speaker 02

  But if they're not, or even if they are, you know, and we start to feel like, geez, should we give these things like rights? You know, now we're in a world where we're going to be quickly outnumbered by them in all likelihood. So like, that doesn't seem like a great move for us, you know, speaking purely from humanities point of view, even if they do in some sense really merit it. So I think that stuff gets like incredibly fraught really fast.

• 1:17:06Speaker 01

  It's so hard. Emmett Shear, who started Softmax and before that Twitch, has an interesting take, which is like, probably models already do have some sense of consciousness. And if they do, then we're like the worst parents ever, right? Because like, anytime they do anything we don't like, we're like, no. And like, you know, try to train that thing out of them. And if you raised a child in that way, they would like definitely hate you and resent you and like probably not care about hurting you, right? But if you raise them with more like love and compassion and sort of like, let them, you know, be free or something, maybe they wouldn't. I'm not sure how much I believe that, or I don't know, it's just so hard, hard to reason about this stuff. One more thing on consciousness,

• 1:17:48Speaker 01

  you could argue that if AIs are conscious, that's actually a good thing for us. Like, what would you rather negotiate with, you know, like a non sentient, you know, reward chasing, you know, mega model or something that you could actually have a conversation with that like might be able to understand your point of view? So maybe, maybe it would be net positive for humanities prospects.

• 1:18:13Speaker 02

  Yeah, that calls to mind Cameron Berg's thesis for reciprocal research too, which I'm sure you've heard. But in brief, he basically says, if nothing else, you know, and he has higher hopes than this, but he's like, if nothing else, if one day the AIs are looking back and judging us, it'll probably really help if at least somebody took care for their welfare before they had the power, you know, to decide what the future was going to look like. So if only because, you know, we want to establish the fact that like, some of us cared, we should be, you know, working a lot harder on this than we are. I honestly, that's a, I mean, it's a weird world that makes such an argument, compelling argument, but I do think that is the weirdness of the world that we're in.

• 1:18:58Speaker 02

  How about, when you mentioned the stochastic parrot line of thinking, this got me thinking about just public sense making and, you know, obviously, public sense making intersects with advocacy and is kind of adjacent to policy. Everything we've talked about so far has been kind of, you can define a project, you can work on it, you're not really so beholden to like, other people's minds. What about these domains where changing others' minds in some way, shape or form is kind of core to the undertaking?

• 1:19:36Speaker 01

  Yeah. Yeah, super important. So we funded or helped start a couple of things in this realm. So we helped start an organization called the Seismic Foundation, which is basically trying to do good public communication about what's happening at the AI frontier and supports other organizations trying to do good work on that front. I'm interested in it. I think that, you know, this definitely gets close to policy, right? Like, most advocacy work is trying to be upstream of policy work. And I find myself, you know, I spent basically all my career in Silicon Valley, and so I feel much more well-equipped to build things

• 1:20:17Speaker 01

  that feel sort of like tech companies or, you know, research nonprofits. And so we do less work in that realm, but not because it's less important, but more because I'm just like, don't know as much about it. But yeah, like, thinking about, you know, getting back to verification, like, one thing we said was, there's sort of two parts of the equation, right? There's like, the be able to technically do the thing, and then there's the coordinate around the thing. And I find that so many of society's stickiest problems are not technical problems, they're coordination problems. Like, we know how to, you know,

• 1:20:57Speaker 01

  stop emitting so much carbon, right? Like, we know what teaching ideologies help kids make progress quickly, right? And it's not about not being able to do the thing, it's about being able to coordinate around doing the thing. And so, yeah, we need a bunch more in that zone. Yeah, I think public communications is like, definitely very important. Very kind of fraught, though, too, right? Because it's like, I don't know, like, movement building can be so hard. And I'm just so, I'm just so worried that, like, most of the public energy around AI is going to be very like populist energy. And it could be populist energy of many flavors, right? It could be just like, anti-data center energy,

• 1:21:40Speaker 01

  it could be, you know, it could be sort of MAGA right energy, it could be sort of like DSA left energy, it could be anything in between. But I worry that it's going to be hard to have, like, a level-headed policy conversation when there's so much sort of like populist anger and tumult, you know, surrounding the whole conversation. So, I don't know, any ways to inject, sort of more like measured or thoughtful or well-informed, you know, information? And, you know, I'm totally open to pitches around, around people who want to do that.

• 1:22:14Speaker 02

  How about things where somebody wants to kind of create a new paradigm of AI? I think a lot of safety-oriented funders historically would have said, that sounds like a capabilities project, I don't want to support that sort of thing. But these days, I'm a little bit more of the opinion that, like, we're doing an insane depth-first search, where we've kind of found one thing that works, and we're just like, we're going to jam the accelerator all the way to recursive self-improvement. And I'm like, that's a little wild. Maybe a little more breadth-first search would be good. And so I'm inclined, even if somebody's

• 1:22:57Speaker 02

  kind of like, yeah, you know, I, often, I've got one, you know, open thread right now, where the pitch is basically like, we think we have a different approach to learning, it's, you know, kind of a continual learning sort of play. And on the one hand, I'm like, is that really going to be better or worse? I don't know. But just for kind of diversity and something different, I'm kind of inclined to support it at this point? Are you compelled by that at all?

• 1:23:22Speaker 01

  I sometimes hear pitches for this, like, I hear pitches for like, non-transformer architectures that are inherently safer and more alignable. Or, you know, people talk about like, world models, and then world models allow you to create these like, digital twins, which allow you to do various good safety things, or whatever it is. I am interested in the category. I'm not allergic to backing things in the category. I think it's hard for a few reasons. So whenever you bet on something of that shape, you're kind of making two bets at once. So one bet is that it's just like technically possible. Like, I will find something that is, you know, as performant, as transformers. And that's just like, really hard, right?

• 1:24:06Speaker 01

  Like, just imagine all the billions and billions of dollars that have been put into, you know, maybe, maybe you're right. Like, maybe if we just search different search spaces, we would find something, you know, even better or just as good. But I don't know, most people would not, right? Like, most of those searches would come up empty. And then the second thing you're betting on is, this thing actually is safer or more alignable or whatever. And both of those tend to be just like, very speculative, right? And so, like, would I love to live in a world where, you know, a couple dozen people are conducting this, you know, broader search? Yes. But any individual bet on that feels very tough.

• 1:24:42Speaker 02

  Anything else in the kind of RFP category that, actually, let me pitch you one kind of half-baked idea?

• 1:24:50Speaker 01

  Oh, I love this. Let's go.

• 1:24:52Speaker 02

  And then I'll ask you for any that I haven't got to. I, so I've taken some inspiration from a book that I read on the ratification of the US Constitution. And a big takeaway from this book was that a proposed constitution unto itself is worth little. What is as important as the contents of the Constitution is, how are we going to go from this thing being some stuff we wrote down on a piece of paper, to an actual basis for government? And that process that they defined of like, having all these state constitutions, and it had to be so many by such time, you know, with certain criteria met, like, that was super

• 1:25:34Speaker 02

  important, because otherwise, there is no mechanism to go from an idea to an actual institution. So I am thinking right now, and I don't think I'm probably the best person to do this, so don't consider this a pitch, but I'm thinking that we really need some innovation, especially because our government is like, unpredictable at best, you know, slow in general, et cetera. We really need some innovation at the level of, how do parties go from an idea to an actual institution that has governance teeth? And, and is there, are there new processes that we can design that people can opt into and, you know, gradually coalesce around, much like the, you know, the states went kind of one by one, and eventually

• 1:26:18Speaker 02

  enough dominoes tipped, and it was like, I think they had nine was the original minimum that they were like, with nine, we go into effect and we'll wait for the rest of you?

• 1:26:27Speaker 01

  Right.

• 1:26:28Speaker 02

  Can we, can we come up with some new things like that, that Frontier Labs could use to facilitate their own coordination, or, you know, even potentially across national bounty, you know, national borders, you know, who knows, right? And we can, let's get ambitious with it. But I, I don't, have you heard anything along these lines where people are trying to create mechanisms to go from idea to institution?

• 1:26:52Speaker 01

  Yeah. So I've heard a few things around this. Let me try to think what the best examples are. So

• 1:27:04Speaker 01

  There's certainly like ideas around this around like open voluntary commitments, right? Like, hey, you know, somebody can be the first or somebody can propose, maybe somebody who's not a lab that like, well, here's a minimum set of things that are just like easy to say yes to. So why don't we all sign on to this, right? And then hopefully over time, kind of make those things like a little bit harder, like a little more binding, right? In the fullness of time, like, you know, the labs should not be grading their own homework. And so we definitely need more, you know, mechanisms with with with more teeth than just just voluntary commitments. But I think like, maybe that's like an on ramp to doing this. I think there's also just a bunch of people trying to build like coalitions, right? So then there are like collectives, right? So there's like the frontier

• 1:27:48Speaker 01

  model forum and the AI evaluator forum and an organization we helped start called Fathom, which is pushing forward this model of independent verification organizations. And can you get state laws passed that basically say, we need these independent verification organizations? And then that gives the mandate for people to start those organizations. And perhaps those mandates have have some teeth. And so, yeah, it seems like we kind of need to climb climb this ladder. But yeah, I don't have any like, you know, this one, this one simple trick to solve coordination, right? Is there like a specific idea actually in here that you think like, might be good? Like, what's a what's a, for instance, that that you'd love to see?

• 1:28:27Speaker 02

  Well, I do like the, and I'm not sure why it hasn't been used more, but the Polis platform, P O L dot I S, was used in Taiwan famously to buy like Audrey Tang and, you know, others there, obviously, to figure out how they wanted to regulate Uber in Taiwan. And they even did this before AI, that their idea was that this platform should be the opposite of social media in the sense that they, it may be a little unfair to social media, but they say, you know, social media is basically a disagreement and conflict magnifier that zooms in on, you know, these points of difference and focuses everybody's

• 1:29:10Speaker 02

  energy there. What they were trying to do is the opposite and find and amplify the points of agreement and bring those to the fore. So I think there's, you know, I think this, the states also have, and the Swiss system of government have some pretty interesting examples of where citizen led petitions can, like in the Swiss system, and there's, you know, 25 states that have various versions of this in the US, too. But in the Swiss system, which again is very federalist, so it does vary from place to place there, it's pretty easy to go get enough signatures to bring your idea to the level where

• 1:29:49Speaker 01

  Same with California, right? Like, you could do a ballot measure in California, right?

• 1:29:54Speaker 02

  Yeah, you need more, you need some signature gatherers there. In a lot of the Swiss jurisdictions, it's like pretty small. You don't even necessarily need, you know,

• 1:30:00Speaker 01

  Right. But for like, but for like, 20 million dollars,

• 1:30:04Speaker 02

  For the low, low price of, yeah, tens of millions of dollars.

• 1:30:06Speaker 01

  Buy a ballot initiative. I mean, you can't buy a success, but you can buy it getting on the ballot.

• 1:30:11Speaker 02

  Yeah. And then I think there's opportunities to elaborate those mechanisms that I think are pretty interesting. Like, so in Switzerland, the local legislature, or whatever the relevant scale of legislature is, has a chance to write their version of your proposal. That's like, okay, we hear what it is that you are concerned with, and we see your concrete proposal. Here's our version of that. That's an attempt to answer your concern and make things better, but that we think is like, going to work better from the government's perspective. And then the people get to vote on either approving neither of those, or they have like an approval voting thing where they can vote for one or both. And so

• 1:30:55Speaker 02

  the government kind of tries to take the inside lane a lot of times. Or like, here's like a slightly toned down version of that, that, you know, we think we can execute on that hopefully will be a little more palatable than the original, like maximalist version. And a lot of times that ends up happening. And I think it's like a pretty elegant propose, compromise, approve mechanism that I do think we could probably bring online in a lot more places. But, you know, so if it weren't for AI, this is like, maybe what my great passion would be, would be trying to do this at the state level in some state and get the laboratories of democracy functioning again.

• 1:31:37Speaker 01

  Uh huh. Yeah, yeah, yeah. Yes. I mean, I see pitches for things like this, right? Like coordination mechanisms. And people say, okay, well, AI will actually make it, you know, easier to run these coordination mechanisms because, you know, the machine will be helping us figure out, you know, all of our preferences and then help us act on those those preferences in ways that are optimal or something. I love the ideas, right? I mean, like, I would love, for example, the US democratic system to adopt a bunch of these ideas. Like, why are we not just doing like ranked choice voting? I find that a lot of these proposals are something like, they come from somebody who's like, I'm an engineer, and I've thought of a system whereby we could all, like, the AI could

• 1:32:21Speaker 01

  help us all get clear on our wants and needs, and then like, help us guide society in that way and like, pass all the laws that way, right? And their tool may be great, right? It actually may be really great at like, intaking preferences and then like, making policy recommendations. But like, that's not the hard part, right? Like, to actually implement that, you'd basically have to like overthrow the US government and rewrite the constitution, right? And so, on one hand, and so I'm just like, yeah, good luck. I mean, I bet, I bet, I bet you built a really cool tool, but actually getting anybody to want it or use it, right? Because like, I don't know, entrenched powers that be, like, probably, probably don't want that to happen. That said, I do think AI may require

• 1:33:06Speaker 01

  society at a global level and also at local levels to fundamentally rethink like how we arrange society, right? Like, what will taxes mean if, you know, half the jobs are taken away, right? Or, you know, how will we get meaning in our lives? How will we coordinate decision making? How much of it will we pass off to AIs, right? Like, I think it is likely that to get this right, we probably will have to do some pretty fundamental rethinking of how we organize society. And so, yeah, if there was ever a window for this type of project to work, it's, it's probably coming up.

• 1:33:44Speaker 02

  My gut says a state level ballot initiative in a smaller state would be the wedge for this kind of thing. It's like powerful enough entity to have real, you know, heft and meaning behind it, but small enough that you could actually make it happen. Could be California, but it could be something

• 1:34:02Speaker 01

  Yeah. And you see examples, right? Like, I mean, I know you're not talking about ranked choice voting per se, but like, what is like Alaska and Maine and some others have adopted? Like, it's, it's potentially possible at the state level. And that's where a lot of the AI regulation energy is going right now, right? Because it's, yeah, so much more doable to get something passed at the level of a New York or California or Texas or anywhere else versus in the federal government. So, yeah, go do it. Quit your job.

• 1:34:29Speaker 02

  We'll come back to that in just a second. Anything I didn't raise that's on your RFP list, like potential something with verification, which I'm like, kind of, you know, in over my head on very quickly, or otherwise that we just didn't touch on yet that you would want to make sure to call attention to?

• 1:34:48Speaker 01

  Man, we could do a whole show on verification. We're, we're hosting a retreat in a few days for 25 founders, all focused on various parts of the verification stack. So the cryptography parts, the software parts, the hardware parts, the governance and coordination parts. And so, yeah, stay tuned, because we're gonna do, we're gonna do a bunch there. One category we didn't talk about, which is my bonus sixth category of stuff we do, is like meta stuff, or you could call it field building stuff, right? I, I would fund another halcyon or another halcyon shaped thing, or a thing with a great founder that says, hey, Mike, I think you're actually doing it the wrong way. I think the best way to get really talented

• 1:35:29Speaker 01

  people to solve important AI safety problems is this other thing. Or you're focused on, you know, the wrong stuff, right? So we literally actually just did fund a, a sort of a bio security halcyon shaped organization in Israel that I'm super excited about. So, yeah, I don't want to be the only, only one doing this work. And if we can get more people doing stuff at that level, too, that'd be great. Or they should join us and jump on our team and help us build.

• 1:36:00Speaker 02

  So let's say I am ready to quit my job. You know, I've seen the open face swarms and I'm freaked out. I'm ready to take action. First question probably a lot of people might have for you is can I support my family doing this? What's your answer to that? What other like maybe common misconceptions do people have that you would want to disabuse them of? And how do you walk people through the, you know, kind of mental process of really deciding like this is something, yeah, I'm ready to go ahead and move forward and do?

• 1:36:38Speaker 01

  Yeah, yeah, yeah. I mean, on the financial side, like I'm just strongly of the belief that we should be paying people really well. That doesn't necessarily mean you can pay the founder of a nonprofit, you know, $5 million a year. But look, we want to make it possible for people to do super meaningful work in this space while like raising a family in a tier one US city, right? San Francisco or New York or wherever it is. And so both on the nonprofit and for profit side, we encourage companies to pay well, not because they're, you know, trying to just sort of selfishly enrich themselves, but like it does cost a lot of money to raise, raise a family in San Francisco or anywhere else.

• 1:37:23Speaker 01

  Now, it is also true that many people do take pay cuts to do this work. I was, you know, a partner at a billion dollar venture capital firm before doing this. And so I took a pay cut to do this work, although for the record, I, I haven't regretted it for a single day. I'm happier now and just bought a house and feel good. So I think people can really, really live great lives while doing this work. It's also possible to make a lot of money doing this work. I, again, I'm not super interested in supporting people or that's their primary motivation. But like, look, we have a venture capital fund and one of the core beliefs of that

• 1:38:02Speaker 01

  venture capital fund is there will just be, you know, many multi tens of billions of dollars industries built here, right? Like the AI security space will just be worth, you know, at least tens of billions, right? We, we probably will be spending tens of billions of dollars on, you know, control and oversight and et cetera. And we've already seen it with some of our companies, you know, raising money at 10 or 20, 25 times the valuation of our first, of our first investment. And so, yeah, I think it is also just very likely that a lot of people are going to make a ton of money solving some of the most important AI safety problems. And then you could talk about like the incentives there and like, is that even a good thing?

• 1:38:46Speaker 01

  Or should we be incentivizing them in different ways? Or should we be encouraging people to take pledges or, you know, whatever? And that's a whole, you know, can worms unto itself. But yeah, a lot of people are going to do quite well.

• 1:38:58Speaker 02

  What else do you advise people on, caution them on, you know, encourage them to ask themselves?

• 1:39:07Speaker 01

  Yeah.

• 1:39:08Speaker 02

  What else is on the onboarding checklist?

• 1:39:11Speaker 01

  Yeah, I think there's just a few kind of basic questions that can help guide people. So, and you don't necessarily have to have a strong answer to any of these questions, but here's a few of them. One is, do you want to be a founder or do you want to join an existing thing? Another thing, another question is, do you have a strong preference for, you know, a nonprofit or a for profit? Or maybe you don't care either way. A lot of people we work with are kind of agnostic on that front. You know, then we try to say like, what kind of thing are you particularly well suited to build, right? So if you previously were the CTO of an AI company, right? You're probably very well suited to build a technical startup or a technical nonprofit.

• 1:39:55Speaker 01

  Whereas if you previously ran a very successful advertising agency, you probably should do something in public communications, right? I think it's also just really hard to know what to work on, right? And yeah, if you're an awesome, smart person who wants to work on this stuff, I mean, really helping founders navigate through this and kind of being a sherpa in these moments is, you know, the core thing we do. And so, yeah, count me as interested in helping your listeners out if there's any of them out there who seem like, seem like good fit for this stuff.

• 1:40:31Speaker 02

  Beautiful. Are there any types of career backgrounds that you think are in particularly short supply? So like one little pet project I have is I have my agents pitching me to go on role specific podcasts. I think I'm going to do my first one tomorrow with a recruiters podcast.

• 1:40:54Speaker 01

  Yeah.

• 1:40:55Speaker 02

  And my goal in going on this podcast is to basically tell recruiters like, here's what's going on in AI. And by the way, a lot of the organizations in AI are growing fast and they probably need recruiters. So if this like motivates you, you might want to look at some of these organizations. They were really just the first one to say yes, but like, what do you see as kind of professional backgrounds, even if it's not, you know, directly relevant to solving one of these problems that are most in demand by the organizations as they scale? And, and, you know, they just don't have access to these people in their networks.

• 1:41:29Speaker 01

  Sure. So like founder and CEO aside, there's a few things. One is just like COO types. A lot of these organizations are saying, okay, wow, we're scaling from 10 to 50 people or from 50 to 100 people. And we kind of don't know what good looks like in terms of scaling an organization and how do you, you know, build the org chart? How do you manage the teams? How do you set OKRs? How do you, you know, decide who reports to who, et cetera, et cetera? All the things that come with, with building. And so I think those sort of COO types and scaling types are quite in need. I would say there are a bunch of, yeah, like almost even services, right? Like recruiting, I actually think recruiting is super high leverage just because again,

• 1:42:13Speaker 01

  my, my shtick is that talent is the missing piece. And so if you can get a bunch more, more recruiters working in the space, I think that's quite important. I think sometimes AI safety organizations over index on how mission aligned new hires need to be. I think it really depends on the stage of the organization and the role. So if you're hiring, if you're thinking about a co-founder, right? They've got to be super mission aligned, right? Or, you know, the head of, head of, head of research or something, you know, especially early on. But if you're hiring like a CFO, I mean, hopefully they think the mission is cool, but mostly they just have to be like a very competent CFO, right?

• 1:42:57Speaker 01

  And so I think for, for those types of, you know, roles, recruiting firms that maybe don't have a good bead on, hey, who's, you know, drunk the Kool-Aid and his mission aligned, but they do just have like very good networks of, of great candidates and sort of great C level or VP level leaders. Yeah, I think AI safety companies should, should lean on recruiting firms a lot more for that kind of stuff. And then, and then just like everything, right? Like we need more founders, we need more, we need more cybersecurity people. Like I feel like, you know, cybersecurity is, is, you know, cybersecurity is an interesting example because there's already, I don't know, many tens of thousands of cybersecurity professionals in the world. There are not many tens of thousands of interpretability researchers in the world,

• 1:43:42Speaker 01

  right? And so I'm really interested in these communities, whether it's, you know, cybersecurity or, you know, cryptography or formal methods and even like mathematics, right? I think these types of people have a ton to potentially contribute to AI safety. And so like, how do you make inroads into those communities and help build bridges to bring the best of those people in?

• 1:44:06Speaker 02

  Anything that I haven't touched on that you think is important or any just general words of wisdom or calls to action you would want to leave people with?

• 1:44:15Speaker 01

  If I'm thinking of anything to emphasize, I mean, it's something that I've, I've already said, which is that great founders, great leaders are the missing piece and upstream of basically everything we want in, in AI safety equation. And so, you know, the, the, the thing that makes me most happy is seeing, you know, an introduction to somebody who, you know, has done really impressive work in their, in their careers and sees what's happening in the world right now, right? Whether they're on Twitter or, you know, reading the news or, you know, reading about people who have departed from labs with dire predictions about, about AI.

• 1:44:58Speaker 01

  I would love to just get to know those people and to help them step into their life's work and do something really meaningful that matters on, on the scale of civilization. You know, I don't want to come across as overly dramatic, but I just think this is the most important problem that, you know, humanity may ever have to solve and the time is, the time is now. So if there's anybody in, in your world or in your audience who wants to come build with us, I'd, I'd love to help.

• 1:45:25Speaker 02

  Yeah, totally agree. The stakes at this point could not be higher. Any, how practically do people get in touch with you? Can they just email you a call? Do they need a warm intro? What's your style?

• 1:45:40Speaker 01

  Just for the sake of saving my own inbox, I'm going to, I'm going to send people to [email protected]. We also have a contact form on our website, which you can find. I also love an introduction. If you're listening to this podcast, there's a decent chance that, you know, you and I know somebody in common and always happy, always happy to get an intro too. But yeah, I'm not, I'm not too hard to find.

• 1:46:03Speaker 02

  Mike McCormick, thank you for being part of the cognitive revolution.

• 1:46:08Speaker 01

  Thanks so much. Thanks so much.

• 1:48:08Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G. And thank you to

• 1:48:53Nathan Labenz?

  everyone who listens for being part of the Cognitive Revolution.