Ep 8: Ex-DeepMind/WhiteHouse Operator Tantum Collins on AI Regulation and Geopolitical Impacts

Jacob sits down with ex-DeepMind / White House operator Teddy Collins to discuss Sam Altman’s Congressional testimony, how AI can enable direct democracy, state-driven economic planning and better orgs, US-China competition and more.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Jacob Effron?

  Welcome to Unsupervised Learning, Redpoint's podcast on the rapidly developing AI landscape and what it means for businesses and the world. I'm one of your co-hosts, Jacob Effron, and today's episode is a fascinating conversation I had with Teddy Collins. Teddy spent six years at DeepMind, working directly for DeepMind's co-founder and CEO, Demis Tsavvas, and then went to the White House, where he helped craft AI policy. He's one of the few people who deeply understands the policy world and the cutting-edge AI landscape. And we had a really interesting, wide-ranging conversation. Touched on everything from how AI can enable direct democracy, we hit why AI may allow for a more successful round two of state-driven economic planning, and the US-China competition.

• 0:45Jacob Effron?

  Teddy provided a really helpful overview of all things AI regulation, including AI policy across countries, his reflections on Sam Altman's congressional testimony, the feasibility of regulating some of the top AI risks, and lessons that we've learned from regulating nuclear weapons and social media. I hope you enjoy this conversation as much as I did. We've been meaning to do a kind of more policy-oriented version episode of this podcast and can't think of someone better to have on than Teddy. Thanks so much for coming on.

• 1:11Speaker 01

  Yeah, of course. I'm excited to be here. Thanks for having me.

• 1:13Jacob Effron?

  Amazing. Well, you know, to start, I'd be curious, you know, you've obviously had done some pretty interesting things in your career, and going from, you know, writing a book with General McChrystal to DeepMind is probably the first of those interesting moves. How'd you come into the AI world to begin with?

• 1:25Speaker 01

  Yeah, it's kind of an interesting, I feel like everyone who ends up in the AI world has some interesting story as to how they got there. In my case, it's, as much as I'd like to say that it was because of some, like, profound insight I had, it was actually just sort of a random series of events. I was studying philosophy, and I got interested in some of these questions about moral epistemology, and those led me, in this roundabout way, to an interest in... Sort of a whole bunch of approaches to modeling values, including inverse reinforcement learning, which is a technique that's used in the AI safety space to try to model the preferences that are difficult to articulate in some neatly distilled way.

• 2:10Speaker 01

  And so that was this tangent that sort of led me to curiosity about AI. I mentioned this to a friend who actually was one of the people who had worked with me on the book, the General Crystals book. And he said that I should meet another friend of his, who was also in London, which is where I was doing grad school. And because his other friend was also into AI stuff. And he put us in touch. And that actually turned out to be Ademis, Saspus, who I then ended up working for. We just kind of like met up and we hit it off. A good friend of friend to be introduced to. It was a great friend of friend to be introduced to. And importantly, I should say, like, for any of the, you know, other philosophy grad

• 2:51Speaker 01

  students out there, I'm not sure that this trick would work today because this was pre-Alphago. So DeepMind, they'd already been acquired by Google and like there were headlines about it, but it wasn't like the company and Ademis had not become as renowned as they are now. And he, I think, was very kind with his time and like said, you know, sort of agreed to meet with a random philosophy grad student. So I think I sort of like got in the door like at just the right time. I don't know if that would have worked even even six months later because Alphago actually was actually Alphago was my first week at DeepMind was when that was on the big career match happened. So, yeah, so anyway, we hit it off. We began by talking about sort of like, yeah, AI stuff.

• 3:33Speaker 01

  And then, as anyone who sort of has, as my Ademis will attest to, he has a sort of very wide ranging set of interests and like intellectual adventures and curiosities. And so we talked about AI and the Fermi paradox and a whole bunch of other things. And we got along quite well. And in the end, he sort of like pitched me on going to DeepMind. And so that was like how I ended up there. And to me, it just seemed like this incredibly exciting space, but a space that really I knew like almost nothing about. And so it was exciting, both in terms of what it was doing, but also as a learning experience. And so broadly, I sort of had maybe three main stages of being at DeepMind. So I arrived and I did. Research and strategy projects specifically for Demis, and that ranged ton from sort of tactical

• 4:18Speaker 01

  things like helping him prepare or even just reformat like a slide deck to really fun, high-level discussion of like where AI is going in the long run and like what DeepMind's strategy will be in line with that and so on. Then we created this new function to try to track that space as it expanded beyond the realm of the types of developments that people at DeepMind could keep track of just by chatting regularly to their friends. So sort of once the community exceeded this sort of like, I guess, organizational equivalent, interorganizational equivalent of Dunbar's number or something like that, we had to have a more dedicated function. So I focused on that because prior to studying philosophy,

• 5:02Speaker 01

  I did stuff that was sort of mainly in the politics, policy and global affairs space. And then the final chapter of my time at DeepMind was looking at a similar but broader set of questions, but through a quantitative, machine learning-focused lens. So eventually I led a team, I was the research lead in this group, MetaResearch, which looked at how we could use machine learning capabilities that existed within DeepMind to examine questions around management and strategy decision-making. So a lot of this was like analysis of academic, commercial and nation-state AI developments, mainly trying to glean insights from Bibliometric

• 5:45Speaker 01

  data and metadata. And none of that is, that was all internal, none of it's public, but it's quite similar to stuff that people may have seen that, and there have been a number of sort of good public products put out by groups such as CSET, which is a think tank in DC, has done a lot of similar and very interesting work. And you can learn a lot from looking at those trends in terms of like what sub areas are rising and falling, what geographic areas are doing well in specific supplements of machine learning and so on. And then we also built some tools for sort of internal management and resourcing decision-making. So who should work with whom, who should think about reading what papers, what projects are relevant to different people. As the company grew, this became also the kind of thing that couldn't just be done sort of interpersonally or anecdotally.

• 6:26Jacob Effron?

  You know, I mean, it just seems like you had such an interesting, different set of products or projects over at DeepMind. You know, after six years, what made you jump into the government route? Not the most common route for someone, you know, as AI was really kind of coming into the mainstream even more.

• 6:40Speaker 01

  One of the trends that I followed when I was at DeepMind and part of the latter two roles that I just described was sort of looking at the growing nation-state interest in AI. And this was, and still is, like a quite consequential and high-impact and high-uncertainty time, because, I mean, broadly speaking, governments have loads of resources, not a ton of in-house AI expertise on average, and quite a wide range of objectives that could pull their priorities in different directions. So you have objectives in the national security space, as well as in the human rights space, as well as in the economic space.

• 7:21Speaker 01

  And you could imagine applying AI in a fundamentalist way to any one or even a subset of those might pull the government and as a result, the society in a very different direction than if it's another set. So as a result, which of those interests, sort of how those balance out and how the government comes to understand and use AI seemed, and I think still seems like a quite high-consequence set of outcomes, or set of decisions.

• 7:46Jacob Effron?

  Yeah. How does, you know, I think you're kind of in the perfect position to, you know, educate our listeners, like, how does the US, and how should we think about how the US and global governments, you know, regulate AI today? And where does like the office on science and technology policy fit into that?

• 8:01Speaker 01

  Yeah, so I think that the short version of this is that right now, most governments don't really regulate AI much, or even at all. And, you know, there are a couple of reasons for this. It's sort of understandable. AI has come about very quickly. It's something that government hasn't been super-involved in at the cutting edge of research, unlike some previous technologies, so let's say like nuclear and personal computing and encryption and the internet and so on. And also, it's a very, very general-purpose technology, and as a result, it is something that doesn't fit very tidily into...

• 8:38Speaker 01

  Taxonomies that usually, so like to your question, how do governments, how do governments think about regulating things in general? Often, it has to do with what falls within the remit of a specific, like in the US, a specific department or agency, you know, in other countries, a specific ministry. Again, going back to the US case, if it's a White House thing, there are specific, basically organizational functions within, within the White House, these, these components, and then sometimes directorates within those. And sometimes you have stuff that sits in between them, but it's pretty rare to have something that is A, brand new, and B, sits across almost all of them. And that means that the number of directions that things could go in is very, very high.

• 9:18Speaker 01

  AI so far has been quite unregulated, and that is changing, I think mostly for the better. So there have been some interesting developments. On the US side, there are things that are actually not yet, like binding regulation in the sense that that term is usually used, but that the foundations upon which future regulation may be built. So there is this thing, the Blueprint for an AI Bill of Rights, that several colleagues of mine at OSTP led work on, and that I made minor contributions to while I was there. OSTP is the Office of Science and Technology Policy, which is where I worked at the White House. But, and that is a really great document, and it outlines many of the concerns that

• 10:01Speaker 01

  people have about AI, and the expectations that we would like consumers, users, to have when they interact with these systems. But it has no teeth in the sense of if companies or the entities that are providing these models fail to do this, at the moment, there is no real consequence for them. Now, it's still quite helpful, because it's sort of a North Star for a bunch of hopeful subsequent work. And I think a lot of companies themselves like having something that they can tie some of their sort of ethics work to. But it falls short of what we would normally think of regulation as a Then there was a similar document that was put out by NIST,

• 10:43Speaker 01

  and which does basically sort of like standards. And so NIST put out this AI risk management framework. Again, very useful. I spoke to a number of companies that said this is exactly what we need because we wanted to have this checklist that we could work towards. But again, it has it has no sort of actual binding legal consequence. So most stuff in the US to date has been exploratory and non-binding. And that's not true everywhere. So China has done some stuff, which is sort of a mixed bag. But most recently, you know, the CAC, which is their cyberspace administration, put out sort of some guidance on LLMs. And some of that is stuff to avoid it besmirching the party.

• 11:27Speaker 01

  But some of it is stuff that I think actually a lot of people, even outside of China, would agree is good and safety focused. Now, we still have to see how that will be implemented because any of these things that are vaguely worded can just be used to further like domestic political agendas. But there's definitely some language in there that I think would be in line with the types of things we'd like to see elsewhere. And then the big one in the West is the EU, which is working on this AI Act. That's kind of the first really ambitious, wide-ranging Western attempt at binding legislation. And broadly speaking, like the AI Act groups use cases into different levels of risk and these risk categories. And then it applies restrictions depending on what category things fall into.

• 12:10Speaker 01

  And inevitably, some things will slip between the cracks, but it seems like a good first stab at regulating things. And if I had to guess, you know, I think that eventually we'll see something similar in the in the US. I'm happy to get into other questions about sort of like some of the interesting ethical quandaries here, but I'll stop there for now.

• 12:27Jacob Effron?

  It feels like AI regulation has, you know, become even more top of mind in the last month, right? You had Sam Altman last week, I guess, going before the Senate and saying that kind of AI needed to be regulated. And, you know, it feels like it's just kind of entered the public discourse a lot more, you know, your everyday Americans waking up to it. You know, what did you think of Sam's testimony? Obviously, I think a big part of it was the proposal for, you know, there to be licenses, the development of large scale AI models and tests that they would need to pass before being released. Yeah, what did you think?

• 12:55Speaker 01

  Yeah, so I mean, I broadly agree with the direction of his proposals. And and I think there was a fair amount of agreement between him and the other two people who testified at the same time. And I think that that in turn reflects something like a growing consensus. In industry and also in government, which is to say, these things are really powerful. They can do very good stuff, but also potentially quite bad stuff. And as a result, like other high impact, and some people refer to dual use technologies, which means things that can be used for both civilian and military purposes. Like other high impact dual use technologies, this stuff should almost certainly be regulated in some way. And so the question is how to do it. So, you know, a number of people have talked about the idea that there should be domestically,

• 13:40Speaker 01

  within, let's say, the United States, there should be a dedicated agency that focuses on AI issues, which I think makes a great deal of sense. And also that internationally, there should be something that is akin to, I mean, people have different points of comparison, but let's say the IAEA is one that comes up a lot. Inevitably, there are going to be differences between AI and the things.

• 13:57Jacob Effron?

  The International Atomic Energy Agency, which I guess is like, you know, more on the nuclear weapon side.

• 14:02Speaker 01

  The IAEA is just one of them. There's the IPCC, like, there's a whole list, and they all institutionally look somewhat different. So I'd say, yeah, I agree with, I think, the gist of basically everything that was said in those hearings. And I think that that represents an area of rising contencious. One thing I would add to it, which I feel is especially important, and it's very tricky, is with something that is this wide-ranging and this high impact, it feels like this is something that should get more direct democratic input from the many, many people who will be affected by something like this than most areas that get regulated by a specialized technocracy of some kind.

• 14:42Speaker 01

  But it's a very complex technology, and figuring out how do we engage in a meaningful nuanced discussion with like the general public about something that is as speculative and as complex as AI. I mean, AI is something that even the people building it often don't understand how it's working underneath the hood. So there are lots of big questions here. The same group I just mentioned, CIP, is doing some interesting work, and I've been working with them on these sort of assemblies that attempt to bring together groups of people and try to elicit in a scalable way what types of things people are most worried about, what the AI applications are about which they feel most positively, and try to back out from that some stuff that could be useful to labs directly in terms of their priorities and how they approach some model training stuff, and also for policymakers.

• 15:29Jacob Effron?

  Yeah, I'm curious on the policy side, you know, obviously with, you know, talks of increased regulation, I think two concerns that often get raised, you know, the first, I mean, to your point about, you know, putting this up for a general public vote, you know, I think there is just a question of, you know, how much do policymakers actually understand this stuff? And I think everyone likes to pick their, their favorite question from the hearings that like illustrate that maybe policymakers aren't fully up to speed on this. And obviously, I'm sure the government has ways to kind of bring in expertise in the way they regulate things. But, you know, I think there's one, you know, common fear I hear is, you know, do folks really understand this enough to regulate it? And the second is, you know, there's this feeling that I think regulation always just kind of benefits the large entities, right? And that, you know, an OpenAI or, or Google, you know, they'll, they'll be fine. They'll be able to check every box. But that startup that's like, you know, four people on a seed round right now, you know, how are they going to

• 16:14Jacob Effron?

  be able to do that? And so I'm sure these are things you've thought about a lot in your, in your past role and, and current, and, you know, just curious how you'd respond to that.

• 16:21Speaker 01

  Yeah, it is definitely a concern. And that has absolutely, that phenomenon has absolutely manifested in a lot of other domains. I will sketch out one, maybe slightly more utopian for the future thing that gives me some hope, which is, AI is unusual in that the technology itself also furnishes new opportunities to collect democratic input and to handle some forms of regulation. So there is lots of, of cool work in the direction of, sort of,

• 16:59Speaker 01

  how can we use LLMs and related models to gather input from many people in a scalable way and extract from that what their preferences are on a complex topic that, that any one person may understand only partially. You can do that and then you can take sort of, you can essentially take the, the preferences that are elicited through that process and you can combine that with expert knowledge that also sits within the model to try to produce some kind of conclusion that like the maximally informed version of these people would agree to, sort of in a consensus way. There are still a whole bunch of footnotes, like I don't think that we have systems that are doing this in a sufficiently reliable way that we trust them to create policy today.

• 17:42Speaker 01

  But I do think that in the long run, that's something that could be quite promising. I think that similar techniques could be useful for the enforcement of regulation itself. So these are sort of digital systems you can imagine at the moment, your compliance mostly involves a lot of very expensive legal costs and filling out loads of forms and things that require interaction with slow, manual, expensive components of the outside world. I think that will remain the case to some extent, but for a lot of the regulation of digital systems, you could imagine this being something that is as lightweight to run as, let's say, antivirus software or something like that, which is to say, we have a set

• 18:23Speaker 01

  of benchmarks, for instance, against which systems are checked, depending on the areas in which they will be deployed. And that is the kind of thing that could be done, at least almost entirely, in digital form. That hopefully means that the costs imposed on any given player will be smaller and that will disproportionately help smaller players because, of course, big costs will be like a bigger chunk of the expense that is shelled out by a startup than a big existing company. But yeah, in general, I would say there is a big tension between some ethics and safety things that could, if enforced, end up entrenching the power and authority of the big players.

• 19:06Speaker 01

  And so how do we balance this desire to sort of like spread the knowledge, wealth, access, et cetera, while minimizing the risk of things going really off the rails? It is a difficult tension.

• 19:17Jacob Effron?

  Yeah, it's super interesting. I'd love to kind of get into some of the different risks here of this technology and kind of regulatory approaches to them. Maybe we'll start with the most dramatic, which I guess is the species-level extinction threat. Our colleague Logan had, Elliot Zierdowski on his podcast, I think there's obviously an increasing set of folks that think this could mean the end for humanity. I don't know, what are your personal beliefs on that? And maybe we'll start there.

• 19:44Speaker 01

  Yeah, so on the existential risk, which people sometimes refer to as ex-risk front, it's...

• 19:57Speaker 01

  These are, yeah, they're real risks. I think they're not...

• 20:03Speaker 01

  Sometimes we feel tempted, it's reassuring, to say, "Well, you know, that sounds like science fiction, so it's probably as unrealistic as science fiction." There's a ton of uncertainty around these things because it's impossible to predict the future of any transformative technology. But it's certainly, these things are certainly not impossible. Like, there are very plausible scenarios that cause tremendous harm, up to and including, like, I mean, it sounds crazy, but yeah, like, killing all humans. Like, this isn't a complete, a totally deluded fantasy. Like, this is the kind of thing that could actually happen.

• 20:33Speaker 01

  In terms of Eliezer's claims directly, I think, and I should say, he's a very prolific writer, and I haven't read anywhere close to the totality of everything that he's put out, or all the interviews that he's given. And he's very smart, so, I mean, there are probably loads of things that he has taken into account that I'm unaware of. But I would say that for a few, sort of, slightly in the weeds reasons, I'm maybe a little bit less fatalistic than I think he is at the moment, mainly having to do with the, sort of, progress on the safety front, as well as the research direction that ML stuff has gone in in recent years. I think it's like a direction that, for various reasons, is slightly more likely to produce, sort of like, aligned systems, where aligned, in

• 21:18Speaker 01

  this case, I'm sort of fuzzily using to mean, systems that will obey the intent of a user that is interacting with them.

• 21:27Speaker 01

  So, I don't think it's like a, sort of, foregone conclusion that we're all totally screwed, I guess, which is good. Like, I think that the median outcome is actually, like, very, very good. But, you know, even if you have a tiny, tiny probability of something terrible happening, you should take it incredibly seriously. And that's what animates a lot of work in the X-risk space. So yeah, it's a very real thing. There are lots of serious unresolved safety questions. I don't think I'm as pessimistic as Yudkowski, but it is, it's totally possible that, yeah, that this ends up destroying humanity. And even short of that, there's like loads of other types of harm it can do. So we shouldn't like focus solely on that.

• 22:04Jacob Effron?

  Totally.

• 22:04Speaker 01

  I'll just mention one other category. So people usually distinguish between alignment failures and malicious use. And they're related, but they sort of require different, like mitigating them requires different tools. So like on the alignment failures side, you, the typical like worst case scenario there is that, as you know, is that like, you tell a system to do something that's actually kind of innocuous, and it interprets that in some fundamentalist way. And in service of what it thinks is the goal, it does something horrific, like kill everyone. The classic example is you, you, you tell a system to make a bunch of paper clips. And so it just tries to kill everyone because humans could be a threat to paper clip production. And then it, you know, tries to appropriate all matter in the known universe to, to create paper clips. And in 10 trillion years, aliens come in the Milky Way. It's just like a huge cluster of paper clips. So if you want to

• 22:48Speaker 01

  address those issues, then you want to invest in technical safety research. And maybe if you don't think that you can get systems to be safe enough, then you want to consider like a ban on building models above a certain level of sophistication. The second category malicious use is when the systems do do what you want them to do, but they're being used by someone who wants to use them to do something very, very bad, like kill loads of people. And so the mitigation measures for that stuff are a little bit different. They're less about never building sophisticated things. And they're more about making sure that organizations and individuals who might want to use these capabilities to cause harm, don't come into possession of or gain access to the most sophisticated models. And so that you can imagine, you know, there are export controls and licensing regimes and all kinds of stuff there, that is a little bit lighter touch than the things

• 23:32Speaker 01

  that you would use if you were worried that systems couldn't be aligned full stop.

• 23:36Jacob Effron?

  Is there any feasible way to do that, though? I mean, it feels like, you know, at least in the current moment, these open source models catch up to whatever state of the art, like, pretty quickly. And obviously, it becomes quite difficult to, you know, regulate or, or kind of control the usage of some of that stuff.

• 23:49Speaker 01

  There are huge coordination challenges. I think right now, it's very, very difficult to think about, yeah, if you, if you wanted to just like press stop on all AI development, it would be prohibitively difficult because you do have these competitive dynamics. You could imagine a future world where this is feasible, mainly because the research costs do tend to be pretty significant. And to some extent, the success of some of these open source projects has been in part thanks to projects at big companies being released in some form. So papers being published, maybe some code being released, maybe some smaller models being shared, maybe some of the training data

• 24:34Speaker 01

  or the logits being shared. And that enables sort of piggybacking on some of those capabilities, or at least the underlying research insights to produce new systems. You could imagine a future where that becomes more locked down for some combination of security reasons and profit incentives. And these companies don't share as much stuff. I think that then we would expect there to be more of a lag between what you see at the sort of big, very, very well-funded labs and the open source community. But yes, there are still huge coordination challenges, both between companies and also between countries. So even if, something that people will often bring up is, well, even if the US enacted something, and even if its allies gone on board with this,

• 25:18Speaker 01

  and all of these labs agreed to, let's say every lab that attended the recent White House summit, if the heads of all those labs agreed to some set of restraints, then China would just pull ahead. And so that's one argument that people use. I'm like, I think it kind of makes sense, but I'm kind of skeptical of it. So yes, there are major coordination challenges.

• 25:35Jacob Effron?

  Yeah, no, I mean, it, you know, I think it's a good point. If you think about like successful, you know, coordination across state, you know, across nation states, it feels like the International Atomic Energy Agency is the one that's often cited. I think Sam Altman literally dropped a piece yesterday talking about how, you know, that was the best place to draw parallels from for the governance of AGI. Do you think that we, I mean, it feels like we end up needing to have something that looks like that. Are we anywhere near, where are we on the path to that right now?

• 26:05Speaker 01

  Maybe, I don't think we've even arrived at the beginning of the path. I think that we're sort of speculating that maybe we can see a path. Like, I don't think we've taken any real steps towards it. But I applaud the people who are speculating about what such a path might look like, because, you know, those are important early steps. But I, yeah, I'd say it's very early days. There are a number of challenges. There are political challenges in terms of, could you get...

• 26:29Jacob Effron?

  How would it even happen? Like, I guess, you know, not to be the cynic, but it feels like there's just so many hurdles to overcome. Well, like, could you even paint the picture where this does work out and we deregulate it in kind of a reasonable way?

• 26:43Speaker 01

  Yes, so there are many directions this could take. But like, so one picture would be that there is, and there don't seem to be already, actually, like, a growing consensus that stuff is very powerful and it could go very wrong. And we sort of all want to avoid it going wrong. So the average citizen wants to avoid it going wrong. Governments want to avoid it going wrong because it looks bad for the governments. And even companies, which you might think have a local incentive to say, well, just let it rip. I think, as came up in the Senate hearings, people have had a lot of bad experiences with social media, both on the user side and on the company side, because the irresponsibility with which a lot of these systems were deployed led to a huge backlash that, in the long run, like, hurt these companies

• 27:27Speaker 01

  beyond the maybe short-run, sort of like, profits that they gain from getting people to click on fake news articles or whatever. So, like, this is an area where incentives are, to some extent, aligned. And there are international competitive dynamics, but you had that with nuclear capabilities too. And yet, you are able to have things like the IAEA, like bilateral arms control and reduction agreements and so on. So it is feasible, like, politically feasible, that you could get the key entities to buy into something like this. And one small thing that helps here is that the power is concentrated across a relatively small number of major players, at least at the moment.

• 28:11Speaker 01

  There are many ways that their sort of research paradigm could change, but I'll set that aside for now and say, assume that the future more or less looks like the present does. The coordination challenge is... Easier than it could be because the space isn't that fragmented because you need so much money and expertise in order to develop these things competitively. So that means that the the the number of entities across which you have to coordinate isn't like stratospherically high. On the technical side, it is complicated and it's in many ways much harder than other technologies, notably nuclear, because nuclear, doing nuclear stuff requires materials that

• 28:49Speaker 01

  are pretty rare and leave a lot of signatures, which makes them much easier to track than training AI models, which you just use computational power for, which is sort of the same thing that underpins.

• 29:04Jacob Effron?

  Yeah, just like GPU orders is like the only thing you can track.

• 29:07Speaker 01

  Exactly, exactly. And so and and then it's very, very difficult to tell from the outside what types of operations a given facility is performing. So, you know, with nuclear stuff, there aren't that many things that one could want to do with visceral materials. Whereas with computation, the range, I mean, there is sort of an infinite variety of things that one could do with that. And ninety nine point nine nine nine percent of those are totally innocuous and we do not want to interfere with them. And in fact, the some of them are essential for like the functioning of modern society. So it's a very difficult technical problem, how to do compute governance, which is to

• 29:48Speaker 01

  say how to tell when someone is training or using a big model or a model with a certain level of sophistication. It does help that sophistication sort of roughly to date has correlated with model size, because that means that if you could get reliable information about the scale of a training run, that would be a pretty good indicator, somewhat noisy, but a pretty good indicator of whether someone's doing something that might merit some kind of controls.

• 30:17Speaker 01

  I was skeptical for a long time of approaches to compute regulation, and I still am to some degree. But I think there's been some good work recently. So in particular, I'd say that Yonadav Shavitt, he released an interesting paper. He's been thinking about this in, I think, original ways for a while. He released a paper two months ago. So I don't know when this will come out. He released a paper in March that outlined a pretty cool proposal. It uses sort of like a couple of different... It has these sort of... You take these snapshots during training, basically, and then you combine that with supply chain auditing to make sure that people aren't building a... You know, nobody's building like a huge secret facility full of GPUs. But...

• 31:00Speaker 01

  And then he thinks that you can actually balance that with like some semblance of control over personal information for the people who are doing this stuff. Because, of course, that's the problem is how do you reconcile sort of the granularity that you need in order to identify a big training run, on the one hand, with people's desire for privacy?

• 31:18Jacob Effron?

  Right. Well, I imagine another problem is how do you actually enforce it, right? I mean, I think that's what got all the sensational... You know, or all the kind of big headlines from Eliezer's like piece that he wrote about, you know, potentially needing to go bomb GPU clusters that were not compliant, right? It's like ultimately, you know, if some folks or some nation states don't want to comply, it's hard to actually enforce.

• 31:41Speaker 01

  Yes, yes. And I have forgotten he said that. I should have said when I was mentioning, referencing Eliezer's work, that I definitely do not endorse bombing data centers. So that, I think, is a bad idea. And yes, the enforcement is super, super hard. So this really would only work if you... But like essentially, like Yonadav's proposal leans on the fact that there are relatively few providers of some of these things. So like GPUs-wise, you wouldn't need to... Like there aren't that many labs. And I mean, in terms of the companies that make really good GPUs, I mean, it's also very, very limited. And that goes sort of like all the way down the supply chain,

• 32:26Speaker 01

  which in turn is related to sort of how the semiconductor export controls were put into place. Because again, it's these... Each stage is maybe not monopolized, but at least there is like maybe some oligopoly, a small number of players that control almost all of the sort of like cutting edge of HMO. So...

• 32:43Speaker 01

  This would only work if you had nominal political buy-in from the countries and companies in question. So they didn't say no upfront, and if you assume that this relatively consolidated market structure maintains, and if further work was done on some of these technical proposals. Because then you could say, in theory, you could say, okay, everyone in this room, all these countries, all these companies, all these labs, all these universities, whatever, everyone has agreed to, in principle, prohibiting this thing. We have a technical solution that, in theory, can test whether people are violating this, and we are going to roll this out through a combination of purely digital stuff that runs in data centers,

• 33:26Speaker 01

  and some tracking of hardware stuff to make sure that people aren't building secret data centers. That's like a very reductive explanation of it, but that's something that, in principle, could work, but it really does require every single one of those bits to work well. And there are major incentives to try to avoid something like that. And you can think of all kinds of ways that if you're a creative entity, you might obfuscate what you're doing. So, yeah, it's tricky, but I would say many previous, like, lots of arms control agreements and so on, weren't perfectly enforceable, but still, on balance, had a positive impact. So this is an area, as with many areas of policy, where we should not let the perfect be the enemy of the good, because maybe we could make some headway doing something.

• 34:11Speaker 01

  But again, it's pretty speculative at this stage.

• 34:13Jacob Effron?

  I mean, beyond AGI and the existential risk of some of these incredibly large models, it feels like there's a ton of nearer-term issues, right? Whether it's job displacement, or we have our first election coming up that I think will be heavily influenced by a lot of potential AI misinformation. How do you think about, I guess, there are so many of these, I'm sure you thought about a lot of them while you were at the White House, like, how do you prioritize what the main nearer-term issues are, and think about how policymakers should wrestle with them?

• 34:41Speaker 01

  Yes, there are loads. And indeed, this is kind of related to what we were talking about earlier on in the conversation, which is that this is part of why it's so challenging from a policy perspective, because it does touch on so many different areas. And really, nobody has expertise in that full range of areas. So yeah, there's automation stuff, there is education stuff around, you know, like students using generative models. There are like a million ways that all of these things can manifest racist and sexist biases that are really difficult to control, and that crop up all over the place. Honestly, I don't think I'm super well, I don't think I have the requisite expertise to address

• 35:19Speaker 01

  the majority of these things, because in my experience, usually, the best solutions emerge from pretty deep discussions between people who have sort of the relevant specialized policy area, issue area expertise, and people who have so deep AI expertise. And one, only one can really go south, either because they maybe they don't know how AI works, or you know, there are lots of people who know plenty about AI and then have some very simplified and inaccurate model of how some complex social process works. And so they say, well, you just need to roll out this thing. And it turns out to like not work at all. So yeah, I would say this is probably something, this isn't like, this is a bit of a cop out

• 36:03Speaker 01

  of nonsense, but I think this is something that like, needs to take place through sort of democratic processes happening at relatively low levels across all of the like relevant agencies, departments, in relevant communities, maybe with some high level consolidation, if we do end up constructing some kind of AI focused agency.

• 36:21Jacob Effron?

  I mean, given your global affairs past, I can't, I can't let you go without asking, you know, a bit about like the US-China conflict and how AI changes that. And I mean, one, it would be helpful to just get kind of a lay of the land of how you see things, you know, currently. And then kind of the crucial questions that determine how this evolves.

• 36:36Speaker 01

  Yeah, absolutely. So this something comes up a ton in government especially, because there's a, there's a real focus, better or worse, on sort of like what US capabilities look like vis-à-vis China and what the US-China relationship looks like. I'll say first that one thing that's nice about AI is that at least historically, it's been a fairly collaborative and academic space. So even though you do have fierce competition. Between companies and between countries, the US and China, for instance, are each other's biggest collaborators in AI. So there are more collaborations like between US and Chinese authors than between any other two, any other pair of countries in the world. And

• 37:17Speaker 01

  that dynamic produces, yeah, more papers, more citations, et cetera. And so even in this era of like heightened political competition, there is still a lot of like positive sum work that happens. And I think that one avenue that I hope becomes like an area where collaboration strengthens is on AI safety stuff, because it's kind of like everyone has an interest in AI systems doing what the user wants them to do. So with that being said, in terms of the sort of relative capability assessment, I'm going to paint things in very, very broad brushstrokes here. The US and China are across almost every area of AI the biggest players

• 38:02Speaker 01

  by a significant margin. It is almost impossible to produce an aggregate measure of AI capability, but in the current frontier areas like big foundation models, the US has most recently been a bit ahead. There are loads of areas where China routinely produces the most capable systems, including for computer vision. But for these big foundation models, which is the current research craze, they've mainly come out of big Western labs and Chinese labs have struggled, I think to a degree that surprised many people, myself included, to keep pace. They still produce models, they produce some really big ones that will

• 38:46Speaker 01

  sometimes outperform Western ones on Chinese language benchmarks. But in a general sense, there is this feeling that China is a bit behind.

• 38:58Speaker 01

  As a caveat, I should say that broadly, I think a lot of the horse race coverage in this area probably overplays relatively small deviations from parity. So for several years, I began working on a China, well, let me say, I've been doing China-facing analysis in the AI space since about 2017. And for a while, there was this narrative that was widely accepted, that was like China's rise to become the AI hegemon is inevitable, and they have, you know, data is the new oil, and they generate so much data because there are so many food delivery services. You know, there would just be these stats, it would be like crammed into something. I'd be like, and it's in, you know, the State Council put out this thing, and they said they want to do it, and so I guess they'll do it.

• 39:40Speaker 01

  And all of that was directionally correct. The Chinese AI ecosystem has a lot going for it, but it was kind of set with this air of inevitability that I think just failed to account for how difficult to predict research stuff is. And so now, with LOMs, a lot of the coverage has gone 180 degrees in the other direction. And it's like, oh, so everyone in China is panicking because, you know, like Ernie, which is the spy equivalent of ChatGPT, like, isn't that great? And they think that they're years behind the US and so on, and that they'll never catch up, and it's hopeless. And I do think directionally, those things are right. I think there have been some slight shifts. But overall, I think that for several years, it's been one of relatively closely matched capabilities.

• 40:21Speaker 01

  And all of these things are so contingent, depending on what the latest research paradigm is. I remember listening to this guy who was a Russian analyst of the Russian military, and this was an interview that he gave in 2022. And he said that before the Ukraine invasion, there were lots of people in DC who were reading about all of these Russian EW capabilities and cyber capabilities and stuff. And this guy had to explain to them that the Russian army wasn't 10 feet tall. And then, after the invasion, which was so horrifically botched, everyone said, oh, they're a total, they're not even a power anymore, like, they can't do anything. And now he has to convince people that they're not like 2 feet tall. And I think there is a similar thing in terms of these AI capabilities, that we can sometimes

• 41:06Speaker 01

  maybe overly focus on very, very local shifts.

• 41:10Jacob Effron?

  I mean, one thing you mentioned earlier that I think is too juicy, not to dig and do further is, you know, you talked about, you know, potentially using AI in, you know, the way that nation-state kind of interacts with, you know, the broader population. And I think the examples you gave were, you know, having kind of models be able to talk to citizens at scale that we can't do manually to kind of elicit preferences and then, you know, actually potentially making policy based on, you know, aggregating those preferences with expertise. It kind of does beg this larger question. I think you probably thought about this a lot, like, as you zoom out and think, you know, in the future of how the new capabilities of these models and future models, you know, end up changing the role of the state in society, you know, what are some things you see?

• 41:49Speaker 01

  Yeah, so this is, yeah, thank you for asking this. This is a question that I is one of my sort of like nerdy interests and something I've been thinking about lately. So if we take a broad view, I think there are many ways that AI could significantly increase state capacity to do all kinds of things, good things and bad things, things that would enable, for instance, more effective personalized social services, better engagement with the average citizen and so on. But also, as with any centralization of power, could enable all kinds of abuses thereof. And so the sort of, well, maybe to give, to make

• 42:33Speaker 01

  it a little bit less abstract, there are loads of things that AI could make more scalable. And we can think about good things like providing social services and enabling vectors for people to like interact directly with the government. But on the scarier side, things like, let's say, surveillance. And some people have speculated that this could even enable the management of previously intractable, complex systems. So there was this big debate in the 20th century, the socialist calculation debate about economic planning, whether centralized economic planning is possible. And most attempts to do that in an extreme way, in fact, all attempts to do that in extreme way failed. And there are many reasons for this, but

• 43:18Speaker 01

  some of the reasons that economists have pointed to, some of the biggest reasons, are computational and informational limitations, that the market is really good at figuring out locally who needs what and how much that they want it because price signals carry a lot of information. In theory, if you had a huge amount of computational power and a huge amount of very granular, both temporally and physically granular, data and very high-capacity models, you might be able to increment towards having a much more interventionist centrally-planned economy. But there are loads of other complex systems that we can imagine. There are natural ecosystems, there are social systems and so on. And to some people, this is a total dystopia.

• 43:59Speaker 01

  To other people, it promises the realization of like forms of justice that we weren't previously able to instantiate. All of this is still like totally speculative, because no one has built systems that can do this. But conceivably, if you look at the fundamentals, you could imagine a world where maybe it's not like a totally centrally-planned economy, but where the ability to intervene in a more pervasive and fine-grained way in all of these systems that right now we just kind of treat as being more or less uncontrollable. As that ability increases, the sort of space of things that people might want and expect the government to do increase too. And so, the stuff that interests me most is a sort of grouping of questions around how

• 44:42Speaker 01

  can we get ahead of things and think about what increased state capacity might look like? And how can we try to pair that with or balance it with sort of increased binding oversight that the general public can exercise over government? Because as those powers increase, you really do want to avoid a situation where they just lead to entrenched interests and corruption, sort of capturing a huge amount of value. And I think it's very interesting to think about, and some of the sort of projects I'm working on now are focused on this question of what does it look like to bulk up the democratic oversight side of that equation? So that if and when these capabilities do emerge, we don't just have to take people

• 45:26Speaker 01

  or existing institutions at their word when they say, "Trust us, we'll use this responsibly."

• 45:31Jacob Effron?

  Super interesting. I imagine, obviously, one of the challenges of that, too, is, you know, obviously, the incentives become of individual people when, you know, when these, if these models are centrally planning to, you know, try to manipulate them or say, hey, you know, I actually want this much when I want that much. And, you know, I guess you have to train them to be sophisticated enough to respond to, you know, potential manipulations of the models or regulation.

• 45:51Speaker 01

  Yeah, absolutely. I think that there are some really interesting questions around when we're using these sort of very complex nebulous models that we don't have full insight into. How can we come up with a set of benchmarks that give us satisfactory guarantees, or at least confidence, that they are following the kind of rules of fairness that we think are important? So how do we make sure that users and regulators and developers aren't able to game them in some way? So this is this whole area of mechanism design, which is focused on incentive compatibility. How do you make a system where your incentive as a user is always to tell the truth, usually about like your own preferences, like how

• 46:35Speaker 01

  much you'd be willing to pay for something, or how much you value one out of a set of alternatives? And this is an area where it will be very difficult and probably impossible to have the same level of guarantees when using ML systems, because it's difficult to produce proofs that have the same sort of like deductive airtight quality that you can get with simpler systems. But we may be able to come up with some empirical standards that would give us trust, as we do in many other areas, empirical standards that would give us heightened trust that like we actually think we can rely on these systems to do these like high, perform these high impact functions.

• 47:12Jacob Effron?

  I mean, another interesting area of your work, you know, earlier was, you know, obviously, while you're at DeepMind, you were working on using AI to improve organizations. I realize a lot of that was internal, so, you know, the extent you can, you can share kind of some of the larger trends, though, you know, it does, you know, I'm curious what you learned from that work and where you kind of think this field might go in the future. Are we all going to have kind of like AI CEOs, or where is, where is this headed in the workplace?

• 47:35Speaker 01

  Yeah, I think there are loads of interesting possibilities here. Again, very difficult to say with confidence, like which one is the most likely, but.

• 47:45Speaker 01

  If we step back and look at things historically, most general-purpose technologies ushered in these new types of organization. And in fact, those organizational innovations were critical in order to tap the technology's underlying potential. And often, it's the case that a technology would arise, let's say, the steam engine. What's like an example of that? Yeah, so, okay, so like a classic example would be, we can give two. One is forms of industrial organizations. So if you took someone from the 1600s who was working in a given industry, and then you dropped them into the equivalent workplace in 1890, it would look quite alien to them

• 48:26Speaker 01

  because you, the new machines kind of lend themselves to a totally different approach that was much more specialized. You know, you got the assembly line and you had this kind of like, these tailorist organizational structures that were, in some ways, very well-suited to that type of production and that weren't especially well-suited to what came before technologically and that arguably we have remained overly attached to now and they're actually like no longer that sort of organizationally fit. Another classic example, one of my favorite papers is this paper, The Computer and the Dynamo, which looks at how there is often a lag between the invention of an underlying technology and the point when it shows up as boosting productivity.

• 49:11Speaker 01

  And so in the case of the Dynamo, when electrification came about, at first, people tried to substitute it quite simply for pre-existing turbines and so forth that powered factories. And that provided some gains, but only a small fraction of the available gains because the real power of electrification in these sort of production settings came from the fact that you could Make, electrical power could be done in a much more modular way. So you no longer had to have everything attached to one central apparatus. And that meant that when one piece of factory broke down, not everyone else had to, you know, other people could keep doing production. It also meant you could build flatter instead of more vertically. It meant you could put in skylights because you didn't have all of this like equipment on the ceiling.

• 49:56Speaker 01

  That reduced the rate of work at entry. There are a whole host of things that was sort of derived from, but not directly related to the underlying technical innovation. And it took a while before people figured these things out through a lot of expensive trial and error and a lot of capital expense because you have to tear down factories and build new ones and so on. So those would be two examples. And my guess is that with AI, we'll see a very, very similar thing. That I would be very surprised if, let's say in 15 years, the companies that are most effectively using AI look like companies do today, but just with AI stuff on the backend as opposed to legacy software or humans. Instead, I expect that those structures will look as alien to us today

• 50:38Speaker 01

  as the factory of 1890 did to someone from 1650. And there are many different, like I'll avoid like nerding out too much here, but just to give one thing to make it a bit more concrete in terms of like, what might the AI organization look like? A lot of the functions that organizations perform, we can think of as being sort of matching tasks. Where you're trying to match a person to another person or to a project or to some resources or to a meeting to go to. And right now, when you have like a group of 10 people, we do this in a sort of very high dimensional rich way because we know each other very, very well. And when you get to a larger organization, we use these very compressive, like human designed structures, like an organizational chart where someone is put into a given team.

• 51:23Speaker 01

  They're working on a certain project. They sign up for certain email lists. And those things largely dictate how they spend their time.

• 51:31Speaker 01

  And we all know that these are better than nothing, but far worse than ideal. And if we think about this from like a false positive and false negative perspective, almost anyone who's worked at an organization of more than fifteen people knows that even if it's in the scheme of things well-run, they end up spending a bunch of time in meetings that are not relevant to them and reading things that are not relevant to them, and they end up missing a bunch of stuff that is relevant to them. So clearly, this matching, both from a precision and a recall perspective, is not doing a very, very good job. This happens to be something that is a very well-studied ML problem, which is the problem of basically providing recommendations. And so when you use Spotify, or Netflix, or Amazon, or any number of these services...

• 52:10Jacob Effron?

  What's like the training data, though, that you'd use for a model like that?

• 52:13Speaker 01

  Yeah, so this is where it gets tricky, because there are some things where you can imagine getting signal in a pretty straightforward way. So for instance, at DeepMind, we built this system that did paper recommendations for researchers based on the stuff that we knew they were working on, because we had this project management system. So we knew that like you were an expert in or interested in this set of things, and there are a ton of papers that get pushed out, you know, to archive every week, way too many for any person to like look over. And there are recommendation, loads of other recommendation systems out there, but those systems are not plugged into this system that tracks like what you're doing every day. So we built the system, and it was relatively simple in the scheme of things, ML-wise, but it produced these recommendations that people found quite helpful in terms of enabling

• 52:58Speaker 01

  them to navigate this like this difficult space. That's something where you can get a signal pretty directly, because people's sense of what papers are the most useful to them is actually quite a good proxy for like the papers that from the organizational perspective are the best for them to be reading. There are lots of other spaces, though, and I think that your question is getting at this, where it's much trickier to kind of like get a clear reward signal, that you can say, "This thing, we should progress against this thing, because this is a sign that we are doing these pairings well." There's a whole set of things that you could do. So you could do surveys after people meet with each other that said, how valuable was this discussion? You can look at sort of revealed indicators like, you know,

• 53:41Speaker 01

  when you intervene in certain ways and you send people to these meetings versus those meetings, or you replace, you know, normal standing, sort of like regular check-ins with something that is more dynamic. Does productivity in aggregate rise or fall based on stuff like the type of output that people are producing and so on? All of any one of these measures will be flawed. And we, when I was a DB, we did not do all of these things. We only experimented with a small fraction of them. But in aggregate, you can imagine these providing a pretty good picture. And again, I would come back to like, you don't need to construct something perfect. You just need to construct something that's better than the status quo, which if we step back and look at it, is like really bad. Like big organizations, even the best ones,

• 54:25Speaker 01

  are in the scheme of things, really, really bad. They just like don't solve the underlying problem anywhere close to optimally. So the distance between that and perfection is so huge that I think even sort of imperfectly executed versions of this could significantly increase efficiency as well as ideally like employee satisfaction.

• 54:50Jacob Effron?

  Super interesting. Well, I feel like we've got a lot of really interesting directions here. You know, we always like to wrap up our interviews with kind of a standard set of closing questions. So, you know, I guess to kind of kick it off in this quickfire round, what's an area of AI that you think is overhyped and underhyped right now?

• 55:07Speaker 01

  So yeah, I will reveal my sort of relative bullishness here because I think that, well, first let me do the like overhyped. I do think that right now probably like LLM based chatbots are overhyped, which is not to say they're not amazing. Like they are, it's incredible what they can do. But especially ones that are like built on top of an existing foundation model and then specialized for a specific use case without a clear mode. I would guess that a lot of those are just going to get eaten out of the box by. This six months from now, a generation of like general purpose foundation models. I also think that people are rightfully excited about and impressed with language based

• 55:49Speaker 01

  systems, but I I think in the medium to long run, that will only be a sort of like a piece of the picture. Like, I think that we actually prefer communicating in like sort of more multimodal ways. And even within language, I think that the chatbot interface, I would be very surprised if in if in five years that we're interacting with these things that way that, for instance, right now, we interact with G P T four by like typing something into this box and like getting a little text box and reply. In terms of things that are underhyped, this will this will sound crazy, given the level of overall hype, but I actually do think that like the overall space remains somewhat undervalued. I mean, there are loads of like bad pitches and undervalued companies and

• 56:31Speaker 01

  stuff. But I think I am still probably more bullish with high uncertainty, but probably like more bullish in the scheme of things than the market is, because I think that the technical trends do point towards having like even more profound capabilities soon that would produce types of growth that just like macroeconomic indicators suggest have not been priced in. So there's lots of frothing and it's not like I'd buy, you know, stock in every like A I startup if I could. But I do think that the overall people are still undervaluing how disruptive this stuff will be. More specifically, I guess to touch on some of the things that we spoke about earlier, A I for management stuff, I think like, you

• 57:13Speaker 01

  know, what does a machine learning native org structure look like? If it's done wrong, it could be like catastrophic and alienating. If it's done right, I think it could really usher in just a totally different approach to like human organization. I think A I in general for like what we might think of as social scientific applications is kind of undervalued. People tend to jump to hard science applications. There are lots of those which are great. But I think in a way, methodologically, sort of M L is exceptionally well suited to the kind of high dimensional illegible problems that have like stumped the social sciences for ages. A lot of those themselves could be could have like commercial value. So for instance, like lots of government services stuff, I expect there will be a

• 57:56Speaker 01

  ton of value in A I companies handling that, which which government might, I could imagine government would would outsource.

• 58:05Speaker 01

  Yeah, maybe just very quickly, and you can cut whichever of these you don't need. People like, I think privacy preserving machine learning is very cool. I think more modular approaches that avoid doing like a full inference path through these like monolith models are cool. And I think creative approaches like UI and UX, like the stuff that Adept, our friend at David Luhan is doing are very cool.

• 58:27Jacob Effron?

  Awesome. And what's kind of the most exciting AI application that you've seen over the last few months?

• 58:33Speaker 01

  I, this is going to be a sort of a strange one, but I remain, I remain very impressed with generative design tools, which are actually not the, the standard ones that people will be familiar with, so like Dolly and stuff are very, very cool. But really what those are trying to do is like imitate human designed stuff. And that's great. And that is relevant to like most use cases. But there are some other generative design tools that are trying to solve structural problems.

• 59:08Speaker 01

  For instance, in doing the design of like physical components and stuff like that. And those often produce these very strange, alien looking structures that it turns out are sort of like the locally optimal solution to supporting a given set of loads with certain like geometric constraints. And the shape that those take, I think is like really, really beautiful. I think that reflects something about like aesthetics and also like what we've missed in science and engineering historically, where we assume that things can only be optimal if they sort of like take the form of a discrete set of sort of like platonic solids. And that's something that continues to sort of like impress me, even though it's been

• 59:51Speaker 01

  around for a while and is, is not the sort of most in vogue form of generative design.

• 59:56Jacob Effron?

  Super interesting. Well, Teddy, this has been like a fascinating, wide ranging conversation. I feel like there's going to be so many threads that people want to pick up on. You obviously have such a diverse range of interests. So I'll leave the last question to you to kind of, one, folks want to learn more about some things we've discussed. But two, any plugs you want to do at this point for our, for our listeners who are curious about you and some of the things we discussed?

• 1:00:16Speaker 01

  I would say, stay tuned. I am hoping to... So some of the projects I enjoy the most at the moment are with some of these groups that are in the midst of developing things. So there's the Collective Intelligence Project, which I think does very good work, as well as the AI Objectives Institute. And then I'm also doing some solo writing on some of the topics that we discussed. I'll be a fellow at this centre, the Saffra Centre, at the Kennedy School at Harvard, beginning in the fall. And I am hoping to sort of write up in a slightly more formal way a number of topics related to things that we chat about today. And hopefully that will be online in a couple of months.