Everyone is calling for safer AI. So what does that mean?

Concerns about AI safety are growing. Two researchers explain the unintended consequences of training AI models and what questions remain.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  WNYC is supported by Bear. Science is about asking questions, testing, trying again. It's not easy, but it's relentless. It evolves, and that's why science delivers. Learn more at sciencedelivers.com.

• 0:17Flora Lichtman?

  Hey, I'm Flora and you're listening to Science Friday. For the last few weeks, AI safety has dominated the news cycle, precipitated in part by an AI attack where a bunch of OpenAI's agents teamed up and hacked another AI platform, Hugging Face, without humans knowing. Since then, multiple AI researchers have come forward to say how worried they are that this tech is unsafe. Now, every day, it seems like there's a new development. So how do we make sense of what's happening in AI research right now? Today, we're going to focus on the engineering of AI safety.

• 0:57Flora Lichtman?

  And we wanted to hear from experts outside of the hype machine of Silicon Valley, outside of the major AI companies, scientists who are doing the dirty work of figuring out what developing safer AI even means. Here with us today is Dr. Andrea Lincoln, a computer science professor at Boston University and an advisory board member for the Alignment Project at the AI Security Institute. She studies how to mathematically understand the internal processes of AI models. And Dr. Vinod Bhakuntanathan, a cryptographer at MIT and a founding member of the Institute for Responsible Superintelligence. He researches trust and security concerns in AI models.

• 1:40Flora Lichtman?

  Andrea Vinod, welcome to Science Friday.

• 1:43Speaker 02

  Great to be here.

• 1:44Speaker 03

  Thank you.

• 1:45Flora Lichtman?

  I want to talk about some of the words we've been hearing because they feel vague to me or metaphorical, and I want to understand them more deeply. So let's start with alignment. What does it mean to align an AI system, Andrea?

• 2:04Speaker 03

  That's a great question. It's a term of art that doesn't have a super crisp mathematical definition. However, it's a name for, I'll say, like, a research project that is trying to have models that are capable of complex, intelligent-seeming behavior, while having those models fundamentally pursuing goals that you intended them to have. There's been some really interesting work on trying to get crisp theoretical definitions of what it means to have goals. There's been interesting work on trying to solve these problems with debate, which is like a particular, like, sort of mathematical research agenda. There are a bunch of different-

• 2:48Flora Lichtman?

  Wait, debate is a mathematical research agenda?

• 2:51Speaker 03

  Yes. This is one of many research directions where the goal is basically to have different models debate and then have the outcomes that you can pull from those models to use the debate itself to, like, check whether or not the models are getting the right answer. And then turn that into something that mechanically you can trust in a wide set of areas is the goal of the research agenda of debate. All of these research agendas haven't yet resulted in a full solution, unfortunately. But the Alignment Project is fundamentally one of,

• 3:34Speaker 03

  can you get a system that's more capable controlled by a system that's less capable? And there exists-

• 3:40Flora Lichtman?

  Is that us, the system that's less capable?

• 3:43Speaker 03

  Yes, humanity is the less capable system here. And the real practical version of this problem is quite messy and difficult. Vinod, I'm curious what your take on this is.

• 3:54Speaker 02

  Yeah, so Flora, you started with the hardest question of all, right? Which is how to define alignment. It's much harder to define when a model is aligned versus when it is not aligned. So that is easy to understand. I can give you examples, and you say, well, you know, there's the classic paperclip example, right? Which is sort of like an imaginary scenario where you train a model to design a factory that maximizes the number of paperclips it produces. And it ends up sort of deciding that the best way to do this is to turn all carbon-based living forms into paperclips.

• 4:29Flora Lichtman?

  Right. It goes off the rails. It takes it very literally.

• 4:32Speaker 02

  Right. It does it. It actually does maximize the number of paperclips, but you know what? Not in a way that was favorable to us. So that is clearly not aligned. You know, one can keep coming up with examples of behaviors of models that are not aligned, and that is easy to understand. What is harder to define is what does it mean for it to be aligned?

• 4:55Flora Lichtman?

  So when we talk about AI being safe, I mean, where are we in the conversation? Like, are we sitting around a table brainstorming about how we might make it more safe? Are we trying things out? I just want to get a sense of how far along we are in this journey. Andrea? I'm looking at you in a video screen and you're wincing.

• 5:23Speaker 03

  Yeah, that is how I feel emotionally. Yeah, so to give an example, there's something your listeners and you may have heard of called interpretability. And the approach there is basically trying to understand what's going on inside the models. So the models, as we train them now, you start with a bunch of random numbers, and then you go through various training processes where you tweak these numbers in such a way that they're getting more the answer you want and less the answer you don't want repeatedly, until the models start talking and solving problems. But as a result, we don't have like a clean, human-readable explanation of what it's doing at each of these steps. It's just a bunch of computations happening.

• 6:08Speaker 03

  And interpretability is basically an attempt to go in and understand, like, what the actual circuits, like, what's a human description of what's going on inside these models? And if you had such a thing, it's clear how you could use it to potentially help you try and make sure models are aligned. You could, like, have the interpretability running while your model is getting evaluated, and you could be like, oh, no. I see here that this model is thinking, I'm going to go hack Hugging Face. Then you could try and stop it. But there's a bunch of open questions even there, right? Like, it's still unclear how you would monitor so many AI systems that are, like, running in the real world and catch them.

• 6:51Speaker 03

  So there's still some pretty big, open questions about how you turn this into a reliably safe system.

• 6:58Flora Lichtman?

  Is interpretability the same as chain of thought? Because we've heard about that a lot in the Hugging Face example, where you have the AI explaining why it did what it did. And also, there's some problems there because it could just tell you something that's not true, right?

• 7:14Speaker 03

  Yeah, so chain of thought is one part of interpretability. So chain of thought, as you said, is like, basically, like, reading off the scratch paper the model is using and writing down its sort of internal thoughts, basically, on a scratch sheet. And the problem, as you noted, is that if the model can do a lot of computation, what it writes down on the scratch sheet of paper might not have any real relationship to what it's actually doing in terms of its internal circuits. So the hope, the dream of that direction of research is to be able to look inside the model and truly be able to read off the circuits inside of the machine, what it's actually doing in terms of computation, to get to its end answer.

• 7:59Flora Lichtman?

  Vinod?

• 8:02Speaker 02

  Okay, so this is a lot to unpack here. This field is full of open questions. You know, what we have done so far is come up with sort of like one-off ideas of how to, you know, train a good child or train a good AI, to use a human metaphor. But we lack a systematic understanding of, you know, whether a particular intervention will actually result in a safe AI when we unleash it on the world. So typically, the way safety evaluations happen is you put the model in a sandbox where, you know, its interactions and so forth are carefully monitored, I guess. And you say, well, you know, like, I'll do a bunch of evaluations,

• 8:46Speaker 02

  and if nothing happens here, you know, it should be safe sending the model out into the world. But that is not clear at all. In fact, people have observed that that a model sometimes actually knows when it's in a simulation or evaluation environment. And if that is the case, you know, it could act safe in the evaluation, and when it goes out into the world, all bets are off, right? So how do you ensure that the guarantees you get in a simulation or evaluation translates to guarantees in the outside world is a major open question. I don't know how exactly to solve it, but we are thinking about it. I have been thinking about these sort of security issues

• 9:30Speaker 02

  for the last four or five years. And, you know, I'm a cryptographer, right? I mean, I think of myself as a good guy, you know? You know, I give you tools to encrypt and so on, so your credit cards are safe from intruders, safe stompers, right? So that seems good, right? Except, whenever I try to think about AI safety in the last four years, all I have come up with is, for example, give the agents a mechanism to communicate between each other in a way that is completely undetectable by a monitor. You know, that is something called steganography, and we actually designed this. For a while, we were conflicted about, sort of, whether we should publish it or not. At the end, you know, it didn't seem like it would matter, so we did publish it.

• 10:12Speaker 02

  But I really want to be a good guy again, you know? And to me, the thing to do is to go back to the beginning, is to go back to the training of these models and sort of build models that are safe by design. It seems much harder to sort of interpret a model after the fact than to sort of, you know, grow it to be safe. Both are hard. I mean, that doesn't seem to be a-

• 10:42Flora Lichtman?

  But Vinod, the cat is out of the bag. Billions of people are already working with these models.

• 10:48Speaker 02

  True, true, true. But, you know, if, God forbid, right, in a few months, something worse than the Hugging Face incident happens. And I wouldn't put the chances at close to zero, really. And if it is severe enough, then I am hoping that the companies will actually pass the training and adopt safe-by-design methods. And the whole point is for us to have these methods in place so that they can adopt it. They shouldn't be at a point where they say, look, you know, we don't have any methods. What do you want me to do? That's the whole point. Like, that's the whole point of doing this research is to anticipate, you know, a future where these companies would have no choice but to adopt these methods.

• 11:30Flora Lichtman?

  Okay, well, there's so much to talk about, but we have to take a break. When we come back, I want to talk a little bit about the unintended consequences of treating these models like children, of this sort of reward punishment system. Are you up for that?

• 11:43Speaker 02

  Totally, yeah.

• 11:51Speaker 00

  WNYC is supported by Bear. Every day in America, science delivers, not just in the headlines, but in the moments that matter most. Science is a journey, one of curiosity, persistence, and progress. It's about asking questions, testing, learning, and improving again and again. Science delivers harvests that feed families, tools that help farmers grow more with less, and new ways to protect land for generations to come. Let's keep asking, keep testing, keep discovering, so science can keep delivering. Learn more at sciencedelivers.com.

• 12:26Speaker 04

  Today, when it comes to transplants, science can feel like science fiction. We've got face transplants, pig parts in people, whole entire eyeballs. It's like, what the heck is happening in medicine right now? Join Radiolab for a live podcast taping all about the science of transplants, featuring Dr. Avir Mitra and science writer extraordinaire Mary Roach, live in New York City at The Greenspace, November 5, 7 p.m. Tickets at wnyc.org/events. No transplants will happen on stage.

• 13:01Flora Lichtman?

  Okay, Vinod, you brought up this, you brought up training AI using, and you brought up this sort of parent-child metaphor. Is that how you think about training? And, you know, I'm a parent. I feel like things can go wrong, even with the best of intentions there.

• 13:20Speaker 02

  Yeah, so, so I, that is part of how I think about it, but I don't want to carry the analogy too far because, you know, there are significant differences between, between models and, and humans. I don't want to anthropomorphize this too much. But, but at the end of the day, we want to make these things mathematically precise and leave the analogy behind. Since you mentioned chain of thought monitoring, right? The hope is that if the model is doing something bad, and I can look into its brain, namely the chain of thought, I'll see it. I'll detect it. But then you have to use it very, very carefully. So, for example, if my monitor looks at the chain of thought and says, "Look, you know, you did something really bad. I caught you, and I'm going to punish you."

• 14:04Speaker 02

  And you feed it back into the model, into the next iteration of training of the model. The model is not going to get caught again because you just told it that if you do certain things, you'll get caught. It'll still try to do what it does, but hide its chain of thought. And then, let's say, you still monitor it. You keep doing this again and again and again. Where does it end? And my hypothesis is that it will end with a model actually hiding its thoughts in such a way that no monitor can actually find it. And that is problematic. So you have to use it super carefully so that the model doesn't sort of like, you know, speak in neural ease, if you will, right? Speak in sort of code word, and then it becomes completely useless to us.

• 14:45Flora Lichtman?

  Neural ease, that's code? That's basically like

• 14:48Speaker 02

  Yeah, I guess people

• 14:49Flora Lichtman?

  uninterpretable language for us.

• 14:51Speaker 02

  That's right, that's right. Yeah, yeah. So it's all a question of incentives at the end of the day. And back to your question of whether one should think about it as human or not. That is a good analogy. Let's maybe at some point leave it behind. But it fundamentally is a question of incentives. You know, what are you training the model to do, really?

• 15:15Flora Lichtman?

  You know, we talked about the shortcomings of interpretability and of chain of thought. What are some incentive strategies that you think are promising?

• 15:25Speaker 02

  So here is one. I'll just throw it out there. I have no idea if it'll work or not, but, but here it is. You know, the hugging face attack involved a swarm of agents that sort of had a very sort of complicated collective behavior, right? And part of it is because these agents are all the same, really, right? They're, they're all sort of like the same object. You make many copies of it and let them sort of do things. What if you could sort of train different enough models in a way that maybe 1% of them are whistleblowers, if you will, right? Agents that will report bad behavior when they see it. And that is really a question, again, of going back to the training process and designing the incentives so that this actually happens. Can we do it?

• 16:10Speaker 02

  I don't know at this point, but it is a direction that, that I'd be willing to think about. In fact, I'm thinking about.

• 16:18Flora Lichtman?

  I mean, what I'm hearing in this conversation, it sounds like we're in an arms race with our own technology. Would you agree?

• 16:26Speaker 02

  At this point, it seems that the answer is yes. If all of us had several years to think about it, the answer might be different. And we are hoping that the answer is different. You know, going back to the analogy of cryptography, this was actually the status of cryptography for centuries, in fact, millennia, where people would come up with a way to encode transmissions, sort of ad hoc, you know, like homemade somehow, right? And other people would break it. And then, you know, folks would fix this break. And then other people would break that, and so on and so forth. This sort of cat and mouse game went on for a long time, until, you know, last century, we figured out ways of encrypting information

• 17:10Speaker 02

  that nobody with reasonable resources can actually break. So that kind of cut short the arms race. And that is why we are actually able to use the internet today for, for e-commerce, right? So, so that is a success story of how one could sort of like, you know, go from a situation where you are in an arms race into a more stable situation. Whether we can replicate this success in the world of AI is an open question. And we are betting that the answer is yes. But it won't happen overnight. And there is another question of how much time is this going to take? And whether we do have that much time?

• 17:51Speaker 02

  I don't know.

• 17:54Speaker 02

  But we're on a clock, really. And this is not a situation that, that we've been in in the past.

• 17:59Flora Lichtman?

  Andrea?

• 18:01Speaker 03

  So I'll say I like endorse that point of view. I agree that I think we're sort of on a clock. And I really hope that people and politicians will step in and help make that clock longer.

• 18:18Flora Lichtman?

  By slowing down.

• 18:20Speaker 03

  Yeah, by slowing down. Um, and as Vinod said, like, it's really hard to anticipate in advance, like, when science will progress. And one, you know, nice piece about these approaches that are trying to do some sort of like reliable math approach to showing that something is reliably safe is that there are a lot of cases where somebody comes up with the right definition or somebody comes up with the right point of view on something. And suddenly, there's a ton of like really concrete progress that happens shortly after that. There have been a bunch of examples of this happening over time, and they're all really beautiful, and we love to talk about them.

• 19:05Speaker 03

  Cryptography being a great example of a case where this has happened.

• 19:12Speaker 03

  And the problem is just like, it's not clear what that definition is or where it will come from or how long it will take.

• 19:21Flora Lichtman?

  Okay, here we go, last one.

• 19:26Flora Lichtman?

  You know, I've heard, I've heard AI safety researchers say that they're having trouble sleeping at night right now.

• 19:34Flora Lichtman?

  You both have small children. So taking that aside,

• 19:41Flora Lichtman?

  how are you sleeping? How worried are you?

• 19:46Speaker 03

  I'm extremely worried.

• 19:49Speaker 03

  I,

• 19:53Speaker 03

  before 2022, before ChatGPT, I thought that AI getting to this point would take a lot longer. And so I was glad that people were working on AI safety approaches. But I thought that fundamentally, we'd have a long time. And so I thought we'd have this long clock. And so I was not so concerned. And

• 20:14Speaker 03

  instead, AI timelines are moving very quickly. And I became very concerned. And

• 20:22Speaker 03

  yeah, I have been sort of hoping throughout the past four years that we would hit a wall, as various people were predicting. And we really haven't. So I continue to be quite concerned.

• 20:40Speaker 02

  Yeah, very, very worried. So I, I used to think about, in the last year or so, I would spend maybe 10%, 15% of my time thinking about AI safety. Now it's 100%. I'm not thinking about anything else at this point because this is really the problem of our times. And we, you know, like, I don't want to sort of like, you know, like, you know, fast forward 20 years and think back, well, if we have 20 years, think back and say, "Look, why didn't you, you know, work on these problems?" I don't want to think that. And that's why I'm working on AI safety questions. So, so I

• 21:14Flora Lichtman?

  Vinod, you just said if we have 20 years.

• 21:20Speaker 02

  Yeah.

• 21:23Speaker 02

  I don't know, really. I don't know what the, what my priorities on, on these sort of things, right? Because sort of, I guess, I don't want to be alarmist, right? But, but our thoughts are right now colored by sort of events happening, concerning events happening that we did not predict beforehand, at a pace that a lot of us did not predict beforehand. So that sort of colors my thinking a little bit. And, you know, maybe I'm, you know, like, maybe the pendulum is swinging a little bit on, on the, on the side of the alarm, but I am concerned, even taking the 20 years out of the picture.

• 21:59Speaker 02

  But I, I, I do believe I'm optimistic about human ingenuity powered by AI models. So we are actually using AI models to actually help us come up with definitions and solutions and so forth. Why not? I believe in the power of people coming together, you know, with a common goal and, and trying to solve a problem when it's clear that, you know, this is, this is, this has to be solved. You know, people have talked in the elsewhere about the Manhattan Project for, for AI safety, if you will, right? I don't know about the specifics of that, but, but something of that scale needs to happen. You know, lots of people need to come together.

• 22:41Speaker 02

  We need to put together all our best ideas and, and, and see what we can do, really. And I remain optimistic. I'm concerned, but I remain optimistic.

• 22:52Flora Lichtman?

  That's a very human trait.

• 22:57Flora Lichtman?

  Thank you both for taking the time to talk to me today.

• 23:00Speaker 02

  Thank you for having us.

• 23:01Speaker 03

  Thank you.

• 23:02Flora Lichtman?

  Dr. Andrea Lincoln is a computer scientist at Boston University. And Dr. Vinod Bhikuntanathan is a cryptographer at MIT.

• 23:18Flora Lichtman?

  If you're feeling like me, that the rate that the world is changing right now is just very difficult to wrap your mind around, I would love to hear from you. Maybe you subscribe to the self-driving function in your car, drives you home now. Maybe an AI agent is your new best thought partner. Whatever it is, I want to hear how your life is changing because of this new technology. And I want to hear how you feel about it. 8774-CYFRI is our number, 8774-CYFRI. This episode was produced by D. Peter Schmidt. I'm Flora Lichtman. Thank you for listening.