The Ethics of AI with Batya Friedman & Steve Omohundro

How can we ensure technology evolves ethically in a rapidly advancing world? Neil deGrasse Tyson, Chuck Nice & Gary O’Reilly explore the challenges of designing a future where human values and AI coexist with The Future of Life Award’s 2024 recipients, Batya Friedman & Steve Omohundro.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Neil deGrasse Tyson?

  I'm glad somebody's thinking about the future of our civilization and the ethical guardrails it might require.

• 0:06Chuck Nice?

  Yeah.

• 0:07Neil deGrasse Tyson?

  Lest we be the seeds of our own demise.

• 0:11Chuck Nice?

  Well, now that we've had this show, we know the future, and y'all gonna have to watch to find out. Exactly.

• 0:18Neil deGrasse Tyson?

  All right. Coming up, StarTalk special edition.

• 0:24Neil deGrasse Tyson?

  Welcome to StarTalk, your place in the universe where science and pop culture collide.

• 0:33Neil deGrasse Tyson?

  StarTalk begins right now.

• 0:37Neil deGrasse Tyson?

  This is StarTalk special edition. Neil deGrasse Tyson, your personal astrophysicist. And when we say special edition, it means I've got as co-host Gary O'Reilly. Gary.

• 0:48Gary O'Reilly?

  Hi, Neil.

• 0:49Neil deGrasse Tyson?

  All right. Chuck Nice, baby. Hey, hey, what's happening? So I understand today, because our special edition themes are always people, our physiology, our behavior, our conduct, our interaction with the world, and finally, we're talking about technologies being safe and ethical.

• 1:11Gary O'Reilly?

  Yeah.

• 1:11Neil deGrasse Tyson?

  These are three words you'll often see in a sentence: safe, ethical, technology.

• 1:15Gary O'Reilly?

  And this is where we're gonna shine our light.

• 1:16Neil deGrasse Tyson?

  Where you're gonna take us today.

• 1:17Gary O'Reilly?

  It seems we have problems, starting with the letter A. Algorithms, AI, autonomous, well, there's three for you. Is there a Wild West tech bubble in play right now? One with no guardrails, no moral compass, that's run by wannabe Bond villains? Are the best of human values baked into technologies during the design process? Is there anyone working on ethical and safe operating protocols for these new technologies? The answer is yes, and we will meet two people responsible shortly, courtesy of the Future of Life Institute that has this year acknowledged their work. Previous honorees include Carl Sagan for popularizing the science of the nuclear winter,

• 2:01Gary O'Reilly?

  and our first guest who follows shortly is Batya Friedman.

• 2:04Neil deGrasse Tyson?

  So Batya Friedman, welcome to StarTalk.

• 2:07Speaker 03

  Well, thank you.

• 2:08Neil deGrasse Tyson?

  Yeah, if I have my, my, my data here is correct on you. Professor at University of Washington's Information School. UW, I think you guys call it. Is that correct?

• 2:19Speaker 03

  That's right, yep.

• 2:20Neil deGrasse Tyson?

  But you're also co-founder of the Value Sensitive Design Lab. Ooh. You're thinking about That's intriguing. The human condition. You focus on the integration of human values and ethics with new technologies that are being born. Very important. Don't come to it when it's too late. Yeah. When, when we're all extinct, is it maybe we should have done that different? Right, exactly. Yeah, when the robots are like, how do you like me now? That's a little late. So, so Bhattacharya, please explain the focus of your Value Sensitive Design Lab.

• 2:53Speaker 03

  Yeah, sure. Um, you know, I started out as a software engineer a long, long, long time ago. And I wanted to build technologies that worked and were efficient and effective. But I also wanted some confidence that they would do something good in the world. You know, that whatever I made as an engineer would ultimately benefit society, human beings, other creatures on the planet. Design constraints are our friends. They, um, they help us shape the kinds of new technologies we develop and their qualities and characteristics in ways that, um, maybe we want to see.

• 3:33Speaker 03

  And so I think of design constraints as trying to bring together our moral imaginations and our technical imaginations. And that leads to really great engineering design. You know, so if I think about energy technologies, I want energy technologies that will give us lots of power, that will do so in a way that is consistent with how the rest of the biology and planet functions. And has limited risk in terms of generating waste or too much power. So if I give myself those design constraints, you know, as an engineer, as somebody who's developing new materials,

• 4:16Speaker 03

  I start looking at what kinds of sources for energy I might want to evolve. Like, I look a lot at chlorophyll. And I just think, how remarkable is this? All these green things somehow manage to absorb energy that's out there from the sun, it's there, and then transform it into a way in which it can be used. That seems like a really great idea. And there isn't a lot of waste generated that lays around and is dangerous to us for thousands, if not tens of thousands of years.

• 4:48Neil deGrasse Tyson?

  Well, technically, if there is a waste product, it's called oxygen.

• 4:51Chuck Nice?

  Oxygen.

• 4:52Speaker 03

  Yeah, there you go. Not such a bad waste product for us, huh?

• 4:56Neil deGrasse Tyson?

  That's the tree's waste product is oxygen, yeah.

• 4:59Speaker 03

  But that's the way that a design constraint that brings together our moral and technical imaginations can lead us in, I think, yeah, new and powerful directions.

• 5:08Chuck Nice?

  Do you try to consider unintended consequences of design? Or is that part of the process? Or does it just, oh, well?

• 5:17Neil deGrasse Tyson?

  They were unintended.

• 5:18Chuck Nice?

  It was unintended. Why do you think they call them unintended consequences?

• 5:24Speaker 03

  That is such an important question. So, you know, let's be honest. Anything we design and put out into the world, we put out into the world and people are gonna do stuff with it and they're gonna do things with it that we didn't anticipate. Like, the telephone is a great example. The telephone was never expected to be this communication device that people used in their homes and it connected women who were staying at home and created a whole society for them. That was an unintended consequence. Um, or the cookies that are being used on your computer right now. Those are a complete unintended consequence. That was just a little bit of data that was left on your machine to help debugging when browsers were first being developed.

• 6:09Speaker 03

  When that protocol was first being developed.

• 6:11Neil deGrasse Tyson?

  Wow.

• 6:12Speaker 03

  And its more massive impact has been our experience with cookies now. So, yeah, what's the takeaway? We design with our eyes open and then after we deploy something, we keep our eyes open. And we hold ourselves accountable for what happens as people take up these technologies and use them. So the design process goes longer than, oh, I had my big release. The design process follows it out. And when we see new things emerge, we're proactive, we're alert, we're proactive. And we see that as, as part of our responsibility as the technologists and engineers.

• 6:52Neil deGrasse Tyson?

  So allow me to push back on you just a little bit here. First, let me agree, of course, any good engineer loves constraints. Because that's the test of their ingenuity and creativity. Okay, if they say do it for this much money with this much energy, fit it into this volume, that's how you get discovery. Okay. That's how we folded up the James Webb Space Telescope into a rocket fairing. Some engineer said, what, I gotta put an eight-foot telescope, or I would have, did I, the diameter? Into this tiny fairing and they go home and come back and figure out how to do it. Right. It unfurls like the petals of a flower. Right. So, we're all in on that. However, uh, let me just push back here and say, if I'm in the lab about to invent something

• 7:35Neil deGrasse Tyson?

  that could be highly useful to society, or possibly even destructive, but it's just a discovery of the science embedded in some bit of engineering. Why should it be my responsibility to design it how you want me to, rather than your responsibility to convince people how to use it ethically? I can invent a knife. Is there an ethical knife? I don't know. But we want to train people how to use knives. Or any bit of technology and any tool that comes out of the brainchild of scientists put into play in the brainchild of engineers.

• 8:16Neil deGrasse Tyson?

  So I don't know that your constraints in my lab are the right thing for me when I just want the freedom to explore and discover. And let the ethical invocations happen after the fact.

• 8:29Chuck Nice?

  And Bhattacharya, now you know why scientists are going to kill us all.

• 8:33Neil deGrasse Tyson?

  No, stop.

• 8:34Speaker 03

  Well, I'm just, Neil, I'm just gonna mark a word in your comment, which is the word "you." Like, who is the "you" here? Which "you"? And how should we think about those different "yous"? And some of the things I think about when I do think about this question, I think there's discovery of basic knowledge, like fundamental underlying phenomena of the universe.

• 8:58Neil deGrasse Tyson?

  We split the atom. That was basic knowledge.

• 9:00Speaker 03

  And I see that as a different enterprise than the engineering enterprise of tools and technologies that we're gonna deploy in society.

• 9:08Neil deGrasse Tyson?

  I'm with you then.

• 9:09Speaker 03

  So I am a strong proponent of very, very diverse scientific exploration. In fact, I actually would claim that, you know, as a country in the United States, our scientific exploration is far more narrow than what I would like to see. And I would really push hard.

• 9:27Chuck Nice?

  So, based on what you just said, all right, here's an ethical question. There's a scientist who discovers a cure for cancer using a virus that can easily be manipulated as a biochemical weapon that could destroy an entire country in the course of 48 hours. This is the most virulent organism that's ever been placed on Earth.

• 9:58Chuck Nice?

  Would you say, go ahead and make that?

• 10:01Speaker 03

  Right. I'm gonna hold on to that for a minute. And I'm just gonna go back to Neil's comment and then I'll, I'll return to that.

• 10:09Chuck Nice?

  Okay.

• 10:09Speaker 03

  Because I also wanna say to Neil's comment, you know, we have limited, um, time and resources. So it is always the circumstance that we are choosing to do some things and not do other things. So it's, it's really a choice of where am I gonna direct my time and energy? Where am I going to, um, place my imaginative energies and innovation? And which ones am I not gonna do? Right? And we saw in the eighties, for example, a real push of resources towards the development of nuclear energy and away from photovoltaics. Right? So it, it, we live in that kind of resource.

• 10:52Speaker 03

  I don't know if you would call it resource scarce, but at least we don't get to work on everything at full force all at the same time. And we have to recognize we are making choices. So one of the first things I would say is, um, how do we make really good choices there? How do we use the resources we have in a way that will be most constructive for us? My own gestalt on that is, on the basic science side of things, I say, spread those resources across a wide diversity of different kinds of science and, um, different kinds of ideas. Far more diverse than what I think we tend to do in the United States. And on the engineering side,

• 11:35Speaker 03

  and now maybe I shift back Chuck to your question, which is really a great question. I don't tend to see the world in terms of forced choices or design trade-offs in the way that you framed it. I wanna bring back in that notion of constraint. And I wanna bring back in that notion of imagination. I think, likely enough,

• 11:56Speaker 03

  If we understand something about whatever this biology is, or whatever the piece is that might be a prevention against cancer, that if we push hard enough on ourselves, we will be able to invent ways that use that knowledge without having to also risk a really deadly virus. And I think that the propensity to say it's a trade-off, it's X or Y, we really limit ourselves in our abilities. So in the work that we do, we have moved away from the language of design trade-off or value conflict, and we talk about tensions.

• 12:39Speaker 03

  And then we talk about how you resolve tensions. And we talk about trying to populate this space with a whole range of better solutions. They're not necessarily perfect solutions. They're better solutions. And so that would be the approach I would take now. I don't know that science to know where, how it might go, but that would be my intuition.

• 13:04Neil deGrasse Tyson?

  That's a brilliant, that's a great, great answer. And insightful, and that's actually been done many times before. Just, it's a slight tangent to your line of work, but it's related. When they used to do crash tests with pigs. Right. Because you can get a hog that has sort of same sort of body mass as a human, put him in the driver's seat, crash the car, and the hog dies. Plus, it's the nearest thing to human skin. Yeah, yeah, okay. So the hog dies, and you say, "Well, there's no other way to do this," you might say at the time, until you say, "No, think of another way." And then we have the crash test dummy. Yeah. And the crash test dummy is even better than the, because you can put sensors everywhere throughout. And so that's, that, so I agree.

• 13:49Neil deGrasse Tyson?

  Not perfect, but better. Yeah, yes. Or even, maybe better and perfect, right? I agree that it's a false choice to say, "I can only do this if I possibly set loose a virus that kills an entire country." Well, then, maybe you're not clever enough. And keep at it. I was clever enough to kill a whole country. No, I'm joking.

• 14:14Neil deGrasse Tyson?

  Okay.

• 14:25Speaker 04

  I'm Kais from Bangladesh and I support StarTalk on Patreon. This is StarTalk with Neil deGrasse Tyson.

• 14:39Neil deGrasse Tyson?

  I have one last thing before we wrap. One last thing. Earlier you said you'd want the ethical compass to be pointed in a direction that serves us, serves civilization in some way. If we were 170 years ago in the American South, the ethical compass was, oh, let's create something where we can get more work out of the slaves and then we all benefit. That would be the ethical compass working in that time and in that place.

• 15:22Neil deGrasse Tyson?

  So what confidence do you have that whatever line of ethic, whatever ethical direction you want to take something in the room with the inventors, that that's, will still be the ethics that we value five years later, 50 years later, 100 years later?

• 15:39Speaker 03

  So it's a great, a really great question. I'm going to answer it in a couple of different ways. The first thing that I want to remind us all about is that, you know, moral philosophers have been trying to identify a workable ethical theory that cuts across all situations, all times. And we have some really good ideas, but none of them cover all of the situations that our intuitions tell us about. So sometimes a consequentialist theory is good, but it comes up short. And then there's a rights-based theory, but it comes up short. We can go to Buddhist ethics, we can go to Islamic ethics, we can go to... Various, you know, various ways of thinking.

• 16:22Speaker 03

  So the place where we are that we just have to accept is that while we're waiting to figure that out from a conceptual ethical moral point of view, we still live in the world and we still need to act in the world.

• 16:39Gary O'Reilly?

  Mm-hmm.

• 16:40Speaker 03

  And so the work that I've done has tried to take that really seriously, to create a space for ethical theory without explicitly saying which ethical theory and also leaving room for as we learn more that we can bring that in. So that's a little background to what you're saying. So now, what does Value Sense of Design do for the circumstance you're talking about? It puts a line in the sand and says you have to engage with all stakeholders, direct and indirect, who are going to be implicated by your technology. That means that not only do the people who want to benefit from somebody else's labor,

• 17:21Speaker 03

  not only are they stakeholders, but those people who are laboring are stakeholders. And Value Sense of Design says they're legitimate stakeholders and their views come into the design process without giving more power to one than another.

• 17:38Chuck Nice?

  That's incredible. That's highly enlightened. Where were you 170 years ago?

• 17:45Speaker 03

  So these are about practice. This is about practice and about implementing these practices. And so I'm going to tell you a story about a project, a very particular project, and you'll see why and how this actually matters and is practical. It's not pie in the sky. So in the state of Washington where I live, there is something called the Access to Justice Technology Principles that govern how the courts give access to technology, what they are required to do. And they were first about maybe 15 years ago, 20 years ago, and then they wanted to update them. And the committee that updated them came to my lab and they said, "You know, we've done a good job updating them,

• 18:30Speaker 03

  but we don't feel like we've really reached out to diverse groups of people. Can you help us?" My lab developed a method called the diverse voices process for tech policy. And the idea is that, you know, the rubber hits the road with the words on the page. So if we take a tech policy in its sort of polished draft form, and we can let groups that might otherwise be marginalized scrutinize that language, give feedback, and then we can help change those policies responsive to them. Then we can improve things. So we did, we ran panels with people who were formerly incarcerated.

• 19:13Speaker 03

  We ran them with immigrants. We ran them with people in rural communities. And we actually ran them with the people who do the court administration because they're also really key stakeholders. As a result of the work we did, there were two principles that were surfaced. One was about human touch and the other was about language. And people said to us things like, look, if somebody is going to deny me parole and I'm not going to get to be there for my kid's 13th birthday or hang out with them at, you know, their soccer games. You can relate to that, right, Gary? Thank you. I want a human being to look me in the eye

• 19:55Speaker 03

  and tell me that that's what my life is going to be for the next year because my parole is denied. I don't want to hear that from an AI. I don't want to hear that from a piece of technology. I want a human being to tell me that because this is a human experience, right? And so, in fact, we gave that feedback back to the committee. The committee then added in new principles, actually around human touch. And those were approved by the Washington State Supreme Court a couple of years ago. And those access to technology principles are a model that many states in the United States follow. So what I'm talking about is really practical.

• 20:38Speaker 03

  We're talking about how we actually improve things in practice, be it on the technology design side or on the policy side that governs how the technology is used.

• 20:49Neil deGrasse Tyson?

  And I love the fact that a state can do that independently from the federal government and be so good at it or so emulatable that other states will then use that as the model. And then that can spread across the country with or without federal guidance on top of it. Yeah.

• 21:07Gary O'Reilly?

  Excellent.

• 21:08Speaker 03

  Well, I guess if I was gonna say one, one last thing, it's, you know, because we have perhaps stumbled in the past, that's no reason to think we need to stumble in the future or stumble in the same way. You know, so really, my takeaway to everyone would be, hold on to your technical and moral imaginations and hold yourselves and your friends and your colleagues and the technology you buy accountable to that. And we will make progress, some incremental and some perhaps much bigger. But that as a, as a keystone, I think is, is really good guidance for us all.

• 21:47Neil deGrasse Tyson?

  A reminder why you are this year's winner.

• 21:50Chuck Nice?

  Yeah.

• 21:50Neil deGrasse Tyson?

  Future of life award.

• 21:52Chuck Nice?

  Yeah.

• 21:53Gary O'Reilly?

  Congratulations.

• 21:54Neil deGrasse Tyson?

  Thank you for being on StarTalk. Your vision for us all gives us hope, which we need a lot of that right now.

• 22:03Chuck Nice?

  Absolutely.

• 22:03Neil deGrasse Tyson?

  Okay.

• 22:04Gary O'Reilly?

  Thank you.

• 22:04Chuck Nice?

  And let me just say, Batia, as a

• 22:07Speaker 03

  Thank you.

• 22:07Chuck Nice?

  as an avid lover of alcohol, I have stumbled in the past and I am sure to stumble in the future as well.

• 22:12Neil deGrasse Tyson?

  Do we need to end on that note?

• 22:16Gary O'Reilly?

  Some things we can ignore.

• 22:17Neil deGrasse Tyson?

  Okay. Next up, our next Future of Life Award winner.

• 22:22Gary O'Reilly?

  Hmm.

• 22:23Neil deGrasse Tyson?

  Steve Omohundro.

• 22:25Gary O'Reilly?

  Yes.

• 22:25Neil deGrasse Tyson?

  Yes, he, he thinks about AI. There's not enough

• 22:30Gary O'Reilly?

  Not like I think, not the way I think about AI.

• 22:33Neil deGrasse Tyson?

  Differently, yes. There's not enough, it seems to me, there's not enough, whatever he did, there's not enough of them in the world.

• 22:38Gary O'Reilly?

  I believe so.

• 22:39Neil deGrasse Tyson?

  If we're thinking about the ethics of AI.

• 22:40Gary O'Reilly?

  Yes.

• 22:41Neil deGrasse Tyson?

  On everybody's mind.

• 22:42Gary O'Reilly?

  Yeah.

• 22:43Neil deGrasse Tyson?

  Right now.

• 22:44Gary O'Reilly?

  I mean, for sure.

• 22:45Neil deGrasse Tyson?

  Steve, welcome to StarTalk.

• 22:47Speaker 05

  Thank you very much.

• 22:48Neil deGrasse Tyson?

  Yeah, so for those who can see this on video, you're donning an eye patch and you said you had recent surgery, but none of us, none of us believe you. We're not buying it, we think you're training for the next Bond villain. Yes.

• 23:02Speaker 05

  That's very appropriate for the topic we're gonna discuss.

• 23:05Gary O'Reilly?

  You know, autonomous systems, AI, name it eye patch.

• 23:09Neil deGrasse Tyson?

  Ouch.

• 23:10Gary O'Reilly?

  Equals Bond villain, is the equation.

• 23:13Neil deGrasse Tyson?

  So where are we now with establishing AI ethics? Because the AI, it's, it delights some people, myself included. It freaks out other people. And we're all at some level thinking about The ethical invocation of it before AI becomes our overlord. So what is the current status of that right now?

• 23:37Speaker 05

  Well, I think we're right on the edge of some very important developments and very important human decisions. I've been working in AI for 40 years, and for the first half of that, I thought AI was an unabashed good. We'd cure cancer, we'd, you know, solve fusion, all the basic human problems we would solve with AI. But then about 20 years ago, I started thinking more deeply about, well, what's this actually going to happen when, if we succeed? What's going to happen when AIs can really reason about what they want to do? And I discovered that there are these things I call the basic AI drives, which are things that basically any AI which has simple goals, and I used to think about chess-playing AIs, will want to do.

• 24:20Speaker 05

  And some of those are get more resources so it can do more of what it wants to do, make copies of itself, keep itself from being turned off or changed. And so those things in the context of the human world are very risky and very dangerous. We didn't have the AIs 20 years ago that could do that, but we're about to have those in the next probably year or two. So this is a critical moment for humanity, I would say.

• 24:42Gary O'Reilly?

  So where do you stand on this, on the subject of consciousness engineering? Those that want to engineer AI for consciousness and those that want to not? What's the benefit, the good or bad here?

• 24:55Neil deGrasse Tyson?

  Is that the difference between a blunt computer that serves our needs and one that thinks about the problems you are?

• 25:01Gary O'Reilly?

  Mm-hmm, the self-improving algorithm, all those sort of things like that, yeah.

• 25:04Speaker 05

  Exactly. Well, I think long-term, you know, I mean, we may very well want to go there. In the short-term, I think we're nowhere close to being able to handle that kind of a system. So I would say, you know, if you made me, you know, king of the world, we limit AIs to being tools, only tools to help humans solve human problems, and we do not give them agency, we do not allow them to take over, you know, large systems. It's not easy necessarily to do that because many of these systems will want to take over things. And so we need technology to keep them limited, and that's what I'm thinking a lot about right now.

• 25:39Neil deGrasse Tyson?

  And in my field, it's exactly, I mean, we've been enjoying AI for a long, long time, and it's been a tool. It's not.

• 25:48Chuck Nice?

  Yeah.

• 25:49Neil deGrasse Tyson?

  It it's and a brilliant, beautiful tool makes our lives easier. Uh, and uh, once they're trained, we we go to the beach while it does the work. And I'm good with that. Uh, but yeah, we're not working with AI with agency.

• 26:04Chuck Nice?

  Yeah, because then it would be like, so how was the beach?

• 26:09Neil deGrasse Tyson?

  That's AI with attitude.

• 26:10Chuck Nice?

  Yeah, I hope you enjoyed yourself while I was here slaving away over my calculations.

• 26:17Neil deGrasse Tyson?

  AI with attitude.

• 26:18Gary O'Reilly?

  Yeah, so if if we do have AI with agency and then we continue to use it as just a tool, do we not get legal on the phone? And all of a sudden we're into contracts and.

• 26:31Speaker 05

  Oh, yeah, big problems, you know, can they vote? Can they own property?

• 26:35Chuck Nice?

  Right.

• 26:35Speaker 05

  And the latest models are have been discovered that they do do. They do something called sycophancy, which is they're trained to try and give responses that people rate as good. Well, AI is very quickly discovered that if you say, that was a brilliant question, you must be an amazing person. Then people say, yeah, that was a really good response. And so they'll just make up all kinds of stuff like that.

• 26:57Neil deGrasse Tyson?

  So they're ass kissing.

• 26:59Chuck Nice?

  Well, they know that we love that.

• 27:02Speaker 05

  Exactly.

• 27:03Neil deGrasse Tyson?

  So where does it stand now today? Is there a table you should be sitting at where you're not as we go forward on this frontier?

• 27:11Speaker 05

  Yeah, I mean, so so who's going to make these decisions? Well, it has to be somebody who understands the technology. Who understands the technology? The companies do. And so, uh, OpenAI, DeepMind, Anthropic, and Elon Musk's XAI is sort of an emergent one. These are the companies that are building these systems at their leading edge. They call them frontier models. And because they're the ones who know what's going on, they're the ones making these decisions. Now, the government has recently realized, oh, my goodness, we better get involved with this. And so there have been a lot of partnerships announced over the last few months, actually, between governmental agencies, intelligence agencies, defense agencies, and these leading edge AI companies.

• 27:53Speaker 05

  And so I think some kind of a new combination is emerging out of that that's going to make the actual end end decisions.

• 27:59Gary O'Reilly?

  So how do you incentivize these these tech companies? To embrace the safety architecture and not go gung ho and disappear off on their own agendas.

• 28:10Speaker 05

  That is the big challenge. And if we look at the history of OpenAI, it's a little bit of a cautionary tale. It was created, I think, around 2017 in response to Google's DeepMind, which was making great progress at the time.

• 28:23Neil deGrasse Tyson?

  Right.

• 28:24Speaker 05

  And a group of people said, oh, my God, we really have to worry about AI safety. It looks like this is happening quickly. Let's start a special company, which is nonprofit and which is particularly focused on safety. And they did that, and everything was great. Elon Musk was one of the forces behind it. There were internal struggles and so on, and Musk left. Well, when he left, he took away some of the money he was going to give them. So then they decided, oh, we need to make money. And so then they started becoming more commercial, and that process has continued. A group of the researchers there said, wait a minute, you're not focusing on safety. They left OpenAI and they started Anthropic to be even more safety oriented. And now Anthropic is also becoming much more commercial. And so

• 29:07Speaker 05

  the forces, the commercial forces, the political forces, the military forces, they all push in the direction of moving faster and, you know, get more advanced more quickly. Whereas the safety, everybody wants safety, but they sort of compete against these economic and political forces.

• 29:25Neil deGrasse Tyson?

  I was in the UAE a couple of years ago. And if I remember correctly, they have a Minister of AI. And as does China and some other countries sort of emergent on this space. How do we get that kind of ear and audience within our own governmental system? The military does have an AI group that's thinking about this.

• 29:51Speaker 05

  Absolutely.

• 29:52Neil deGrasse Tyson?

  As you would want them to.

• 29:53Speaker 05

  Exactly.

• 29:53Neil deGrasse Tyson?

  Yeah. But in terms of Policy and laws and legislation, do we need a cabinet member who's Secretary of AI or Secretary of Computing, something, some structural change?

• 30:06Speaker 05

  Yeah, this is the biggest change to humanity and to the planet ever, and it looks like it's happening, you know, sometime over the next decade, and many are predicting very short timelines. And so we, as a species, humanity is not ready for this.

• 30:23Neil deGrasse Tyson?

  Right.

• 30:23Speaker 05

  And so how do we deal with it? And people, many people are starting to wake up to that fact. And so there are lots and lots of meetings and organizations and groups. Um, it's still pretty incoherent, I would say.

• 30:35Gary O'Reilly?

  So Steve, if you've, if you've got this talking shop going on where something may or may not get done, are we misplaced focusing exactly on AI when we've still got quantum computing on the horizon?

• 30:47Neil deGrasse Tyson?

  Ooh, good one. Yeah, we, how much of this is premature?

• 30:52Chuck Nice?

  But won't, won't they go hand in hand? So it's like whatever, whatever problems you have with AI and whatever considerations you're making with AI, you're just gonna have to transfer them over to quantum computing.

• 31:03Neil deGrasse Tyson?

  Magnified.

• 31:03Chuck Nice?

  So you should really start dealing with it now.

• 31:05Gary O'Reilly?

  But if you're not in on the ground floor, not in at all.

• 31:09Neil deGrasse Tyson?

  Well, let's just, Steve hit this.

• 31:10Speaker 05

  Yeah, Steve. Well, so I'll give you an example. So quantum computing, you know, if it were successful, would break much of the public key cryptography that's used in the world today. And so NIST has been busily trying to create a post-quantum cryptography where they create new algorithms which wouldn't be vulnerable to, to quantum computing. But Meta, for example, has a group which is using the latest AI models to break these post-quantum uh, algorithms. And they've been successful at breaking some of them. And so, like you say, the two are going hand in hand. AIs will be much better at creating quantum algorithms than humans are. And that may lead to some great advances. It may also lead to, you

• 31:50Speaker 05

  know, current cryptography not, not withstanding that. And so, uh, that's another horror wave of, of, uh, transformation that's likely to happen.

• 31:59Chuck Nice?

  We, we just make every password one, two, three, four. No AI would ever go for that. They'd be like, oh, yes. So ridiculous.

• 32:08Neil deGrasse Tyson?

  And, and listening to you, Steve, it, it reminds me, was it Kurt Vonnegut in one of his stories? I don't remember which. He said, these are the last words ever spoken in the human species.

• 32:18Chuck Nice?

  Yes, yeah.

• 32:19Neil deGrasse Tyson?

  Two scientists saying, let's try it this other way. That's the end.

• 32:24Speaker 05

  Yeah, there you go. And that was it.

• 32:26Neil deGrasse Tyson?

  Yeah, yeah. Let's, let's try AI in this other mode.

• 32:30Speaker 05

  Right.

• 32:30Neil deGrasse Tyson?

  Boom, that's the end of the world, right, right there. So you can set ethical guidelines, but that doesn't stop bad actors out there.

• 32:37Speaker 05

  No.

• 32:37Neil deGrasse Tyson?

  That means a bad actor can take over the world while the rest of us are obeying ethical guides. So what are the guardrails put in place for something like that?

• 32:45Speaker 05

  I think that's one of the greatest challenges. We now have open source language models, open source AIs that are almost as powerful as the, the ones in the lab. And far more dangerous. And they're being downloaded hundreds of millions of times. And so you have to assume every actor in the world now, uh, China is now using, uh, Meta's latest models for their military AI. And so I believe we need hardware, uh, controls to limit the, the capabilities of, uh, so right now, the biggest AIs require these GPUs, uh, that are quite expensive and quite large. The, the latest one is the Nvidia H100. It's about $30,000 for a chip. Yeah. The US, uh, put an embargo on selling those to China, but apparently China has

• 33:30Speaker 05

  found ways to get the chips anyway.

• 33:32Neil deGrasse Tyson?

  Hmm.

• 33:33Speaker 05

  Um, people are gathering up these chips, gathering huge amounts of money, hundreds of, well, certainly hundreds of millions of dollars, billions of dollars. And now they're even talking about trillion dollar data centers over the next few years. And so the good news is if it really costs a trillion dollars to build the system that will host the super duper AI, then very few actors can, uh, pay that and therefore it'll be limited in, in its extent. You just described where the next frontier of warfare will exist. Yeah. Absolutely, absolutely. One thing, you know, it's pretty obvious these data centers are going to be a target and they don't seem to be building them in a very hardened way. So I think that's something people need to start thinking about, maybe underground data centers.

• 34:16Gary O'Reilly?

  Steve, are we?

• 34:19Gary O'Reilly?

  Looking at something in terms of the safety aspect here that's doable? Or are we just the kings of wishful thinking?

• 34:26Neil deGrasse Tyson?

  I want to make sure we got the good thoughts here.

• 34:29Gary O'Reilly?

  Yeah.

• 34:29Neil deGrasse Tyson?

  I don't want to leave this conversation with you completely bumming us out, okay?

• 34:33Speaker 05

  Yeah, I hope not to do that.

• 34:35Neil deGrasse Tyson?

  Yeah, yeah, Steve, give us a place where we can say, "Thank you, Steve, for being on our show."

• 34:39Gary O'Reilly?

  And be able to sleep tonight, yeah.

• 34:41Neil deGrasse Tyson?

  Yes, go.

• 34:42Speaker 05

  Well, so the truly safe technology needs to be based on the laws of physics and the law and the mathematical proof. So the only two things that we can be absolutely sure can't be subverted by a sufficiently powerful AI. And AIs are getting very good at both of those. They're becoming able to model physical systems and design physical systems with whatever characteristics we want. And they're also able to perform mathematical proof in a very good way. And so it looks to me like we can design hardware that puts constraints on AIs of whatever form we want, but that we need AI to design this hardware and that if we can shift humanities technological infrastructure.

• 35:27Neil deGrasse Tyson?

  You say, "AI, please design your own prison cell that we're going to put you in." That's what you just said.

• 35:32Speaker 05

  Exactly. And so

• 35:33Gary O'Reilly?

  And then it's going to design a way to get out.

• 35:35Speaker 05

  We certainly don't want an agent to do that because then they'll find some way to hide a back door or something. But by using mathematical proof, we can get absolute guarantees about the properties of systems. And we're just on the verge of that kind of technology. And so I'm very hopeful that probably the next two or three years, you know, there are several groups who are building superhuman mathematicians, and they're expecting to be at the level of, say, human graduate students in mathematics by the end of this year. Using those AIs, we can build designs for systems that have properties that we are very, very confident in. And so I think that's where real safety is going to come from. But it builds on top of AI. So we need them both.

• 36:19Chuck Nice?

  I was going to say, the good thing about what you just said, even though it sounds crazy to have the inmate design its own cell, is that without agency

• 36:29Chuck Nice?

  At this point, it's just a drone carrying out an order.

• 36:32Neil deGrasse Tyson?

  Yeah.

• 36:32Chuck Nice?

  So that's, that's the encouraging part. You know, so, yeah. Whereas if it were sentient in any way, or if it had some kind of agency, it could very well say, yeah, I'm also gonna design a backdoor and a trapdoor.

• 36:47Neil deGrasse Tyson?

  And I'm not gonna tell you.

• 36:48Chuck Nice?

  And I'm not gonna tell you.

• 36:49Neil deGrasse Tyson?

  Close enough. Steve, first, congratulations on winning this award. You are exactly the right kind of person who deserves such an award. That give us hope for the future of our relationship with technology.

• 37:03Chuck Nice?

  Yeah, yes.

• 37:03Neil deGrasse Tyson?

  And the health and wealth and security of civilization as we go forward.

• 37:08Chuck Nice?

  Yeah.

• 37:09Neil deGrasse Tyson?

  So.

• 37:09Chuck Nice?

  Thank you so much. And, and I look forward to the day where an AI beats you out for this award.

• 37:14Speaker 05

  Oh, great, yeah. That's a great point. Maybe next year, it'll be an AI that wins.

• 37:21Chuck Nice?

  I'm joking, by the way.

• 37:22Neil deGrasse Tyson?

  Steve Almohondro, winner of this year's Future of Life Award.

• 37:27Chuck Nice?

  Yeah.

• 37:27Neil deGrasse Tyson?

  And deservedly so. Thank you. Thank you very much.

• 37:31Gary O'Reilly?

  Before we jump to our next segment, I need to acknowledge the third honoree, James Moore, who is now sadly deceased. His paper in 1985, What Is Computer Ethics? Established him as a pioneering theoretician in this field. His policy vacuum concept created the guidelines to address the challenges of emerging technologies. His work profoundly influencing today's policymakers and researchers. Gone, but not forgotten.

• 38:20Neil deGrasse Tyson?

  So. Seems to me that for ethical principles to work at all, they have to be everywhere at all times and capable of evolving with the technology itself. I, I can't foresee the, the ethics panel getting together and from on high.

• 38:43Neil deGrasse Tyson?

  Declaring what is ethical and what isn't. And then everyone has to obey that for the next 10 years.

• 38:48Gary O'Reilly?

  You put 10 people in a room, nearly get 12 opinions. Right? That's a basic human nature. Then you've got to get all of these components, all of these nation states or whatever investment groups.

• 38:59Neil deGrasse Tyson?

  Demographics?

• 39:00Gary O'Reilly?

  Demographics with their own agendas to buy into the same principles. Because on a Wednesday, the principle's not the same for them. They're going to think in a different direction. Oh.

• 39:12Neil deGrasse Tyson?

  But that doesn't, that doesn't even scare me. What scares me more than anything? China, Russia, North Korea.

• 39:17Gary O'Reilly?

  Yeah.

• 39:18Neil deGrasse Tyson?

  It's that's it. Serious. I'm not even gonna, this is China, Russia, North Korea. Yeah, we can put out any constraints we have on ourselves. There you go. Doesn't mean anybody else is paying attention. And that's the problem.

• 39:26Gary O'Reilly?

  And you're herding cats. Good luck.

• 39:28Neil deGrasse Tyson?

  Yeah, that's, well, that's what makes it so scary. Well, well, herding cats with nuclear.

• 39:34Gary O'Reilly?

  Yeah.

• 39:34Neil deGrasse Tyson?

  Yeah. We're herding nuclear cats. Herding nuclear, right. Exploding nuclear cats. Nuclear cats. It's the newest, it's the newest game sweeping the internet. It's a whole other meaning to Schrodinger's cat. Yeah. Dead or alive.

• 39:46Gary O'Reilly?

  I mean, then you've got autonomous systems. And if it's geared to say, if it's human, kill it. That's problematic.

• 39:55Neil deGrasse Tyson?

  Here's another little known fact. When we signed with the Soviet Union, the Nuclear Test Ban Treaty. Mm hmm. That was progress. Absolutely. This was, you will no longer test nuclear weapons. Because at the time, from the late 1950s into the early 1960s, there were several tests a day. In some, in some years, there were several tests a day. Right. Okay, somewhere in the world. Right. And. Which made for such great video. Okay, so we said, so we, we said, well, this has to stop. All right.

• 40:28Gary O'Reilly?

  Yeah.

• 40:29Neil deGrasse Tyson?

  What is, little known fact, and we write about this in the accessory to war, the Unspoken Alliance Between Astrophysics and the Military book. In that book, we highlight the fact that we agreed to that around the same time that computing power was good enough to calculate the results of what would be a test. So we didn't really stop testing.

• 41:00Neil deGrasse Tyson?

  Not philosophically, not morally.

• 41:03Chuck Nice?

  Mm hmm.

• 41:04Gary O'Reilly?

  Was that where MAD came from? Mutually assured destruction?

• 41:06Neil deGrasse Tyson?

  Oh, that was later.

• 41:07Gary O'Reilly?

  Yeah.

• 41:07Neil deGrasse Tyson?

  That was later. I'm not convinced, based on my read of history, that anyone, any one nation can utilaterally say, oh, we're gonna just do nice things and moral and ethical things with this new technology.

• 41:20Chuck Nice?

  Right.

• 41:20Gary O'Reilly?

  Yeah.

• 41:21Neil deGrasse Tyson?

  I don't boo. Yes, let's say you do that, but no one else does it, then...

• 41:27Chuck Nice?

  What difference does it make?

• 41:28Neil deGrasse Tyson?

  What difference does it make?

• 41:29Gary O'Reilly?

  You know, you gotta play by the same rulebook, but we know that's not likely to happen.

• 41:33Chuck Nice?

  Yeah.

• 41:33Gary O'Reilly?

  I mean, what was interesting-

• 41:34Neil deGrasse Tyson?

  The history of our species is, offers great evidence for that impossibility.

• 41:38Gary O'Reilly?

  But when you, when you listen to Batya talking, there's such a strength in the points that she makes. You would hope that would, people will go, you know what, yeah, and the majority come online, and then these guys sit in isolation, testing, you know, intercontinental ballistic missiles.

• 41:57Neil deGrasse Tyson?

  Yeah, but, but the MAD concept, mutually assured destruction.

• 42:00Chuck Nice?

  Right.

• 42:01Neil deGrasse Tyson?

  Just think about that. That brought the United States and the Soviet Union to the table.

• 42:06Chuck Nice?

  Yes.

• 42:07Neil deGrasse Tyson?

  Not because they thought nuclear weapons were bad, but they realized they couldn't win.

• 42:14Chuck Nice?

  Right, right, and that's the problem.

• 42:15Neil deGrasse Tyson?

  The war.

• 42:16Chuck Nice?

  When you can't win. Now, and what that does-

• 42:18Neil deGrasse Tyson?

  That doesn't mean they didn't, weren't thinking about it.

• 42:20Chuck Nice?

  Right.

• 42:20Neil deGrasse Tyson?

  Or if they could win, they would.

• 42:22Chuck Nice?

  And it also doesn't mean that they've taken into account for what I call the Nero scenario.

• 42:28Neil deGrasse Tyson?

  What's that?

• 42:29Chuck Nice?

  So what did Nero do? He fiddled while Rome burned.

• 42:32Neil deGrasse Tyson?

  Oh.

• 42:32Chuck Nice?

  He did, he burned it down. He didn't care. So what happens if, you know, you're still in a position where the danger is ever present?

• 42:40Neil deGrasse Tyson?

  So, just because I've spent enough time hanging around military people, I don't talk about, I'm not talking about hawks that, you know, that just want, I'm just talking about people who think about the history of conflict in this world.

• 42:52Chuck Nice?

  Right.

• 42:53Neil deGrasse Tyson?

  And the behavior of other members of our species.

• 42:55Chuck Nice?

  Not, not just one guy standing there going, "Just smell that, son? That smell, do you smell it?"

• 43:02Gary O'Reilly?

  I know where that came from. Yeah, the apocalypse. But you see, you-

• 43:07Neil deGrasse Tyson?

  Apocalypse now.

• 43:07Gary O'Reilly?

  Yeah, you speak to- The generals and the majors, you find invariably they're students of war. They've understood strategies, they understood histories, the provocations and the outcomes.

• 43:19Neil deGrasse Tyson?

  And most of them are not the warmongers we stereotype them to be.

• 43:23Gary O'Reilly?

  Exactly, because of that knowledge, that understanding.

• 43:25Neil deGrasse Tyson?

  Correct. And so I just, I don't have the confidence. I mean, I wish I was, I was as hopeful as Batya is. I want to be that hopeful. I will aspire to be that hopeful.

• 43:39Gary O'Reilly?

  So I just wonder when, when she talks, how far ahead of a story in terms of a technology's development are they? And how far are they playing catch up? And, you know, are they being able to bake it in from the get-go? Or are they just trying to retro engineer what's gone wrong?

• 43:56Neil deGrasse Tyson?

  It could be a new emergent philosophy where everyone knows to bake it in from the beginning. That would be a shift in our conduct and in our awareness. The kind of shift, for example, dare I harp on this yet again, that when we went to the moon to explore the moon, we looked back and discovered Earth for the first time.

• 44:18Gary O'Reilly?

  Yes.

• 44:20Neil deGrasse Tyson?

  Around the world, people started thinking about Earth as a planet. Earth as a holistic entity that has interdependent elements. There's no one island distinct from the rest of anything else that's going on on this planet.

• 44:35Gary O'Reilly?

  There's no boundaries.

• 44:37Neil deGrasse Tyson?

  No boundaries. We share the same air molecules, water molecules. And that was a firmware upgrade to our sensibilities of our relationship with nature. And that's why to this day, people all around the world say, we gotta save Earth. Nobody was saying that before we went to the moon and looked at Earth in the sky. All the peaceniks at the time, in the 1960s, they were just anti-war. They weren't, let's save the Earth. Nobody had that kind of sensibility. So maybe it's a sensibility upgrade that's waiting to happen on civilization. Lest we all die at the hands of our own.

• 45:15Neil deGrasse Tyson?

  Discoveries. Yeah, I'm going with the last part. I'm just saying that, you know, you talk about Earth Day, you talk about we went to the moon, and there are people who think we didn't go to the moon and that the earth is flat. Yeah, we are, we're screwed. And by the Earth Day, the first Earth Day was 1970. Right. While we were going to the moon.

• 45:33Gary O'Reilly?

  And the irony is-

• 45:33Neil deGrasse Tyson?

  Could have been 1960, but it wasn't. No. Might have been delayed, 1980. No, while we were going to the moon, first Earth Day. Right.

• 45:39Gary O'Reilly?

  So is the irony that we lean into AI to get it to help us create ethical and safety architecture?

• 45:47Neil deGrasse Tyson?

  Help it save us from ourselves. I like that. Maybe that's the way to flip the table. Right.

• 45:54Gary O'Reilly?

  Yeah.

• 45:55Neil deGrasse Tyson?

  And that should be it. And say, AI, they're bad actors among humans who are trying to use AI to get rid of humans. Now kill them.

• 46:07Neil deGrasse Tyson?

  Chuck! This is the first one?

• 46:09Gary O'Reilly?

  This is where they live.

• 46:11Neil deGrasse Tyson?

  Hit their dress, dox them.

• 46:13Gary O'Reilly?

  This is their daily routine.

• 46:18Neil deGrasse Tyson?

  Google knows your daily routine.

• 46:20Gary O'Reilly?

  We really are-

• 46:21Neil deGrasse Tyson?

  Android. That is for sure. Knows who you've been Googling. That's for sure. Knows everything.

• 46:24Gary O'Reilly?

  We really are bad people.

• 46:27Neil deGrasse Tyson?

  Maybe it's the good AI.

• 46:29Gary O'Reilly?

  Yes.

• 46:29Neil deGrasse Tyson?

  Against, that's the future battle.

• 46:30Gary O'Reilly?

  Oh.

• 46:31Neil deGrasse Tyson?

  Good AI against- Versus bad AI, evil AI. Evil AI.

• 46:35Gary O'Reilly?

  But then again, the bad AI, there's a bad-

• 46:36Neil deGrasse Tyson?

  With a goatee.

• 46:37Gary O'Reilly?

  The bad AI will tell you that the good AI is the bad AI. And then you cut, the first casualty of war?

• 46:43Neil deGrasse Tyson?

  Right.

• 46:44Gary O'Reilly?

  Is always the truth.

• 46:46Neil deGrasse Tyson?

  Ooh. Yeah.

• 46:48Gary O'Reilly?

  Thank you. I don't know who authored that, but that's brilliant.

• 46:50Neil deGrasse Tyson?

  That was deep. Yeah.

• 46:52Gary O'Reilly?

  Yep.

• 46:52Neil deGrasse Tyson?

  Oh, that's deep.

• 46:54Gary O'Reilly?

  And truthful.

• 46:55Neil deGrasse Tyson?

  I wish it weren't true.

• 46:56Gary O'Reilly?

  Exactly.

• 46:57Neil deGrasse Tyson?

  Stop speaking the truth. Why don't you lie to us every now and then? Yeah, like all, like everybody else. Yeah, dude.

• 47:04Gary O'Reilly?

  You could be giving a new program.

• 47:06Neil deGrasse Tyson?

  All right, this has been our Future of Life installment.

• 47:09Gary O'Reilly?

  Hmm.

• 47:10Neil deGrasse Tyson?

  Of StarTalk Special Edition.

• 47:12Gary O'Reilly?

  Yeah.

• 47:12Neil deGrasse Tyson?

  Yeah, I enjoyed this. Thank you.

• 47:13Gary O'Reilly?

  Yeah, and congratulations to the award winners. They are the people that we need out there.

• 47:18Neil deGrasse Tyson?

  Yes. Let's, we not be around.

• 47:20Gary O'Reilly?

  Hmm.

• 47:21Neil deGrasse Tyson?

  To even think about that problem. Yes, well, yeah. All right, Gary, Chuck.

• 47:26Gary O'Reilly?

  Pleasure.

• 47:27Neil deGrasse Tyson?

  Always good to have you. Neil deGrasse Tyson here, as always, bidding you to keep looking up.