#144 – Michael Littman: Reinforcement Learning and the Future of AI

Michael Littman is a computer scientist at Brown University. Please support this podcast by checking out our sponsors: – SimpliSafe: https://simplisafe.com/lex and use code LEX to get a free security camera – ExpressVPN: https://expressvpn.com/lexpod and use code LexPod to get 3 months free – MasterClass: https://masterclass.com/lex to get 2 for price of 1 – BetterHelp: https://betterhelp.com/lex to get 10% off EPISODE LINKS: Michael’s Twitter: https://twitter.com/mlittmancs Michael’s Website: https://www.littmania.com/ Michael’s YouTube: https://www.youtube.com/user/mlittman PODCAST INFO: Podcast website: https://lexfridman.com/podcast Apple Podcasts: https://apple.co/2lwqZIr Spotify: https://spoti.fi/2nEwCF8 RSS: https://lexfridman.com/feed/podcast/ YouTube Full Episodes: https://youtube.com/lexfridman YouTube Clips: https://youtube.com/lexclips SUPPORT & CONNECT: – Check out the sponsors above, it’s the best way to support this podcast – Support on Patreon: https://www.patreon.com/lexfridman – Twitter: https://twitter.com/lexfridman – Instagram: https://www.instagram.com/lexfridman – LinkedIn: https://www.linkedin.com/in/lexfridman – Facebook: https://www.facebook.com/LexFridmanPage – Medium: https://medium.com/@lexfridman OUTLINE: Here’s the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time. (00:00) – Introduction (07:43) – Robot and Frank (10:02) – Music (13:13) – Starring in a TurboTax commercial (23:26) – Existential risks of AI (41:48) – Reinforcement learning (1:07:36) – AlphaGo and David Silver (1:17:15) – Will neural networks achieve AGI? (1:29:42) – Bitter Lesson (1:42:32) – Does driving require a theory of mind? (1:51:58) – Book Recommendations (1:57:20) – Meaning of life

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation with Michael Littman, a computer science professor at Brown University, doing research on and teaching machine learning, reinforcement learning, and artificial intelligence. He enjoys being silly and lighthearted in conversation, so this was definitely a fun one. Quick mention of each sponsor, followed by some thoughts related to the episode. Thank you to Simply Safe, a home security company I use to monitor and protect my apartment; ExpressVPN, the VPN I've used for many years to protect my privacy on the internet; MasterClass, online courses that I enjoy from some of the most amazing humans in history; and BetterHelp, online therapy with a licensed professional.

• 0:45Lex Fridman?

  Please check out these sponsors in the description to get a discount and to support this podcast. As a side note, let me say that I may experiment with doing some solo episodes in the coming month or two. The three ideas I have floating in my head currently is to use one, a particular moment in history; two, a particular movie; or three, a book, to drive a conversation about a set of related concepts. For example, I could use 2001: A Space Odyssey or Ex Machina to talk about AGI for one, two, three hours. Or I could do an episode on the, yes, rise and fall of Hitler and Stalin, each in a separate episode,

• 1:30Lex Fridman?

  using relevant books and historical moments for reference. I find the format of a solo episode very uncomfortable and challenging, but that just tells me that it's something I definitely need to do and learn from the experience. Of course, I hope you come along for the ride. Also, since we have all this momentum built up on announcements, I'm giving a few lectures on machine learning at MIT this January. In general, if you have ideas for the episodes, for the lectures, or for just short videos on YouTube, let me know in the comments that I still definitely read, despite my better judgment and the wise, sage advice of the great Joe Rogan.

• 2:16Lex Fridman?

  If you enjoy this thing, subscribe on YouTube, review it with five stars on Apple Podcast, follow on Spotify, support on Patreon, or connect with me on Twitter at Lex Fridman. As usual, I'll do a few minutes of ads now and no ads in the middle. I try to make these interesting, but I give you timestamps. So if you skip, please still check out the sponsors by clicking the links in the description. It is the best way to support this podcast. This show is sponsored by SimpliSafe, a home security company. Everyone wants to keep their home and family safe. That's what they told me to say, so it must be true. Whether it's from a break in, a fire, flooding, or a medical emergency,

• 2:57Lex Fridman?

  SimpliSafe home security got your back day and night, ready to send police, fire, or EMTs when you need them most, straight to your door. I'm pretty sure if you suffer an AGI robot takeover, they will also allegedly send spot robots from Boston Dynamics for a full on robot on robot battle. However, small caveat, I haven't tried this aspect of the service yet myself, so I can't tell you if it's a good idea or not. They have sensors and cameras that protect every inch of your home. All it takes is a simple 30 minute setup. I have it set up in my apartment, but unfortunately, anyone who tries to break in

• 3:39Lex Fridman?

  will be very disappointed by the lack of interesting or valuable stuff to take. Some dumbbells, pull a bar, and some suits and shirts. That's about it. You get a free security camera and a 60 day risk free trial when you go to simplisafe.com/lex. Again, that's simplisafe.com/lex. This episode is also sponsored by ExpressVPN. Earlier this year, more than 100 Twitter users got their accounts hacked into, passwords, email address, phone numbers, and more. The list included Elon Musk and Kanye West. Can't believe they gave me those two options. ExpressVPN can help avoid that. I use it to safeguard my personal data online.

• 4:25Lex Fridman?

  Did you know that for 20 years, the Permissive Action Link PAL access control security device that controls access to the United States nuclear weapons had a password of just eight zeros? That's it. Apparently, this was a protest by the military to say that PAL systems are generally a bad idea because they're hackable and so on. Also, the most popular leaked password of 2020 are 123456, 123456789, picture one, password, and 12345678. If you have one of these passwords, please perhaps make it a new year

• 5:07Lex Fridman?

  resolution to change them. Anyway, ExpressVPN encrypts your data unless you surf the web safely and anonymously. Get it at expressvpn.com/lexpod to get extra three months free. That's expressvpn.com/lexpod. This show is also sponsored by MasterClass. $180 a year for an all-access pass to watch courses from literally the best people in the world on a bunch of different topics. Let me list some I've enjoyed watching in part or in whole. Chris Hadfield on space exploration, Neil deGrasse Tyson on scientific thinking and communication, Will Wright, creator of Simcity and Sims on game design, Carlos Santana

• 5:48Lex Fridman?

  on guitar, Garry Kasparov on chess, Daniel Negrono on poker, Neil Gaiman on storytelling, Martin Scorsese on filmmaking, Jane Goodall on conservation, and many more. By the way, you can watch it on basically any device. Sign up at masterclass.com/lex to get 15% off the first year of an annual subscription. That's masterclass.com/lex. This episode is also sponsored by BetterHelp, spelled H E L P, Help. They figure out what you need and match you with a licensed professional therapist in under 48 hours. I chat with the person on there and enjoy it. Of course, I also have been talking to David Goggins over the past few months, who's definitely not

• 6:31Lex Fridman?

  a licensed professional therapist, but he does help me meet his and my demons and become comfortable to exist in their presence. Everyone is different, but for me, I think suffering is essential for creation, but you can suffer beautifully in a way that doesn't destroy you. Therapy can help in whatever form that therapy takes. BetterHelp is an option worth trying. They're easy, private, affordable, available worldwide. You can communicate by text anytime and schedule weekly audio and video sessions. You didn't ask me, but my two favorite psychiatrists, Zigmund Freud and Carl Jung, their work was important in my intellectual development. Anyway, check out betterhelp.com/lex. That's

• 7:16Lex Fridman?

  betterhelp.com/lex. And now here's my conversation with Michael Legman.

• 7:43Lex Fridman?

  I saw a video of you talking to Charles Isbell about Westworld, the TV series. You guys were doing a kind of thing where you're watching new things together, but let's rewind back. Is there a sci-fi movie or book or shows that you that was profound, that had an impact on you philosophically, or just like specifically something you enjoyed nerding out about?

• 8:08Speaker 01

  Yeah, interesting. I think a lot of us have been inspired by robots in movies. The one that I really like is, there's a movie called Robot and Frank, which I think is really interesting because it's very near term future, where robots are being deployed as helpers in people's homes. And it was, and we don't know how to make robots like that at this point, but it seemed very plausible. It seemed very realistic or imaginable. And I thought that was really cool because they're awkward, they do funny things. It raised some interesting issues, but it seemed like something that would ultimately be helpful and good if we could do it right.

• 8:44Lex Fridman?

  Yeah, he was an older cranky gentleman, right?

• 8:46Speaker 01

  He was an older cranky jewel thief, yeah.

• 8:49Lex Fridman?

  It's kind of funny little thing, which is, you know, he's a jewel thief. And so he pulls the robot into his life, which is like, which is something you could imagine, taking a home robotics thing and pulling into whatever quirky thing that's involved in your existence.

• 9:08Speaker 01

  Meaningful to you, exactly so. Yeah. And I think, I think from that perspective, I mean, not all of us are jewel thieves. And so when we bring our robots into our lives.

• 9:15Lex Fridman?

  Speak for yourself, yeah.

• 9:16Speaker 01

  It explains a lot about this apartment, actually. But no, the idea that, that people should have the ability to, you know, make this technology their own, that it becomes part of their lives. And I think that's, it's hard for us as technologists to make that kind of technology. It's easier to mold people into what we need them to be. And just that opposite vision, I think is really inspiring.

• 9:38Lex Fridman?

  And then there's a anthropomorphization where we project certain things on them. Because I think the robot was kind of dumb. But I have a bunch of Roombas I play with and they, you immediately project stuff onto them. Much greater level of intelligence. Probably do that with each other, too. Much, much greater, much greater degree of compassion.

• 9:57Speaker 01

  One of the things we're learning from AI is where we are smart and where we are not smart.

• 10:01Lex Fridman?

  Yeah. You also enjoy, as people can see, and I enjoyed myself, watching you sing and even dance a little bit, a little bit, a little bit, a little bit of dancing.

• 10:13Speaker 01

  A little bit of dancing, that's not quite my thing.

• 10:16Lex Fridman?

  As a, as a method of education, or just in life, you know, in general, so, easy question, what's the definitive, objectively speaking, top three songs of all time? Maybe something that, you know, to walk that back a little bit, maybe something that others might be surprised by, the three, three songs that you kind of enjoy.

• 10:40Speaker 01

  That is a great question that I cannot answer, but instead, let me tell you a story. So.

• 10:45Lex Fridman?

  Pick a question you do want to answer.

• 10:47Speaker 01

  That's right. I've been watching the presidential debates and vice presidential debates, and turns out, yeah, it's really, you can just answer any question you want. So, so.

• 10:54Lex Fridman?

  Let me interrupt you.

• 10:55Speaker 01

  It's a related question.

• 10:58Speaker 01

  Yeah, well said. I really like pop music. I've enjoyed pop music ever since I was as very young, so 60s music, 70s music, 80s music, this is all awesome. And then I had kids, and I think I stopped listening to music, and I was starting to realize that the, like, my musical taste had sort of frozen out. And so I decided, in 2011, I think, to start listening to the top 10 Billboard songs each week. So I'd be on the, on the treadmill, and I would listen to that week's top 10 songs, so I could find out what was popular now. And what I discovered is that I have no musical taste whatsoever. I like what I'm familiar with. And so, the first time I'd hear a song, it's the first week that was on the charts, I'd be like, ugh. And then the second week, I was into it a little bit, and the third

• 11:40Speaker 01

  week, I was loving it, and by the fourth week is like, just part of me. And so, I'm afraid that I can't tell you the most, my favorite song of all time, because it's whatever I heard most recently.

• 11:50Lex Fridman?

  Yeah, that's interesting. People have told me that.

• 11:56Lex Fridman?

  There's an art to listening to music as well. Then you can start to, if you listen to a song just carefully, explicitly just force yourself to really listen. You start to, I did this when I was part of jazz band and fusion band in college. Is there's they, you, you start to hear the layers of the instruments. You start to hear the individual instruments. And you, you start to, you can listen to classical music at orchestra this way. You can listen to jazz this way. And it's funny to imagine you now to walk in that forward to listening to pop hits now as like a scholar. Listening to like Cardi B or something like that, or Justin Timberlake. Is he, no, not Timberlake, Bieber.

• 12:37Speaker 01

  They've both been in the top 10 since I've been listening.

• 12:39Lex Fridman?

  They're still, they're still up there. Oh, my God, I'm so cool.

• 12:43Speaker 01

  If you haven't heard Justin Timberlake's top 10 in the last few years, there was one song that he did where the music video was set at essentially NURIPS.

• 12:52Lex Fridman?

  Oh, wow. Oh, the one with the robotics. Yeah, yeah, yeah, yeah, yeah.

• 12:55Speaker 01

  Yeah, yeah, it's like at an academic conference and he, and he's doing a demo.

• 12:58Lex Fridman?

  He was presenting, right?

• 13:00Speaker 01

  It was sort of a cross between the Apple, like Steve Jobs kind of talk and NURIPS.

• 13:05Lex Fridman?

  Yeah.

• 13:06Speaker 01

  So I, you know, it's, it's always fun when AI shows up in pop culture.

• 13:09Lex Fridman?

  I wonder if he consulted somebody for that. That's, that's really interesting. So maybe on that topic, I've seen your, your celebrity in multiple dimensions. But one of them is you've done cameos in different places. I've seen you in a TurboTax commercial as like, I guess, the, the brilliant Einstein character. And the, the point is that TurboTax doesn't need somebody like you.

• 13:36Lex Fridman?

  It doesn't need a brilliant.

• 13:37Speaker 01

  Very few things need someone like me. But yes, they were specifically emphasizing the idea that you don't need to be a, like a computer expert to be able to use their software.

• 13:45Lex Fridman?

  How'd you end up in that world?

• 13:47Speaker 01

  I think it's an interesting story. So I was teaching my class. It was an intro computer science class for non-concentrators, non-majors. And I, sometimes when people would visit campus, they would check in to say, hey, we want to see what a class is like. Can we sit on your class? So a person came to my class who was the daughter of the brother of the hus- husband of the best friend of my wife.

• 14:18Speaker 01

  Anyway, basically, a family friend came to campus to check out Brown and asked to come to my class and came with her dad. Her dad is, who I've known from various kinds of family events and so forth, but he also does advertising. And he said that he was recruiting scientists for this ad, this TurboTax set of ads. And he said, "We wrote the ad with the idea that we'd get the most brilliant researchers, but they all said no. So can you help us find like B-level scientists?" And I'm like, "Sure, that's who I hang out with, so that should be fine." Yeah.

• 15:00Speaker 01

  So I put together a list, and I did what some people called a Dick Cheney. So I included myself on the list of possible candidates, with a little blurb about each one and why I thought it would make sense for them to do it. And they reached out to a handful of them, but then ultimately they YouTube stalked me a little bit, and they thought, "Oh, I think he could do this." And they said, "Okay, we're going to offer you the commercial." I'm like, "What?" So it was such an interesting experience, because they have another world. The people who do nationwide ad campaigns and television shows and movies and so forth, it's quite a remarkable system that they have going. Is it like a set?

• 15:40Speaker 01

  Yeah, so I went to, it was just somebody's house that they rented in New Jersey, but in the commercial, it's just me and this other woman. In reality, there were 50 people in that room, and another half a dozen kind of spread out around the house in various ways. There were peoples whose job it was to control the sun. They were in the backyard on ladders, putting filters up to try to make sure that the sun didn't glare off the window in a way that would wreck the shot. So there was like six people out there doing that. There was three people out there giving snacks, the craft table. There was another three people giving healthy snacks, because that was a separate craft table. There was one person whose job it was to keep me from getting lost.

• 16:24Speaker 01

  And I think the reason for all this is because so many people are in one place at one time, they have to be time efficient, they have to get it done. The morning, they were gonna do my commercial, in the afternoon, they were gonna do a commercial of a mathematics professor from Princeton. They had to get it done, no wasted time or energy. And so there's just a fleet of people, all working as an organism. And it was fascinating. I was just the whole time, just looking around, like, this is so neat. Like, one person whose job it was to take the camera off of the camera man, so that someone else, whose job it was to remove the film canister. Because every couple's takes, they had to replace the film, because, you know, film gets used up. It was just, I don't know, I was geeking out the whole time, it was so fun.

• 17:07Lex Fridman?

  How many takes did it take? It looked the opposite, like there was more than two people there, it was very relaxed.

• 17:12Speaker 01

  Right, yeah. I mean, the person who I was in the scene with is a professional, she's a, you know,

• 17:19Lex Fridman?

  She's an actress?

• 17:19Speaker 01

  Improv comedian from New York City. And when I got there, they had given me a script, as such as it was. And then I got there and they said, "We're gonna do this as improv." I'm like, "I don't know how to improv, like, this is not, I don't know what this, I don't know what you're telling me to do here." "Don't worry, she knows." I'm like, "Okay, okay, we'll see how this goes."

• 17:38Lex Fridman?

  I guess I got pulled into the story, because like, where the heck did you come from? I guess in the scene. Like, how did you show up in this random person's house? I don't know.

• 17:47Speaker 01

  Yeah, well, I mean, the reality of it is, I stood outside in the blazing sun, there was someone whose job it was to keep an umbrella over me, because I started to schwitz, I started to sweat. And so I would wreck the shot, because my face was all shiny with sweat. So there was one person who would dab me off, had an umbrella. But yeah, like, the reality of it, like, why is this strange stalkery person hanging around outside somebody's house?

• 18:08Lex Fridman?

  We're not sure, we'll have to look in, we'll have to wait for the book. But are you, so you make, like you said, YouTube, you make videos yourself, you make awesome parody, sort of parody songs that kind of focus in a particular aspect of computer science. How much, those seem really natural, how much production value goes into that? Do you also have a team of 50 people?

• 18:32Speaker 01

  The videos, almost all the videos, except for the ones that people would have actually seen, are just me. I write the lyrics, I sing the song, I generally find a...

• 18:43Speaker 01

  Like a backing track online because I'm like you can't really play an instrument. And then I do, in some cases, I'll do visuals using just like PowerPoint. Lots and lots of PowerPoint to make it sort of like an animation. The, the most produced one is the one that people might have seen, which is the overfitting video that I did with Charles Isbell. And that was produced by the Georgia Tech and Udacity people because we were doing a class together. It was kind of, I usually do parody songs kind of to cap off a class at the end of a class.

• 19:13Lex Fridman?

  So that one you're wearing, so it was just a thriller.

• 19:17Speaker 01

  Yeah.

• 19:17Lex Fridman?

  You're wearing the Michael Jackson, the red leather jacket. The interesting thing with podcasting that you're also into is that I really enjoy is that there's not a team of people.

• 19:32Speaker 01

  Hmm.

• 19:33Lex Fridman?

  It's kind of more, because you know, the, the, there's something that happens when there's more people involved than just one person, that just the way you start acting, I don't know, there's a censorship, you're not given, especially for like slow thinkers like me, you're not, and I think most of us are if we're trying to actually think, we're a little bit slow and, and careful. It, it kind of, large teams get in the way of that. And I don't know what to do with that. Like that's the, to me, like if, you know, it's very popular to criticize quote unquote mainstream media. I, but there is legitimacy to criticizing them, the same, I love listening to NPR, for

• 20:18Lex Fridman?

  example, but every, it's clear that there's a team behind it, there's a commerce, there's constant commercial breaks, there's this kind of like rush of like, uh, okay, I have to interrupt you now because we have to go to commercial, just this whole, it, it creates, it destroys the possibility of nuanced conversation.

• 20:38Lex Fridman?

  Yeah, exactly. Evian, which Charles Isbell, who I talked to yesterday, told me that Evian is naive backwards, which the fact that his mind thinks this way is just, is quite brilliant. Anyway, there's a freedom to this podcast.

• 20:54Speaker 01

  He's Dr. Awkward, which by the way is a palindrome. That's a palindrome that I happen to know from other parts of my life. And I just.

• 21:01Lex Fridman?

  You just throw it out.

• 21:02Speaker 01

  Well, you know.

• 21:03Lex Fridman?

  It's gonna stick.

• 21:03Speaker 01

  Use it against Charles. Dr. Awkward.

• 21:06Lex Fridman?

  So what, what was the most challenging parody song to make? Was it the Thriller one?

• 21:12Speaker 01

  Hmm, no, that was really fun. I wrote the lyrics really quickly and then I gave it over to the production team. They recruited a a cappella group to sing. That went really smoothly. It's great having a team because then you can just focus on the part that you really love, which in my case is writing the lyrics.

• 21:28Lex Fridman?

  Yeah.

• 21:29Speaker 01

  For me, the most challenging one, not challenging in a bad way, but challenging in a really fun way, was I did, one of the, one of the parody songs I did is, is about the halting problem in computer science, the, the fact that you can't create a program that can tell for any other arbitrary program, whether it's actually going to get stuck in an infinite loop, or whether it's going to eventually stop.

• 21:50Lex Fridman?

  Yes.

• 21:51Speaker 01

  And so I, I did it to an 80s song because that's, I hadn't started my, my new thing of learning current songs. And it was Billy Joel's The Piano Man.

• 22:02Lex Fridman?

  Nice.

• 22:03Speaker 01

  Which is a great song.

• 22:04Lex Fridman?

  Great song.

• 22:05Speaker 01

  Yeah, yeah. And.

• 22:07Lex Fridman?

  Sing me a song. You have the piano man?

• 22:09Speaker 01

  Mm hmm.

• 22:10Lex Fridman?

  Yeah.

• 22:11Speaker 01

  Yes. So the lyrics are great because first of all, it rhymes. Not all songs rhyme. I did, I've done Rolling Stones songs, which turn out to have no rhyme scheme whatsoever. They're just sort of yelling and having a good time, which it makes it not fun from a parody perspective, because like you can say anything. But this, you know, the lines rhymed and there was a lot of internal rhymes as well. And so figuring out how to sing with internal rhymes, a proof of the halting problem, was really challenging. And it was, I really enjoyed that process.

• 22:40Lex Fridman?

  What about, last question on this topic, what about the dancing in the Thriller video? How many takes that take?

• 22:46Speaker 01

  So I wasn't planning to dance. They had me in the studio and they gave me the jacket. And it's like, well, you can't, if you have the jacket and the glove, like, there's not much you can do. So I, I think I just danced around and then they said, why don't you dance a little bit? We, there was a scene with me and Charles dancing together.

• 23:03Lex Fridman?

  In that video?

• 23:03Speaker 01

  They did not use it in the video, but we recorded it.

• 23:05Lex Fridman?

  I would have remembered.

• 23:06Speaker 01

  Yeah, yeah, no, it was, it was pretty funny.

• 23:10Speaker 01

  And Charles, who has this beautiful, wonderful voice, doesn't really sing. He's not really a singer. And so that was why I designed the song with him doing a spoken section and me doing the singing.

• 23:20Lex Fridman?

  It's very like Barry White.

• 23:21Speaker 01

  Yeah, it was smooth baritone. Yeah, yeah, it's great.

• 23:24Lex Fridman?

  That was awesome. So one of the other things Charles said is that, you know, everyone knows you as like a super nice guy, super passionate about teaching and so on. What he said, don't know if it's true, that despite the fact that you're, you are-

• 23:42Speaker 01

  Killed the man in cold blood. Like, okay, all right, I will admit this finally for the first time. That was, that was me.

• 23:48Lex Fridman?

  It's the Johnny Cash song, Kill the Man in Reno, just to watch him die. That you actually do have some strong opinions on some topics. So if this in fact is true, what strong opinions would you say you have? Is there ideas you think maybe in artificial intelligence, machine learning, maybe in life, that you believe is true that others might, you know, some number of people might disagree with you on?

• 24:17Speaker 01

  So I try very hard to see things from multiple perspectives. There's, there's this great Calvin and Harp's, Calvin and Hobbes cartoon where Cal, do you know? Okay, so Calvin's dad is always kind of a bit of a foil and he, he talked Calvin into, Calvin had done something wrong. The dad talks him into like seeing it from another perspective and Calvin, like this breaks Calvin because he's like, oh my gosh, now I can see the opposite sides of things. And so the, it's, it becomes like a Cubist cartoon where there is no front and back. Everything's just exposed and it really freaks him out. And finally he settles back down. It's like, oh, good. No, I can make that go away. But like, I'm that, I'm that. I live in that world where I'm trying to see everything from every perspective all the time.

• 24:59Speaker 01

  So there are some things that I've formed opinions about that I would be harder, I think, to disavow me of. One is, The superintelligence argument and the existential threat of AI is one where I feel pretty confident in my feeling about that one. Like, I'm willing to hear other arguments, but like, I am not particularly moved by the idea that if we're not careful, we will accidentally create a superintelligence that will destroy human life.

• 25:27Lex Fridman?

  Let's talk about that. Let's get you in trouble and record you on video. It's like Bill Gates, I think he said like, some quote about the internet, that that's just gonna be a small thing. It's not gonna really go anywhere. And then I think Steve Ballmer said, I don't know why I'm sticking on Microsoft, that's something that like, smartphones are useless. There's no reason why Microsoft should get into smartphones, that kind of... So, let's get, let's talk about AGI. As AGI is destroying the world, we'll look back at this video and see. No, I think it's really interesting to actually talk about, because nobody really knows the future, so you have to use your best intuition. It's very difficult to predict it. But you have spoken about AGI and the existential risks around it, and

• 26:12Lex Fridman?

  sort of based on your intuition that we're quite far away from that being a serious concern relative to the other concerns we have. Can you maybe unpack that a little bit?

• 26:23Speaker 01

  Yeah, sure, sure, sure. So, so, as I understand it, that, for example, I read Bostrom's book and a bunch of other reading material about this sort of general way of thinking about the world. And I think the story goes something like this, that we will at some point create computers that are smart enough that they can help design the next version of themselves, which itself will be smarter than the previous version of themselves, and eventually bootstrapped up to being smarter than us, at which point we are essentially at the mercy of this sort of more powerful intellect, which in principle,

• 27:06Speaker 01

  we don't have any control over what its goals are. And so if its goals are at all out of sync with our goals, like, for example, the continued existence of humanity, we won't be able to stop it. It'll be way more powerful than us, and we will be toast. So there's some, I don't know, very smart people who have signed on to that story. And it's a, it's a compelling story. I once, huh, now I can really get myself in trouble. I once wrote an op ed about this, specifically responding to some quotes from Elon Musk, who has been, you know, on this very podcast more than once. And

• 27:45Lex Fridman?

  AI summoning the demon.

• 27:47Speaker 01

  That's a thing he said, but then he came to Providence, Rhode Island, which is where I live, and said to the governors of all the states, you know, you're worried about entirely the wrong thing. You need to be worried about AI. You need to be very, very worried about AI. So, and journalists kind of reacted to that. They wanted to get people's, people's take. And I was like, okay. My, my, my belief is that one of the things that makes Elon Musk so successful and so remarkable as an individual is that he believes in the power of ideas. He believes that you can have, you can, if, you know, if you have a really good idea for getting into space, you can get into space. If you have a really good idea for a company or for how to change the way that people drive,

• 28:30Speaker 01

  you just have to do it and it can happen. It's really natural to apply that same idea to AI. You see these systems that are doing some pretty remarkable computational tricks, demonstrations, and then to take that idea and just push it all the way to the limit and think, okay, where does this go? Where is this going to take us next? And if you're a deep believer in the power of ideas, then it's really natural to believe that those ideas could be taken to the extreme and, and kill us. So I think, you know, his strength is also his undoing because that doesn't mean it's true. Like, it doesn't mean that that has to happen, but it's natural for him to think that.

• 29:08Lex Fridman?

  So another way to phrase the way he thinks, and I find it very difficult to argue with that line of thinking. So Sam Harris is another person from neuroscience perspective that thinks like that, is saying, well, is there something fundamental in the physics of the universe that prevents this from eventually happening? And Nick Bostrom thinks in the same way, that kind of zooming out, yeah, okay, we humans now are existing in this like time scale of minutes and days. And so our intuition is in this time scale of minutes, hours and days. But if you look at the span of human history,

• 29:51Lex Fridman?

  Is there any reason we you can't see this in in a hundred years? And like, is there is there something fundamental about the laws of physics that prevent this? And if it doesn't, then it eventually will happen. Or we'll we will destroy ourselves in some other way. It's very difficult, I find, to actually argue against that.

• 30:11Speaker 01

  Yeah. Me too.

• 30:15Lex Fridman?

  And not sound like not sound like you're just like rolling your eyes. Ah, like, I have like.

• 30:23Speaker 01

  It's science fiction. We don't have to think about it.

• 30:24Lex Fridman?

  But even even worse than that, which is like, I don't have kids, but like, I gotta pick up my kids now. Like, this, okay.

• 30:30Speaker 01

  I see. There's more pressing short term.

• 30:32Lex Fridman?

  Yeah, there's more pressing, short term things that like stop it with this existential crisis. We have much, much shorter things like now, especially this year, there's COVID. So like, any kind of discussion like that is like, there's, you know, there's pressing things today. It's it's and then so the Sam Harris argument, well, like, any day, the exponential singularity can can occur. It's very difficult to argue against. I mean, I don't know.

• 30:58Speaker 01

  Well, part of his story is also he's he's not going to put a date on it. It could be in a thousand years. It could be in a hundred years. It could be in two years. It's just that as long as we keep making this kind of progress, it's ultimately has to become a concern. I I kind of am on board with that. But the thing that the the piece that I feel like is missing from that that way of extrapolating from the moment that we're in is that I believe that in the process of actually developing technology that can really get around in the world and really process and and and do things in the world in a sophisticated way, we're going to learn a lot about what that means, which that we don't know now because we don't know how to do this right now. If you believe that you can just turn on a deep learning network and eventually give it enough compute and it'll eventually get there, well, sure, that seems really scary because we won't.

• 31:44Speaker 01

  We won't be in the loop at all. We won't. We won't be helping to design or or target these kinds of systems. But I don't I don't see that.

• 31:54Speaker 01

  That feels like it is against the laws of physics because these systems need help, right? They need, they need to surpass the, the, the difficulty, the wall of complexity that happens in arranging something in the form that, that, that will happen in.

• 32:07Lex Fridman?

  Yeah.

• 32:08Speaker 01

  Like, I believe in evolution. Like, I believe that the, that, that, there's an argument, right? So there's another argument, just to look at it from a different perspective, that people say, well, I don't believe in evolution. How could evolution, it's, it's sort of like a random set of parts assemble themselves into a 747, and that could just never happen.

• 32:26Lex Fridman?

  Yeah.

• 32:27Speaker 01

  So it's like, okay, that's maybe hard to argue against, but clearly, 747s do get assembled. They get assembled by us. Basically, the idea being that there's a process by which we will get to the point, the point of making technology that has that kind of awareness. And in that process, we're going to learn a lot about that process, and we'll have more ability to control it, or to shape it, or to build it in our own image. It's not something that is going to spring into existence like that 747, and we're just gonna have to contend with it completely unprepared.

• 32:59Lex Fridman?

  That's very possible that in the context of the long arc of human history, it will in fact spring into existence, but that, that springing might take, like, if you look at nuclear weapons, like, even 20 years is a springing in, in the context of human history, and it's very possible, just like with nuclear weapons, that we could have, I don't know what percentage you want to put at it, but the, the, the possibility of-

• 33:25Speaker 01

  We could have knocked ourselves out.

• 33:26Lex Fridman?

  Yeah, the possibility of human beings destroying themselves in the 20th century with, with nuclear weapons, I don't know, you can, if you really think through it, you could really put it close to like, I don't know, 30, 40%, given like, the certain moments of crisis that happen. So like, I think-

• 33:45Lex Fridman?

  One like fear in the shadows that's not being acknowledged is it's not so much the AI will run away is is that as it's running away, we won't have enough time to think through how to stop it.

• 34:01Speaker 01

  Right, the fast takeoff or boom.

• 34:04Lex Fridman?

  Yeah, I mean, my much bigger concern, I wonder what you think about it, which is we won't know it's happening. So I kind of think that there's an AGI situation already happening with social media, that our minds, our collective intelligence of human civilization is already being controlled by an algorithm. And like, we're we're already super like, the the level of a collective intelligence, thanks to Wikipedia, people should donate to Wikipedia to feed the AGI.

• 34:37Speaker 01

  Man, if we had a super intelligence that that was in line with Wikipedia's values, that it's a lot better than a lot of other things I could imagine. I I trust Wikipedia more than I trust Facebook or YouTube as far as trying to do the right thing from a rational perspective.

• 34:53Lex Fridman?

  Yeah.

• 34:53Speaker 01

  Now, that's not where you were going, I understand that, but it it it does strike me that there's sort of smarter and less smart ways of of exposing ourselves to each other on the internet.

• 35:03Lex Fridman?

  Yeah, the interesting thing is that Wikipedia and social media have very different forces. You're right. I mean, Wikipedia, if AGI was Wikipedia, it'd be just like this cranky, overly competent editor of articles. You know, there's there's something to that, but the social media aspect is is is not. So the vision of AGI is as a separate system that's super intelligent, that's super intelligent. That's one key little thing. I mean, there's the paperclip argument that's super dumb, but super powerful systems. But with social media, you have a relatively like algorithms, we may talk about today, very simple algorithms that when, so something Charles talks a lot about, which is interactive

• 35:48Lex Fridman?

  AI, when they start like having at scale, like tiny little interactions with human beings, they can start controlling these human beings. So a single algorithm can control the minds of human beings slowly to where we might not realize it can start wars, it can start, it can change the way we. Think about things, it feels like in the long arc of history, if I were to sort of zoom out from all the outrage and all the tension on social media, that it's progressing us towards better and better things. It feels like chaos and toxic and all that kind of stuff, but-

• 36:25Speaker 01

  It's chaos and toxic, yeah.

• 36:27Lex Fridman?

  But it feels like actually the chaos and toxic is similar to the kind of debates we had from the founding of this country. You know, there was a civil war that happened over that, over that period. And ultimately, it was all about this tension of like, something doesn't feel right about our implementation of the core values we hold as human beings, and they're constantly struggling with this. And that results in people calling each other, like just, just, just being shitty to each other on Twitter. But I, ultimately, the algorithm is managing all that. And it feels like there's a possible future in which that algorithm controls us to, into the direction of self-destruction, whatever that looks like.

• 37:11Speaker 01

  Yeah. So, so, all right, I do believe in the power of social media to screw us up royally. I do believe in the power of social media to benefit us, too. I do think that we're in a, yeah, it's sort of, almost got dropped on top of us, and now we're trying to, as a culture, figure out how to cope with it. There's a sense in which, I don't know, there's, there's some arguments that say that, for example, I guess, college-age students now, late college-age students now, people who were in middle school when, when social media started to really take off, may be, may be really damaged. Like, this may have really hurt their development in a way that we don't have all the implications of quite yet. That's the generation who, if, and I hate to make it somebody else's

• 37:56Speaker 01

  responsibility, but like, they're the ones who can fix it. They're the ones who can, who can figure out, how do we keep the good of this kind of technology without letting it eat us alive? And if they're successful, we move on to the next phase, the next level of the game. If they're not successful, then, yeah, then we're, we're gonna wreck each other, or we're gonna destroy society.

• 38:20Lex Fridman?

  So you're gonna, in your old age, sit on a Porsche and watch the world burn? Because of the TikTok generation that-

• 38:26Speaker 01

  I, I believe, well, so like, this is my kid's age, right? And certainly my daughter's age. And she's very tapped in to social stuff, but she's also- She's trying to find that balance, right? Of participating in it and in getting the positives of it, but without letting it eat her alive. And I think sometimes she ventures. I hope she doesn't watch this.

• 38:47Speaker 01

  Sometimes I think she ventures a little too far and is consumed by it. And other times she gets a little distant.

• 38:55Speaker 01

  And if, you know, if there's enough people like her out there, they're gonna, they're gonna navigate this choppy waters.

• 39:00Lex Fridman?

  That's, that's an interesting skill, actually, to develop. I talked to my dad about it. You know, I've now somehow this podcast in particular, but other reasons has received a little bit of attention. And with that, apparently in this world, even though I don't shut up about love and I'm just all about kindness, I have now a little mini army of trolls. It's kind of hilarious, actually, but it also doesn't feel good. But it's a skill to learn to not look at that. Like to moderate, actually, how much you look at that. The discussion I have with my dad is similar to, it doesn't have to be about trolls. It could be about checking email, which is like,

• 39:45Lex Fridman?

  if you're anticipating, you know, there's, my dad runs a large institute at Drexel University, and there could be stressful, like emails you're waiting. Like there's drama of some kinds. And so like, there's a temptation to check the email. If you send an email and you kind of, and that pulls you in into, it doesn't feel good. And it's a skill that he actually complains that he hasn't, I mean, he grew up without it. So he hasn't learned the skill of how to shut off the internet and walk away. And I think young people, while they're also being quote unquote, damaged by like, you know, being bullied online, all of those stories, which are very like horrific. You basically can't escape your bullies these days when you're growing up.

• 40:29Lex Fridman?

  But at the same time, they're also learning that skill of how to be able to shut off the, like disconnect with it, be able to laugh at it, not take it too seriously. It's fascinating. Like, we're all trying to figure this out. Just like you said, it's been dropped on us and we're trying to figure it out.

• 40:45Speaker 01

  Yeah, I think that's really interesting. And I guess I've become a believer in the human design, which I feel like I don't completely understand. Like, how do you make something as robust as us? Like, we're so flawed in so many ways. And yet, and yet, you know, we dominate the planet. And we Do seem to manage to get ourselves out of scrapes eventually. Not necessarily the most elegant possible way, but somehow we get to the next step. And I don't know how I'd make a machine do that. I generally speaking, like if I train one of my reinforcement learning agents to play a video game and it works really hard on that first stage over and over and over again, and it makes it through it, succeeds on that first level.

• 41:29Speaker 01

  And then the new level comes and it's just like, okay, I'm back to the drawing board. And somehow humanity, we keep leveling up and then somehow managing to put together the skills necessary to achieve success, some semblance of success in that next level too. And, you know, I hope we can keep doing that.

• 41:49Lex Fridman?

  You mentioned reinforcement learning, so you have a couple of years in the field? No, quite, you know, quite a few, quite a long career in artificial intelligence broadly, but reinforcement learning specifically. Can you maybe give a hint about your sense of the history of the field? And in some ways it's changed with the advent of deep learning, but has a long roots. Like, how is it weaved in and out of your own life? How have you seen the community change or maybe the ideas that it's playing with change?

• 42:23Speaker 01

  I've had the privilege, the pleasure of being, of having almost a front row seat to a lot of this stuff, and it's been really, really fun and interesting. So when I was in college in the eighties, early eighties, the neural net thing was starting to happen. And I was taking a lot of psychology classes and a lot of computer science classes as a college student. And I thought, you know, something that can play tic-tac-toe and just like learn to get better at it, that ought to be a really easy thing. So I spent almost, almost all of my, what would have been vacations during college, like hacking on my home computer, trying to teach it how to play tic-tac-toe.

• 43:01Lex Fridman?

  Programming language.

• 43:02Speaker 01

  Basic. Oh, yeah. That's, that's, I was, that's my first language. That's my native language.

• 43:07Lex Fridman?

  Is that when you first fell in love with computer science? Just like programming, BASIC on that? What was the computer? Do you remember?

• 43:14Speaker 01

  I had a TRS 80 Model 1 before they were called Model 1s because there was nothing else. I got my computer in 1979. So I would have been bar mitzvahed, but instead of having a big party that my parents threw on my behalf, they just got me a computer because that's what I really, really, really wanted. I saw them in the mall in Ria's Shack and I thought, "What? How are they doing that?" I would try to stump them. I would give them math problems like 1 plus and then in parentheses 2 plus 1. And I would always get it right. I'm like, "How do you know so much?" Like I've had to go to algebra class for the last few years to learn this stuff and you just seem to know.

• 43:57Speaker 01

  So I was smitten and got a computer. And I think ages 13 to 15, I have no memory of those years. I think I just was in my room with the computer.

• 44:09Lex Fridman?

  Listening to Billy Joel.

• 44:10Speaker 01

  Communing, possibly listening to the radio, listening to Billy Joel. That was the one album I had on vinyl at that time. And then I got it on cassette tape and that was really helpful because then I could play it. I didn't have to go down to my parents' wifi or hi-fi, sorry. And at age 15, I remember kind of walking out and like, "Okay, I'm ready to talk to people again. Like I've learned what I need to learn here." And so, yeah, so that was my home computer. And so I went to college and I was like, "Oh, I'm totally going to study computer science." And I opted, the college I chose specifically had a computer science major. The one that I really wanted, the college I really wanted to go to didn't, so bye-bye to them.

• 44:49Lex Fridman?

  Which college did you go to?

• 44:50Speaker 01

  So I went to Yale. Princeton would have been way more convenient and it was just beautiful campus and it was close enough to home. And I was really excited about Princeton and I visited. I said, "So, computer science major?" They're like, "Well, we have computer engineering." I'm like, "Oh, I don't like that word engineering."

• 45:06Speaker 01

  I like computer science. I really, I want to do, like, you're saying hardware and software? They're like, yeah, I'm like, I just want to do software. I couldn't care less about hardware.

• 45:13Lex Fridman?

  And you grew up in Philadelphia?

• 45:14Speaker 01

  I grew up outside Philly. Yeah, yeah.

• 45:16Lex Fridman?

  Okay.

• 45:16Speaker 01

  So the, you know, local schools were like Penn and Drexel and Temple. Like, everyone in my family went to Temple, at least at one point in their lives, except for me. So, yeah, Philly, Philly family.

• 45:29Lex Fridman?

  Yale had a computer science department, and that's when you, it's kind of interesting, you said 80s and neural networks. That's when neural networks was a hot new thing, or a hot thing, period. So what, is that in college, when you first learned about neural networks?

• 45:42Speaker 01

  Yeah, yeah.

• 45:42Lex Fridman?

  When she learned, like, how did you?

• 45:43Speaker 01

  And it was in a psychology class, not in a CS class.

• 45:46Lex Fridman?

  Oh, wow.

• 45:46Speaker 01

  Yeah.

• 45:47Lex Fridman?

  Was it psychology or cognitive science? Or like, do you remember, like, what context?

• 45:51Speaker 01

  It was, yeah, yeah, yeah. So, so I was a, I've always been a bit of a cognitive psychology groupie. So, like, I'm, I studied computer science, but I like, I like to hang around where the cognitive scientists are, because, I don't know, brains, man, they're like, they're wacky, cool.

• 46:07Lex Fridman?

  And they have a bigger picture view of things. They're a little less engineering, I would say. They're more, they're more interested in the nature of cognition and intelligence and perception and how like the vision system work. They're asking always bigger questions. Now, with the deep learning community, there, I think, more, there's a lot of intersections. But I do find that the neuroscience folks, actually, and cognitive psychology, cognitive science folks are starting to learn how to program, how to use your own, artificial neural networks. And they are actually approaching problems in like, totally new, interesting ways.

• 46:43Speaker 01

  Yeah.

• 46:43Lex Fridman?

  It's fun to watch that grad students from those departments, like, approach a problem of machine learning.

• 46:49Speaker 01

  Right. They come in with a different perspective.

• 46:51Lex Fridman?

  Yeah. They don't care about, like, your image net data set or whatever. They, they want, like, to understand the, the, like, the basic mechanisms at the, at the neuronal level, at the functional level of intelligence. It's kind of, it's kind of cool to see them work. But yeah, okay. So you always, you were always a groupie of cognitive psychology.

• 47:12Speaker 01

  Yeah, yeah. And so, so it was in a class by Richard Garrick. He was kind of my, my favorite psych professor in college. And I took, like, three different classes with him. And, yeah, so that, we, they were talking specifically, the class, I think, was kind of a...

• 47:29Speaker 01

  There was a big paper that was written by Steven Pinker and Prince. I'm blanking on Prince's first name, but Pinker and Prince. They wrote kind of a... They were at that time, kind of like...

• 47:44Speaker 01

  I'm blanking on the names of the current people. The cognitive scientists who are complaining a lot about deep networks.

• 47:50Lex Fridman?

  Oh, Gary.

• 47:52Speaker 01

  Gary Marcus.

• 47:53Lex Fridman?

  Sorry, Marcus?

• 47:54Speaker 01

  Yeah.

• 47:55Lex Fridman?

  And who else? I mean, there's a few, but Gary. Gary's the most feisty.

• 47:59Speaker 01

  Sure, Gary's very feisty. And with his co-author, they're kind of doing these kind of takedowns where they say, "Okay, well, yeah, it does all these amazing things, but here's a shortcoming. Here's a shortcoming. Here's a shortcoming." And so the Pinker-Prince paper is kind of like that generation's version of Marcus and Davis, right? Where they're trained as cognitive scientists, but they're looking skeptically at the results in the artificial intelligence neural net kind of world and saying, "Yeah, it can do this and this and this, but it can't do that, and it can't do that, and it can't do that." Maybe in principle, or maybe just in practice at this point. But the fact of the matter is, you've narrowed your focus too far to be impressed. You're impressed with the things within that circle, but you need to broaden that circle a little bit.

• 48:44Speaker 01

  You need to look at a wider set of problems. And so I was in this seminar in college that was basically a close reading of the Pinker-Prince paper, which was like, really thick. There was a lot going on in there. And it talked about the reinforcement learning idea a little bit. I'm like, "Oh, that sounds really cool." Because behavior is what is really interesting to me about psychology, anyway. So making programs that... I mean, programs are things that behave. People are things that behave. Like, I want to make learning that learns to behave.

• 49:17Lex Fridman?

  In which way was reinforcement learning presented? Is this talking about human and animal behavior? Or are we talking about actual mathematical constructs?

• 49:25Speaker 01

  Ah, that's a great... So that's a good question. Right. So this is... I think it wasn't actually talked about as behavior in the paper that I was reading. I think that it just talked about learning. And to me, learning is about learning to behave. But really, neural nets at that point were about learning, like supervised learning. So, learning to produce outputs from inputs. So, I kind of tried to invent reinforcement learning. I, when I graduated, I joined a research group at Bell Core, which had spun out of Bell Labs recently at that time because of the divestiture of the, of long distance and local phone service in the 1980s, 1984. And I was in a group with Dave Ackley, who was the first author of the Boltzmann machine paper.

• 50:07Speaker 01

  So, the very first neural net paper that could handle XOR, right? So, XOR sort of killed neural nets, the very first, the zero with order.

• 50:15Lex Fridman?

  The first winter.

• 50:16Speaker 01

  Yeah. The, the perceptrons paper. And Hinton, along with his student, Dave Ackley, and I think there was other authors as well, showed that, no, no, no, with Boltzmann machines, we can actually learn nonlinear concepts. And so, everything's back on the table again. And that kind of started that second wave of neural networks. So, Dave Ackley was, he became my mentor at, at Bell Core. And we talked a lot about learning and life and computation and how all these things fit together. Now, Dave and I have a podcast together. So, so I get to kind of enjoy that sort of, his, his perspective, once again, even, even all these years later. And so, I said, so I said, I was really interested in learning, but in the concept of behavior.

• 51:00Speaker 01

  And he's like, oh, well, that's reinforcement learning here. And he gave me Rich Sutton's 1984 TD paper. So, I read that paper. I honestly didn't get all of it, but I got the idea. I got that they were using, that he was using ideas that I was familiar with in the context of neural nets and, and like sort of backprop. But with this idea of making predictions over time. I'm like, this is so interesting, but I don't really get all the details, I said to Dave. And Dave said, oh, well, why don't we have him come and give a talk? And I was like, wait, what? You can do that? Like, these are real people? I thought they were just words. I thought it was just like ideas that somehow magically seeped into paper. He's like, no, I, I, I know Rich. Like, we'll just have him come down and, and he'll give a talk.

• 51:47Speaker 01

  And so I was, you know, my mind was blown. And so Rich came and he gave a talk at Belkor. And he talked about what he was super excited, which was they had just figured out at the time, Q learning. So Watkins had visited the Rich Sutton's lab at UMass or Andy Barto's lab that Rich was a part of. And he was really excited about this because it resolved a whole bunch of problems that he didn't know how to resolve in the in the earlier paper. And so

• 52:18Lex Fridman?

  For people who don't know TD, temporal difference, these are all just algorithms for reinforcement learning.

• 52:23Speaker 01

  Right. And TD, temporal difference in particular is about making predictions over time. And you can try to use it for making decisions, right? Because if you can predict how good a future action or an action outcomes will be in the future, you can choose one that has better. But the theory didn't really support changing your behavior. Like, the predictions had to be of a consistent process if you really wanted it to work. And one of the things that was really cool about Q learning, another algorithm for reinforcement learning, is it was off policy, which meant that you could actually be learning about the environment and what the value of different actions would be while actually figuring out how to behave optimally. So that was a revelation.

• 53:02Lex Fridman?

  Yeah. And the proof of that is kind of interesting. I mean, that's really surprising to me when I first read that and then in Rich Sutton's book on the matter. It's kind of beautiful that a single equation can capture.

• 53:13Speaker 01

  One equation, one line of code, and like, you can learn anything.

• 53:16Lex Fridman?

  Yeah, like.

• 53:17Speaker 01

  If you have enough time.

• 53:18Lex Fridman?

  So equation and code, you're right. Like, you can, the code that you can arguably, at least, if you like squint your eyes, can say, this is all of intelligence, is that you can implement that in a single, I think I started with Lisp, which is, shout out to Lisp, with like a single line of code, key piece of code, maybe a couple, that you could do that. It's kind of magical. It's feels too good to be true.

• 53:49Speaker 01

  Well, and it sort of is.

• 53:50Lex Fridman?

  Yeah.

• 53:52Speaker 01

  It seems they require an awful lot of extra stuff supporting it. But nonetheless, the idea is, the idea is really good. And as far as we know, it is, it is a very reasonable way of trying to create adaptive behavior, behavior that gets better at something over time.

• 54:08Lex Fridman?

  Did you find the idea of optimal at all compelling? That you could prove that it's optimal? So like, one part of computer science that it makes people feel warm and fuzzy inside is when you can prove something like that a sorting algorithm, worst case runs and n log n and it makes everybody feel so good. Even though in reality, it doesn't really matter what the worst case is. What matters is like, does this thing actually work in practice on this particular actual set of data that I that I enjoy? Did you?

• 54:38Speaker 01

  So here's that. Here's a place where I have maybe a strong opinion.

• 54:41Lex Fridman?

  Uh oh.

• 54:41Speaker 01

  Which is like, you're right, of course, but no, no. Like, so, so what makes worst case so great, right? If you have a worst case analysis so great is that you get modularity. You can take that thing and plug it into another thing and still have some understanding of what's going to happen when you click them together. Right? If it just works well in practice, in other words, with respect to some distribution that you care about, when you go plug it into another thing, that distribution can shift. It can change and your thing may not work well anymore. And you want it to, and you wish it does, and you hope that it will, but it might not. And then, ah.

• 55:18Lex Fridman?

  So you're, so you're saying you don't like machine learning?

• 55:25Speaker 01

  But we have some positive theoretical results for these things. You know, you can come back at me with, yeah, but they're really weak. And yeah, they're really weak. And, and you could even say that, you know, sorting algorithms, like if you do the optimal sorting algorithm, it's not really the one that you want. And that might be true as well, but.

• 55:43Lex Fridman?

  But it is, the modularity is a really powerful statement.

• 55:46Speaker 01

  I really like that.

• 55:46Lex Fridman?

  As an engineer, you can then assemble different things. You can count on them to be, I mean, it's interesting. It's, it's a balance, like with everything else in life. You don't want to get too obsessed. I mean, this is what computer scientists do, which they tend to like get obsessed. They over optimize things that where they start by optimizing them, they over optimize. So it's, it's easy to like, get really granular about this thing. But like the step from an n squared to an n log n sorting algorithm is a big leap for most real world systems, no matter what the actual. Behavior of the system is that's a big leap.

• 56:26Lex Fridman?

  And the same can probably be said for other kind of first leaps that you would take in a particular problem. Like it's the picking the low hanging fruit or whatever the equivalent of doing the not the dumbest thing, but the next to the dumbest thing.

• 56:44Speaker 01

  I see, picking the most delicious reachable fruit.

• 56:46Lex Fridman?

  Yeah, most delicious reachable fruit.

• 56:48Speaker 01

  I don't know why that's not a saying.

• 56:49Lex Fridman?

  And yeah, okay. So, so you then this is the eighties and this kind of ideas such as the percolator of uh.

• 56:59Speaker 01

  Yeah, well, at that point, I got to meet, I got to meet Rich Sutton. So everything was sort of downhill from there. And that was, that was really the pinnacle of everything. But then I, you know, then I felt like I was kind of on the inside. So then as interesting results were happening, I could like check in with, with Rich or with Jerry Tassaro, who had a huge impact on kind of early thinking in, in temporal difference learning and reinforcement learning and showed that you could do, you could solve problems that we didn't know how to solve any other way. And so that was really cool. So it was good things were happening. I would hear about it from either the people who were doing it or the people who were talking to the people who were doing it. And so I was able to track things pretty well through, through the nineties.

• 57:40Lex Fridman?

  So what, wasn't most of the excitement on reinforcement learning in the nineties era with, what is it TD gamma? Like, what's the role of these kind of little like fun game playing things and breakthroughs about, you know, exciting the community? Was that like, what were your, because you've also built a crossword or part of building a crossword puzzle solver.

• 58:08Speaker 01

  Program, yeah.

• 58:09Lex Fridman?

  Solving program called proverb. So, so you were interested in this as a problem, like informing, using games to understand how to build. Intelligent systems. So like, what did you think about TD Gamm? Like, what did you think about that whole thing in the 90s?

• 58:28Speaker 01

  Yeah, I mean, I found the TD Gammens result really just remarkable. So I had known about some of Jerry's stuff before he did TD Gamm. He did a system just more vanilla, well, not entirely vanilla, but a more classical backprop-y kind of network for playing backgammon, where he was training it on expert moves. So it was kind of supervised. But the way that it worked was not to mimic the actions, but to learn internally an evaluation function. So to learn, well, if the expert chose this over this, that must mean that the expert values this more than this. And so let me adjust my weights to make it so that the network evaluates this as being better than this. So it could learn from from human preferences. It could learn its own preferences.

• 59:14Speaker 01

  And then when he took the step from that to actually doing it as a full-on reinforcement learning problem, where you didn't need a trainer, you could just let it play. That was remarkable. Right? And so I think as as humans often do, as we've done in the recent past as well, people extrapolate. It's like, oh, well, if you can do that, which is obviously very hard, then obviously you could do all these other problems that we that we want to solve, that we know are also really hard. And it turned out very few of them ended up being practical, partly because I think neural nets, certainly at the time, were struggling to be consistent and reliable. And so training them in a reinforcement learning setting was a bit of a mess. I had,

• 59:59Speaker 01

  I don't know, generation after generation of like, master students who wanted to do value function approximation, basically reinforcement learning with neural nets. And over and over and over again, we were failing. We couldn't get the good results that Jerry Tesauro got. I now believe that Jerry is a neural net whisperer. He has a particular ability to get neural networks to do things that other people would find impossible. And it's not the technology, it's the technology and Jerry together.

• 1:00:33Lex Fridman?

  Yeah, which I think speaks to the role of the human expert in the process of machine learning.

• 1:00:40Speaker 01

  Right. It's so easy. We're so drawn to the idea that that it's the technology that is where the power is coming from. That I think we lose sight of the fact that sometimes you need a really good, just like, I mean, no one would think, hey, here's this great piece of software. Here's like, I don't know, GNU Emacs or whatever. And doesn't that prove that computers are super powerful and basically going to take over the world? It's like, no, Stoman is a hell of a hacker, right? So he was able to make the code do these amazing things. He couldn't have done it without the computer, but the computer couldn't have done it without him. And so I think people discount the role of people like Jerry, who have just a particular, a particular set of skills.

• 1:01:21Lex Fridman?

  On that topic, by the way, as a small side note, I tweeted Emacs is greater than Vim yesterday and deleted, deleted the tweet 10 minutes later when I realized

• 1:01:32Speaker 01

  You were on fire.

• 1:01:33Lex Fridman?

  It started a war.

• 1:01:34Speaker 01

  Yeah.

• 1:01:35Lex Fridman?

  I was like, oh, I was just kidding. I was just being, and I'm gonna walk, walk, walk back on it. So people still feel passionately about that particular piece of

• 1:01:45Speaker 01

  Yeah, I don't get that, because Emacs is clearly so much better. I don't understand. But, you know, why do I say that? Because I, because like, I spent a block of time in the 80s, making my fingers know the Emacs keys. And now, like, that's part of the thought process for me. Like, I need to express. And if you take that, if you take my Emacs key bindings away, I become

• 1:02:09Lex Fridman?

  Yep, yep.

• 1:02:10Speaker 01

  I can't express myself.

• 1:02:11Lex Fridman?

  I'm the same way with the, I don't know if you know what it is, but it's a Kinesis keyboard, which is this butt-shaped keyboard.

• 1:02:17Speaker 01

  Yes, I've seen them.

• 1:02:18Lex Fridman?

  Yeah.

• 1:02:19Speaker 01

  And they're very, I don't know, sexy, elegant?

• 1:02:23Lex Fridman?

  Yeah.

• 1:02:23Speaker 01

  Beautiful.

• 1:02:24Lex Fridman?

  Yeah, they're gorgeous, way too expensive. But the, the problem with them, similar with Emacs, is when, once you learn to use it.

• 1:02:36Speaker 01

  It's harder to use other things.

• 1:02:37Lex Fridman?

  It's hard to use other things. There's this absurd thing where I have like, small, elegant, lightweight, beautiful little laptops. And I'm sitting there in a coffee shop with a giant Kinesis keyboard and a sexy little laptop. It's absurd. But it, you know, like, I used to feel bad about it, but at the same time, you just kind of have to, sometimes it's back to the Billy Joel thing. That throw that Billy Joel record and throw Taylor Swift and Justin Bieber to the wind. So.

• 1:03:05Speaker 01

  See, but I like them now because I because again, I have no musical taste like like now that I've heard Justin Bieber enough. I like, I really like his songs and Taylor Swift. Not only do I like her songs, but my daughter's convinced that she's a genius. And so now I basically have I'm signed on to that. So.

• 1:03:20Lex Fridman?

  So, yeah, that that speaks to the back to the robustness of the human brain. That speaks to the neuroplasticity that you can just you can just like a mouse. Teach yourself to or probably love dog, teach yourself to enjoy Taylor Swift. I'll try it out. I don't know.

• 1:03:36Lex Fridman?

  I try, you know what? It has to do with just like acclimation, right? Just like you said, a couple of weeks.

• 1:03:41Speaker 01

  Yeah.

• 1:03:41Lex Fridman?

  That's an interesting experiment. I'll actually try that like.

• 1:03:44Speaker 01

  That wasn't the intent of the experiment, just like social media. It wasn't intended as an experiment to see what we can take as a society, but it turned out that way.

• 1:03:51Lex Fridman?

  I don't think I'll be the same person on the other side of the week, listening to Taylor Swift, but let's try.

• 1:03:55Speaker 01

  No, it's more compartmental. Don't be so worried like it's like, I get that you can be worried, but don't be so worried because we compartmentalize really well. And so it won't bleed into other parts of your life. You won't start, I don't know, wearing red lipstick or whatever like it's it's fine.

• 1:04:10Lex Fridman?

  It's changed fashion and everything.

• 1:04:11Speaker 01

  It's fine, but you know what? The the thing you have to watch out for is you'll walk into a coffee shop. Once we can do that again.

• 1:04:17Lex Fridman?

  And recognize the song.

• 1:04:18Speaker 01

  And you'll be no, you won't know that you're singing along until everybody in the coffee shop is looking at you. And then you're like, that wasn't me.

• 1:04:28Lex Fridman?

  Yeah, that's the, you know, people are afraid of AGI. I'm afraid of the.

• 1:04:33Speaker 01

  The Taylor Swift takeover.

• 1:04:34Lex Fridman?

  Yeah. And I mean, people should know that TD Gammon was I get, would you call it? Do you like the terminology of self play by any chance?

• 1:04:42Speaker 01

  Sure.

• 1:04:43Lex Fridman?

  So like systems that learn by playing themselves. Just, I don't know if it's the best word, but.

• 1:04:50Speaker 01

  So what's what's the problem with that term?

• 1:04:53Speaker 01

  Okay.

• 1:04:53Lex Fridman?

  So it's like the Big Bang, like it's it's like talking to serious physicists. Do you like the term Big Bang? And when when it was early, I feel like it's the early days of self-play. I don't know, maybe it was used previously, but I think it's been used by only a small group of people.

• 1:05:09Speaker 01

  Oh.

• 1:05:09Lex Fridman?

  And so like, I think we're still deciding, is this ridiculously silly name a good name for the concept, potentially one of the most important concepts in artificial intelligence?

• 1:05:19Speaker 01

  Okay, it depends how broadly you apply the term. So I used the term in my 1996 PhD dissertation.

• 1:05:24Lex Fridman?

  Oh, you, wow, the actual terms of.

• 1:05:26Speaker 01

  Yeah, because because.

• 1:05:27Lex Fridman?

  Oh, I didn't know.

• 1:05:28Speaker 01

  Tsaros paper was something like training up an expert backgammon player through self-play. So I think it was in the title of his paper.

• 1:05:35Lex Fridman?

  Okay.

• 1:05:36Speaker 01

  If not in the title, it was definitely a term that he used. Now, there's another term that we got from that work is rollout. So I don't know if you do you ever hear the term rollout? That's a backgammon term that has now applied.

• 1:05:46Lex Fridman?

  Whoa.

• 1:05:47Speaker 01

  Generally in computers, well, at least in in AI.

• 1:05:49Lex Fridman?

  Yeah.

• 1:05:50Speaker 01

  Because of TD gammon. Yeah.

• 1:05:52Lex Fridman?

  That's fascinating.

• 1:05:53Speaker 01

  So how is self-play being used now? And like, why is it, does it, does it feel like a more general powerful concept? Sort of the idea of, well, the machine is just going to teach itself to be smart.

• 1:06:01Lex Fridman?

  Yeah, so that's, that's where, maybe you can correct me, but that's where, you know, the continuation of the spirit and actually like literally the exact algorithms of TD gammon are applied by DeepMind and OpenAI to learn games that are a little bit more complex. That when I was learning artificial intelligence, Go was presented to me with artificial intelligence, the modern approach. I don't know if they explicitly pointed to Go in those books.

• 1:06:28Speaker 01

  I think so.

• 1:06:30Lex Fridman?

  As like unsolvable kind of thing, like implying that these approaches hit their limit in this, for these particular kind of games. So something, I don't remember if the book said it or not, but something in my head before was the professors instilled in me the idea like this is the limits of artificial intelligence of the field. Like it it instilled in me the idea that if we can create a system that can solve the game of Go, we've achieved AGI. That was kind of, I didn't explicitly like say this, but it that was the feeling. And so from, I was one of the people that it seemed magical when a learning system was able to to beat a a human world champion at the game of Go. And even more so.

• 1:07:16Lex Fridman?

  From that, that was AlphaGo, even more so with AlphaGo Zero than kind of renamed and advanced into AlphaZero, beating a world champion or world-class player without any supervised learning on expert games, doing only through by playing itself. So that is, I don't know what to make of it. I think it would be interesting to hear what your opinions are on just how exciting, surprising, profound, interesting, or boring the breakthrough performance of AlphaZero was.

• 1:07:57Speaker 01

  Okay, so AlphaGo knocked my socks off. That was, that was so remarkable.

• 1:08:02Lex Fridman?

  Which aspect of it?

• 1:08:04Speaker 01

  That, that, that they, they got it to work, that they actually were able to leverage a whole bunch of different ideas, integrate them into one giant system. Just the software engineering aspect of it is mind-blowing. I don't, I, I've never been a part of a program as complicated as the program that they built for that. And, and just the, you know, like, like Jerry Tsarao is a neural net whisperer, like, you know, David Silver is a kind of neural net whisperer too. He was able to coax these networks and these new way-out-there architectures to do these, you know, solve these problems that, as you said, you know, when we were learning from AI, no one had an idea how to make it work.

• 1:08:44Speaker 01

  It was, it was remarkable that these, you know, these, these techniques that were so good at playing chess and that could beat the world champion in chess, couldn't beat, you know, your typical Go-playing teenager in Go. So the fact that, that, you know, in a very short number of years, we kind of ramped up to trouncing people in Go, just blew me away.

• 1:09:08Lex Fridman?

  So you, you're kind of focusing on the engineering aspect, which is also very surprising. I mean, there's something different about large, well-funded companies. I mean, there's a compute aspect to it too.

• 1:09:19Speaker 01

  Sure.

• 1:09:19Lex Fridman?

  Like, that, of course, I mean, that's similar to Deep Blue, right? With, with IBM. Like, there's something important to be learned and remembered about a large company taking the ideas that are already out there and investing a few million dollars into it, or, or more. And So you're kind of saying the engineering is kind of fascinating, both on the with AlphaGo is probably just gathering all the data, right, of the expert games, like organizing everything, actually doing distributed supervised learning. And to me, see, the engineering I kind of took for granted.

• 1:10:01Lex Fridman?

  To me, philosophically, being able to persist in the in the face of like, long odds, because it feels like, for me, I'll be one of the skeptical people in the room, thinking that you can learn your way to beat Go. Like, it sounded like, especially with David Silver, it sounded like David was not confident at all. It's like, it was, like, not, it's funny how confidence works.

• 1:10:30Speaker 01

  Yeah.

• 1:10:30Lex Fridman?

  It's like, you're not, like, cocky about it, like, but...

• 1:10:36Speaker 01

  Right, because if you're cocky about it, you kind of stop and stall and don't get anywhere.

• 1:10:40Lex Fridman?

  Yeah, but there's like a hope that's unbreakable. Maybe that's better than confidence. It's a kind of wishful hope and a little dream, and you almost don't want to do anything else. You kind of keep doing it. That's, that seems to be the story, and...

• 1:10:55Speaker 01

  But with enough skepticism that you're looking for where the problems are and fighting through them.

• 1:11:00Lex Fridman?

  Yeah.

• 1:11:00Speaker 01

  Because you know, there's got to be a way out of this thing, yeah.

• 1:11:03Lex Fridman?

  And for him, it was probably, there's, there's a bunch of little factors that come into play. It's funny how these stories just all come together. Like, everything he did in his life came into play, which is like a love for video games, and also a connection to, so the 90s had to happen with TD Gammern and so on.

• 1:11:20Speaker 01

  Yeah.

• 1:11:20Lex Fridman?

  In some ways, it's surprising, maybe you can provide some intuition to it, that not much more than TD Gammern was done for quite a long time on the reinforcement learning front.

• 1:11:31Speaker 01

  Yeah.

• 1:11:31Lex Fridman?

  Is that weird to you?

• 1:11:33Speaker 01

  I mean, like I said, the students who I worked with, we tried to get, basically apply that architecture to other problems, and we consistently failed. There were a couple, a couple really nice demonstrations that ended up being in the literature. There was a paper about controlling elevators, right, where it's, it's like, okay... Can we modify the heuristic that elevators use for deciding, like a bank of elevators for deciding which floors we should be stopping on to maximize throughput, essentially. And you can set that up as a reinforcement learning problem, and you can have a neural net represent the value function so that it's taking where are all the elevators, where are the button pushes, this high dimensional, well, at the time, high dimensional input, a couple dozen dimensions,

• 1:12:17Speaker 01

  and turn that into a prediction as to, oh, is it going to be better if I stop at this floor or not? And ultimately, it appeared as though for the standard simulation distribution for people trying to leave the building at the end of the day, that the neural net learned a better strategy than the standard one that's implemented in elevator controllers. So that was nice. There was some work that Satinder Singh et al. did on handoffs with cell phones, deciding when should you hand off from this cell tower to this cell tower?

• 1:12:50Lex Fridman?

  Oh, okay. Communication networks, yeah.

• 1:12:51Speaker 01

  Yeah. And so a couple of things seemed like they were really promising. None of them made it into production that I'm aware of. And neural nets as a whole started to kind of implode around then. And so there just wasn't a lot of air in the room for people to try to figure out, okay, how do we get this to work in the RL setting?

• 1:13:10Lex Fridman?

  And then they found their way back in 10 plus years. So you said AlphaGo was impressive, like it's a big spectacle. Is there?

• 1:13:19Speaker 01

  Right. So then AlphaZero. So I think I may have a slightly different opinion on this than some people. So I talked to Satinder Singh in particular about this. So Satinder was like Rich Sutton, a student of anti-barto. So they came out of the same lab, very influential machine learning, reinforcement learning researcher, now at DeepMind, as is Rich, though different sites, the two of them.

• 1:13:43Lex Fridman?

  He's in Alberta.

• 1:13:45Speaker 01

  Rich is in Alberta, and Satinder would be in England, but I think he's in England from Michigan at the moment. But the but he was, yes, he was much more impressed with AlphaGo Zero, which is didn't didn't get a kind of a bootstrap in the beginning with human-trained games. It just was purely self-play. Though the first one, AlphaGo, was also a tremendous amount of self-play. Right? They started off, they kick-started the the action network that was making decisions, but then they trained it for a really long time using more traditional temporal difference methods.

• 1:14:20Speaker 01

  So so as a result, I didn't, it didn't seem that different to me. Like, it seems like, yeah, why wouldn't that work? Like, once it works, it works. So what, but he found that that removal of that extra information to be breathtaking. Like, that's a game changer. To me, the first thing was more of a game changer.

• 1:14:39Lex Fridman?

  But the open question, I mean, I guess that's the assumption, is the expert games might contain within them a humongous amount of information.

• 1:14:51Speaker 01

  But we know that it went beyond that, right? We know that it somehow got away from that information because it was learning strategies. I don't think it's, I don't think AlphaGo is just better at implementing human strategies. I think it actually developed its own strategies that were that were more effective. And so from that perspective, okay, well, so it made at least one quantum leap in terms of strategic knowledge. Okay, so now maybe it makes three, like, okay, but that first one is the doozy, right? Getting it to to work reliably and and for the networks to to hold on to the value well enough. Like, that was, that was a big step.

• 1:15:27Lex Fridman?

  Well, isn't, maybe you could speak to this on the reinforcement learning front. So the starting from scratch and learning to do something, like the first, like, like random behavior to

• 1:15:42Lex Fridman?

  Like crappy behavior to like somewhat okay behavior. It's not obvious to me that that's not like impossible to take those steps. Like i- if you just think about the intuition, like how the heck does random behavior become somewhat basic intelligent behavior? Not not human level, not superhuman level, but just basic. But you're saying to you, kind of the intuition is like, if if you can go from human to superhuman level intelligence on the uh on this particular task of game playing, then so you're good at taking leaps. So you can take many of them.

• 1:16:20Speaker 01

  That the system, I believe that the system can take that kind of leap. Yeah, no, and and also I think that that beginner knowledge in Go, like you can start to get a feel really quickly for the idea that um, you know, certain parts of the being in certain parts of the board seems to be more associated with winning, right? Because it's not it's not stumbling upon the concept of winning. It's told that it wins or that it loses. Well, it's self-play, so it both wins and loses. It's told which which side won. And the information is kind of there to start percolating around to make a difference as to um, well, these things have a better chance of helping you win and these things have a worse chance of helping you win.

• 1:17:02Speaker 01

  And so, you know, it can get to basic play, I think, pretty quickly. Then, once it has basic play, well, now it's kind of forced to do some search, to actually experiment with, okay, well, what gets me that next increment of of improvement?

• 1:17:16Lex Fridman?

  How far do you think? Okay, this is where you kind of bring up the the Elon Musk and the Sam Harris, is right? How far is your intuition about these kinds of self-play mechanisms being able to take us? Because it feels, one of the ominous but stated calmly things that when I talked to David Silver, he said, is that they have not yet discovered a ceiling for Alpha Zero, for example, on the game of Go or Chess.

• 1:17:44Speaker 01

  Oh, interesting.

• 1:17:45Lex Fridman?

  Like, it's it keeps, no matter how much they compute, they throw at it, it keeps improving. So it's possible, it's very possible that you, if you throw, you know, some like 10x compute, that it will improve by 5x or something like that. And when stated calmly, it's so like, oh, yeah, I guess so. But like, and then you think like, well, are can we potentially have like? Continuations of Moore's Law in totally different way, like broadly defined Moore's Law.

• 1:18:16Speaker 01

  Right.

• 1:18:16Lex Fridman?

  Not the

• 1:18:17Speaker 01

  Exponential improvement.

• 1:18:18Lex Fridman?

  Exponential improvement, like, are we going to have an alpha zero that swallows the world?

• 1:18:25Speaker 01

  But notice, it's not getting better at other things. It's getting better at Go.

• 1:18:28Lex Fridman?

  Yeah.

• 1:18:28Speaker 01

  And I think it's a, that's a big leap to say, okay, well, therefore it's better at other things.

• 1:18:34Lex Fridman?

  Well, I mean, the question is, how much of the game of life can be turned into?

• 1:18:39Speaker 01

  Right. So that's a, that I think is a really good question. And I think that we don't, I don't think we as a, I don't know, community really know the answer to this, but. So, okay, so, so I went, I went to a talk by some experts on computer chess. So in particular, computer chess is really interesting because, you know, for, of course, for a thousand years, humans were the best chess playing things on the planet. And then computers like edged ahead of the best person and they've been ahead ever since. It's not like people have, have overtaken computers. But, but computers and people together have overtaken computers.

• 1:19:18Lex Fridman?

  Right.

• 1:19:19Speaker 01

  So at least last time I checked, I don't know what the very latest is, but last time I checked that there were teams of people who could work with computer programs to defeat the best computer programs.

• 1:19:29Lex Fridman?

  In the game of Go?

• 1:19:30Speaker 01

  In the game of chess.

• 1:19:31Lex Fridman?

  In the game of chess.

• 1:19:32Speaker 01

  Right. And so using the information about how these things called Elo scores, this sort of notion of how strong a player are you? There's, there's kind of a range of possible scores and this, you, you increment in score. Basically, if you can beat another player of that lower score, 62% of the time or something like that. Like there's some threshold of if you can somewhat consistently beat someone, then you are of a higher score than that person. And there's a question as to how many times can you do that in chess? Right? And so we know that there's a range of human ability levels that cap out with the best playing humans. And the computers went a step beyond that and computers and people together have not gone, I think, a full step beyond that.

• 1:20:17Speaker 01

  It feels, the estimates that they have is that it's starting to asymptote, that we've reached kind of the maximum, the best possible chess playing. And so that means that there's kind of a finite strategic depth, right? At some point, you just can't get any better at this game.

• 1:20:33Lex Fridman?

  Yeah. I mean, I, I don't, so I'll actually check that. I think.

• 1:20:39Lex Fridman?

  It's interesting, because if you have somebody like Magnus Carlsen, who's using these chess programs to train his mind, like to learn about chess.

• 1:20:49Speaker 01

  To become a better chess player, yeah.

• 1:20:50Lex Fridman?

  And so like, that's a very interesting thing, because we're not static creatures. We're learning together. I mean, just like we're talking about social networks, those algorithms are teaching us, just like we're teaching those algorithms. So that's a fascinating thing. But I think the best chess-playing programs are now better than the pairs. Like, they have competition between pairs, but the, it's still, even if they weren't, it's an interesting question, where's the ceiling? So the David, the ominous David Silver kind of statement is like, we have not found the ceiling.

• 1:21:24Speaker 01

  Right. But so the question is, okay, so I don't, I don't know his analysis on that. My, from talking to Go experts, the depth, the strategic depth of Go seems to be substantially greater than that of chess, that there's more kind of steps of improvement that you can make, getting better and better and better. But there's no reason to think that it's infinite.

• 1:21:44Lex Fridman?

  Infinite, yeah.

• 1:21:45Speaker 01

  And so it could be that it's, that what David is seeing is a kind of asymptoting, that you can keep getting better, but with diminishing returns. And at some point, you hit optimal play. Like, in theory, all these finite games, they're finite. They have an optimal strategy. There's a strategy that is the minimax optimal strategy. And so at that point, you can't get any better. You can't beat that, that strategy. Now, that strategy may be, from an information processing perspective, intractable, right? That you need, that all the situations are sufficiently different that you can't compress it at all. It's this giant mess of hardcoded rules, and we can never achieve that. But, but that still puts a cap on how many levels of improvement that we can actually make.

• 1:22:31Lex Fridman?

  But the, the thing about self-play is if you, if you put it, although I don't like doing that, in a broader category of self-supervised learning.

• 1:22:40Lex Fridman?

  Is that it doesn't require too much or any human

• 1:22:43Speaker 01

  Human labeling, yeah.

• 1:22:44Lex Fridman?

  Yeah, human label or just human effort, the human involvement past a certain point. And the same thing you could argue is true for the recent breakthroughs in natural language processing with language models.

• 1:22:57Speaker 01

  Oh, this is how you get to GPT three.

• 1:22:59Lex Fridman?

  Yeah, see how that did the.

• 1:23:01Speaker 01

  That was a good, good transition. Yeah.

• 1:23:03Lex Fridman?

  I practiced that for days, uh, leading up to this. Good now, uh, but like, that's one of the questions is can we find ways to formulate problems in this world that are important to us humans, like more important than the game of chess, that, uh, to which self-supervised kinds of approaches could be applied, whether it's self-play, for example, for like, maybe you could think of like autonomous vehicles in, in simulation, that kind of stuff, or just robotics applications and simulation, or, uh, in the self-supervised learning where un- unannotated data or data that's generated

• 1:23:47Lex Fridman?

  by humans naturally without extra cost, like the Wikipedia or like all of the internet can be used to, to learn something about, to create intelligent systems that do something, uh, really powerful, that pass the Turing test, or that do some kind of superhuman level of performance. So what's your intuition, like trying to stitch all of it together about our discussion of AGI, the limits of self-play, and your thoughts about maybe the limits of neural networks in the context of language models? Is there some intuition in there that might be useful to think about?

• 1:24:29Speaker 01

  Yeah, yeah, yeah. So, so first of all, the, the whole transformer network. Family of things is really cool. It's really, really cool. I mean, for, you know, if you've ever, if back in the day you played with, I don't know, Markoff models for generating text and you've seen the kind of text that they spit out and you compare it to what's happening now, it's, it's amazing. It's so amazing. Now, it doesn't take very long interacting with one of these systems before you find the holes, right? It's, it's not smart in any kind of general way. It's really good at a bunch of things and it does seem to understand a lot of the statistics of language extremely well. And that turns out to be very powerful.

• 1:25:13Speaker 01

  You can answer many questions with that. But it doesn't make it a good conversationalist, right? And doesn't make it a good storyteller. It just makes it good at imitating of things it has seen in the past.

• 1:25:24Lex Fridman?

  The exact same thing could be said by people who are voting for Donald Trump about Joe Biden supporters and people voting for Joe Biden about Donald Trump supporters is, uh, you know,

• 1:25:34Speaker 01

  That they're not intelligent? They're just following the-

• 1:25:37Lex Fridman?

  Yeah, they're following things they've seen in the past and, uh,

• 1:25:40Speaker 01

  So-

• 1:25:40Lex Fridman?

  It's very, it doesn't take long to find the flaws in their, uh, in their like natural language generation abilities.

• 1:25:48Speaker 01

  Yes, yes.

• 1:25:49Lex Fridman?

  So we're being very-

• 1:25:50Speaker 01

  That's interesting.

• 1:25:51Lex Fridman?

  Critical of AI systems.

• 1:25:53Speaker 01

  Right, so, so I've had a similar thought, which was that the stories that GPT-3 spits out are amazing and very human-like. And it doesn't mean that computers are smarter than we realize necessarily. It partly means that people are dumber than we realize or that much of what we do day to day is not that deep. Like, we're just, we're just kind of going with the flow. We're saying whatever feels like the natural thing to say next. Not a lot of it is, is, is creative or meaningful or intentional. But enough is that we actually get, we get by, right? We, we do come up with new ideas sometimes and we do manage to talk each other into

• 1:26:35Speaker 01

  things sometimes and we do sometimes vote for reasonable people sometimes.

• 1:26:41Speaker 01

  But it's really hard to see in the statistics because so much of what we're saying is kind of rote. And so our metrics that we use to measure how these systems are doing don't reveal that because it's in the interstices that is very hard to detect.

• 1:26:59Lex Fridman?

  But is your, do you have an intuition that with these language models, if they grow in size, it's already surprising that when you go from GPT-2 to GPT-3, that there is a noticeable improvement. So the question now goes back to the ominous David Silver and the ceiling.

• 1:27:15Speaker 01

  Right. So maybe there's just no ceiling. We just need more compute. Now, I mean, okay, so now I'm speculating.

• 1:27:22Lex Fridman?

  Yes.

• 1:27:22Speaker 01

  As opposed to before, when I was completely on firm ground.

• 1:27:25Lex Fridman?

  Yeah.

• 1:27:25Speaker 01

  All right. I don't believe that you can get something that really can do language and use language as a thing that doesn't interact with people. Like, I think that it's not enough to just take everything that we've said, written down, and just say, that's enough. You can just learn from that and you can be intelligent. I think you really need to be pushed back at. I think that conversations, even people who are pretty smart, maybe the smartest thing that we know, not, maybe not the smartest thing we can imagine, but we get so much benefit out of talking to each other and interacting. That's presumably why you have conversations live with guests, is that there's something in that interaction that would not be exposed by, oh, I'll just write you a story and then

• 1:28:09Speaker 01

  you can read it later. And I think, I think because these systems are just learning from our stories, they're not learning from being pushed back at by us, that they're fundamentally limited into what they could actually become on this route. They have to, they have to get, you know, shut down. Like, we have to have an argument that, they have to have an argument with us and lose a couple of times before they start to realize, oh, okay, wait, there's some nuance here that actually matters.

• 1:28:35Lex Fridman?

  Yeah, that's actually subtle sounding, but quite profound, that the interaction with humans is essential. And the limitation within that is profound as well, because the time scale, like, the bandwidth at which you can really interact with humans is very low. So it's costly. So you can't, one of the underlying things about self-play is it has to do. You know, a very large number of interactions. And so you can't really deploy reinforced learning systems into the real world to interact. Like you couldn't deploy a language model into the real world to interact with humans,

• 1:29:16Lex Fridman?

  because it was just not get enough data relative to the cost it takes to interact. Like the time of humans is expensive, which is really interesting. That's that good. That takes us back to reinforced learning and trying to figure out if there's ways to make algorithms that are more efficient at learning, keep the spirit in reinforced learning and become more efficient. In some sense, this seems to be the goal. I'd love to hear what your thoughts are. I don't know if you got a chance to see a blog post called Bitter Lesson.

• 1:29:47Speaker 01

  Oh, yes.

• 1:29:49Lex Fridman?

  By Rich Sutton that makes an argument, and hopefully I can summarize it. Perhaps, perhaps you can. Yeah.

• 1:29:55Speaker 01

  Oh, OK.

• 1:29:57Lex Fridman?

  So I mean, I could try and you can correct me, which is he makes an argument that it seems if we look at the long arc of the history of the artificial intelligence field, it calls, you know, 70 years, that the algorithms from which we've seen the biggest improvements in practice are the very simple, like dumb algorithms that are able to leverage computation. And you just wait for the computation to improve. Like all of academics and so on have fun by finding little tricks and congratulate themselves on those tricks. And sometimes those tricks can be like big, that feel in the moment like big spikes and breakthroughs, but in reality, over the decades, it's still the same dumb algorithm that just waits for the compute to get faster and faster.

• 1:30:43Lex Fridman?

  Do you find that to be an interesting argument against the entirety of the field of machine learning as an academic discipline?

• 1:30:52Speaker 01

  That we're really just a subfield of computer architecture.

• 1:30:55Lex Fridman?

  Yeah.

• 1:30:56Speaker 01

  We're just kind of waiting around for them to do their next thing.

• 1:30:58Lex Fridman?

  Really don't want to do hardware work, so like.

• 1:31:00Speaker 01

  That's right. I really don't want to.

• 1:31:01Lex Fridman?

  We're procrastinating.

• 1:31:03Speaker 01

  Yes, that's right. Just waiting for them to do their jobs so that we can pretend to have done ours. So.

• 1:31:09Speaker 01

  Yeah, I mean, the argument reminds me a lot of, I think it was a Fred Jelinek quote, early computational linguist, who said, you know, "We're building these computational linguistic systems, and every time we fire a linguist, performance goes up by 10%." Something like that. And so the idea of us building the knowledge in, in that case, was much less, he was finding to be much less successful than get rid of the people who know about language as a, you know, from a kind of scholastic academic kind of perspective, and replace them with more compute. And so I think this is kind of a modern version of that story, which is, okay, we want to do better on machine vision, you could build

• 1:31:50Speaker 01

  in all these, you know, motivated part-based models that, you know, that just feel like obviously the right thing that you have to have, or we can throw a lot of data at it, and guess what, we're doing better with a lot of data. So I hadn't thought about it until this moment in this way. But what I believe, well, I've thought about what I believe. What I believe is that, you know, compositionality and, what's the right way to say it, the complexity grows rapidly as you consider more and more possibilities, like explosively. And so far, Moore's Law has also been growing explosively, exponentially.

• 1:32:32Speaker 01

  And so it really does seem like, well, we don't have to think really hard about the algorithm design, or the way that we build the systems, because the best benefit we could get is exponential, and the best benefit that we can get from waiting is exponential, so we can just wait. It's got, that's got to end, right? And there's hints now that Moore's Law is, is starting to feel some friction, starting to, the world is pushing back a little bit. One thing I, I don't know, do lots of people know this? I didn't know this. I was, I was trying to write an essay. And yeah, Moore's Law has been amazing, and it's been, it's enabled all sorts of things, but there's a, there's also a kind of counter Moore's Law, which is that the development cost for each successive generation of chips

• 1:33:17Speaker 01

  also is doubling. So it's costing twice as much money. So the amount of development money per cycle or whatever is actually sort of constant. And at some point, we run out of money. So, or we have to come up with an entirely different way of, of doing the development process. So- Like, I guess I was always a bit skeptical of the, "Look, it's an exponential curve, therefore, it has no end. Soon, the number of people going to NURIPS will be greater than the population of the Earth. That means we're going to discover life on other planets." No, it doesn't. It means that we're in a sigmoid curve on the front half, which looks a lot like an exponential. The second half is going to look a lot like diminishing returns.

• 1:33:58Lex Fridman?

  Yeah, the, I mean, but the interesting thing about Moore's Law, if you actually, like, look at the technologies involved, it's hundreds, if not thousands, of S-curves stacked on top of each other. It's not actually an exponential curve. It's constant breakthroughs. And then what becomes useful to think about, which is exactly what you're saying, the cost of development, like, the size of teams, the amount of resources that are invested in continuing to find new S-curves, new breakthroughs. And, yeah, it's an interesting idea. You know, if we live in the moment, if we sit here today, it seems to be the reasonable thing to say that exponentials end. And yet, in the software realm, they

• 1:34:43Lex Fridman?

  just keep appearing, keep happening. And it's so, I mean, it's so hard to disagree with Elon Musk on this, because it, like, I've, you know, I used to be one of those folks, I'm still one of those folks, I studied autonomous vehicles, that's what I worked on. And it's like, you look at what Elon Musk is saying about autonomous vehicles, well, obviously, in a couple of years, or in a year, or next month, we'll have fully autonomous vehicles. Like, there's no reason why we can't. Driving is pretty simple. Like, it's just a learning problem, and you just need to convert all the driving that we're doing into data, and just have, you know, with the trains on that data. And, like, we use

• 1:35:28Lex Fridman?

  only our eyes, so you can use cameras, and you can train on it. And it's like, yeah. That's that what that should work. And then you put that hat on like the philosophical hat. And but then you put the pragmatic hat and it's like, this is what the flaws of computer vision are like. This is what it means to train at scale. And then you you put the human factors, the psychology hat on, which is like, it's actually driving is a lot, the cognitive science or cognitive, whatever the heck you call it. Is it's really hard. It's much harder to drive than than we realize. There's much larger number of edge cases. So building up an intuition around this is uh, is around exponentials is really difficult.

• 1:36:11Lex Fridman?

  And on top of that, the pandemic is making us think about exponentials, making us realize that like, we don't understand anything about it. We're not able to intuit exponentials. We're either That's true. ultra terrified, some part of the population and some part is like, uh, the opposite of whatever the carefree. And we're not managing it very well. Blase. Wow, wow, that's that's French. I assume so. It's got an accent. So it's a it's fascinating to think what what the limits of this exponential

• 1:36:51Lex Fridman?

  growth of technology, not just Moore's law, it's technology, how that rubs up against the bitter lesson and GPT three and self play mechanisms. Like, it's not obvious. I used to be much more skeptical about neural networks. Now at least give a slither possibility that will be all that will be very much surprised. And also, you know, uh, caught in a way that like, we are not prepared for. Like in applications of um, social networks, for example. Sure. Because it feels like really good transformer models that are able to do some kind of.

• 1:37:39Lex Fridman?

  Like very good natural language generation are the same kind of models that can be used to learn human behavior and then manipulate that human behavior to gain advertiser dollars and all those kinds of things.

• 1:37:51Speaker 01

  Sure.

• 1:37:51Lex Fridman?

  If the capitalist system and.

• 1:37:53Speaker 01

  Right, and they arguably already are manipulating human behavior.

• 1:37:56Lex Fridman?

  Yeah.

• 1:37:57Speaker 01

  Yeah, so but not for self preservation, which I think is a big, that would be a big step. Like if they were trying to manipulate us to convince us not to shut them off,

• 1:38:09Speaker 01

  I would be very freaked out, but I don't see a path to that from where we are now. They, they, they don't have any of those abilities. That's not what they're trying to do. They're trying to keep people on, on the site.

• 1:38:22Lex Fridman?

  But see, the thing is, this, this is the thing about life on Earth is they might be borrowing our consciousness and sentience like, so like, in a sense, they do, because the creators of the algorithms have like, they're not, you know, if you look at our bodies.

• 1:38:38Speaker 01

  Yeah, okay.

• 1:38:38Lex Fridman?

  We're not a single organism. We're a huge number of organisms with like tiny little motivations. We're built on top of each other. In the same sense, the AI algorithms that are, they're not like.

• 1:38:48Speaker 01

  It's a system that includes the companies and corporations, right? Because corporations are funny organisms in and of themselves that really do seem to have self preservation built in. And I think that's at the, at the design level. I think they're designed to have self preservation be a focus. So you're right, in that, in that broader system that we're also a part of and can have some influence on, it's, it's, it is much more complicated, much more powerful. Yeah, I agree with that.

• 1:39:18Lex Fridman?

  So people really love it when I ask what three books, technical, philosophical fiction had a big impact on your life, maybe you can recommend. We went with movies. We went with Billy Joel and I forgot what you, what music you recommended, but.

• 1:39:36Speaker 01

  I didn't, I just said, I have no taste in music. I just like pop music.

• 1:39:39Lex Fridman?

  That was actually really skillful, the way you avoided that question.

• 1:39:43Speaker 01

  I was, I'm gonna try to do the same with the books.

• 1:39:46Lex Fridman?

  So do you have a skillful way to avoid answering the question about three books you would recommend?

• 1:39:51Speaker 01

  I'd like to tell you a story.

• 1:39:54Speaker 01

  So my first job out of college was at Belcor. I mentioned that before, where I worked with Dave Ackley. The head of the group was a guy named Tom Landauer. And I don't know how well-known he's known now, but arguably, he's the inventor and the first proselytizer of word embeddings. So they developed a system shortly before I got to the group. Yeah. That was called latent semantic analysis, that would take words of English and embed them in a multi-hundred dimensional space and then use that as a way of assessing similarity and basically doing reinforcement learning, not, sorry, not reinforcement, information retrieval, you know, sort of pre-Google information retrieval.

• 1:40:35Speaker 01

  And he was trained as an anthropologist, but then became a cognitive scientist. So I was in the cognitive science research group. It's, you know, like I said, I'm a cognitive science groupie. At the time, I thought I'd become a cognitive scientist, but then I realized in that group, no, I'm a computer scientist, but I'm a computer scientist who really loves to hang out with cognitive scientists. And he said, he studied language acquisition in particular. He said, you know, humans have about this number of words of vocabulary, and most of that is learned from reading. And I said, that can't be true, because I have a really big vocabulary and I don't read. He's like, you must. I'm like, I don't think I do. I mean, like, stop signs, I definitely read stop signs, but like, reading books is not

• 1:41:19Speaker 01

  a thing that I do a lot of. Do you really, though? It might be just visual, maybe the red color. Do I read stop signs? Yeah. No, it's just pattern recognition at this point. I don't sound it out. Yeah. So now I do. I wonder what that... Oh, yeah, stop the guns. So... That's fascinating. So you don't... So I don't read very... I mean, obviously, I read, and I've read plenty of books. But like, some people, like Charles, my friend Charles, and others, like, a lot of people in my field, a lot of academics, like, reading was really a central topic to them in development. And I'm not that guy. In fact, I used to joke that when I got into college, that it was on kind of a...

• 1:42:04Speaker 01

  Help out the illiterate kind of program because I got to like I in my house, I wasn't a particularly bad or good reader, but when I got to college, I was surrounded by these people that were just voracious in their reading appetite. And they were like, have you read this? Have you read this? Have you read this? And I'd be like, no, I'm clearly not qualified to be at this school. Like, there's no way I should be here. Now I've discovered books on tape, like audiobooks. And so I'm much better. I'm more caught up. I read a lot of books.

• 1:42:32Lex Fridman?

  A small tangent on that. It is a fascinating, open question to me on the topic of driving. Whether, you know, supervised learning people, machine learning people think you have to like drive to learn how to drive. To me, it's very possible that just by us humans, by first of all, walking, but also by watching other people drive, not even being inside cars as a passenger, but let's say being inside the car as a passenger.

• 1:43:01Speaker 01

  Yeah.

• 1:43:01Lex Fridman?

  But even just like being a pedestrian and crossing the road. You learn so much about driving from that. It's very possible that you can, without ever being inside of a car, be okay at driving once you get in it. Or like watching a movie, for example. I don't know, something like that.

• 1:43:20Speaker 01

  Have you, have you taught anyone to drive?

• 1:43:23Lex Fridman?

  No.

• 1:43:23Speaker 01

  So.

• 1:43:24Lex Fridman?

  Except myself.

• 1:43:25Speaker 01

  I have two children and I learned a lot about car driving because my wife doesn't want to be the one in the car while they're learning. So that's my job.

• 1:43:34Lex Fridman?

  Yeah.

• 1:43:35Speaker 01

  So I sit in the passenger seat and it's really scary. I have, you know, I have wishes to live and they're, you know, they're figuring things out. Now, they start off very, very much better than I imagine like a neural network would, right? They, they get that they're seeing the world. They get that there's a road that they're trying to be on. They get that there's a relationship between the angle of the steering, but it takes a while to not be very jerky. And so that happens pretty quickly, like the ability to stay in lane at speed, that happens relatively fast. It's not zero-shot learning, but it's pretty fast. The thing that's remarkably hard, and this is, I think, partly why self-driving cars are really hard, is the degree to which driving is a social interaction activity.

• 1:44:20Lex Fridman?

  Yes.

• 1:44:21Speaker 01

  And that blew me away. I was completely unaware of it until I watched my son learning to drive. And I was realizing that he was sending signals to all the cars around him. And those, in his case, he's always had social communication challenges. He was sending very mixed, confusing signals to the other cars, and that was causing the other cars to drive weirdly and erratically. And there was no question in my mind that he would have an accident, because they didn't know how to read him. There's things you do with the speed that you drive, the positioning of your car, that you're constantly like in the head of the other drivers. And seeing him not knowing how to do that and having to be taught explicitly, "Okay, you have to be thinking about what the other driver

• 1:45:06Speaker 01

  is thinking," was a revelation to me. I was stunned.

• 1:45:10Lex Fridman?

  So creating kind of theories of mind of the other.

• 1:45:15Speaker 01

  Theories of mind of the other cars.

• 1:45:16Lex Fridman?

  Yeah.

• 1:45:16Speaker 01

  Yeah, yeah. Which I just hadn't heard discussed in the self-driving car talks that I've been to. Since then, there's some people who do consider those kinds of issues, but it's way more subtle than I think.

• 1:45:28Lex Fridman?

  There's a little bit of work involved with that when you realize, like when you especially focus not on other cars, but on pedestrians, for example. It's literally staring you in the face.

• 1:45:38Speaker 01

  Yeah, yeah, yeah.

• 1:45:39Lex Fridman?

  So that when you're just like, how do I interact with pedestrians?

• 1:45:43Speaker 01

  Pedestrians, you're practically talking to an octopus at that point. They've got all these weird degrees of freedom. You don't know what they're gonna do. They could turn around any second.

• 1:45:50Lex Fridman?

  But the point is, we humans know what they're gonna do. Like, we have a good theory of mind. We have a good mental model of what they're doing. And we have a good model of the model they have of you, and the model of the model of the model. Like, we're able to kind of reason about this kind of, the social game of it.

• 1:46:11Lex Fridman?

  All the hope is that it's quite simple, actually, that it could be learned. That's why I just talked to the Waymo. I don't know if you know that company. It's Google South Africa. They I talked to their CTO about this podcast and they like I rode in their car. And it's quite aggressive and it's quite fast and it's good and it feels great. It may it also, just like Tesla, Waymo made me change my mind about like, maybe driving is easier than I thought. Maybe I'm just being hu- speciest, human centered. Maybe.

• 1:46:45Speaker 01

  It's a speciest argument.

• 1:46:47Lex Fridman?

  Yes, I don't know. But it it's fascinating to think about like the same as with reading, which I think you just said. You avoided the question, though I still hope you answered it somewhat. We avoided it brilliantly. It is there's blind spots as artificial intelligence that artificial intelligence researchers have about what it actually takes to learn to solve a problem.

• 1:47:10Speaker 01

  Have you had Anka Dragan on?

• 1:47:12Lex Fridman?

  Yeah.

• 1:47:13Speaker 01

  Okay.

• 1:47:13Lex Fridman?

  She's one of my favorites. So much energy.

• 1:47:15Speaker 01

  She's right. Oh, she yeah.

• 1:47:17Lex Fridman?

  She's amazing.

• 1:47:17Speaker 01

  Fantastic. And and in particular, she thinks a lot about this kind of I know that you know that I know kind of planning. And the last time I spoke with her, she was very articulate about the ways in which self-driving cars are not solved. Like what's still really, really hard.

• 1:47:34Lex Fridman?

  But even her intuition is limited. Like, we're all like new to this. So in some sense, the Elon Musk approach of being ultra confident and just like plowing.

• 1:47:42Speaker 01

  Put it out there.

• 1:47:42Lex Fridman?

  Putting it out there, like some people say it's reckless and dangerous and so on. But like partly, it's like it seems to be one of the only ways to make progress in artificial intelligence. So it's a it's, you know, these are difficult things. Like, you know, democracy is messy. Implementation of artificial intelligence systems in the real world is messy.

• 1:48:05Speaker 01

  So many years ago, before self-driving cars were an actual thing you could have a discussion about, somebody asked me like, what if what if the what if we could use that robotic technology and use it to drive cars around? Like, isn't that aren't people going to be killed? And then it's now, you know, blah, blah, blah. I'm like, that's not supposed to happen, I said with confidence, incorrectly, obviously. What I think is going to happen is we're going to have a lot more like a very gradual kind of rollout where people have these cars in like closed communities, right? Where it's somewhat realistic, but it's still in a box, right? So that we can really get a sense of what. What are the weird things that can happen? How do we, how do we have to change the way we behave around these vehicles?

• 1:48:47Speaker 01

  Like, it obviously requires a kind of co-evolution that you can't just plop them in and see what happens. But of course, we're basically popping them in to see what happens. So I was wrong, but I do think that would have been a better plan.

• 1:48:59Lex Fridman?

  So that's, but your intuition, it's funny, just zooming out and looking at the forces of capitalism. And it seems that capitalism rewards risk takers and rewards and punishes risk takers. And like, try it out. The academic approach to let's try a small thing and try to understand slowly the fundamentals of the problem. And let's start with one, then do two, and then see that, and then do the three. You know, the capitalist like startup entrepreneurial dream is let's build a thousand and let's

• 1:49:38Speaker 01

  Right, and 500 of them fail, but whatever, the other 500, we learn from them.

• 1:49:42Lex Fridman?

  But if you're good enough, I mean, one thing, it's like your intuition would say like, that's going to be hugely destructive to everything. But actually, it's kind of the, the forces of capitalism, like people are quite, it's easy to be critical, but if you actually look at the data, the, at the way our world has progressed in terms of the quality of life, it seems like the competent, good people rise to the top. This is coming from me from the Soviet Union and so on. It's like, it's interesting that somebody like Elon Musk is the way you, you push progress in artificial intelligence.

• 1:50:20Lex Fridman?

  Like it's forcing Waymo to step their, their stuff up and Waymo is forcing Elon Musk to step up. It's fascinating, because I have this tension in, in my heart and just being upset by

• 1:50:36Lex Fridman?

  The lack of progress in autonomous vehicles in within academia. So there's huge progress in the early days of the DOPR challenges. And then it just kind of stopped like at MIT, but it's true everywhere else with with an exception of a few sponsors here and there is like it's not seen as a sexy problem. Times like the moment artificial intelligence starts approaching the problems of the real world, like academics kind of like, ah, all right, let, let the

• 1:51:12Speaker 01

  They get really hard in a different way.

• 1:51:13Lex Fridman?

  In a different way.

• 1:51:14Speaker 01

  And that's right. I think, yeah, right. Some of us are not excited about that other way.

• 1:51:19Lex Fridman?

  But I still think there's fundamentals problems to be solved in those difficult things. It's not, it's still publishable. I think like we just need to, it's the same criticism you could have of all these conferences in Europe, CVPR, where application papers are often as powerful and as important as like a theory paper, even like theories just seems much more respectable and so on. I mean, in machine learning community is changing that a little bit. I mean, at least in statements, but it's, it's still not seen as the sexiest of pursuits, which is like, how do I actually make this thing work in practice as opposed to on this toy data set? All that to say, are you still avoiding the three books question? Is there something

• 1:52:03Lex Fridman?

  on audiobook that you can recommend?

• 1:52:06Speaker 01

  Oh, I've, yeah, I mean, I, yeah, I've read a lot of really fun stuff. In terms of books that I find myself thinking back on that I read a while ago, like that have stood the test of time to some degree, I find myself thinking of program or be programmed a lot by Douglas Rosch, Roschkopf, which was, it basically put out the premise that we all need to become programmers in one form or another. And it was in analogy to once upon a time, we all had to become readers, we had to become literate. And there was a time before that when not everybody was literate, but once literacy was possible, the people who were literate had more of a say in society than the people who weren't. And so we made a big effort to get everybody up to speed.

• 1:52:51Speaker 01

  And now it's, it's not a hundred percent universal, but it's quite widespread. Like the, the assumption is generally that people can read. The analogy that he makes is that programming is a similar kind of thing, that we need to have a say in, right? So being a reader, being literate, being a reader means you can receive all this information, but you don't get to put it out there. And programming is the way that we get to put it out there. And that was the argument he made. I think he specifically has now backed away from this idea. He doesn't think it's happening quite this way. And that might be true that it didn't, society didn't sort of play forward quite that way. But I still believe in the premise.

• 1:53:34Speaker 01

  I still believe that at some point, we have, the relationship that we have to these machines and these networks has to be one of each individual can, has the wherewithal to make the machines help them, do the things that that person wants done. And as, you know, as software people, we know how to do that. And when we have a problem, we're like, okay, I'll just, I'll hack up a Perl script or something and make it so. If we lived in a world where everybody could do that, that would be a better world. And computers would be, have, I think, less sway over us. And other people's software would have less sway over us as a group.

• 1:54:09Lex Fridman?

  Yeah, in some sense, software engineering, programming's power.

• 1:54:12Speaker 01

  Programming is power. Right. It's, yeah, it's like magic. It's like magic spells. And, and it's not out of reach of everyone, but at the moment, it's just a sliver of the population who can, who can commune with machines in this way. So, I don't know. So that book had a big, big impact on me. Currently, I'm, I'm reading The Alignment Problem, actually, by Brian Christian. So I don't know if you've seen this out there yet.

• 1:54:35Lex Fridman?

  Is it similar to, to Russell's work with The Control Problem?

• 1:54:38Speaker 01

  It's in that same general neighborhood. I mean, they take, they have different emphases that they're, they're concentrating on. I think, I think Stuart's book did a remarkably good job, like a, just a celebratory good job at describing AI technology and sort of how it works. I thought that was great. It was really cool to see that in a book.

• 1:54:57Lex Fridman?

  Yeah. I think he has some experience writing some books.

• 1:55:01Speaker 01

  That's, you know, that's probably a possible thing. He's maybe thought a thing or two about how to explain AI to people.

• 1:55:07Lex Fridman?

  Yeah.

• 1:55:08Speaker 01

  Yeah, that's a really good point. This book so far has been remarkably good at telling the story of the

• 1:55:16Speaker 01

  Sort of the history, the recent history of some of the things that have happened. I'm in the first third. He said this book is in three thirds. The first third is essentially AI fairness and implications of AI on society that we're seeing right now. And that's been great. I mean, he's telling those stories really well. He went out and talked to the frontline people who, whose names are associated with some of these ideas. And it's been terrific. He says the second half of the book is on reinforcement learning. So maybe that'll be fun. And then the third half, third third, is on the superintelligence alignment problem. And I suspect that that part will be less fun for me to read.

• 1:55:56Lex Fridman?

  Yeah.

• 1:55:58Lex Fridman?

  Yeah, it's, it's an interesting problem to talk about. I find it to be the most interesting, just like thinking about whether we live in a simulation or not. As a, as a thought experiment to think about our own existence. So in the same way, talking about alignment problem with AGI is a good way to think. Similarly, like the trolley problem with autonomous vehicles. It's a useless thing for engineering, but it's a, it's a nice little thought experiment for actually thinking about what are like our own human ethical systems, our moral systems to, to, uh, by thinking how we engineer these things, you start to understand yourself.

• 1:56:37Speaker 01

  So sci fi can be good at that, too. So one sci fi book to recommend is Exhalations by Ted Chang, a bunch of short stories. This Ted Chang is the guy who wrote the short story that became the movie Arrival.

• 1:56:51Speaker 01

  And all of his stories, just from a, he was a computer scientist, actually he studied at Brown. They all have this sort of really insightful bit of science or computer science that drives them. And so it's just a romp, right? To just like, he creates these artificial worlds with these, by extrapolating on these ideas that, that we know about, but hadn't really thought through to this kind of conclusion. And so his stuff is, it's really fun to read. It's mind warping.

• 1:57:20Lex Fridman?

  So I'm not sure if you're familiar, I seem to mention this every other word. Is I'm from the Soviet Union and I'm Russian. Read way too much of the Soviet.

• 1:57:30Speaker 01

  My roots are Russian too, but a couple of generations back.

• 1:57:34Lex Fridman?

  Well, it's probably in there somewhere. So maybe you can. We can pull at that thread a little bit of the existential dread that we all feel. You mentioned that you, I think somewhere in the conversation, you mentioned that you don't really pretty much like dying. I forget in which context, it might have been a reinforcement learning perspective. I don't know.

• 1:57:54Speaker 01

  I'd know, you know what it was? It was in teaching my kids to drive.

• 1:57:58Lex Fridman?

  That's, that's how you face your mortality. Yes. From a human being's perspective, or from a reinforcement learning researcher's perspective, let me ask you the most absurd question. What's, what do you think is the meaning of this whole thing? The meaning of life on this spinning rock?

• 1:58:18Speaker 01

  I mean, I think reinforcement learning researchers maybe think about this from a science perspective more often than a lot of other people, right? As a supervised learning person, you're probably not thinking about the sweep of a lifetime, but reinforcement learning agents are having little lifetimes, little weird little lifetimes. And it's, it's hard not to project yourself into their world sometimes. But, you know, as far as the meaning of life, so I, when I turned 42, you may know from, that's, that is a book I read, the, the

• 1:58:48Lex Fridman?

  Hitchhiker's Guide to the Galaxy.

• 1:58:50Speaker 01

  That that is the meaning of life. So when I turned 42, I had a meaning of life party where I invited people over and everyone shared their meaning of life. We, they, we had slides made up and so we had, we all sat down and did a slide presentation to each other about the meaning of life. And mine

• 1:59:09Lex Fridman?

  That's great.

• 1:59:10Speaker 01

  Mine was balance. I think that life is balance. And so the activity at the party, for a 42 year old, maybe this is a little bit non-standard, but I, I found all the little toys and devices that I had that where you had to balance on them. You had to like stand on it and balance, or a pogo stick I brought, a rip stick, which is like a weird two wheeled skateboard.

• 1:59:35Speaker 01

  I got a unicycle, but I didn't know how to, I didn't know how to do it. I now can do it.

• 1:59:40Lex Fridman?

  Love watching you try.

• 1:59:41Speaker 01

  Yeah, I'll send you a video. I'm not great, but I, but, but, but I managed. And so, so balance, yeah. So, so my, my wife has a really good one that she sticks to and is probably pretty accurate. And it has to do with Healthy relationships with people that you love and working hard for good causes. But to me, yeah, balance, balance in a word, that's, that, that works for me. Not too much of anything, because too much of anything is iffy.

• 2:00:12Lex Fridman?

  That feels like a Rolling Stones song. I feel like they must be.

• 2:00:15Speaker 01

  You can't always get what you want, but if you try, sometimes you can strike a balance.

• 2:00:21Lex Fridman?

  Yeah, I think that's how it goes.

• 2:00:24Speaker 01

  I'll write you a parody.

• 2:00:26Lex Fridman?

  It's a huge honor to talk to you. This is really fun.

• 2:00:29Speaker 01

  The honor's mine.

• 2:00:29Lex Fridman?

  I've been a big fan of yours, so can't, can't wait to see what you do next in the world of education, the world of parody, and the world of reinforcement learning. Thanks for talking today.

• 2:00:41Speaker 01

  My pleasure.

• 2:00:42Lex Fridman?

  Thank you for listening to this conversation with Michael Littman, and thank you to our sponsors. SimpliSafe, a home security company I use to monitor and protect my apartment. ExpressVPN, the VPN I've used for many years to protect my privacy on the internet. MasterClass, online courses that I enjoy from some of the most amazing humans in history. And BetterHelp, online therapy with a licensed professional. Please check out these sponsors in the description to get a discount and to support this podcast. If you enjoy this thing, subscribe on YouTube, review it with five stars on Apple Podcasts, follow on Spotify, support it on Patreon, connect with me on Twitter at Lex Fridman.

• 2:01:24Lex Fridman?

  And now, let me leave you with some words from Groucho Marx. If you're not having fun, you're doing something wrong. Thank you for listening and hope to see you next time.