Brian Christian – How to Live with Computers - [Invest Like the Best, EP.140]

My guest this week is Brian Christian, the author of two of my favorite recent books: Algorithms to Live By and The Most Human Human. Our conversation covers the present and future of how humans interact with and use computers. Brian’s thoughts on the nature of intelligence and what it means to be human continue to make me think about what works, and life, will be like in the future. I hope you enjoy our conversation. For more episodes go to InvestorFieldGuide.com/podcast. Sign up for the book club, where you’ll get a full investor curriculum and then 3-4 suggestions every month at InvestorFieldGuide.com/bookclub. Follow Patrick on Twitter at @patrick_oshag Show Notes 1:11 - (First Question) – Summarizing his collection of interests that led to his three books 2:59 – Biggest questions in AI 3:43 – Defining AGI (Artificial General Intelligence) and its history 5:18 – Computing Machinery and Intelligence 7:54 – The idea of the most human human 9:59 – Tactics that have changed the most in learning to be the most human human 16:10 –Tests for measuring AGI and updates made to them 20:12 – Concerns for once we have AGI 26:06 – Self-awareness as a threshold for AGI 31:58 – Skeptics’ take on AGI 37:14 – Advice for people building careers and how AGI will impact work 38:16 – Explore/Exploit trade-off 44:57 – How to explore/exploit applies to business concepts 49:16 – Impacts of AGI on the economy 52:40 – Highlights from his second book 57:39 – Kindest thing anyone has done for Brian Learn More For more episodes go to InvestorFieldGuide.com/podcast. Sign up for the book club, where you’ll get a full investor curriculum and then 3-4 suggestions every month at InvestorFieldGuide.com/bookclub Follow Patrick on twitter at @patrick_oshag

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Patrick O'Shaughnessy?

  I know firsthand how complex the tech stack is for asset management firms. And seemingly every new tool and data source makes the problem even worse, adding more complexity, more headcount and more risk. RidgeLine offers a better way forward. One unified platform that automates away the complexity across portfolio accounting, reconciliation, reporting, trading, compliance and more, all at scale. RidgeLine is revolutionizing investment management, helping ambitious firms scale faster, operate smarter and stay ahead of the curve. See what RidgeLine can unlock for your firm. Schedule a demo at ridgeline.ai.

• 0:36Patrick O'Shaughnessy?

  Hello and welcome, everyone. I'm Patrick O'Neal and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, methods, stories, and of strategies that will help you better invest both your time and your money. You can learn more and stay up to date at investorfieldguide.com.

• 0:56Speaker 02

  Patrick O'Neal is the CEO of O'Neal Asset Management. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of O'Neal Asset Management. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of O'Neal Asset Management may maintain positions in the securities discussed in this podcast.

• 1:22Patrick O'Shaughnessy?

  My guest this week is Brian Christian, the author of two of my favorite recent books, Algorithms to Live By and The Most Human Human. Our conversation covers the present and future of how humans interact with and use computers. Brian's thoughts on the nature of intelligence and what it means to be human continue to make me think about what work and life will be like in the future. I hope you enjoy our conversation.

• 1:43Patrick O'Shaughnessy?

  I'd love to begin with some framing from you around just what your general interest is that unites the first two books that you've written and published and the third that you're working on now. How would you sum up your set of interests?

• 1:55Speaker 03

  I think I have this very on paper eclectic academic background, where in college I studied computer science and philosophy and then I went to graduate school for creative writing. And I remember at the time raising a lot of eyebrows at family reunions and so forth, people saying, so you're studying computer science, this very precise, exacting engineering discipline and philosophy, like this abstract, vague field. How do these two things connect? Increasingly, people don't ask me that question. And I think we're perhaps just coincidentally living through the great synthesis of computer science and philosophy, that as the sort of computational metaphor for mind becomes increasingly literal, we've developed these

• 2:39Speaker 03

  neural networks, for example, starting in the 1940s, through this analogy to the way that the nervous system worked, and are now discovering in the last 10 years that this is the best mechanism that we know of for actually implementing AI, which maybe shouldn't surprise us that we are rediscovering what evolution found after millions of years of trial and error. Computation gives us a way of asking questions about what does it mean to think? What does it mean to have a mind? What does it mean to make rational decisions? And I found, as I was studying these two things in undergraduate, the questions that I was interested in asking in philosophy about the mind were in some ways better answered through the tools and the vocabulary of computer science than they were in philosophy

• 3:24Speaker 03

  itself. So that's the sort of collision of those two fields has been kind of the overriding interest of my whole professional career.

• 3:31Patrick O'Shaughnessy?

  What do you think the biggest open questions are today in AI and computer science?

• 3:37Speaker 03

  I think it's clear that we're on the road to building AGI. And the open question that I think is on the minds of many of the people in the technical research community is how many significant breakthroughs away are we? I was just at an AI conference a few weekends ago, and we were talking about this. And I said, I think we're zero to two major breakthroughs away from AGI. And that got a general consensus of hmm and nodding. And I think that's, in some ways, the holy grail. That's the biggest question hanging over the field. So I think that Is going to be one of the great things that we'll learn in the next couple of decades.

• 4:15Patrick O'Shaughnessy?

  We're going to talk about the most human human in some detail in your own kind of journey, which was so interesting to define sort of what it is that makes us human, that that line, that goal keeps moving, the goalpost keeps moving. But can you define AGI for people listening? What sort of the history of that idea is? What is like the equivalent Turing test? Is it still the Turing test that would lead us to believe we've achieved this? And why might the answer be zero?

• 4:37Speaker 03

  Right, okay, so AGI is artificial general intelligence, which is a fairly recent term, but the underlying idea has been around for a long time. And it basically means a system that is human level competent or beyond at not just a specific thing that humans do, but basically everything that humans do. So we've long had systems that could outperform humans at arithmetic since the 40s, and then things like chess since the 1990s. And we're seeing these particular domain-specific milestones kind of fall into the rearview mirror, most more recently Go, for example. But there's still this question of the kind of flexible general intelligence where you can ask someone to write you a poem, you can ask them to add some figures,

• 5:21Speaker 03

  you can ask them to ruminate on their childhood. What do we need in order to build a system capable of competence with that degree of flexibility? So that's the question of AGI. And I think the Turing test, I mean, Alan Turing was, of course, one of the great founding fathers of computer science and was amazingly prescient in thinking about some of these philosophical questions decades before other people were thinking about it. Already by 1950, he writes this famous paper called Computing Machinery and Intelligence, and the opening line is, "Can machines think?" Of course, you have to remember, this was back in the time when the computer as we know it was this like room-sized thing processing punched cards. But he was already asking these philosophical questions.

• 6:04Speaker 03

  He was kind of seeing the long view ahead. And he answered the question not as a philosopher, but as an engineer. He said, "I don't know, I don't purport to know what thinking actually is, but let's just imagine a practical test that we can run. We will sequester a random person in a room down the hall, and we'll be sending text messages back and forth with that person. And we'll also have this computer." That we're sending text messages back and forth with the computer. And the computer claims that it is the random human being hidden in a room. And that the other person is the computer. Of course, the human is claiming the same thing. Turing says that, in his view, by the year 2000, he predicted, we'd reach a point where,

• 6:47Speaker 03

  after five minutes of interaction, the program could fool a panel of judges 30% of the time. And that, as a result, we would come to speak of machines as being intelligent without expecting to be contradicted. He wasn't making a metaphysical point. He was making a cultural point. And this is one of the famous predictions of computer science that didn't come true. So, starting in the early 1990s, a series of annual Turing test competitions were inaugurated, famously by this quirky New Jersey millionaire who had made his fortune selling plastic, portable, roll-up, lighted disco dance floors. And decided to bankroll this Turing test competition and name it after himself.

• 7:32Speaker 03

  So it's called the Lobner Prize. Starting in the 1990s, they actually started doing these competitions. They'd convene people. They'd have people sending these texts back and forth for five minutes. And the Turing test threshold of 30% was not crossed by the year 2000. But I got very interested in this in the year 2008 because the top computer program at that year's competition managed to fool 25% of the judges. So it was just a single vote short of hitting this famous threshold. You were seeing newspaper headlines at the time, like, "Humanity Dodges a Bullet." There was this thinking that perhaps the next year would be the one that machines finally breached that. And so I got involved in my first book by being one of the human participants in the 2009 competition.

• 8:16Speaker 03

  So I was the guy sequestered in a room down the hall. With five minutes on the clock to persuade a panel of judges that I was the human being, not the mechanical impostor.

• 8:26Patrick O'Shaughnessy?

  So, we're talking about it now, so let's just go right there. I mean, I thought this story was so fascinating. So, a lot of people will be familiar with the binary Turing test. And I want to come back to that binary line for AGI, past which things get very interesting if we do cross it. But first, I bet that most people won't know this idea of the most human human, that there's not just a binary prize yes or no, with a percentage of judges fooled. There's another thing, which was the subject of your book. So, define what that is and tell us the story of your participation in this event.

• 8:52Speaker 03

  There's this very bizarre aspect to the competition which you're highlighting, which is that after each conversation, the judge will not only make a binary decision about whether they thought they were talking to the human or the computer, but they will indicate a degree of confidence and basically say, on a scale of zero to 100, how sure are you in making that determination? And so, every year, there is a computer program that gets the highest confidence score out of all of the other computer programs, and you win this title called the most human computer, and you get a bronze medallion and a few thousand dollars. But there is also, of course, every year, a real person who gets the highest confidence score out of all of the real people. So, they were better able to persuade the judges that they were a human being than the other human beings.

• 9:36Speaker 03

  And this person wins the so-called most human human award. And so, when I learned about this, I was riveted. And indeed, I found myself kind of approaching my role in the competition from the perspective that I was not only competing against the machines, but I was also, in fact, competing against the other real people. And it opens up these bizarre and, to me, fascinating questions of, what do you actually do from a tactics perspective if you have five minutes to act as human as possible in a competitive scenario? What does that actually look like? And so, for me, that's a case of this... Philosophical question where the rubber is hitting the road. What are the distinctive qualities of the human mind?

• 10:20Speaker 03

  What is inimitable by software, at least at the current time? And that all has to operationalize in an actual strategy you're going to bring to what seems like this kind of high-pressure, small-talk scenario.

• 10:31Patrick O'Shaughnessy?

  What changed the most? So when you first encountered this problem, you're trying to figure out how to be the most human human, and then sort of the journey ends at some point. Maybe it's ongoing still, you're still thinking about it. But what were the tactics that changed the most? What were some surprising findings about things that were effective at trying to be the most human human?

• 10:49Speaker 03

  So I would say there was at least one big change between the early 90s and when I participated in 2009, and then at least one major change since. So one of the biggest differences between the early 90s and the late 2000s was that in the early 90s, there was this idea that you had to make the conversation topic-specific. The original Logner Prize competitions kind of handicapped the human players by forcing every conversation to a predefined topic as a way to make it just a little bit easier for the programmers. And this led to very strange judging decisions of if your ostensible topic is hockey, can

• 11:28Speaker 03

  you talk about the Cold War implications of the famous 1980s US versus Soviet Union Olympic hockey game, or are we now into politics? So it was, of course, very difficult to police the boundaries of what was or wasn't on topic. There was sort of a classic signature that a program might demonstrate pretty impressive competence within a prescribed kind of channel of speech. As long as you kind of followed the predictable path through that subject matter, you were likely to encounter things for which the programmers had prepared. But if you tried to push against that envelope... The only possible thing that would happen to you would either be you'd get just nonsense

• 12:10Speaker 03

  or you'd get this very noticeable attempt to direct you back to the kind of main thread of conversation. That started to change in the 2000s because of the internet. Suddenly, it became possible to approach the writing of a chatbot program not as if you were writing a choose-your-own-adventure book where you had to manually key in all of these different conversational branches, but to think about it more like something like a wiki where you can kind of scrape text from the web and assemble it into this kind of giant compendium. So I would say the software program in the 2000s that really exemplified this is what's called Cleverbot where it just hung out on the web talking to people.

• 12:52Speaker 03

  And the ingenious trick that the programmer used was that everything you said to Cleverbot was going into the database of things that humans say. So if you show up and you're like, "Howdy, how's it going?", it tags that in its database as "this is a conversational opener that humans tend to use."

• 13:09Patrick O'Shaughnessy?

  Yeah, it's almost like supervised learning in like some kind of interesting way.

• 13:12Speaker 03

  Yeah, and every time it would then say something to you, you'd react and it would tag that in the database as "this is a human reply to this sort of thing." And you would essentially combat these bots in a totally different way. If you were trying to explore the breadth of knowledge, you would be amazed at what they could competently respond to. It's basically anything that someone on the internet talked about over an 11-year span is in there. So if you say, "What's the capital of France?", it would say, "Paris." If you said, "What's 2 plus 2?", it would say, "4." There's an example that I give in the book, which is I asked it, "What is the capital of Romania?", and it said, "Budapest? I don't know." Which, and the real answer is Bucharest.

• 13:56Speaker 03

  And so that's the kind of graceful degradation that you expect from a person but not a computer. And if you gave it popular culture things, if you said a line of Bohemian Rhapsody, it would give you the next line. You'd say, "Thunderbolt and lightning?", it would say, "Very, very frightening." And this was just kind of dazzling at the time, just the breadth. But the tell in this case was... That you got the sense that you were not talking to the same person over time. Because indeed, you weren't. It was kind of this mosaic of the entire internet. And so sometimes you would see it spell the word flavor with the British U, sometimes with the American spelling. Sometimes it would claim to be female, sometimes male, and so forth.

• 14:37Speaker 03

  And so the maxim here is that you get the sense, not that you aren't talking to a person, but that you aren't talking to a person. And this was something when I was taking the Turing test, that I was hyper-conscious of trying to come across, not only to give good answers to the questions that I was posed, but to give answers that fit together and presented a picture of a single coherent individual, where the things I knew or didn't know made sense as part of this single life history. Now, I think we are starting to move into what I would think of as kind of a recognizable third era of text systems, where you have, particularly with the rise of deep learning and neural networks,

• 15:21Speaker 03

  you can feed huge amounts of text into a system in a much more nuanced and sophisticated way than simply cataloging it as a database. So in this case, you can have a text system that is perfectly capable of saying things that it had never explicitly seen before, just by using this neural network model that had been trained on this corpus of text. And if you train it on a reasonably coherent corpus, then it's going to appear reasonably coherent. And the significant milestone on my mind when I'm thinking about this is a system that, just a couple of months ago, was published by OpenAI called the GPT-2.

• 16:02Speaker 03

  And famously, they did not publish their code because they were worried about misuse. So this is a significant milestone for the publication norms of the AI field as we start to inch closer to something like AGI. There are now these ethical questions around Doing science where the OpenAI team felt the potential for bad actors to get a hold of this system, which can generate infinite amounts of very plausibly human seeming text on any subject and further destabilizing online discourse and so forth. And I think that risk is totally real.

• 16:41Patrick O'Shaughnessy?

  Before you answer the question about why potentially the number might be zero breakthroughs through two, I'm just curious if you think we need an updated test from the Turing test and or if there already is one out there in the world where we'd be able to say, okay, this is AGI, this clearly isn't some binary threshold. And part of this question is, is there such a test like conceptually, theoretically? Because it seems to me, as we said earlier, like the goalposts are always moving, like things that we would have said are AI 20 years ago happen and we're like, no, no, no, that's not AI. We get that now. It's still clearly not a human. So talk about what that threshold looks like and whether you think we can actually ever get to it.

• 17:16Speaker 03

  The Turing test as a threshold is very controversial and many people have argued over the last few decades that it needs replacing with one of any other various proposals. There's one alternative test that's called the Winograd Schema.

• 17:29Patrick O'Shaughnessy?

  Not as catchy.

• 17:30Speaker 03

  It's not quite as catchy, but these things often have the form of brain teasers. So it'll be like, what is bigger, the moon or a pizza? Things that are obvious, but if you've spent your entire life living on the internet, never experiencing anything firsthand, it might be trickier than it would seem. There are questions that involve like resolution of ambiguous pronouns. So if I say, I opened the oven that had the pizza in it and I put it on the table, what did I put on the table? The oven or the pizza? There's all sorts of questions like this, or like the police arrested the protesters because they were being violent. Do I mean the police were being violent? Do I mean the protesters were being violent?

• 18:11Speaker 03

  There's a lot of these interesting questions that come up even in machine translation around ambiguous grammar that turn out to be surprisingly deep questions that require world knowledge, actual experience of the world, or at least that's the idea. What I found Eerie about the result that OpenAI published with this system GPT-2 was that with a single generic model that they trained, they were able to get world-class, state-of-the-art results in something like 18 out of 20 of the leading computational linguistics tests. So, things like I described, things like you read a short story and then it asks you which of four sentences is a better summary of that story,

• 18:55Speaker 03

  things that almost look like an SAT type of a question. There are 20 different benchmarks out there in the linguistics community, and a single model was able to get record-breaking performance on 18 out of those 20 benchmarks. So, that is the kind of thing that gives me some evidence that we're inching towards something like AGI. And I think I've stood up for the Turing test as a valuable benchmark. I know many people within computer science and beyond are dubious about whether it has ever made sense as a benchmark or still makes sense as a benchmark. But I think Turing saw pretty clearly that language is a medium for accessing all different kinds of intelligence. You can ask someone verbally to do a math problem.

• 19:39Speaker 03

  You can ask someone verbally to produce some kind of art, and then they can produce verbal art. That it doesn't necessarily capture the entirety of the human experience. There are certain things arguably that can only be expressed through dance or whatever. But it sure seems to capture a lot. And so, I think that from a practical perspective, I think the Turing test has been passed. When you are using Twitter or Reddit, it really isn't clear whether the speech that you're reading was written by the person that the account is claiming to be, or was it written by that person's publicist? Or was it written by a bot who has hijacked that person's account? Or what's happening? I think we're kind of past that point now.

• 20:24Speaker 03

  But the broader idea of the Turing test, that language is this nearly universal channel for tapping into all of these different types of intelligence, I think that's as true now as it was then. And I think That will continue to be kind of one of the main ways that we will demonstrate something like AGI in the future.

• 20:44Patrick O'Shaughnessy?

  Let's plan an axiom and say that we achieve it, that kind of how we think about AGI is achieved, and now we have this tool. Let's talk about what the very interesting positives and very interesting potential negatives of this technology could be. As you already mentioned, OpenAI is guarding this for fear of nefarious use. So maybe we'll start with the negative because it's visceral. So what are the specific concerns? I know you're writing about some of these ideas now. I always loved Isaac Asimov's rules. So what is sort of the state of the art of thinking, whether it's from people like Nick Bostrom, like yourself, about things we should really be concerned about as a species when it comes to AGI? And then we'll also talk about the great potential benefits.

• 21:20Speaker 03

  The way that the great worry has been framed by people like Nick Bostrom, people like Eliezer Yudkowsky, is through this idea that you have this all-powerful maximizer that will maximize whatever objective function you want to give it. However you're keeping score, it will find some way to maximize that score. And the field of AI has a long and colorful history of totally demented ways of finding some loophole that maximizes the score while nonetheless doing nothing like the behavior that you originally intended. There are examples where, OpenAI published an example where they were trying to train a program to play this boat racing game.

• 22:02Speaker 03

  And the objective of the boat racing game is to win the race, but it's very hard to encode what it means to be winning the race. You have to somehow get the program to recognize, I'm on a lap. What is a lap? Where are the other people? Because it's circular, they might seem like they're ahead of you, but you're actually about to lap them. So there's a lot of nuance that goes into actually defining that, what it means to win. So instead, they use the proxy of score. So they said, well, just try to maximize the points that you win in this game. And their program found this little weird cul-de-sac where it could just spin around doing donuts forever, collecting these power-ups while everyone just passed by. And you tune in hours later, and it's just doing these donuts, collecting these power-up points infinitely. So there are a lot of cautionary tales of things like that.

• 22:47Speaker 03

  I mean, earlier in the Nineties, there was a group that was working on sort of evolutionary artificial life where they would create these kind of rudimentary organisms that would do various things, kind of simple geometric environments. And one of them was they wanted to evolve an organism for fast land travel. So they would reward it for how quickly it could reach some predefined goal point that was like 20 meters ahead horizontally. And the organism that developed was like this giant column that just tipped over. And so it hadn't learned anything about land travel, but it had learned a very quick way of getting 20 meters in front of you, which is just be 20 meters high and fall on your face.

• 23:27Speaker 03

  And there is just this catalog of these sorts of things where a programmer says, okay, that's not what I meant. You did what I asked, but not what I meant. And so there's a fear among people that as these programs get increasingly competent, the stakes for making that kind of mistake will go up to a potentially catastrophic level. So there's this famous thought experiment of the paperclip maximizer, which the Humble Paperclip Factory develops this super powerful maximizer. And they say, we'd like you to boost our paperclip output this quarter. And one thing leads to another, and now every atom in the invisible universe has been turned into a paperclip and there's no sentient life anywhere. It's all just paperclips.

• 24:08Speaker 03

  And Eliezer Yudkowsky has this famous quote, "The AI does not hate you, nor does it love you. You're simply made out of atoms that it can use for something else." And I think this sort of concern of how are we going to get the objective function right went from being perceived as sort of a fringe idea to increasingly becoming a standard part of the way that machine learning engineers are being trained.

• 24:35Patrick O'Shaughnessy?

  Like a kill switch, basically.

• 24:36Speaker 03

  I mean, that's one idea, and there are a number of ideas. So the idea that there should be some sort of off switch. There's a lot of technical literature that's being written, starting around kind of 2015 through 2017, that were a flurry of papers on if the AI has an off switch, but it has a way of preventing you from hitting it, what do you do to incentivize this? Robot to allow you to switch it off. And one of the critical ideas there is that the AI needs to be uncertain about what it is that you want, such that if you try to turn it off, it can interpret that as evidence that it has the wrong idea about what you wanted it to do. Otherwise, if it's totally confident that it knows what you want it to do,

• 25:20Speaker 03

  then any attempt of yours to switch it off, it will be like, no, no, no, no, I'm helping you. Let me help you. So this idea that the objective function that you give the system needs to encode some degree of uncertainty is increasingly becoming, I think, pretty widely accepted. Certainly in the technical AI safety community, which is a subset of AI more generally. But people are, I think, motivated by thought experiments like this paperclip maximizer to rethink some of the fundamental premises of the field. The field has this premise that you have this objective function, which you maximize as aggressively as you can. And it's up to the human engineers to be wise about crafting that objective function

• 26:04Speaker 03

  just so that they've eliminated all of these possible loopholes. Increasingly, the field is moving towards rethinking some of these fundamental assumptions and saying, what does it mean to build a system that doesn't have an objective function as such, but observes the way that I behave and tries to make inferences about the things that I'm trying to accomplish based on what it sees me doing and tries to be helpful, but maintains some uncertainty over exactly what I want or what I appear to be doing. And as I said, that uncertainty turns out to be very critical. So there's some promising work that's now happening on these things.

• 26:38Patrick O'Shaughnessy?

  So you've mentioned a number of terms, which I didn't expect to go as far down this rabbit hole as we have, but I'm glad we did. You mentioned awareness, experience, some interesting terms. And we do a lot of work with optimization. And there's objective functions, there's also constraints. And you've talked about some of these constraints already. Asimov famously, that was maybe the strength of his system was that it cannot harm, you can't harm humans in whatever you do. So you can't maximize paperclips if it's going to kill off humanity. So my question is, the degree to which self-awareness and experience that the machine may have or not have factor into this. And putting your philosopher hat on, if it does have self-awareness, let's say that's a threshold, a machine becomes aware of itself. Might that then self-regulate?

• 27:25Patrick O'Shaughnessy?

  Such that it perceives some objective good and bad spectrum in the world. And if it's told to maximize paperclips, but becomes self-aware, it stops because it realizes this is in violation of just like objective good in the universe or something. I realize this is an insane sounding question, but self-awareness maybe is the true threshold for AGI, whether or not that holds credence.

• 27:45Speaker 03

  Yeah, there's a lot of different threads that intersect in what you're saying. So I mean, there's one thing is this question of what's called moral realism, which is are moral truths, objective facts that exist in the world, the way that physical and mathematical truths exist. Some philosophers are moral realists, some are not.

• 28:03Patrick O'Shaughnessy?

  Sam Harris would be a moral realist, right? With the moral landscape?

• 28:06Speaker 03

  Yeah.

• 28:06Patrick O'Shaughnessy?

  Where there's peaks of the, yeah, okay.

• 28:09Speaker 03

  Right. The argument that we can use science to uncover

• 28:12Patrick O'Shaughnessy?

  Objective goods.

• 28:13Speaker 03

  Facts about ethical behavior. If that's true, then I've heard people make the argument, I'm not necessarily sympathetic to it myself, but I've heard people make the argument that let's just let AGI do its thing because it will be smarter than us and so it will inevitably discover these moral truths that we don't know about and we should just let it rip. I don't find that persuasive for a number of reasons, but that's a position that people take. The self-awareness question is interesting to me. And so there's two threads that I would kind of identify within that. So one is the question of consciousness and I don't even know where to begin on that. It may well be the case that we, by the end of this century, have systems that are, without

• 28:56Speaker 03

  any argument, as cognitively flexible as we are, and then some, but we have no idea if anyone's home, if there's any light on, so to speak. It may well be that we have to live in a very weird limbo of not knowing whether these systems are, do any sort of moral respect. So in ethics, this is known as being a moral patient. So we may have to live with a lot of kind of cognitive dissonance around the question of, do we owe these systems anything and do we need to treat them well? I think that's a very interesting question and I, in some ways, I would be just as surprised if we got the answer to that question as if we didn't. It would be very weird to live in a world in which these systems were everywhere and we didn't know if they were conscious or not. They maybe claimed to be, but we had no way of knowing.

• 29:41Speaker 03

  It would also be very weird if science just told us consciousness is this. This process or this pattern is consciousness. We've figured it out. That would be very strange also. So, one of two very strange things will happen and we'll have to find out which of those it is.

• 29:57Patrick O'Shaughnessy?

  I would encourage people interested in this, we'll stop here on consciousness, but if people are interested to read Hofstetter and read Dan Dennett, and there's some fascinating stuff on this question that you can spend a whole lifetime trying to figure this out and maybe get nowhere, but you'll certainly be interested along the way.

• 30:11Speaker 03

  Yeah, for me, I don't know where to even begin to get traction on that argument, but it's fascinating. And the other side of what you're talking about with self-awareness is this idea of how does a system, setting consciousness aside, take its own existence and its own behavior into account when it thinks about how it interacts with the environment? And that has been another one of these long-held assumptions within the field of machine learning that people are just now starting to poke at. So typically, the framework that's used in machine learning treats the learning agent itself as kind of outside of the environment. The environment can't do things to it. It can't be changed in dramatic ways by the things that it encounters.

• 30:56Speaker 03

  But that's not true of real life. You can do things that kill you. You can do things that change what you believe or what you value. And when you're in a situation, it's important to understand the role that you yourself play in that situation. You're not just confronting it as if from the outside. And so there's been a research agenda. I know people at the Machine Intelligence Research Institute have an entire vein of research now into what they call embedded agency, which is what does it mean for a system to reason about itself as part of the environment in which it's behaving? And that's the kind of thing which will become... Increasingly relevant as AI systems become increasingly powerful, they will take up more space in the environment that they're operating in,

• 31:41Speaker 03

  and they will need to pay more attention to themselves in that way. And so this is, I think, very nascent. There's not a lot that we know at this point. I would say the folks at Miri have been working on this for a handful of years at this point, and I think they're starting to build some track record of work in this area, but it's really early. And so, I mean, this is part of what's so exciting for me about the point that we find ourselves at in the history of AI, is we've made enough progress in doing things the classical way that we're starting to see the weaknesses in the classical paradigm. And we're saying, OK, does it need to have an objective function? Does it need to be treated as exterior to its own environment?

• 32:25Speaker 03

  What does it mean to reimagine what a system could be like if we relax some of those?

• 32:30Patrick O'Shaughnessy?

  What would be the kind of skeptics' take on all this? So, as I mentioned, I do a lot of work with the fanciest of these new algorithms. And very often what we find is that this stupid, simple, old linear regression does almost just as well as the coolest deep neural net or whatever. And that I love the idea of AGI, both positive and negative. Thinking about it, it's just like an objective function maximizer. It's sort of this magic box. You feed it an objective function, and it just gets you as much of that as possible using all the available resources, like on the internet or in the world. It's a really interesting thought experiment. Our experience is that magic boxes don't exist today, that it's incredibly hard to build a system that gets you the thing you want. It often takes a lot of smart people a very long time. So what would be like the most skeptical viewpoint on AGI,

• 33:14Patrick O'Shaughnessy?

  where someone says either it's impossible, it'll never happen, or it's way further away than we think it is today?

• 33:20Speaker 03

  This sort of comes back to this question of are we zero to two or more significant breakthroughs away? There are many Obvious criticisms that can be made of the current state of AI in terms of the things that it can't do. Humans are really good at operating in situations where the number of actions you could possibly take is almost infinite. So, one of the great achievements of the last 20 years was going from a system Deep Blue that was superhuman at chess to a system AlphaGo that was superhuman at Go. One of the biggest differences between those two games, not the only difference, but one of the major differences is what's called the branching factor. So, in chess, you have at any one point in time about 30 moves available to you, and then your opponent has about 30 replies.

• 34:04Speaker 03

  In Go, you have, on average, 200 or more possible moves, and then your opponent has 200 or more possible replies. And so, the exponential just blows up way faster. And we've now got methods that are impressively capable of dealing with that kind of complexity. But you compare that to something like just existing as a human being, having a conversation. How do you even cap the number of possible things that could be said next? I mean, there's a nearly unlimited amount of actions that you have to choose from. So, I think there's a reason to feel skeptical about that, that we don't really have great methods for dealing with infinite action spaces. Some of the classical algorithms

• 34:48Speaker 03

  that are useful in game-playing scenarios, like there's one called Q-Learning, which basically says, "Try every possible action in every possible situation nearly infinite number of times, and just build up a memory of which actions tend to lead to high-expected rewards." Well, that algorithm makes absolutely no sense if you're living a normal life where most critical junctions only come around once, or you have a nearly infinite number of things that you could do in any situation. So, the idea of just try everything an infinite number of times and see what happens, that just doesn't make sense as a strategy. So, that's kind of a non-starter. So, I think that's part of the skeptics' argument,

• 35:30Speaker 03

  is like, these sorts of things are really cute in the context of Atari or even Go, but they quickly disintegrate when faced with the real world. I think there's some truth to that. Humans are really good at what's called hierarchical planning. So, when we think about what we want to do... We think about it in this very abstract level, not at the motor command level. So it's like when I woke up today, I wasn't planning like the motor actions that I needed to achieve in order to come here and talk to you. And even now, I'm not thinking about the motor actions that I'm taking and informing words and so forth. That we plan in this hierarchical way. Okay, first I need to make coffee, then I need to get the subway, then I need to, you know, whatever.

• 36:14Speaker 03

  We don't really have good systems for doing this kind of hierarchical planning. We have systems that are pretty good for the equivalent of muscle memory. So anything that you can do sort of instinctively, we're pretty good at. And AI had an earlier history of working on these sorts of more deductive planning-based systems. A lot of that work has kind of fallen by the wayside in favor of the deep learning stuff, which has its own strengths and weaknesses. I think a lot of people are saying there needs to be some grand synthesis of these two research programs that have until now basically existed in two different silos. The skeptic, I think, would say there's a hell of a lot of work to be done at reconciling

• 36:56Speaker 03

  those two things, and we're only at the very beginning of doing that. I think the, whatever's the opposite of the skeptic, the booster, their response would be with a big enough neural net and enough time and enough compute, the net will just figure out how to do hierarchical planning, if that is important. And maybe it's not that important, you just think it is. That's the gauntlet that's been thrown down is how much can you do with a big net, a lot of examples, and a lot of electricity? Is it just a matter of pumping more electricity into enough GPUs to train the net? I think that's a pretty sharp dividing line between research communities right now in AI, that there are some that think, yeah, it's just a matter of enough compute.

• 37:40Speaker 03

  Other groups saying, you're just going to get the wrong answer faster. But we need to like totally rethink the actual structure of how we are building these.

• 37:46Patrick O'Shaughnessy?

  So a very practical question. With everything that you know about all of this that we've talked about, everything you've researched, what advice would you give to people building careers? So we're in a political cycle now where things like basic income are being discussed. There's all sorts of fascinating books, Brittenley, Rise of the Machines or Robots, or I can't remember the name of that book, but fascinating, thinking about how all of this is going to affect work in the future. So really, I guess what I'm asking is, in your view, what are like the most defensible areas of human activity? Whether that's some sort of creativity or asking great questions, coming up with the objective functions that you then feed the machines, like what would you recommend people focus on as they think about either early or late in their career? Sort of adding value?

• 38:30Speaker 03

  There are sort of two ways that I can approach this question. So my second book, which we haven't really touched on at all.

• 38:36Patrick O'Shaughnessy?

  We're gonna get to, yeah.

• 38:37Speaker 03

  Yeah, it's called Algorithms to Live By, and it looks at things like career decisions from an explicitly algorithmic perspective. So we can get into this more later if you're interested, but there's this paradigm called the explore-exploit trade-off, which is how much of your energy do you spend gathering information? How much do you spend just sort of committing based on the information?

• 38:55Patrick O'Shaughnessy?

  Can we go there now? Because that was my favorite chapter in the book. I like prescribe it to everybody I ever meet. I think it's the most interesting of the many interesting algorithms in that book. So could you explain that one in some detail?

• 39:04Speaker 03

  Yeah, so there's a number of decisions that we face throughout life that take the form of a tension or a balance between trying new things and committing to the things that seem to be the best. Whether that is where to go out to eat, do we go to our favorite restaurant? Do we try a new restaurant? Who do we spend time with? Do we reach out to a new acquaintance we'd like to get to know better? Do we spend time with our close family or our best friend? And I think the same thing is true in investing. The same thing is true in managing your time and your career. So. I think the structure of this problem, kind of this iterated decision that you get to make over and over again about, do you continue to put energy into the things that

• 39:48Speaker 03

  have seemed promising, or do you spend your energy trying new things, is applicable across a wide range of areas in human life. You can even think of, for example, a clinical trial as having that same structure. And indeed, the FDA has been increasingly interested in looking over the disciplinary fence at the computer scientists and saying, maybe those algorithms that you're using to optimize ads could also be used to optimize human lives. So the way that a computer scientist approaches this question is through something that's called the multi-armed bandit problem. And it's a sort of a weird name. It comes from the slang for a slot machine as a one-armed bandit. So in the multi-armed bandit problem, you walk into a casino that has all these different slot machines. Some of them pay

• 40:33Speaker 03

  out with a higher probability than others, but you don't know which are which. And so quite simply, what strategy do you employ to try to make as much money in the casino as you can? Well, it's going to necessarily involve some amount of exploration, trying out different machines to see which ones appear to pay out more than others, and exploitation, which to a computer scientist doesn't have the negative connotation that it has, you know, in regular English. Exploitation, meaning just leveraging the information you've gained so far to crank away on those machines that do seem to be the best. Now intuitively, I think most of us would recognize that you need to do some amount of both. But it's not totally obvious what that balance should look like in practice. And indeed, for much

• 41:16Speaker 03

  of the 20th century, this was considered not only an unsolved problem, but an unsolvable problem, and sort of career suicide to think about it. During World War II, the British mathematicians joked about dropping the multi-armed bandit problem over Germany as like- Just to occupy them. Yeah, the ultimate intellectual sabotage, just waste the brain power, like nerds snipe all of the German mathematicians, essentially. And in many ways, to the field's own surprise, there came a series of breakthroughs on the multi-armed bandit problem through the second half of the 20th century. And now we have a pretty good idea of what exact solutions look like, given a number of constraints, but also what sort of more general flexible algorithms look like.

• 41:58Speaker 03

  So the critical, I would say, insight into thinking about this problem is that your strategy should depend, in some ways entirely, on how long you plan to be in the casino. So if you feel that you have a long time ahead of you, then it's worth it to invest in exploration, because if you do find something great, it has a long horizon to pay out into the future. On the other hand, if you feel that you are about to leave, then the return that you would get on making a great new discovery is going to be much smaller, because you have fewer opportunities to sort of crank away on that handle once you find it.

• 42:40Speaker 03

  So for a number of reasons, we should naturally segue or transition from being more exploratory at the beginning of a process to more exploitative at the end. And I think that's an intuition that makes sense, but the math bears that out very concretely. And it's been interesting to see this idea that emerges in computer science in the late 50s through the 70s is now getting picked up by psychologists and cognitive scientists who are interested in human decision-making. So, for example, Alison Gopnik at UC Berkeley, who studies infant cognition, has been thinking about the explore-exploit trade-off as a framework for how the infant mind works.

• 43:23Speaker 03

  That if you think about how children behave, we have all these stereotypes about children are just kind of random, they're generally incompetent at things, and there's a huge literature that shows that they have what's called a novelty bias, they're just relentlessly interested in the next thing, and the next thing, and the next thing. And rather than viewing that as just kind of low willpower or attentional control, you can view it as the optimal strategy. And if you've just burst through the doors of life's casino and you have 80 years ahead of you, or whatever, it really does make a lot of sense to just run around wildly pulling handles at random. And the same is true for being in the later years of one's life, that we have a lot of stereotypes about older people, that they're very set in their ways, very habitual, resistant to change.

• 44:08Speaker 03

  There's a psychology literature that shows that older adults maintain fewer social connections than younger people. And it's tempting to view that kind of pessimistically as, oh, I guess it must be kind of sad getting older or lonely or something. In fact, if you build an argument from the mathematics, you can see that older adults are simply in the exploit phase of their life. And they are, again, doing the optimal thing, given where they are in that interval of time. And so you have psychologists like Stanford's Laura Carstensen appealing to the explore-exploit trade-off to make this argument that, no, older adults know exactly what they're doing and they're very rationally choosing a strategy that makes sense, given where they are. They have a lifetime's exploration behind them. They know what they really like. They know the people and the connections that matter to them.

• 44:53Speaker 03

  And they have a finite amount of time left to reap the fruits of some new connection or a new discovery. So they're very deliberately enacting this strategy. And the math should predict that, on average, older adults are happier than young people, despite our preconceptions. And her research bears out that that appears to be the case.

• 45:12Patrick O'Shaughnessy?

  So if you translate this kind of fascinating trade-off to career and life advice, you've done some of it already. Obviously, the bias would be, when you're young, try a lot of stuff. And maybe when you're older, think more about using the best solutions that you've found versus trying to find new ones. Fascinating how this maps on to like the life cycles of businesses. For example, in the business context, explore might be innovation and exploit might be run the same playbook to earn high returns on capital or something, on something you know works. And it seems like you kind of always want to be handing off to a next batch of exploration or innovation, while thoughtfully maintaining something that you know works. And in a business context, if you want to survive for a very long time.

• 45:52Speaker 03

  Yeah, there's a couple of things here that I think are interesting in a business context. One is that implicitly, the casino framing that I've described.

• 46:00Speaker 03

  Assumes that those probabilities are stable and fixed. And of course, we know that the world is not stable and not fixed, that things change over time. And this is true in our personal lives as well. The restaurant that you love gets a new line cook and the burger's not as good. These things shift. And so, this is known in the mathematical literature as the restless bandit problem. So, how do you play this game when these probabilities are, let's say, on a random walk, so they're drifting? And this is a very interesting case where the theory has not yet consolidated, but humans seem to, in practice, have no problem, like if you put people in a lab and give them a restless bandit problem. They have no trouble making choices within that environment, but we don't yet know what

• 46:44Speaker 03

  the mathematics of the optimal solution looks like. So, here's a case where the computer scientists and the mathematicians are asking the cognitive scientists, "What are your models for how humans are actually approaching this? Because there may be some insight that we can use from the theory side." One of the implications of thinking in this way, that I think is particularly relevant in a business setting, is if the interval of time you perceive yourself to be on determines the strategy that you should employ, then it should be the case that if you observe someone else's strategy, you can infer the interval that they're optimizing over. So, we give the example in the book of Hollywood. So, most people have noticed, it feels like we're living through this deluge of sequels.

• 47:29Speaker 03

  X-Men 12 and The Avengers 7 and whatever. So, I spent a while digging through the data and it turns out that this is objectively true. There's a sea change in Hollywood that, I want to say in 1980, two of the top 10 grossing films were sequels. By 1990, it was like six. By the year 2000, it was eight. And I think most recently, it was like all 10, or something like this. And we can infer from that, that Hollywood has taken like a very hard turn towards an exploitative strategy, that they are milking their existing franchises, rather than investing money speculatively to try to develop new franchises that will last them into the next few decades. So...

• 48:15Speaker 03

  From that, I think it's reasonable to infer that, for example, movie ticket sales are declining, which turns out to be the case, that Hollywood perhaps correctly perceives itself to be at the waning time of this kind of golden era of cinema going. And if that's true, then they really should invest all of their money into just squeezing everything they can out of the existing franchises. So that's an example that I think is applicable more broadly. So you can look at different industries, different corporations and see, oh, they've really cut their R and D budget and they've given that money to marketing, let's say. Well, that'd be an indication that they feel that that area has matured or has plateaued or something.

• 49:00Patrick O'Shaughnessy?

  Yeah, that's a fascinating research idea that we've sniffed around. It's very hard to draw firm conclusions, even though there's a lot of companies, sample sizes actually aren't all that big. But fascinating idea that you can observe trends and we've certainly seen one. One exploit strategy in public markets might be paying dividends, buying back shares, consolidating the share count versus spending on capex or R and D. What's interesting is those firms actually outperform. They do much better than the ones that are spending what we would call recklessly, which maybe is a bit counterintuitive. But I love this idea of the broad numbers of backwards and forwards, applications of exploring versus exploiting. You mentioned that that was sort of one of two ways to approach my original question of kind of advice for a career. What would the second one be?

• 49:43Speaker 03

  Well, the second avenue is sort of totally different from this way of thinking, which is just what will the impacts of something like AI or AGI be on the economy? And I'm reminded of McKinsey did a report on which jobs they thought would be the most robust. The big picture thing that was interesting to me is that it cuts across the traditional class lines. It's not a white collar versus blue collar thing. It's not an upper middle class versus lower middle class thing. It's very sector dependent. And I remember that what they identified as the most resilient or robust jobs, at the top end, it was gardener, legislator, and psychotherapist. I thought that was very fascinating and it sort of. This eclectic mixture of things.

• 50:27Speaker 03

  I don't think of myself necessarily as a prognosticator about these sorts of things. But my way of thinking about it is that there is a lot of kind of human machinery around how capital moves, how laws get made, how licensing and permitting happens. That's still done as sort of a human negotiation level. I know a guy, I'll talk to Joe and we'll sort it out. I think humans will maintain oversight of these flows of power and capital, even if the actual value is being created by software.

• 51:07Speaker 03

  So that's my somewhat off the top of my head thought is position yourself closer to the flow of that value than the actual creation of the value, which is maybe somewhat counterintuitive. I don't know as far as the question of UBI. I don't have a great intuition for that. I am interested in Norway has this sovereign oil wealth fund. What would something like that look like? I don't know. In many ways, I think there is already a restlessness in the labor force. That a lot of the careers that employed some of the most numbers of people are the most vulnerable. So people who drive cars or trucks, people who work in warehouses doing sort of pick

• 51:51Speaker 03

  and pack things, a lot of those jobs are just one innovation away. And it's not clear to me. I mean, there's also going to be a political response as well as just a pure sort of economic response. So I grew up in New Jersey where there's, at least in my childhood, there was a really robust toll collector union. And they had machines where you could toss your change in a bin and it would automatically sort your change and give you whatever you needed back from that. And there was an effective effort to unionize the toll collectors so that you still had a human being in the booth counting out your quarters and so forth. And that's an example where it's not for lack of technology. We had a coin sorting machine, but there was kind of a political process

• 52:33Speaker 03

  that was directing the actual level of implementation. So I often encourage people to remember that. People will.

• 52:43Speaker 03

  Fight to use licensing requirements and regulation to maintain those things despite the actual technological capability having radically changed. So it's very hard for me to know which areas will look shockingly different than the world looks today and which things will be in some ways shockingly backwards for their time because we've had for political reasons to sort of hold the line. But as to what exactly that looks like, I'd say my guess is as good as anyone's.

• 53:13Patrick O'Shaughnessy?

  So to complete our arc of starting very theoretically and ending more practically, I'd love to close with one, two, maybe three final examples from your second book, Algorithms to Live By. It's certainly the one that I've recommended like crazy and I think people would just all love it, even if it's one of those beautiful books where you should read the whole thing, but it's also feasible to skip ahead a chapter if one thing isn't resonating or whatever. So we talked about explore exploit. Thinking back on it, are there a couple that you found most interesting and or useful? Those are sort of the lenses that I'd love you to to apply to the many chapters and maybe close with two examples.

• 53:46Speaker 03

  One thing that comes to my mind is the idea of what's called optimal stopping. The multi-armed bandit problem and the explore exploit trade-off presumes a framing that's highly iterative. You can pull the handles again and again and again. You can kind of fluidly go from one machine to another and back. There are many decisions in life where you are forced to make a single binding commitment. That could be anything as banal as pulling into a parking space. It could be something like purchasing a house or signing a lease. It could be something like marrying your spouse. And there's kind of a separate mathematics of cases where you need to find the right moment in time to go all in and commit to an option and no longer gather any further information.

• 54:30Speaker 03

  And I think that mathematics is very kind of instructive, both in a specific way, but also as a broader set of principles. There's this very famous result called the 37% rule, which is that, let's say you're looking for an apartment and it's a really competitive marketplace. I'm from San Francisco, so it's this kind of hellishly competitive situation where you have to just force the check into the landlord's hands. If you're in a situation like that, where you encounter a series of options one by one, and at each point in time you must either immediately commit and then never know what else might have been out there. Or decide to walk away and keep exploring your options, but lose that opportunity forever. What do you do to try to end up with the best thing possible,

• 55:14Speaker 03

  even though you you won't necessarily know at the time whether you've found the best option that might be out there? There's this beautifully elegant result that says that you should spend the first 37% of your search, or 1 over E, noncommittally exploring your options. Don't bring your checkbook. Don't commit to anything, no matter how good it seems. You're just purely setting a baseline. And then after that 37%, whether it's 37% of the time that you've given yourself to make the decision, or 37% of the way through the pool of options, be prepared to immediately commit to the very first thing you see that's better than what you saw in that first 37%. This is not just an intuitively satisfying balance between looking and leaping. This is the mathematically optimal result.

• 55:58Speaker 03

  And there are strategies like that that I think are wonderfully crisp in the recommendation that you give. But they, of course, rest on this bed of many different assumptions about exactly how the problem is structured and exactly what your goals are.

• 56:12Patrick O'Shaughnessy?

  And specifically, it's better than anything you had seen in the prior 37%?

• 56:17Speaker 03

  Yeah.

• 56:17Patrick O'Shaughnessy?

  Got it. Okay.

• 56:18Speaker 03

  Yeah. This rule presumes that your entire goal is to maximize the chance that you get the very best thing in the entire pool. But it comes with a 37% chance, of course, that you have nothing at all because you've passed it by. Yeah, and you'll never find something that's good. And now many people would find that unacceptable. And so, of course, we can go down the rabbit hole of how do you modify this? And the solutions get less and less clean as you wiggle the assumptions around. But I think there's broadly just this intuition that one of the highest level takeaways for me from working on the book and just thinking in computational terms about decisions in my own life is some decisions are just hard. That the classical optimal stopping problem,

• 57:03Speaker 03

  it turns out, due to a weird mathematical symmetry, that if you follow the 37% rule, you will only succeed 37% of the time. The other 63% of the time, you'll fail. And that is the best possible strategy you could enact in that situation. So there's no better that you could possibly do than failing 63% of the time. In a weird way, that's some measure of consolation because often in real life, when we find ourselves not getting the outcome we wanted, we can rake ourselves over the coals or try to reconstruct our entire thought process and say, where did I go wrong? I think it's some comfort that computer science and mathematics more broadly can, in effect, certify that you were just up against a hard problem. And I think broadly, that, for me,

• 57:48Speaker 03

  is some measure of comfort, that if you have the kind of the vocabulary to understand the type of problem that you're facing and you have some intuitions about the general shape of what optimal solutions look like, then even when you don't get the outcome that you wanted, you can, in some sense, rest easy because you knew that you followed the appropriate procedure or the appropriate process for dealing with that situation.

• 58:11Patrick O'Shaughnessy?

  So that's an elegant place to end. And so I'll move to my final closing question that I ask everybody, which is for the kindest thing that anyone's ever done for you.

• 58:18Speaker 03

  The kindest thing that anyone has ever done for me? I want to offer some very concrete thing, but I would have to think pretty seriously about that. What comes to my mind is just the idea of being loved unconditionally by my parents. I think most of the difficult things that I've faced in my life in one form or other took the form of not being totally comfortable with who I was, whether it was in school, feeling self-conscious about being a nerd and not answering a question in class even though I knew the answer because I sort of wanted to fit in or didn't want to seem like I was enthusiastic about school because it wasn't cool to act like that.

• 59:01Speaker 03

  Throughout my life, I think some of the most meaningful things to me have been ways that I've come to just accept who I am. And so I think one of the kindest things that you can do for someone is to see them exactly as they are and accept them. And to be able to do that for oneself is tremendously powerful.

• 59:18Patrick O'Shaughnessy?

  Yeah, I haven't heard it put quite so eloquently as you just did, but unconditional love, or some version thereof, is a common answer, and I think for good reason.

• 59:25Speaker 03

  Yeah.

• 59:26Patrick O'Shaughnessy?

  So thanks for that great answer. Thanks for a fascinating conversation. I love the ones that barely touch investing, and this has been a great example. So I hope we can do a round two someday, maybe after the next book. It's been a pleasure.

• 59:37Speaker 03

  Likewise. Thank you.

• 59:40Speaker 04

  Hey, everyone. Patrick here again. To find more episodes of Invest Like the Best, go to investorfieldguide.com/podcast. If you're a book lover, you can also sign up for my book club at investorfieldguide.com/bookclub. After you sign up, you'll receive a full investor curriculum right away, and then three to four suggestions of new books every month. You can also follow me on Twitter, @Patrick_OShag, O-S-H-A-G. If you enjoy the show, please leave a quick review for us on iTunes, which will help more people discover Invest Like the Best. Thanks so much for listening.