Kenneth Stanley: Set The Right Objectives

Artificial intelligence researcher and author Kenneth Stanley has argued that “as soon as you create an objective, you ruin your ability to reach it.” So what should you consider when thinking about your objectives, and what will set you up for success? On this episode Stanley discusses how to set the right objectives for your life, why we’re too tied to accomplishments, what role accountability plays in our education system, the value of peer review, why transformative innovations are always counter intuitive, and so much more. Stanley is the co–author of Why Greatness Cannot Be Planned: The Myth of the Objective, as well as the former Head of Core AI Research at Uber AI and the Open-Endedness Team Leader at OpenAI. He has also served as the Charles Millican Professor in Computer Science at University of Central Florida. -- Want even more? Members get early access, hand-edited transcripts, member-only episodes, and so much more. Learn more here: https://fs.blog/membership/ Every Sunday our Brain Food newsletter shares timeless insights and ideas that you can use at work and home. Add it to your inbox: https://fs.blog/newsletter/ Follow Shane on Twitter at: https://twitter.com/ShaneAParrish Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  You might like assessment and accountability and objectives and metrics because they make you feel better because it feels like we're making sure that nothing bad will happen. But you have to recognize it's just a security blanket. It doesn't work. Take off the straitjacket, get rid of the security blanket and just acknowledge how reality works. And it'd be better, I'd prefer if reality worked where if I set an objective, like I could just get to it as long as I'm sufficiently determined to do it. And that'd be really convenient, but it's just not the way reality works.

• 0:43Shane Parrish?

  Welcome to the Knowledge Project Podcast. I'm your host, Shane Parrish. This podcast is about mastering the best what other people have already figured out so that you can apply their insights in life and business. If you're listening to this, you're missing out. If you'd like special member-only episodes, access before anyone else, hand-edited transcripts, including my personal highlights, and other member-only content, you can join at fs.blog/membership. Check out the show notes for a link. Today, I'm speaking with Ken Stanley. Ken is the author of Why Greatness Cannot Be Planned: The Myth of the Objective, a book I read and loved, so I reached out and wanted to chat with him.

• 1:26Shane Parrish?

  Ken is currently deciding his next adventure, but recently led a research team at OpenAI. In this episode, we're going to take a simple idea and take it seriously. That idea is the question of objectives. They seem good when they're modest, but things get a lot more complicated the more ambitious they get. We also explore the growing obsession with matrification, why we accept failure in science, but not when it comes to things like politics or education, why trying harder doesn't always help you achieve the outcomes you see, why you can't be so tied to your destination that you're not open to the unexpected and unplanned, and why you should avoid ideas that make too much sense. It's time to listen and learn.

• 2:16Shane Parrish?

  You wrote a book about questioning the value of objectives, which revealed a surprising paradox that objectives are good when they're modest, but things get more complicated when they're ambitious. Can you expand on that?

• 2:28Speaker 00

  Yeah, this is something that people don't talk about that often. People don't talk about the problem with setting objectives. There are some controversies in society. This is currently not one of them. But yet, if you think about it, this is something that we do all the time. Basically, one of the, like, I think deepest facets of our culture is that we think of accomplishment and achievement and discovery in terms of setting an objective and then pursuing it. And I do research in artificial intelligence, my normal job. And in the course of doing that research, we just started to see undeniable evidence that that approach to achievement has some serious flaws.

• 3:11Speaker 00

  And that realization at first was mostly just a kind of algorithmic realization, like, okay, well, this has, this has applications and implications for artificial intelligence. But it dawned on us over time that actually it has a lot more implications than just for artificial intelligence, because it's not just something people do in the algorithms of artificial intelligence, but in basically in life and in our culture. It's what we do all the time. And it started to seem to me, maybe almost urgent, that this is actually brought into like a public conversation of some sort. So that's why myself and my co-author, Joel Lehman, decided to kind of take the unconventional road of writing a book like this, which is not an AI book, and kind of try to provoke

• 3:55Speaker 00

  at least, at least a conversation, if not some kind of change in the way that institutions and people structure what they're doing.

• 4:02Shane Parrish?

  I think we're going to get into sort of some of the drawbacks to this approach and maybe some of the nuances around it. Before we do, you sort of mentioned that some great ideas were never objectives for anyone, at least until they were discovered. Rock and roll, I think penicillin is another one. Can you say more on that?

• 4:21Speaker 00

  Yeah, I mean, it's related to the idea of serendipity. And, you know, in serendipitous kinds of discoveries.

• 4:30Speaker 00

  You weren't expecting to make it. And I think the insight is that this is something that is much more common than the kind of narrative that we tell ourselves about how discoveries, inventions, innovations are made. And actually, like, a lot of what we do that facilitates making these kinds of important discoveries is to actually set us up, set ourselves up for having effective serendipity, which is not the way that we, you know, talk about things when you talk about setting an objective and just moving towards it. So, of course, something like rock and roll, which is a good example, it's not the kind of thing that you could set as an objective because it doesn't even exist as an idea until you run into it.

• 5:11Speaker 00

  And yet, somehow, you know, the pieces were in place, like when Elvis was there on the scene, for him to kind of run into this. And the question, I think, that's interesting is why that happens? Like, what kind of situation leads to that? And what kind of person will take advantage of that kind of situation?

• 5:28Shane Parrish?

  Are all objectives the same?

• 5:31Speaker 00

  No. And it's an important caveat to what I'm saying, that there are many objectives that, I guess I would characterize them as just modest. Like, for example, you know, I wanna be able to run for longer, or I wanna lose some weight, or maybe even, like, I want to get a degree in accounting. Nothing against accounting. But the reason I call those things modest is just because, well, they've been done many times. Like, we know that these are achievable things. And I think those need to be distinguished from what I would call ambitious objectives. So by ambitious, I mean, these are things we don't know how to do. We're not sure how they're gonna get accomplished, even though we want to accomplish them. Curing cancer, or achieving artificial general intelligence, or something like that.

• 6:16Speaker 00

  Like, those are really ambitious. And so, when I critique objectives, or when the book does, and it tries to make it clear, it's really the ambitious ones that we're talking about here. Like, having modest objectives, it shouldn't be affected by this critique. And it would be verging on kind of nutty, cranky type of behavior to try to get rid of those. Of course, you can set modest objectives. But you have to remember, though, that, like, a lot of our society runs on the ambitious ones. Like we are banking on innovation to save us from all kinds of problems and also to deliver us into like a new world of different types. And so we depend on this kind of ambitious stuff to happen. And the fact that we run things as if they actually happen through

• 6:59Speaker 00

  objectives is perhaps self deception that is then really grinding down our efficiency, like ability to really take advantage of the resources that we have to make these kinds of discoveries by recognizing how they actually work.

• 7:12Shane Parrish?

  So the core problem with the ambitious objectives then is that in many cases, trying harder won't help you achieve the outcomes you're seeking. And sort of like a follow on to that, you can't be so tied to your vision of accomplishment that you're not open to the unexpected and unplanned?

• 7:31Speaker 00

  Yeah, so it is true that like one of the principles in the book is that you can actually block your own ability to reach an objective by setting it, which is paradoxical. So grappling with that is, you know, hard and important and something that the book tries to discuss, like how to grapple with that. But yes, we're actually causing ourselves to achieve less in these ambitious cases by setting very ambitious objectives. And one of the things that we do when we set an ambitious objective conventionally is that we would also set some metrics up to measure progress towards that objective. And that's where I think things really get tripped up is these metrics or assessments. And we

• 8:15Speaker 00

  love assessment in our culture. We have a very big assessment culture. And the assessment is basically trying to give us a security blanket so we can feel like we're moving towards the objective and that we're actually making progress. And the problem is that a lot of the time, even if your score on a metric is going up in the short run, it doesn't mean it will all get all the way to the point that you want it to get. It's a fundamental problem. It's called deception and it's a fundamental problem of all complex problems. And so the fact that we rely so much on these assessments and metrics is very deceiving. And can ultimately cause us to invest a lot in a deceptive path, which is actually going to lead to a dead end. And that's why it can actually be, although it's counterintuitive,

• 8:59Speaker 00

  it can actually be bad for you to have a very strict objective that you're assessing movement towards.

• 9:05Shane Parrish?

  Let's sort of make that tangible. One of the examples you give in the book around that is schools and improving student performance. And I'm thinking here that's, you know, both hard to argue with and it also never seems to improve despite billions, if not trillions of dollars. And so progress sort of can't be packed into a single metric. But what should we do instead for something like that? Where we, we also need accountability on behalf of sort of politicians or decision makers who have some sort of skin in the game for their choices.

• 9:38Speaker 00

  That, that word accountability always comes up. Is it, is it sort of like why we feel like we have to have these metrics and assessments? And education is a great example of this. So the, because the education system has, does have an objective. And it's a little bit fuzzy. It's not usually stated explicitly, but you know, I characterize it something like the objective of the education system is for everyone that's in the education system, all of the students to score perfectly on a bunch of assessment tests. That would be ideal if everybody was perfect. Of course, we're never going to get quite to there, but that would be the ultimate perfect objective. What leads you to that outcome? We don't come close to that outcome right now.

• 10:19Speaker 00

  We have many people whose performance is way below what we want to see. So that's our problem. That's what, the, the problem with the education system is. And so we're trying to solve it by being objective. So we say, okay, well, what we need then is we need some standardized tests. We're going to blanket like the entire country, basically, with these standardized tests and other countries to do something similar. And then we have a universal measure or metric that can be used, like to decide whether progress is going well in, in local locations. Like if, like what's going on in your local school district is still being assessed with this kind of like a global test that is used across the nation. You know, then we can compare things and see if things are moving in the right direction in different locations.

• 11:03Speaker 00

  And the problem is that this is subject to this deceptive problem, or what I call this objective paradox, which is that you can look like, you know, your metrics are going up a little bit from year to year, but that has nothing to do with necessarily getting to a point where everybody is scoring perfectly, or even above the threshold that we would consider acceptable. And that's borne out by history, like it never happens. We keep on trying to revise, like every decade or something, like there's a whole new like push to like, let's really do this seriously this time. And just the same thing over and over again, like we don't learn from this mistake. It's that the problem isn't that the assessment is somehow flawed, the problem is with assessment itself. We cannot make

• 11:45Speaker 00

  progress in certain kinds of extremely complex problems, which education is one of those, simply by just laying out some assessment system and then trying to follow it towards this global objective, which is just incredibly complex to get to. But what comes into play then is what you said, which is this accountability issue. If you get a critique, like what I just said, at first, I mean, people make critiques, obviously, against standardized tests. And the response is usually like, okay, well, where's accountability going to come from? This is the way to hold people responsible. And the problem is that it's not necessarily a dichotomy, the way it's being presented, where like you either have accountability or you have no assessment at all. There is a possibility of having accountability

• 12:30Speaker 00

  with a different approach. But we need to have an approach that recognizes how you actually make innovative progress in an extremely complex problem, which is what the book tries to get into, in general, how those kinds of discoveries are made. And really, what happens in these kinds of problems is that because we don't know the stepping stones, like this is the key thing, the stepping stones that we need to traverse through in order to get to the outcome that we finally want, which in this case is like this universal high achievement, we don't actually know what we need to cross through. And what we can almost be sure is that one of the things that's not a stepping stone is everybody getting just a tiny bit better next year. Like, that's not probably going to happen. And if it did happen, that not probably going to lead to this kind of universal global

• 13:15Speaker 00

  high achievement. And so it's very deceptive. And so the stepping stones are likely things that are counterintuitive. And this is what where these metrics start to break down like that we use across all these institutions. Is that if the if the actual stepping stones that lead to where we want to go are counterintuitive. In other words, they're not what you would expect. Then the metrics are useless, right? Because they won't detect those stepping stones because they don't look like what the metrics are trying to detect. And if you think about it, of course, they're going to be counterintuitive. It's like a rule. Because the thing is, if the stepping stones are not counterintuitive, then it's not a hard problem. So we would have solved it already. Like, that's basically what makes a problem hard is you don't know what the stepping stones are. If you do, then you don't have a problem. You don't even need to do the assessments, just follow the stepping stones.

• 14:00Speaker 00

  And so because we don't know the stepping stones, what we need to do is we need to proliferate stepping stones. Like stepping stone candidates, things that could lead to something interesting, but we don't know which one. It's a lot, this is related in some ways to investing. It's a lot like investing. It's like you have this portfolio of ideas and you don't know which one is going to pay off. But you need to have that portfolio because you can't a priori make that kind of prediction. And so we will have some, we would need to have some stepping stones that we check into that that don't ultimately pay off. But if we have a portfolio, then some will pay off and eventually branch to more stepping stones. And some of those will lead to this, you know, ultimate holy grail.

• 14:42Speaker 00

  But the thing is that obviously we're gonna, we're gonna have some risk and we're gonna have some, some things that don't work out. And we need to tolerate that. And that's because that's what allows stepping stones to proliferate. And generally, these kinds of assessment and accountability cultures don't allow that. You know, if you have this accountability culture, you're not going to be willing to tolerate having things that don't look like they're succeeding. Um, with respect to your naive metrics. And so you need a completely different kind of a culture. And you still want accountability because people just can't live without accountability. But I think accountability needs to be much more nuanced. It needs to recognize that what makes something valuable is if it's an interesting stepping stone, not whether a metric is going up.

• 15:24Speaker 00

  It's like if there's a teacher out there in some obscure town in the middle of Alabama or something who does something interesting. Like, the key to getting the education system overall to improve is to disseminate that through the social network. Like, what that teacher did, it's not necessarily the solution to everything, all the problems you have in education. But it will be the solution for some people and they can follow up on that and go to the next stepping stone and see where it leads. Maybe some of these things lead to dead ends, but we can't find out if it doesn't disseminate through the social network, in this case, the network of teachers. There's nothing set up whatsoever to facilitate that in our system. Everything is centralized and globalized so that everything is assessed with respect to the same kind of criteria. So if something interesting happens in some obscure place, nobody's going to know about

• 16:09Speaker 00

  it, nobody can follow up on it, nobody can think about it, discuss it. But in the new version of assessment, that should be recognized and rewarded in some way. I can think of ways, but I won't, I try not to take up too much time with this. We could discuss if we want, but there are ways we can imagine that peer review and things like that could allow us to recognize interesting things. Do we still have assessment? Like, we don't allow completely crazy things to go on. You know, like if somebody proposes, like, let's just not do anything in school and let the kids run around or something. Okay, this would, this would be, this would get caught by something like peer review. But we do need to be able to recognize things that are interesting, which means things that are not objectively detectable through the usual assessment techniques.

• 16:48Shane Parrish?

  I have four thoughts that came from that. Wouldn't, wouldn't peer review necessarily push back on anything that's counterintuitive?

• 16:56Speaker 00

  So there is a cultural issue, actually, with that. Like, if, if we live in a culture where you're basically under the gun all the time, so people are basically, your boss is looking at you and saying, if you don't like walk the narrow line that we consider to be like, like the well, the accountable line, you're in trouble, like big trouble, like you could lose your job or something, you could lose funding, then the peer review system will also suffer from that culture. People will be like trying to kind of patrol the culture to make sure that it's being, being adhered to. But I don't think that has to be the way it works. I think that peer review is, what it's supposed to do is allow individuals to speak from their individual perspective.

• 17:38Speaker 00

  So it basically disentangles the individual from this large kind of global monolithic view of what is, what it means to be doing something good. And, and that happens. That is not an impossible vision, but because, you know, people, generally how innovation actually happens is individual connections. Most people see some idea somebody had and say, that doesn't fit with the usual paradigm. Like, that's not actually a good idea. Like, we know the right way now. We're all very sure right now what we need to be doing. But there's some one, you know, contrarian or something who sees that. It's like, actually, like, I could see a lot of potential here. And if you give them the space to express that.

• 18:19Speaker 00

  In fact, express means not just to write a critique, which they could, but also to follow up and actually try it themselves and build on it. Because they see the spark there of potential, which other people don't see. Then, yeah, I think that that is how ideas percolate through networks. Like, there's people, individuals make decisions that are somewhat disentangled from like the large mass of consensus around the field. And I think that peer review can facilitate that, but you have to start giving people permit, like, make it clear that this is a change. Like, we're not in this universal assessment culture anymore. Like, that's not how this is going to work. But we're still going to use the peers as accountability. Like, your peers will see what you do, so you can't just go crazy and do something stupid.

• 19:03Speaker 00

  And they will, you know, report it if it's something absolutely intolerable. But they have an opportunity to make unique assessments, not telling them what they should like. It's just like in academic publishing, you know, you do get peer review. Like, that's a place where peer review does happen as sort of like a standard aspect of the culture. You don't tell the reviewers, like, how to think about, like, the reviewers are the experts in their field. Like, so a paper has been submitted to a journal, and we have some reviewers or other scientists. You know, tell them, okay, well, here's how you should think about this. Like, they get to think about it how they want to think about it. But that doesn't mean that there aren't, like, big problems with peer review. And in fact, I think peer review in scientific publication and also in like the assessment of proposals for getting grants

• 19:47Speaker 00

  is flawed also because of objective thinking. But it is a place where I think we can, if we structure it right, which would be somewhat different than how it's structured even there. I think we can start to escape this kind of a global thinking.

• 20:01Shane Parrish?

  It's so interesting, because like the peer review system is sort of also not caught the replication crisis that we've. Oh, crisis is an overuse of the term, but it hasn't caught that sort of mistake. But it sounds like what you're really saying is that, you know, with ambitious goals, they're far off in the future. We don't know what the next stepping stone is. And so it's better to almost take like an evolutionary approach where we're, we're creating these mutations or copying errors or sort of, you know, we're trying all these little experiments and then we see which of those experiments leads to some interesting insights or conclusions.

• 20:40Shane Parrish?

  And then the idea being we take those conclusions or that interesting insight and then we propagate it to all the other nodes, almost like nature, sort of rewarding variations. And we do this blindly because we, we don't know sort of what will yield the best results. Is that, how do you think of it?

• 21:00Speaker 00

  Yeah, that's a good characterization. Actually, it's not, it's not a coincidence that you bring up evolution because the, the background behind the genesis of this discussion in the book is that, you know, I was working in evolutionary algorithms. And this is like in a branch of artificial intelligence and I was trying to understand what actually allows evolution in nature to make the kinds of incredible innovations that it makes. You know, if you think about it like from a computer science perspective, like as opposed to biology perspective, like what evolution is, it's a very unique thing in the sense that it's kind of like a search or like a learning algorithm that discovered everything that was ever created in nature in a single run.

• 21:43Speaker 00

  This is very different from like what you see in typical machine learning, which is like, we're going to try to solve a very hard problem and just that one problem and all the resources go to that one problem and that's the objective and that's the run. It's a very, very unique that you would have a single run discover the solution to every problem. And so what I mean by problems, like how do you get flight to work? How do you get photosynthesis to work? How do you get human level intelligence? Like it's all one run. And I was trying to understand this from an algorithmic perspective, not like a biology textbook, but like, can we actually? Create algorithms that work in some analogous way to what nature has done. And, and so it comes from some of the revelations that, that came from that work.

• 22:27Speaker 00

  Like this, like other, this discussion, which seems somewhat remote from that, but it's, it's very connected actually. Um, because it's based on a recognition of like how nature was, was innovating. And one of the, the keys that you mentioned, I just wanted to highlight, like the word interesting. You know, when we, we talk about selection in nature, it's like, who survives? Like usually the word interesting isn't like what comes up. You usually hear the word fitness. But actually, um, I think like, we won't go into all detail, but you could, you could take an alternative interpretation of nature. That the way it's set up is actually a way of detecting things that are interesting in some sense. You know, because of the fact that what everybody has to be, some, how I sometimes put it is like a walking Xerox machine.

• 23:11Speaker 00

  Like you have to have within your guts, something that will make a copy, which is extremely complex. Or else that lineage will not persist. And that means that like, pretty much everything's going to be interesting in some sense. It's a very abstract and weird sense, but like you can't degenerate into complete meaninglessness. Cause everybody has to be a walking Xerox machine. And this keeps things honest and kind of interesting. Now, when we move to other paradigms, like, uh, you know, like, like education or something like that. Or invention of another type of human invention, civilization, like how we progress through this space of possible inventions. Then like, it's a different, because like here, we, we care about our view of what's interesting. You know, evolution serves as an arbitrary concept of what's interesting. It just happens to be the way nature is structured.

• 23:55Speaker 00

  But we care about things that we find interesting as people. And so really the crux of the matter is to really delve into the issue of what is interesting. You know, the, the education example is an example, you, you know, you could look at for reference. Like what does it mean to have an interesting teaching technique, independent from like the immediate assessment implication of it? And we are afraid of that conversation, I think, as a culture. We do not like talking about whether something's interesting, because again, it seems to somehow veer around this accountability issue. We don't want to know your subjective view of why your little thing is interesting, your pet idea, because I can't assess it in some objective sense. And so I don't like to hear about this. Also, I think is one of the problems with peer reviews that

• 24:37Speaker 00

  you don't allow it to really discuss interestingness. But the truth is, interestingness is the magic sauce. And that's what I think humans are really good at. That is why civilization has actually created all of the amazing artifacts and genres and, you know, musical and artistic and literary genres that it's gone through over the eons is because, you know, I mentioned those just because I don't want this only to be about technology. It's about everything. And it's because we have a nose for the interesting. We're super, super sensitive to what's interesting. We're just, we've gotten to a point in our culture where we're not allowed to talk about it. And I just want to point out that it's not unprincipled to let people talk about what's interesting, because we're talking about experts here. You know, we're not saying

• 25:20Speaker 00

  that like, okay, in the field of education, we're just going to go up to some random person on the street and ask them whether some random teacher somewhere did something interesting. We're talking about the people who are experts in education, who have a history, who like actually been in the field for years. The idea that those people, that their opinions about what's interesting are invalid, is like completely throwing away, like I think, decades of societal investment, like all of the education that you put into that person from the time they went into kindergarten, all the way up to the point that they like got out of graduate school. Like, what was the point of all that if we don't trust their judgment on anything subjectively? It's actually the subjective judgments are the interesting ones, because like the objective judgments are easy. You don't need a degree

• 26:02Speaker 00

  to just measure something. You know, some kid takes some test, you get a score, you can average them across everybody at the school. Who needs to have a degree for that? Like, anybody could look at that and just tell you how it's going. It's the interestingness judgments that require education, experience, like deep insight. And so the fact that we are paranoid and afraid and unable to engage with the question of what's interesting, I think, is crippling, like to our ability to innovate.

• 26:26Shane Parrish?

  That's fascinating in a couple of ways. One of the ways is that you mentioned the word judgment, judgment's subjective, and we're hesitant to say or do anything that's subjective, and we grab on to these metrics, right? So, an example would be sort of like during COVID, we If we were mutating, we could have done something where, you know, the best teacher in your state, or your province, or your country, the best grade five teacher is now teaching all grade fives, right? This could have been set up, and it could have been, you know, you could have added access to the best teachers in the world at whatever level that you're at, in whatever subject you're at, or even the best teachers in your state, or your country, or even your city. I mean, we could have done this at any of those levels, and we didn't, because that

• 27:09Shane Parrish?

  would require, A, it would require sort of like trying something that might fail, which I want to talk about in a second, but B, it would require us saying something subjective, that this is a better teacher than somebody else, and we're so hesitant to do that. Can you, what are your thoughts when I say that?

• 27:27Speaker 00

  Yeah, I agree with the point. Like, we could, we could debate about whether we should have one teacher, like, teaching everybody in New York State, or something like that. I mean, I'm not saying we shouldn't, but we could debate about it, obviously. But the, but the point that you're making, that like, we can't even, like, consider it because of the structure of the system, that is a problem. Why can't we debate about it? Like, in an official sense? I mean, we could, you and I could debate about it, and it will have no effect on anything, but, but like, the actual official system, the formal system that actually has the gatekeepers inside of it, will not ever go through that debate. Because, yeah, it's like, it's incompatible with the assessment system, and it's just like, way too unwieldy, given that bureaucratic system, the way it's set up.

• 28:10Speaker 00

  And I think that's an example of what is impeding our ability to collect stepping stones, and try different things, and see their outcomes, and really be flexible and innovative, and sort of like proliferate ideas, in the spirit of nature. But it's, it's, I think it's multifaceted, so it's not only, you, you can't point to just one reason for it, it's like pervasive across all kinds of levels of the culture, that we're, we're afraid of this kind of engagement and exploration, including what you alluded to briefly, just that we are afraid to take risks. I mean, that, that's part of it. Like, what if it doesn't work? You know, like, this could be the end of, the end of the world, or something. And I, I think it depends on the domain, whether risk is tolerable.

• 28:52Speaker 00

  Like, we should acknowledge that risk sometimes isn't worth it, you know? Like, there are some places you don't want to explore necessarily. And, and so maybe there are some places you can't innovate, because you just can't tolerate the risk. This may be true for an individual, like, you know, you've got a family to take care of, and you can't really go try some crazy thing right now. Or it could be like at a societal level, like we can't try a whole new economic system. You know, it might be interesting, like it would cause potentially so much devastation that we just can't take the risk. So we have to acknowledge that like some risks are too risky. But I think that's that's orthogonal to the idea that we can't discuss what's interesting. Like, I think we can still discuss what's interesting. We can also talk about whether the risk is too high.

• 29:37Speaker 00

  But there are certainly systems where where risk is tolerable. And it's and it won't be too devastating. And, you know, we have to thread that needle depending on the system. Like, we don't want like kids to go through a year of school and learn absolutely nothing. But that doesn't mean we can't try anything interesting whatsoever either. And so we have to thread that needle carefully depending on the domain. Some domains, like there's, there's lots of room for risk, like, for example, science funding. I mean, the whole point of it is risk, as far as I can see. Like, we don't expect most of these things to work out, I think. I don't know if that's the view of the National Science Foundation, but my view is it's okay if lots of things don't work out. So in a situation like that, like, there's not like any real downside, like to really, really like just doubling down on the way I'm

• 30:19Speaker 00

  advocating, which is being a lot less objective, a lot more subjective and a lot more willing to talk about what's interesting. And then in more kind of like brittle situations, like the national economy, we have to be a little more cautious about that. Because there's only one national economy. But but we can do it, I think. We just, we just have to be clear eyed about what what the consequences might be.

• 30:39Shane Parrish?

  It it's like nature or evolution, if you you want to call it that, has no concept of loss aversion. So it'll just keep trying things. And if it's fit, it'll reproduce. And if it's, you know, not, it'll eventually wean itself out. And then it'll keep trying the same things over and over again. And it doesn't, it doesn't have a consciousness, so it doesn't think about being wrong or the consequences of being wrong or, you know, moving backwards to move forward. It doesn't, it doesn't have any of those concepts where we do, like, we accept failure in science, but we won't accept failure when it comes to an education system. So any anybody who comes in and tries something else, they have this really weird equation, right, where you have a very linear, small upside

• 31:23Shane Parrish?

  and a very exponential downside if things go wrong. And so it prevents us from trying. Uh, anything new. And so that's why everything always looks like what was before it with slight wording differences or nuances around it.

• 31:39Speaker 00

  Yeah, yeah, it's true. That's a good point about nature, how like there isn't this kind of assessment. There's there's nothing like, you know, should this mutation go forward? You know, like, let's have a committee look at this first before we check into it. It just it just happens. And, you know, sometimes will will be bad and it won't that lineage won't persist. And but what's interesting is you look at it in aggregate, like the system is obviously absolutely prolifically creative. I mean, nature is like the ultimate creative genius. And so it's because of that. That's why it's like not afraid to try things. And it's willing to invest resources into things. But, you know, of course, we can't be that laissez faire. Like, we could just let anything happen anywhere. I understand

• 32:22Speaker 00

  that and acknowledge, you know, this is people's lives can be affected. But we certainly can swing the pendulum a little bit away from the current assessment accountability paranoia. And especially in some domains, like, like, science research, where it's like, very natural to do that. Like, there's no reason to be paranoid in these kinds of domains. In other domains, like education, you have to be a little more careful. But I think that it's not working anywhere. Like, that's one of the points here, I think, is that, look, you might like assessment and accountability and objectives and metrics because they make you feel better because it feels like we're making sure that nothing bad will happen. But you have to recognize it's just a security blanket. It doesn't work. I mean, look at the education system. Like, it never gets any better. So like, it's just making you

• 33:06Speaker 00

  feel better, but it's not actually doing anything productive. So maybe it would be worth it, and actually not any worse, if we did allow some more risk taking, which is not just about risk, it's about actually following interesting things. That's really what it's about on the positive side. Risk sounds like it's all negative, but we're talking about following the interesting. Well, maybe things actually be better if we did that. To take off the straitjacket, get rid of the security blanket, and just acknowledge how reality works. And it's unfortunate that reality works this way. You know, it's like, it'd be better, I prefer if reality worked where if I set an objective, like, I could just get to it as long as I'm sufficiently determined to do it. You know, that'd be really convenient. But it's just not the way reality works. And so, reality is inconvenient and difficult and scary.

• 33:47Shane Parrish?

  How do you map that to sort of like Elon Musk saying something like crazy, or what seems like crazy, which is like, we're gonna go to Mars in the next eight years, which is this objective, and we don't have all those stepping stones. We might have a little bit, but I mean, he's building custom engines and, you know, there's a little bit of technology that exists that maybe, but there's also, there's a psychological value, I think, in some ways of pulling people toward a pursuit that might never even be reached, right? Mars might be one example, but CEOs and politicians do this too. And at its best, it sort of unites us. And it pulls us through sometimes when we're having hard times or, and we feel part of something larger than ourselves. We feel part of something meaningful.

• 34:31Shane Parrish?

  So there's a psychological angle to pursuing these big, large ambitions.

• 34:36Speaker 00

  Yeah, that is an interesting question, because we see these quests that are set up. The self-driving cars is one of those, which, from the perspective of this objective critique, are really interesting, because they basically are very ambitious objectives being set. So they're an example of what I'm critiquing, basically, like, just like, completely plainly that. And they, they often are, like, not successful for the, I would, I would claim for the reasons that are in the critique. Like, so the self-driving car thing, like, back in 2016 or so, I don't know the exact year, it's around 2016. People like Musk, but others too, not, not just him,

• 35:18Speaker 00

  were saying like, this is around the corner, like, one or two years, like, you're gonna start seeing these services. And it didn't happen.

• 35:25Speaker 00

  And, you know, the, like, the interpretation through what I'm saying would be that, well, that's not a surprise. This isn't how innovation actually happens. Like, you don't set some extremely ambitious objective where we don't yet know the stepping stones, and then just like, double down and throw all your money at it, and it's gonna happen. That's not how things actually work. And of course, what the crux of the argument has to do with, well, but are the stepping stones actually there? That's the real question. Like, I think often visionaries are interpreted as people who make these statements. Like, we're gonna go to Mars. Like, there's a visionary, like, let's, you know, let's, let's put the halo on that person, like, they're, they're a visionary. But the thing is that, that's just speculation, because we don't know what the stepping stones are to getting humans on Mars yet. We don't know that.

• 36:10Speaker 00

  I think a visionary is somebody... In contrast, who has recognized when the stepping stones actually have snapped into place. Now, that's a person you should follow, and that's a very interesting and unique kind of person. It's a different kind of a person, and I think that's more like a Steve Jobs type of a person. Like, the Elon Musk kind of thing is, I agree with you, though, on the, on the point that it might rally interest in an area, and that connects to the word interestingness. Like, that could be a positive thing. So it's not clear there's all negative, just somebody saying we're gonna go to Mars. Like, there could be a positive side because it basically moves resources and people's interest into an area which might matter. And so, we could still say, "Well, actually, the, the predictions here are wrong,

• 36:54Speaker 00

  but, but the, the social effect, like cultural effect, actually is positive." I, I could see that, and I think that, that makes things a little, a little hard because, you know, it's complicated because of that. Like, it's not like we can just fully critique somebody like that and, like, dismiss them. But they're not doing that. So I think, to be fair, like, in terms of who you want to lionize, I think you have to be fair, like, to, to at least acknowledge that, like, that's not what they're actually doing. They're not just saying, like, "This is culturally, like, a good place for us to be interested." They're making claims that are not really well-founded, and so, it's not really visionary to make these kinds of claims. And so, you don't want to go too far in kind of embracing and lionizing this kind of thing. But you, you can say, "Well, it, it might have had a positive effect," and we can concede that.

• 37:37Speaker 00

  But I think what's really interesting is the people who recognize when the stepping stones are there. Because even the visionaries, the so-called visionaries, are so bad at that. Like, they're the ones who tell us, like, "Right around the corner, this is gonna happen, that's gonna happen." Never actually ends up being that way. But it's like this rare kind of a person who's like, "Huh, you know, look at what the things that we have in this iPad. Look at the technology that we have, like, screen technology, you know, like, right now, like, we could actually make this iPhone concept for real. Like, it's actually possible to do it." And that's not just speculation. It's not like, "I'm gonna predict, like, ten years from now, there's gonna be this phone thing that does all these magical things." That's like, "I actually realize this is, now is the time." That's very hard. Like, people have visions all the time of, like, really cool stuff that might happen, like, flying.

• 38:21Speaker 00

  Like, people were trying to build planes for hundreds of years. They're not, like, particularly impressive people because they said, like, "We might be able to build a flying machine." And they were wrong, like, about how long it was gonna take, what's gonna go into it. But you know, the Wright Brothers are in the right place at the right time. So you know, I view the Wright Brothers more in that way, like it's like they saw that the stepping stones had now snapped into view. And so they actually saw this actually is the time where this can happen. Those are the people to be impressed with and to follow, I think.

• 38:47Shane Parrish?

  But but hold on, like at a meta level, and I might be completely wrong here. Isn't that a variation? Like, we're trying this idea now and it doesn't work. And just because it doesn't work, doesn't mean that we shouldn't try it, because that in and of itself is a variation that we're, you know, this is like how nature proceeds.

• 39:04Speaker 00

  So yeah, I think this gets into some subtlety. Like, is it ultimately, you know, actually harmful to just try something, even if like it's really unrealistic? And like, I think it depends on what your motivations. Like, if you're going in that direction because it's interesting, it might not be harmful. I think the field of AI is kind of like that. Like, at some level, there's like this really grandiose conception of like some human-like computer. And like, that is, I think, a naive objective right now. Like, you just don't want to know how to do that. We don't know what the stepping stones are that lead to that, although they're getting closer, perhaps, but they're still not close, I would say.

• 39:47Speaker 00

  But at the same time, like, the the point that, like, investigating around the area of like, algorithms that have intelligent like qualities, that's still valuable, I think, because it's interesting. Like, one thing that happens, I think, if you do this, you're unearthing stepping stones that could lead to something else that isn't artificial general intelligence, but still really valuable. That's not usually how it's thought of. I guess in some way, that would be disappointing, if it's like, well, I made some progress, but it's never going to get to the AI that I'm envisioning, but it's still caused something cool to happen. But in effect, that's really what's happening. You know, like, the effect on industry of AI is significant, but it's not because human-level intelligence has emerged, because it hasn't.

• 40:32Speaker 00

  It's because these things have other implications that are in the short run quite useful and interesting. And so it's sort of a side effect of the fact that people have this kind of grandiose vision that a lot of interest has now focused in and like now caused a lot of stepping stones to be uncovered. And we don't really know where those stepping stones lead. They may not lead to the human level, but they do lead to interesting things. That that so you have to, I think you have to distinguish which type of thing are we talking about? Like when somebody says something like AI, like a self-driving car, flying car, you know, whatever it is. Are we talking about like this is an interesting space to play around in because we're going to find some interesting stuff? Or are we literally believing that like, okay, within two years, we're going to be on Mars? And or even 10 years?

• 41:15Speaker 00

  And it's not like, you know, the answer is like extremely critical. Like, I mean, you know, you could be wrong and still have gotten a lot of people on some interesting stepping stones. But at least I would want to for myself, try to disentangle those that question and decide which which type of visionary is this? And what type of vision is this? Because it I think it guides the realisticness. And also it it takes out a lot of inefficiency, like to recognize why you're doing something. You know, like, because when I investigate AI, I do, I think of myself as just basically looking at stepping stones because they're interesting. I don't necessarily think this leads to AI. This doesn't lead to AI because I don't know. It's like way far off. I can't tell you.

• 41:53Shane Parrish?

  So so what do you tell your boss?

• 41:55Speaker 00

  Well, that's a

• 41:55Shane Parrish?

  Not you, but like somebody listening to this. So how do I go to my boss and say, you know what? These objectives, they're really destroying my creative thinking here. How do I step back and, you know? What do I say?

• 42:07Speaker 00

  That is a serious problem. That actually is one of the big reasons that we wrote wrote the book. Because it's kind of like the book is, I think, a weapon.

• 42:18Shane Parrish?

  It becomes an argument.

• 42:19Speaker 00

  Yeah, it's trying to empower people to make an argument to their boss, you know, because it's it's really scary, I think, to argue like this. Like this sounds wacky, actually. Like if you don't have any context, you haven't listened to this show, read the book or anything. And if you just go to your boss and you're just like, "Look, um, I'm just doing this because it's interesting and um, there's no assessment, you know, we're gonna drop the assessment." Uh, just let me do it though, because interestingness is really important. I mean, your boss is gonna, yeah, you're risking your job. Your boss is gonna freak. And it's not only because your boss is like a jerk, because he's not necessarily a jerk, or she is not necessarily a jerk. The real problem for your bosses, they have a boss. How are they gonna explain it to their boss? The problem is that this culture percolates through everything. So like, everybody's trapped

• 43:02Speaker 00

  and even if they believe in what I'm saying, like I've said, I mean, I've talked to a lot of people about this, a lot of different organizations. Like, the book has caused me to come before like, all kinds of audiences I never would have encountered. And a lot of the time, I meet people who are gatekeepers. And they're like, you know, people who decide what should go forward, what should not go forward. And they love, they love the, the principle of the argument. Like, they're like, "This is so inspiring. Like, I'd love to change things around here." But, and there's always this caveat, like, "I answer to this, that, and the other thing. And this is gonna be really hard, like, to, to, to explain this to these people. And uh, so I'm not really sure what I can do." And then, basically, it doesn't really lead to any changes. And, and so I'm hoping that the book will actually empower, through this kind of discussion, becoming more mainstream, hopefully. I mean,

• 43:45Speaker 00

  that's idealistic. People being able to make this argument without being ridiculous. Like, this is serious. Like, this is principled, it's based on research. Um, and there's a reason that justifies doing things like this.

• 43:57Shane Parrish?

  But, but isn't this sort of like the self, like, not, the, the destructive nature of capitalism, or, you know, that it just like, sort of feeds on itself? And then, so you have an interesting idea at work, you can't pursue it, that becomes a startup. Uh, so you find the next stepping stone, you find something interesting for you, your, your workplace won't necessarily allow you to explore it, so you quit your job, and you find a couple like-minded people, and now you get to explore it.

• 44:22Speaker 00

  Yeah. I mean, I'm, I'm only talking about improving things, really. It's not like nothing ever works at all. Obviously, we, we, we see progress. Like, rock and roll was invented. Cool stuff happens, you know, people create startups that change the world. It's not like nothing can happen at all. At least we're not in this kind of like horrible entrenched situation. And then there are, you know, dictatorships and things where that effectively is like that. But like we at least have a certain amount of flexibility in our society. But I'm basically saying it could work a lot better. And in fact, like it shouldn't be the case that like in an organization or a business, for example, or like a corporation where like there is a part of the corporation,

• 45:03Speaker 00

  which is supposed to be facilitating innovation, that the only way to actually do something innovative is to actually quit and start a startup company. You know, I don't feel bad that that happened because like the startup ended up being really cool. But something is wrong though. Like what is going on inside of that organization? Like they could have captured that idea, but they let it go. And this is, I think pervasive all over the place because of our objective culture. And so things could be a lot better and life could be a lot better. Because like dropping out of your job and having to start something new and risking your career and your finances kind of sucks. And it really shouldn't be necessary, like because this is actually a principled thing to do, especially if you're part of the innovative component. Like I, I just want to acknowledge there are parts of companies where you shouldn't necessarily be about innovation.

• 45:48Speaker 00

  It's not every single thing that's being done is not about innovation. Like some things need to be conservative, some things need to be conserved. But there are the parts of companies explicitly, you know, supposedly assigned to innovate. And they actually work in this objective way, which is just like completely backwards. And so does, so do like huge agencies like the National Science Foundation, which is I think very objectively run. Like if you look at the criteria for funding, it's very objective. It's like you have to propose to this committee by telling them what your objectives are and then assessing whether you can get to those objectives and being very clear on what the assessment will be. And it's completely objective and then it's consensus driven by a committee, which is not also, which is another thing I would talk about, which is not great for doing this kind of thing.

• 46:31Speaker 00

  And so like, yeah, there, it still might happen. Like I myself have pursued projects that like were rejected by those committees because I basically was pissed off and thought, screw them, I'm going to just do it. But that shouldn't, that's not very ideal. Like it should work better than that. That shouldn't, it shouldn't be like you always have to like be a rebel in order to do something which is principled.

• 46:52Shane Parrish?

  You had two sort of insights on decision making that you mentioned before we we got on here. And the first was that you told me one rule of thumb that you use for deciding on projects is to try to avoid ideas that make too much sense. Can you double click on that?

• 47:05Speaker 00

  Yeah, this is it's kind of fun because when I say it, people look at me like I'm insane. It's like somebody says, look at this great idea. Like it's usually something in science. It's like, uh, this is, isn't this, isn't this exciting? Or maybe even we should follow up on this or do something. And I'm just like, well, I do think it's a, it's a good idea, but it makes too much sense. So I really don't find it that exciting. And so that's a little personal heuristic, which is related to what we're discussing. And it's sort of what it is for me, or a rule of thumb, is that like I recognize because of all this, that when you talk about stepping stones that lead somewhere really revolutionary, they're going to be counterintuitive.

• 47:47Speaker 00

  You know, and, and if you think about it, that makes a lot of sense because if they were just intuitive, like obvious, then we would have crossed them anyway, you know, like it's not a problem. Getting to things that are really important or interesting will cross through counterintuitive stepping stones. And so what that means is that like they won't make sense at first. That's basically the definition of counterintuitive. Like it's true that in hindsight, they'll make sense because like at some point you look back and you say, oh, well, I see why this led to that. But looking forward, they will be strange and counterintuitive and basically seem like they don't make sense. And it's like even like if you think about what the theme that we're talking about here, which is in effect, that, um, that, uh, to achieve our highest goals, we should be willing to abandon them.

• 48:32Speaker 00

  Doesn't sound like it makes sense at first, you know, which is why it's a good stepping stone because they're always going to be like that. They don't make sense at first. But in hindsight, you can look at it like what we've discussed, if you're starting to be convinced and it does make sense actually, but only in hindsight. At first you proposed, when you hear a proposal like that, you're like, what the heck is that? That's crazy. And those are the ones I'd really rather pursue, you know, because that are the ones that are going to lead to something revolutionary. And so if you give me something that makes a lot of sense, I basically think, well, someone's going to follow it. It's guaranteed. It makes sense. But it's not exciting enough for me because like I know it's going to get followed. Someone else can do it. Like the ones that aren't going to get done are the ones that don't make sense. So I'd rather think of something like that and that would be more exciting.

• 49:14Shane Parrish?

  That goes with your sort of second heuristic, which is trying to imagine the people you know. And if they can predict what you would do next, then you probably shouldn't do that. Because if it's predictable, somebody else can and probably will do it.

• 49:25Speaker 00

  Yeah, yeah. So that's, that's another heuristic. Yeah, like, um, I'm at least I feel much more pleased if I do something next that isn't what you would have expected me to do. That means, yeah, trying to think about upfront, like what is the natural next thing that you would think I would do? And then not doing it. So trying to run away from it. And this is related to the concept of novelty. Like basically that was a large concept in the book is novelty, because novelty is a very large component of interestingness. It's not all of it, but like just about anything that's interesting is novel. It's other things too, but it's at least novel.

• 50:02Shane Parrish?

  What do you mean when you say novel?

• 50:04Speaker 00

  Yeah, novel means it hasn't been tried before. It doesn't look like things that have come before. It's different in some very fundamental way from other things that have existed in the past. And, and so those are like, like, think about things like, you know, what's, what's novel changes over time? Of course, like something like, you know, the idea of like a little room that's like on top of wheels and can move you from A to B. Like, you know, a hundred years ago, a hundred, 20 years ago, that would be like a super interesting thing. Like if you actually could get something like that. That would be like dinner conversation. Like, could you imagine? How will this affect society that like this little thing on wheels can like go anywhere? Now today, that's not very novel. Like it's completely not novel.

• 50:48Speaker 00

  And that's why it's not interesting. Like this is not good dinner conversation. Like because it's been done and it's been done for a hundred years. And so like it changes. And so generally speaking, like this, the next stepping stones, the ones that are going to be interesting are also going to be novel. So that's generally a rule of thumb. And so like running away, like novelty means running away from where you've been in the past. You know, a lot of objective problem solving is the opposite. It's about converging along the same path. Convergence is basically what it means to optimize. You converge towards that global optimum if you're doing well. But it means you're basically sticking to the same path. You're not trying to get away from it. Now, this is the exact opposite. It's like I'm getting away from where I've been in the past. And that's uncomfortable, of course, by nature and risky and everything.

• 51:30Speaker 00

  I mean, because it's like if I've been successful in some path that I've taken in the past and now I'm in a situation where I'm now trying to take a different path. Well, of course, that's going to be like super uncomfortable for me. Like I'm leaving behind like everything that makes me feel comfortable and safe. But then again, it's a heuristic for innovation, you know, because it will be through the novel and the interesting that innovation actually happens. And so I think that's why generally I, I, I think, well, it is, you can kind of predict what I would do next. You look at a certain trajectory that I've taken, but like, I guess the way I think is that then someone will do it, you know, like it's, it's kind of obvious. You don't need to be me anymore. Like this path has been laid and I'm no, I'm sort of like pushed myself out of relevance because the path is now clear.

• 52:16Speaker 00

  I think this is why like luminaries kind of get stuck in a rut as they get older. It's part of it, I think, is that it's really, it's really scary like to leave that path because that's what you're known for. That's what everybody respects you for and you're comfortable in it. But the problem is it becomes predictable. And so you look less innovative, the, the farther down the path you go and the less you deviate. And so it takes a lot of energy to actually intentionally deviate. But it's, I think it's a good heuristic for if you really want to continue to be innovative.

• 52:49Shane Parrish?

  Yeah, it's a good place to end this conversation, Ken. I want to thank you for your time today and like it was a fruitful discussion.

• 52:56Speaker 00

  Thank you so much. Yeah, it was really great to be here.

• 53:04Shane Parrish?

  Thanks for listening and learning with us. For a complete list of episodes, show notes, transcripts and more, go to fs.blog/podcast. We're just Google The Knowledge Project. Until next time.