AI Expert Dr. Michael Littman: This is Why Everything You Know About AI is Wrong

What if everything you thought you knew about AI was wrong? Today on Digital Disruption, we’re joined by Dr. Michael Littman, Division Director for Information and Intelligent Systems at the National Science Foundation. The Information and Intelligent Systems division is home to the programs and program officers that support researchers in artificial intelligence, human-centered computing, data management, and assistive technologies, as well as those exploring the impact of intelligent information systems on society. He is also University Professor of Computer Science at Brown University, where he studies machine learning and decision-making under uncertainty. He has earned multiple university-level awards for teaching and his research has been recognized with three best-paper awards and three influential paper awards. Littman is a Fellow of the Association for the Advancement of Artificial Intelligence and the Association for Computing Machinery. Summer of 2025, he will become Brown University's inaugural Associate Provost for Artificial Intelligence. Dr. Michael Littman sits down with Geoff Nielson to explore the misconceptions about AI, its real-world applications, and what the future holds. Dr. Littman challenges common fears about AI replacing humans, emphasizing the importance of human-AI collaboration rather than competition. He talks about ethical considerations, the evolution of AI decision-making, and the balance between automation and human oversight. They unpack what is next in the world of artificial intelligence and why understanding the technology's limitations is just as important as embracing its potential. In this episode: 0:00 Intro 0:50 How AI technology has evolved 3:24 Is the public ready for AI 6:30 Sci-fi view of AI 7:46 The underlying fear 12:40 Society is adaptive 16:00 This current round of AI gives us new tools 20:15 New era of computer science 23:16 AI is creating new roles 27:25 The different kinds of education 32:40 Dissemination mechanism 34:36 How AI works as an organization 36:45 Importance of AI education 38:28 How the National Science Foundation supports AI research 43:23 What about AGI 48:27 Future predictions 52:34 The goal is to help organizations make the best use of this moment 54:00 AI in the 80s to now Connect with Dr. Michael Littman: Website: https://www.littmania.com/ LinkedIn: https://www.linkedin.com/in/michael-littman-b26351/ X: https://x.com/mlittmancs Visit our website: https://www.infotech.com/ Follow us on YouTube: https://www.youtube.com/@InfoTechRG

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Geoff Nielson?

  I'm super excited to talk to Dr. Littman today. You know, this guy is an educator. He works for the National Science Foundation in the US. So he sits at, you know, an intersection of understanding what needs to be done. Where are we going with AI? And how do we actually disseminate that knowledge? How do we teach people and bring everyone along for the ride? So it should be a fantastic discussion.

• 0:25Geoff Nielson?

  Welcome to Digital Disruption. I'm Jeff Nielsen and joining me today is Dr. Michael Littman, who is a computer scientist, an educator, a researcher and an author. Michael, thanks so much for joining today.

• 0:37Speaker 01

  Oh, it's a pleasure to be here.

• 0:38Geoff Nielson?

  Amazing. You know, I wanted to ask you, you've been in, you know, the machine learning game for, you know, a long time, certainly enough time that it predates this kind of, you know, modern bonanza, if I can call it that. What are you seeing that's different now? And, you know, what's kind of surprised you about how this technology has evolved?

• 0:57Speaker 01

  Yeah, that's a great question. I mean, so I kind of, I mean, I've been tracking AI as an intense area of interest since like the early 80s, late 70s. And so that's a, you know, I wasn't in, I was a, I was a high school student, but you know, nonetheless, it was something that was really interesting to me. And when I first became a researcher, it was during the previous wave of interest in neural networks, artificial intelligence. And it was very exciting. A lot of people jumped on the bag when it had a lot in common with what we're seeing now, but then it all fizzled out. And I think when it all started up again in the last, I don't know, five, 10 years, I had the same kind of thought that you have, which is how is this similar and different from, let's say, the last time?

• 1:42Speaker 01

  And what struck me is each time we've seen kind of AI gain a lot of attention and then lose that attention, often what happens is there's been some kind of really neat breakthrough in the lab that people start to rush to apply to real world problems. And when AI and the real world meet, basically what we saw is that the real world would win and AI would lose. And so it would kind of recede back because it just wasn't ready to take on the, the messiness of the real world. And I think what we're seeing this time is that AI and the real world are coming into contact and kind of AI is winning. And so And I mean that in two ways. One is that it the the current wave of technology does seem up to

• 2:27Speaker 01

  the challenge of being able to work with the messy real world, which I think is great. It's fascinating. It's it's exciting. But also, what we're finding is the real world wasn't quite ready for AI to be effective in that way. And I think what we're doing now, what a lot of people are focused on now, is what can we do in the real world to make it so that things are kind of more in equilibrium again, that the AI can be useful and helpful to people, but at the same time, we're not kind of undermining the the foundations of society.

• 2:59Geoff Nielson?

  So that, you know, AI readiness or public sentiment toward AI is a really interesting area. And, you know, you talk about who, you know, who's winning AI or the real world. You know, you participated in AI 100. And, you know, there was some content there at the time about public sentiment. And, you know, one of the key challenges is getting the public ready for AI. Where are we on that journey? Are you are we ready? How has it evolved over the last few years? And, you know, what's left to do?

• 3:32Speaker 01

  Yeah, outstanding question. I think this is very much up in the air, and it's very much kind of a focus of a lot of people's attention right now. There are concerns that that the real benefits to bringing AI into to solving various kinds of real world problems, and to help people do what they're doing, but be more effective in doing it, do it more accurately, do it more efficiently. And to the extent that people as a whole are wary and concerned, then that maybe that won't happen, right? So we'll kind of lose this opportunity if we don't make sure that people are on board. On the other hand, there are some real concerns. And so making sure that people are appropriately wary is the challenge, right? That there shouldn't be this

• 4:14Speaker 01

  sort of blind, let's all rush into this because, you know, the glorious future awaits. But at the same time, if we're so afraid that we're not willing to even try out some of these ideas, and and, you know, risk some maybe bumps early on, but then figure out how to get through them, then I think we really are missing out on an exciting opportunity. So one of the things that I do in my current role at the National Science Foundation in the US is we look at various ways of supporting AI research as a whole. But we also are very concerned about education around AI. And that goes all the way from educating, you know, Medical doctors to help them figure out how they can be more effective in their use of AI

• 4:55Speaker 01

  in their practice, all the way down to elementary school students and how they can be learning what they need to learn so they can contribute to this. And one of the areas that I personally am really excited about, and one of the reasons I was so glad that you guys reached out to have me on the podcast, is the notion that we need to be getting, kind of getting the word out to the public, right? So in NSF, we call it the informal education. But the sort of idea that there's stuff that is important that we should learn, that we're not learning in school. And I know that, you know, so my institution is really focused on trying to make that happen. But I think there's a lot of people worldwide who are realizing that this is a critical time for people to find out what they need to find out.

• 5:40Speaker 01

  And so where are we on that journey? I mean, I think we're really close to the beginning of that journey. I think, you know, we're seeing lots of news stories about AI. We're not seeing a lot of, I don't know, like kid shows around AI. Like, I'm not really seeing people laying the foundations the way that traditionally folks have done in other science areas. I think there's, yeah, there's a lot more that we could do there.

• 6:07Geoff Nielson?

  Yeah, and I'm just, you know, kind of processing, you know, this, you know, absence of education right now and this, you know, reluctance right now for, you know, people to adopt it, or maybe there's an angst about it. My sense, Michael, is that, you know, if you've been following AI since the 80s, you probably started with almost like a sci-fi view of AI. Is that, is that fair?

• 6:35Speaker 01

  I do love, I do love a good sci-fi, that's for sure. But that was, that was never really my motivator. I mean, I think early on, this is maybe saying too much about myself. But, but I read a book in the 80s that was, it was a bit of a, I mean, in retrospect, it was, it was highly, not inflammatory, but it was like, it was getting people excited in a way that probably people didn't need to be excited. But it was about understanding how folks in Japan were Embracing the idea of artificial intelligence and how they were going to like get ahead of us in the West and and we were going to be left behind and and all these amazing things that were going to happen weren't going to happen. And so from my perspective, it's it's always been about the excitement of the technology, but also the impact that it has on our society.

• 7:19Speaker 01

  And so um, I don't know. I mean, I always thought it seemed like a really cool impactful idea. And now I think other people agree with me.

• 7:30Geoff Nielson?

  No, that's that's that's really interesting. And that, yeah, the reason, I mean, the reason I ask, I'm so, it's awesome to hear about that excitement and, you know, it's contagious. It feels like there's this underlying fear, right, of like, you know, what is the dark side of this, you know, what, you know, how worried do I have to be? Um, and and the excitement is, is, you know, in some ways really refreshing. And so, you know, to what degree is, you know, kind of this like existential fear? A barrier and, you know, what what is kind of the the fact versus fiction here? Like, are you are you worried? You know, about us getting to like a singularity or about like, you know, an AI overthrow or or is that like?

• 8:14Geoff Nielson?

  Is that so far removed from reality that, you know, people, you know, it's it's not a rational, you know, point of discussion?

• 8:22Speaker 01

  No, I definitely don't want to call it irrational. I have some very, uh, very close colleagues who are, uh, focused on that particular issue. So the the the concern, just to just to kind of put it on the table, the concern that a lot of people have is that to the extent that what makes the human species so powerful in the history of our planet? And, uh, arguably, our intelligence isn't that much greater than, say, that of chimps. But at the same time, if human beings don't decide that they don't really care about chimps anymore, chimps are going to go extinct in in favor of us, right? So it's you only need a little bit of extra intelligence to make it so that you control the destiny of of other entities on our planet.

• 9:03Speaker 01

  And so the concern is, well, if AI is really all that smart and it's it's heading in a direction that's going to be able to kind of outthink us and maybe maybe even become something that we don't completely understand. Where does that leave us as human beings in terms of our our ability to control our own destiny? So I.

• 9:22Speaker 01

  I think it's it's fine to think about that question and you were you were making a connection to sci fi before. I think that that science fiction has been a remarkable way for people to kind of grapple with some of those fears. I personally have always been more drawn to things like R2D2 and C3PO than say the Terminator, right? The notion that they're that these are machines that are here to help us do what it is that we do and not something that's trying to subvert us and and kind of get above us. In principle, it's it's a possibility. I don't think that it's something that we should be very worked up about. I in in 2016, that was kind of at the very beginning of this current round of excitement around neural networks.

• 10:04Speaker 01

  There was at least in the within the field, it was obvious that something was new here, something was exciting here. And folks at the New York University in in the US put together a workshop where they tried to bring a bunch of AI thinkers together and to talk about this this concern, this fear. And

• 10:27Speaker 01

  what my impression at that time was that, first of all, that a lot of the people who are really pushing this idea were pushing this idea because they they wanted to get attention around it. They wanted people to be thinking about it. But they were overplaying their hand in some ways and they knew it. They they were making this out to be something that was an imminent existential threat. And then when you actually talk to them in detail, they weren't really sure. They were they were they weren't but they thought it was very important from a from a gathering attention standpoint to do this. But my reaction at the time was, if you think this problem is important, and if you think our best minds should be thinking about how to solve it, fear is not a good motivator for creativity,

• 11:07Speaker 01

  right? Playfulness, I think, is a much better mindset to be in if you're trying to make Conceptual breakthroughs. And so I don't think it's really smart for any of us to be so worked up and fearful about this that we stop thinking creatively. And so, so my perspective on it is, yeah, we should be paying attention to issues like this, but we shouldn't let them like keep us up at night and dominate our thinking. We should be thinking about all the various kinds of implications that, that, that you get when you think about, oh my gosh, intelligence is such a defining feature of who we are as a species. What, what does it mean for something other than us to be intelligent?

• 11:47Geoff Nielson?

  With, with that lens of excitement though, right? Which is, which is, you know, so interesting versus, you know, coming at it from a place of fear.

• 11:55Speaker 01

  Yeah, no, exactly right. Exactly right. And I think there are plenty of things that we should be concerned about. We, I don't think this is like party time that we should just be thinking, oh my gosh, all our problems are solved. We can just like ride it out and have a great time. I think we have to be very, very thoughtful and I think engaging people in that process is critical for us as a society. But yeah, I don't, I don't think the fear, I mean, I think, I don't think it's as bad as the people who think things are super bad think it is. And that's across the board. I think in terms of these kinds of sci-fi outcomes, I think those are overblown. And I think it's some of the societal outcomes are, are overblown as well, that we're actually very adaptive as this, as a species and as a society.

• 12:36Speaker 01

  And, and as long as we take things in stride and figure out how to integrate them with what we actually care about, we're going to be fine. We're going to be great. We're going to be better than, than we've been up to this point.

• 12:47Geoff Nielson?

  So, so let me maybe push on that for a minute, Michael. You know, which societal outcomes do you think are, you know, overblown? And are, are there any that you don't think are overblown and, and, you know, we should actually be concerned about for better or for worse?

• 12:58Speaker 01

  Yeah. I mean, so one of the stories that I think people are grappling with tremendously is the notion that, well, if we were to create, if humanity was to create a kind of artificial general intelligence, something that was smart about the world, the way that people are smart about the world, then in principle, they could do the same kinds of work that people do, right? And, and we'd be in a situation where there wouldn't really be any point for us working anymore, which could be great or could be. Completely horrible, depending on how things are framed and how things are structured. What I'm seeing is that I think there's a lot of misunderstanding about what people bring to work, like what it is that they're actually doing.

• 13:41Speaker 01

  And we are not in a situation now where much of that can be replaced. I think that for the foreseeable future, the tools that we have are much better suited to supporting people who are doing work than it is for supplanting people that are doing work. And so that's an example where I feel like it's really like every single new idea that comes out, some journalist stands up and says, oh, my gosh, this projecting this forward, there's nothing for us to do anymore. Humanity is doomed. And I think that's it's really not appreciating the complexity of what it is that people need to do to act in the real world.

• 14:16Geoff Nielson?

  So I'm glad you brought up, you know, the work angle because, you know, certainly, you know, in my line of work, this is a conversation that comes up again and again is, you know, fear of replacement or, you know, why does an organization need employees if they can have AI? Now, you know, as a computer scientist, I know you teach and you work with, you know, developers and certainly, you know, code gen, you know, QA, you name it. There's a, there's a number of augmentative, you know, capabilities that AI can do right now. How are you seeing AI, you know, transform what it means to be, you know, in the computer science space, you know, what, what is it? What's changed for the, you know, the outlook of that role?

• 14:56Speaker 01

  Yeah, let me start with the smallest thing that is kind of a big deal, which is the way that we organize our classes in computer science and generally involve explaining to students, we would, you know, we would like you to write a piece of code that would carry out the following functionality. It's now possible to take these homework assignments verbatim and hand them to a chatbot and it will write the code for you. And so that, at the very least, needs to have an impact on how we're doing this teaching, right? It's not, no one's going to learn anything if all they're doing is, is cut and pasting from the homework assignment into a chatbot and then cut and pasting the program back in the other direction. But that's a, that's a small thing and there's plenty of ways to kind of work around that. I think, so my, my personal interest in programming has always been about thinking of it

• 15:42Speaker 01

  as a way of empowering people to take the tasks that they want carried out and then to convey them to another entity that can then carry it out on their behalf. And so that's always been my focus. That's always been what's exciting to me about computing. And I think this current round of AI gives us all new suite of tools that are super exciting for making it possible for all of us to express ideas to the computer and have the computer carry them out on our behalf. And so that's one of the things that I'm studying in my own research is, well, okay, what does that actually look like? What does it mean? I think a lot of people for forty plus years, there have been people sending the message that, wow, programming is hard. We have to work really hard to teach people programming.

• 16:24Speaker 01

  It's very arcane in its, in its kind of communicative style, right? There's lots of parentheses and braces and commas that have to be all put into, into structure very, very, very precisely. I think what we're starting to see, and this is again what I'm focused on in my research, is the notion that if you take all that off the table, programming is still hard. It's still hard for people to kind of conceptualize what it is that they want done and then to convey it accurately enough that, that another person could do it, let alone a computer. Right? So being able to converse with a computer in natural language is extremely powerful, but it, it raises the question, okay, well, how good are people at telling other people what to do to carry out a task on their behalf?

• 17:08Speaker 01

  And one of the neat, the neat things that we're finding is that, at least for the kinds of tasks that we've been experimenting with, the computers are really good at carrying out, people who are trained in programming are better express, better at expressing in English, in natural language, what it is they want to another person and have that other person carry it out than people who have not gone through that kind of instruction, right? Have not really reflected on the process of, of breaking something down into tasks and then conveying it to, well, the computer. So I, I think it's actually going to be really neat because I think we're gonna, it's going to be easier for people to program because it's going to be less arcane and less symbolic, but I think we're gonna, what's revealed by that is that it's something that

• 17:50Speaker 01

  we all have to actually work on. We have to practice. We have to think hard. It doesn't make it go away. It's not like just because the computer can understand you that you don't have to put in effort to figure out what it is that you want done.

• 18:04Geoff Nielson?

  Yeah. And, and you know, that understanding part is so interesting, right? Because now... Yeah, I mean, as I said, I've done a bit of coding, but there's the and to me, there's always been kind of a misconception, at least in the business world about coding that like, you know, 90% of coding is just like hands on keyboard, you know, like typing in lines of code, how many lines of code can you type? When, you know, it's always like the what separates the best from the rest is like the caliber of that logical reasoning. And how can you, how can you frame what you want? And, you know, if I understand you correctly, it sounds like in some ways that's actually more important than ever in this world of AI.

• 18:44Speaker 01

  That's exactly what I'm starting to see. Yeah. And it's, it's, as you say, logical thinking is part of it, but it's part of it is just being really organized, right? So if you've ever, I see, so my, so my daughter, for example, is a theater person and television production person. And I've watched her organize a group of people to put on a musical. And so much of what she had to do to just to organize that group and to figure out what it looks so much like programming to me, right? Because they, you have to kind of conceptualize what all the different modules are, what their responsibilities are, what they're tasked with, what they need to worry about and what's different from what others have to worry about and how they're going to communicate with each other.

• 19:22Speaker 01

  That's it's so much, that's so much of what it means to, to, to build big, complicated programs is just getting that organizational structure right. And so again, I think that like as a species, programming is just one of the things that some of us do. But organizing and working with other people is something we all do. It's sort of inherent in the DNA of our species. And so I'm hoping that this ability, like if we, if we put more effort, not into teaching people to code in the sense of getting all your parentheses to balance, but teaching people to code in the sense of creating organizations that are effective at doing whatever it is you need them to do. That's going to be good for us on the tech side, but also really good for us on the social side. It'll help us work together.

• 20:07Speaker 01

  And all the really hard problems in our world right now boil down to we need to figure out how to work together more effectively.

• 20:15Geoff Nielson?

  Yeah, wow, it's such a it's such a mindset shift from what you traditionally think of as computer science, or maybe you've been this way all along, and you know, the rest of us are just catching up. But you know, as you put that into practice, you know, as an educator, I mean, has that already started? And you know, what are you finding as you know, students go through this new process?

• 20:36Speaker 01

  It's a yeah, fantastic question. So my own journey at the moment is a lot of this stuff really started to become a big deal when ChatGPT was released, and it became obvious that anybody could interact with this sort of technology and get something interesting out of it. I was already at the the doing my rotation at the National Science Foundation at that time. So I haven't taught a single class since ChatGPT has come out, but I am in touch with my colleagues back at the university. And it's challenging, right? It's there's there's the the kind of just nuts and bolts of, as I mentioned before, how can we restructure our homework assignments so that they are still meaningful. But then it really does, as I think you're raising, raise the question of what do we need to teach people and how

• 21:21Speaker 01

  can we teach it most effectively? And I think those conversations are still quite, quite early. I know that, you know, my colleagues at the National Science Foundation are really interested in in understanding that as well, so that we can help support those efforts with with grants, and you know, various kinds of organizational support so that people can study this. But I think across the world, people are trying to address that question in their own way. So I I don't have any answers yet. It is the case, though, that when when I rotate out of the National Science Foundation this summer, I'm going to be going back to my university, and I will be serving as the inaugural Associate Provost for Artificial Intelligence. So as few people who know what AI is, even fewer people know what provosts

• 22:06Speaker 01

  are. So so I don't know exactly what that's going to entail. But the effort is about trying to coordinate at the university level, all these kinds of questions. What is it that people need to know in the computing area? But what do they need to know in in history as well? Like, how does AI kind of impact that? I am, yeah, super jazzed about working with my colleagues over the whole planet to try to figure out what's the best way of of making this this transition that we're having now as effective as possible, like really bring out our humanity. What is it that we want? And how can we use this technology to bring it about?

• 22:42Geoff Nielson?

  Yeah, so so as we, you know, my my wheels are turning and I'm thinking about, you know, how you put that into action and, you know, the the intersection of, you know, the the education piece, but then also the workforce and what organizations need in this new world. Um, you know, one of the the phrases or roles that's gotten a decent amount of attention these days is prompt engineer. You know, in this age of AI, suddenly we need prompt engineers. Is that a role? Is that a skill? Is it technical? Is it non-technical? Is it a fad? What's kind of your take on that?

• 23:16Speaker 01

  Yeah, it's really interesting. So

• 23:20Speaker 01

  Okay, so so what we're seeing on the technical side is, and what we've seen over the, you know, the past maybe seven-ish years. So sort of pre chat GPT, once these language models were starting to get built, they weren't they weren't, they were surprising, really amazing from a technical standpoint early on, but they weren't really good at what they were doing in the beginning, right? It wasn't anything that was ready for prime time for for regular people to use. But what the researchers started to discover is that phrasing questions to them in different forms could have a huge impact on how effective they were. They were actually, in some ways, very, very brittle, like if you if you didn't get, if you didn't ask the question the right way, you got nothing good.

• 24:00Speaker 01

  If you could just, hmm, if you could just phrase it, if you could just prompt it in the right way, suddenly it unlocks all this potential, all this power. And so that was visible to people, again, I don't know, maybe 10 years ago, that that that concept was starting to come into play. We're still seeing it today. It is still the case that there are people who can make better use of a chatbot than others by being smart about the way that they phrase things.

• 24:27Speaker 01

  I think in the long run, what we see is it comes back to what I was saying before. It's it's people. What is it about the prompting that's really helpful? Some of it is just a trick, like some of these models have certain words that they just are very responsive to. But part of it is taking what it is that you want it to do and laying it out really clearly. Right? So you could, I think in the long run, what we now call prompt engineering, or what is starting to be called prompt engineering, just turns into expressing yourself clearly. Yeah. And if you've if you've ever tried to be a writer, as you mentioned before, like I I wrote a book. I think a lot of people in in journalism have spent a lot of time trying to think of how to hone that craft. Writing is hard. What is it that's hard about writing?

• 25:11Speaker 01

  It's hard because you're actually taking all this complicated stuff that's going on in your head and like linearizing, right, like turning it into a string of tokens. So that when they land in the head of someone else, it triggers the right thoughts, the right ideas, the right expectations. That's remarkably hard, right? Writing is hard. Prompt prompting is hard. Prompting other people is hard. And I think in the limit prompting machines is hard in the same way. It's it's really about putting together your thoughts in a way that paint the right picture. What it's just, I mean, it's it's mind blowing to me that we can have this conversation now because it wasn't that many years ago that people in my field didn't have to think about that.

• 25:53Speaker 01

  It was not the case that we needed to figure out how to tell the computer something in a way that it would that would like land, like that would be expressive to the machine. But that's how people are thinking about it now, that it's it's very, very, very similar to, or maybe the best model that we have is that it's like talking to a person where you have to be clear and articulate and bring up the right metaphors potentially. And that's that's a new space for us to be in. But I think that's, I think at the end of the day, that's that's where all this is heading.

• 26:25Geoff Nielson?

  Yeah, it's interesting because you know, when you were talking about the, you know, the brittleness of it and you have to ask the right question, you know, reflecting on like, some of it feels new, some of it feels not new, like I, you know, garbage in, garbage out, right? Like, we've been saying that, you know, probably since the 80s, if not longer. But it's just never been like this before, right? It's never been, you know, that's been, you know, this kind of abstract, technical thing versus this like conversational, you know, non-technical piece. So, I mean, it's probably too early to ask you this question, but as you, you know, as you end up in your new role and as you think about the field

• 27:08Geoff Nielson?

  of computer science and what it means to be a computer scientist, or someone who uses these technologies, does that field need to expand? Does it need to shift? Like, what could that look like, you know, two to five years from now?

• 27:25Speaker 01

  Yeah, so one of the ways that we conceptualize that, that very question at the US National Science Foundation is to think about education and different kinds of education, different audiences for the education, right? So there's always going to be a need to get people to really understand the guts of the technology, right? So the technology isn't going to mature, it isn't going to grow, it isn't going to turn into other wonderful things, unless there's some people who really know it at that nuts and bolts level. Then there are folks who need to understand and appreciate the technology because they're using it to solve other highly technical, difficult problems. And so we have to get them to understand it well enough that they know its limitations, they know its affordances, right? Sort of

• 28:08Speaker 01

  what it's good at and what you can lean into. And so that's another sort of level of education that we have to pay attention to. Like, what do biologists need to know about this? What do doctors need to know about this? And then there's the, just what does everybody need to know, right? They're not necessarily doing technical work, but so much of the infrastructure of our society is like coded in this way. And so it's, you can't really be an effective person unless you have some level of understanding about what's going on. And so all those levels are really important, and they're all different, right? Because the expectations of what people know coming into the conversation are different, and what they want to use it for is different. And so that means really tailoring the education to these different levels.

• 28:49Geoff Nielson?

  Yeah. So as you tailor that education to different levels, you know,

• 28:55Geoff Nielson?

  As you said, you've got everything from people who just need general education to, you know, specialized education. You've got people who are afraid of AI, people are excited about AI.

• 29:07Geoff Nielson?

  How are you, how are you clustering this? Or like, what are the key ways you're conceptualizing of the different groups to be educated? Is it like, you know, is it age brackets? Is it use case? Like, how do we, what are the macro level patterns we're seeing?

• 29:24Speaker 01

  No, that's, that's really interesting. I mean, the, to me, the top level is what I described before, the kind of, what level of depth about the technology do you really need to have mastered to be able to be effective about what you're trying to do? But you're absolutely right that cutting across that are things like, like age, right? So, so it is the case that, that, you know, K-12, elementary level, primary school, people need some exposure to these things so that they have the right conceptual structures moving forward so that they can bifurcate into one of these, say, three groups as adults. So there is a, there is a difference at the, at the kid level. I think there's potentially issues at the, at the nation level as well, right? So different countries, different socioeconomic groups

• 30:08Speaker 01

  require different interventions to really help them kind of get on board with some of this stuff. And then one of the things that I've personally been really excited about and hope to try to find some way to support is the notion that it's, it's not like you educate people about AI and then they know AI and then we never have to talk to them again. It's, it's, if, if nothing else, and you know, disruption, right? Digital disruption, I like, I really believe very deeply in the idea of digital disruption. And one of the things that AI is providing is a set of capabilities that are useful today, but also the ability to do continual digital disruption, right? The world can keep changing, keep being updated. New, new ways of applying this are coming out all the time.

• 30:52Speaker 01

  And so the idea that we can just educate people and then stop doesn't make any sense to me. We need. Almost a like knowledge distribution network, a two-way knowledge distribution network, right, where really important ideas about those kind of three levels of the technology and the use of the technology and the kind of engagement with the technology are constantly being updated as new ideas come out and information is flowing back from all those groups to say, these are the things that aren't really working for us. Or, you know, this change happened, but now suddenly we can't do this other thing anymore. So I kind of think of it as as an analogy with the way that we distribute software today, right? In the old days, it used to be you write a piece of software, you send it out, people

• 31:35Speaker 01

  use it. You know, maybe someday someone else will write another piece of software. But that's not how it's done today. Now, a piece of software goes out and then security bugs are discovered or just behavioral bugs are discovered. And so there's a way to for the developers to make a change to the code and then make sure that that change propagates out through the network so that the people who are depending on that code have the best version of it. And so to me, I see an analogy between that and the notion of like the AI knowledge that needs to go out there, right? That we need a way of distributing that and we need a way of like doing bug reports, right? The things that aren't working need to make it back to the developers, like the people who are actually creating this

• 32:14Speaker 01

  technology, so that there's this constant dialogue, right? So that we can always be disrupted in the sense that things change, but not disrupted in the in the sense that it derails us and makes us ineffective.

• 32:27Geoff Nielson?

  Right. And the, you know, that dissemination, that ongoing, you know, kind of updating, has there been any discussion about like how you apply that? You know, what does that look like in practice? And is it, you know, is it public sector? Is it higher ed? Is it the actual tech companies? How do you make that work?

• 32:47Speaker 01

  I don't know. That's a really good question. So this is like, I've been taking every opportunity that I have to kind of bounce that idea off people because I'm hoping that somebody will see an angle of it that will be really helpful. I, um, yeah, I'm not sure. I think we have various mechanisms today, things like, like, like journalism, like, like newspapers and magazines and podcasts and scientific papers, right? We have all these mechanisms for getting information out. I just think that they're not. Like they they show us some of the positives, some of the ways that ideas can actually propagate. Because ideas do propagate, but I think the time constant is off, right? Relative to how fast this technology can change.

• 33:26Speaker 01

  We don't have good dissemination mechanisms that um that that can get the news out as quickly as we want. And I so I think, yeah, I think that's sort of on all of us to to be thinking about, okay, well, is it? Is it like so one of the ways I think about it is, could we have like intermediate experts, right? There's like the I talked about those those those three classes of people before, three kind of levels of of of depth of of people with the technology. Could we have folks that kind of sit in the interestances between some of those? So folks that are not developing the technology, but they know it well enough that they can serve as an intermediary to the people who are putting it into practice. Right? So something to kind of fall in that crack. Like, could that be a new, almost professional sector, right?

• 34:10Speaker 01

  Where we support people to help be those intermediaries? That's what we have, like in computer networks, we have computers that just are sitting there, helping to move the information back and forth. They're great computers, but that but that's their job is to make that connection. And so maybe we can have people like that.

• 34:24Geoff Nielson?

  I I'm gonna take this in a weird direction, Michael, but like i- is there potentially a world where like? Like the parallel I saw is like, are there like high priests of AI? You know, like, is it almost like a religion distribution network where, you know, you've got the Pope or you've got, you know, Cardinals? But then you have, you know, local churches and you've got people who, you know, in some way get, is that kind of what you're thinking of? That there's these kind of nodes?

• 34:47Speaker 01

  I had not made the connection to religion before. I will have to kind of think on that a little bit, but um but I think a lot of our human organizations are like that, right? So we have um we have sort of nation governance, but then there's like region governance as well. There's like provinces in the in the in in Canada and there's states in the United States. The European Union has countries, right? But whatever, but but then then down to cities and down to kind of local areas within those cities and the organizations within those cities. So we have, yes, we absolutely have things like that. I think that's what we have to leverage. Is the fact that people do seem to naturally organize themselves into those kinds of patterns, but to to really enable those patterns, right, to make it so that the communication

• 35:28Speaker 01

  between the levels is is sort of faster and more efficient and more accurate so that so that we really we really can activate people relatively quickly. I mean, there's some things that are just that we don't know how to speed up, like it takes a while for people to absorb a new idea. It will take me a while to think about your religion comment, right? I can't do it in the on the fly. And so that's gonna that's gonna set the pace in many ways. But I think we're far from that at the moment. We're not at the level of how quickly can individuals absorb things. We're at the level of how quickly can an organization kind of distribute the knowledge and and the organization absorb things. And I think we can make that actually considerably faster than it is now.

• 36:07Geoff Nielson?

  So so there's a there's a flip side there. You know, the whole area is super interesting. There's a flip side there. And and what triggered me to it was, you know, you were talking about journalism, you're just talking about kind of, you know, information and knowledge in general. Um, you know, to what degree is there like an AI literacy angle that needs to be in place here? You know, we talk about sharing the information. You are you concerned about that being like misappropriated or something like, do we need to instill this kind of these baseline, yeah, literacies in people to be able to absorb this?

• 36:45Speaker 01

  Yeah, no, I definitely believe that. All right. I definitely think that this is this is the role of informal education to some degree, early education to another degree, continuing education to another degree. But it has to be the case that our our public institutions are supportive, right, of that sort of thing. And and like, I don't want to get too close to like, how do you organize a society to be effective?

• 37:10Geoff Nielson?

  It's fair. It's slippery slope. Yeah.

• 37:12Speaker 01

  But I but I think it really does come down to that. It's like, how do we organize ourselves as people? And if we if we do that in a way that is Conducive to this kind of knowledge sharing, then I think we're okay. But yeah, language, like all these technologies are really powerful, all the way down to just like language itself, right? Language itself can be used for good and for evil. And it's really important that we all as individuals are constantly on on guard for, okay, is this, you know, am I ready to absorb this language? Is it really going to make things better? Or is this going to do something bad to me? The same thing is true about knowledge about these kinds of technologies. And so I, I, I think it's maybe acuter, like it is something that we have, but it's not new. It's not unheard of.

• 37:57Speaker 01

  It's just something that we've always had to be be watchful about.

• 38:00Geoff Nielson?

  Yeah, that makes complete sense. I could follow this trail for a very, very long time, Michael. I'm trying to be, you know, conscious of your time as well. I wanted to shift gears a little bit and talk more about the, the work you're doing with the National Science Foundation. You know, you've got a natural excitement, you know, about all of this. Through, through your work there, were there any findings that, you know, were particularly exciting for you that you, you know, were maybe not exposed to before you got down this line of work?

• 38:27Speaker 01

  Yeah, that's a really interesting question. So, so there's, the way that the, the National Science Foundation supports AI research is quite broad spectrum, right? So there's, there's individual projects where, where a single researcher and that researcher's students are funded to like study something that is connected with AI in some way. At that, that's at a very sort of fine-grained level. At a much broader level, there are things like the, the AI institutes, there's national AI institutes that got kicked off around 2020, when, when, when language models were really starting to become a thing, it was sort of obvious that AI was going to be disruptive. And there wasn't really a way for folks outside of computing to

• 39:11Speaker 01

  Participate in the process. And so the NSF created these, these, these large scale twenty million US dollar institutes where each one is focused on some kind of thematic area. And that's been really neat because there's folks, for example, studying AI in agriculture. We have multiple AI institutes that are studying agriculture, they're studying weather, AI in the weather, so meteorology and communicating about meteorology, AI and decision making in the context of natural disasters. Like, there's all sorts of really interesting angles. And it's been exciting to see how some of these institutes have really like taken the ball and run with it and done very, very exciting things. Another

• 39:52Speaker 01

  thing that I've been kind of very pumped about is the notion that, my understanding is that Canada has already been doing something like this, and I need to understand more about it. But in the US, we don't really have the ability for researchers to say, hey, I'm doing this kind of breakthrough AI work, but I need more computers, I need more data, I need more resources to be able to do this kind of work. And I think lots of folks have noticed that some of the splashiest new findings have been coming out of industry where they have the resources to just throw at the problem and just scale it up massively. What can we do to help academics, researchers, sort of people in the public eye to kind of keep

• 40:37Speaker 01

  pace with that and maybe even drive some of the direction of that research? And so there's envisioned a national AI research resource. So basically, an ability for people to apply for resources and then get those resources to apply to the work that they're doing. And this is something that I, I've only been at the NSF for, it'll come in on three years. I've seen it come from like basically an idea that was written down in a report to a pilot that we're actually running right now jointly with a bunch of

• 41:07Speaker 01

  US government agencies and also tech companies working together to try to make resources available to the to the the research public. And I'm hoping before I go to see it actually turn into a fully funded, scaled resource.

• 41:22Geoff Nielson?

  That's that's super cool. I had no idea. So it's almost like, yeah, it's like a partnership or like it's like almost like a marketplace of like, where do people need resources? Who's willing to help them out? And how can we, you know, collaborate to, you know, drive the technology forward?

• 41:35Speaker 01

  Yeah. And it's really interesting to me as a computing person, because it turns out that computing has been supporting other sciences like biology and geology and engineering, mathematics, physics for a long time. Like that we there's actually like a sub community of computing researchers who just like help run the computers that are needed for the other scientists to do to make their breakthroughs, right? To analyze, help analyze their data, move the data around, get insight into the data. And computing has generally said, no, thank you. We don't need that. Like the people who are doing research on computing itself.

• 42:06Geoff Nielson?

  Yeah.

• 42:06Speaker 01

  For various reasons, you know, some of it is like, well, that's not us, you know, and we're we could do it better than you guys can do it. But I think what we're starting to realize is that the the resources needed to do this kind of scaling up of data and extracting value out of the data really requires shared resources. And so we're seeing, again, it just in the three years that I've been at the NSF, kind of a a culture shift. We're moving towards this notion that we could actually be sharing large scale resources. And so we can invest as a as a large community in creating things kind of like giant telescopes, like the astronomers are really good at sharing their resources. So they have a telescope that everybody can make use of to make their discoveries. We we're starting to learn that we need that in computing as well.

• 42:48Geoff Nielson?

  I I I love that. Um, you know, for me, one of the most exciting pieces of that is one of my personal concerns with AI is it feels like it's become more and more sort of like walled gardens inside some of these like, you know, tech mega corps, if if you'll call it that and indulge me. And the idea that we can, you know, as a broader community, start to get those economies of scale, that's, I mean, that's massive. And then it really feels like a force for good.

• 43:16Speaker 01

  I feel the same way. So I'm hopeful that this is something that's going to move forward.

• 43:21Geoff Nielson?

  Oh, that's awesome. That's so amazing. Before I forget, we did briefly talk about AGI as something that maybe, you know, proverbially around the corner, maybe not. You know, do you have an outlook? Is it realistic? Is it coming soon? You know, is it time to think about it seriously? Where are we sitting?

• 43:41Speaker 01

  It is definitely an exciting time. So when I first became an AI researcher,

• 43:48Speaker 01

  let's call it decades ago,

• 43:51Speaker 01

  it was never talked about. Like in the very early days of AI, when the community was first, when it first named itself, there was a sense in which, yeah, obviously the goal is that we're going to create something that is just, it's just generally intelligent. It has the ability to have perception and decision-making and language. Like, it's all going to be integrated into one system. And then the field went out and sort of balkanized, right? It was too hard to solve that whole problem all at once. And it made much more sense to kind of drill down and say like, I'm just going to make one that's really good at math. It's not going to be able to recognize a picture of a dog. Yeah, but it's going to be really great at mathematical reasoning. And so all these different kind of stovepipe style

• 44:35Speaker 01

  work, research and discoveries took place. What we're starting to see today is that the same technology, this neural net technology that underlies chatbots, is actually really useful for perception and decision-making and action and language. It's one kind of fabric that actually cuts across all these different areas of artificial intelligence. And so it's very sensible to start asking, oh, well, maybe we can bring all these pieces together again and have a kind of a unified intelligence of the kind that the founders of the field first envisioned. And that's what I think a lot of people think of when they're, at least people in the field, think of when they hear AGI.

• 45:13Speaker 01

  It is, you know, it's absolutely the right time to be thinking about that kind of integration. It's perfect. Is this something that's going to?

• 45:23Speaker 01

  Kind of change society and the way that that that people live? Possibly, like some people should be thinking about that, too, for sure. But is this really right around the corner? I again, I mean, what we've seen time and time again in the history of artificial intelligence is there's a lot more to being an intelligent person than it appears on the surface. So much of what we do is outside our ability to kind of reflect on it, that it just seems like it's just there, it's just easy, right? For the longest time in AI, it was easier to make a chess program that could play with the the the the strongest players in the history of humanity than it was to be able to have a to like to read a children's story and understand what the moral was.

• 46:07Speaker 01

  Right? Things that we think of as really easy have traditionally been very hard for machines and and vice versa. So um, this this notion that we've actually kind of maybe started to crack that and we can we can start to address some of the harder problems is valid. But I don't think we've gotten to the bottom of, well, how hard is this really to do this in a way that is is really as as rich and sophisticated as the way people do it? I I don't think we're there yet.

• 46:33Geoff Nielson?

  Well, you know, what what I'm hearing as well, Michael, is that it's it it sounds like we don't even understand our own intelligence and how that works well enough to make an educated guess. Like if we would have to be able to say this is when we can have our, you know, AGI to answer that question. We have to know exactly like what intelligence is, and we're not close enough to that answer to.

• 46:56Speaker 01

  I think that's right. And I think I think there's some people, some sort of boosters of the idea who are like, well, we don't really have to understand it at all. We just have to build the machine bigger and it's going to happen. And that may be true, but I think the way you said it is great, which is, but we won't really know that it's happened unless we have some understanding of what this landscape actually is, right? So we may create it by accident, but the more likely scenario is that we're going to fall well short of it because we don't know what the terrain actually looks like. And so to me, that's kind of the next wave of of exciting breakthroughs that we're going to see is that now that we have this this. Artifact, right? This machine that can do so much of what we think of as human intelligence, but not everything.

• 47:38Speaker 01

  It's it allows us to ask in a more scientific way. Okay, what did we really mean when we said intelligent? Right? It's it's been the kind of purview of philosophers. It can now become something much more.

• 47:50Speaker 01

  Empirical, right? And so I think I'm hoping that we come out of that with a better understanding of ourselves. Because I think for a lot of us, that's why we do science in the first place.

• 48:00Geoff Nielson?

  Yeah, yeah, it's crazy. It's a it's a crazy prospect. It's so. It's so interesting. And, you know, to think that we're kind of at that frontier point where, you know, whether we get there or not, we're still pushing the boundaries of. Understanding ourselves. It's it's super, super cool.

• 48:16Speaker 01

  That we're at, yeah, that we're out of we're at a place where we can start to formulate the question. Right? Which is not all the way there to an answer, but it's way farther than we were not too long ago.

• 48:27Geoff Nielson?

  Right. Do you have any predictions, Michael, for, you know, what is the next chat GPT? Like, not necessarily like the exact piece of technology, but but what kind of, you know, functionality or approach? Is going to be the next kind of breakthrough in this space?

• 48:42Speaker 01

  Yeah, I mean, a lot of people these days are talking about agents as if that was a new thing, though I've been. I mean, colleagues of mine have been working in the area of of intelligent agents for decades. I I think what the the notion of agency is, is about these chatbots not just having a conversation with you, but being able to actually make a decision and then act in the world in some way, like help you make a reservation or help you kind of organize. More real world things. And so I do think that that's kind of something that we're moving towards. I think that the. The bigger breakthrough is when we start to have these machines that are structured in a way, I mean.

• 49:22Speaker 01

  So think about starting a new job. So you're smart, you're educated, say you went to college even. You're now going to go out into the world and get a job. It's not like you walk in the door and you just start to do that job, right? It's not like somehow you knew all the pieces of the job. There's kind of getting on board with the job. Like, how do we talk about it here? What is it that we're actually trying to do? How do we weigh these things against one another? And we don't yet have these machines that could that can kind of enculturate like that, right? That can actually be dropped into a new circumstance and then start to take on the values and the goals and the objectives of the organization that they sit in. Because that involves a kind of intermediate level of learning.

• 50:07Speaker 01

  We're really good now at pre-training, where we actually create a neural network that's trained on like the entire internet, like all this data, but that doesn't learn anything new. Right? Because once it's trained, it's trained, and it knows what it knows at that point. But then some of these systems can actually kind of learn, they call it in-context, where you actually, in the context of going back and forth with them, you can give them a piece of information, and it can use that piece of information for some time. But what we don't really have is this kind of intermediate level of learning, where you can learn about a thing, and then six months from now, you still know that thing, right? The new fact, or a new process, or a new relationship can be maintained. It's just not, it's not there yet. And so I think that's going to require

• 50:53Speaker 01

  the companies that are doing this sort of stuff, and the researchers who are doing this stuff, to really engage with what it means for an organization to onboard a new person. I think that's going to be, I don't know, much less sci-fi, right? It's sort of kind of blah, like, oh yeah, onboarding, like with HR. Like, it's like the most boring thing possible. But that's where the value actually starts to come into play. And it's not just a cool toy, but something that is helping people get their work done. That's, I think, we haven't reached that stage yet.

• 51:27Geoff Nielson?

  Yeah, and it's super, super interesting. And you taught me a new word along the way there as well, which is like you said, "inculturation", which to me is like, it's that cultural localization of AI, right? It's not just, oh, I operate out here, and you can have that in your organization. It's What do you need? What does it actually mean here? And yeah, I mean, if we can get to that point and that and that's part of like a gentic AI, that's going to be, that's going to be huge. Michael, do you want to talk at all about what you're going to be doing in your new mandate at Brown? Or is that still a bit too early?

• 52:03Speaker 01

  No, I'm happy to mention it. I think I don't think there's much to say in the sense that it's very, very much like the conversation that we've been having. I think a lot of organizations are excited about AI, maybe a little bit afraid of AI, and they're not really sure how to take it on. And so all the individuals in the organization are doing, you know, what they need to do to get their work done. But there isn't a lot of coordination at the at the organization level. And so I think that's, that's mainly what I see my role as, is to help the organization as a whole, make the best use of this moment. Right? And so I think that's, I think there's a lot of, I mean, the questions you've been asking are fantastic questions, right? These are the exact things that we're all trying to grapple with. I'd

• 52:46Speaker 01

  like that that Brown's take on it, to the my home institution's take on it is, it's, it's very holistic, right? So it includes not just the research about AI, which I think is obviously critical. This is what I spent my career doing, but also the implications of AI for teaching, the implications of AI for other fields of study, right? So what is it, if you're not studying AI, how can AI help you do the work that you do do better? And then things about just the operations of the campus as a whole, right? So there's admissions, there's, there's, I don't even know, I don't know all the names yet, but like, like, there's, there's a physical plant, there's like, maintenance of the buildings, there's the energy of the buildings. These are all things that could benefit from better

• 53:31Speaker 01

  decision making, better data flow, AI, essentially. And so I'm really excited to see some of these ideas that we've, again, talked about and been motivated by in the field for so, so long, we might actually get to see some of the benefits soon.

• 53:45Geoff Nielson?

  Yeah, that's, that's so fantastic. And it's so exciting. And yeah, I'm excited to catch up with you, you know, I don't know how long down the road, when, you know, you can talk a little bit more about the impact that it's having, because it's just, it's such an exciting time. And, you know, it, there's no shortage, it seems like, of opportunities. And yeah, I just, I love, love the education piece that you're doing. It just seems like it's, it's ripe with opportunity for impact.

• 54:11Speaker 01

  Outstanding. Thanks.

• 54:13Geoff Nielson?

  So having been, you know, working in this space, or at least interested in this space for, you know, since the, since the eighties, you know, could you ever have envisioned, you know, the, the trajectory of this, the pace of this, you know, how does it compare with, you know, your, your wildest dreams when you first got into this space?

• 54:31Speaker 01

  Artificial intelligence is a field that forever was 10 years away from the big breakthrough, right? And so that was a, that was like, after I had been in the field for 10 years and realized, wait a second, that wasn't true. It's still 10 years away. Exactly. The fact that we're, that those, those, that horizon actually does seem to have meaningfully shrunk in the last years is stunning. That said, it's really interesting because if you, if you reread, or sorry, sorry, if you read the, um, some of the early work of Alan Turing, who was one of the founders of all of computer science, but also a, a, a leading thinker about what artificial intelligence was, was envisioned to be back in the forties and around there.

• 55:15Speaker 01

  He, um, he lays out in a couple of places like, well, I don't know how to do this, but if I had to do it, here's how I would do it. It's so similar to what actually is happening, right? There wasn't, the field spent a long time trying to take the, the ideas of what we think of as intelligence. This is the way we build a chess playing program is we take what we think is the strategy and we kind of build it into the, the program, right? We actually just program it. We say, okay, well, you should kind of look ahead, play, play out the game, a bunch of steps in advance and see how that, what that leads to, and then propagate back to figure out which move you should take right now. Like we, we took our very literal thought process and tried to encode it as programs. And what Turing said early on is, well, that's going to be too hard. Like there's

• 55:58Speaker 01

  no way that we're going to get all the details of that right. So what we need to do is we, the people, write evolution. We write the, um, kind of the creation of the overall machine that then becomes like a baby and it learns everything. And so this notion of like we the programmer, what we're responsible for is kind of the the platform on which learning appears. And then it's through exposure to actual experiences in the world that intelligence really starts to arise. And you know, son of a gun, that's kind of the structure that we're using that people, human beings wrote code that actually is is really focused on how to learn from data. And then we just hit it with all the data that we can find in the world.

• 56:38Geoff Nielson?

  Yeah.

• 56:39Speaker 01

  And sort of organize, I mean, it's tricky to organize that in exactly the right way so that it all ends up, you know, coming up with something that's interesting. But nonetheless, like, yeah, it's a long time in coming. But the form of it, we saw it in advance and this is really starting to play out. You know, it took sufficient, and he touring even spelled out like, computers would need to be about this powerful for that to happen.

• 57:04Geoff Nielson?

  Wow.

• 57:04Speaker 01

  And yeah, it's like within order of magnitude, he got it. Yeah, yeah. So so it's it's not. It's not shocking, right? But it is, it is very exciting.

• 57:14Geoff Nielson?

  Exciting.

• 57:15Speaker 01

  And it did take a lot of us by surprise. I talked to some other folks in the field who like the first person who taught me machine learning. I got it. He happened to have been making a presentation at the National Science Foundation. So I pulled him aside at the end of it and I was like, you know, when you were teaching that class, why did we not see this? Why did we not see how this was going to play out? Right? Like, we talked about it and we even, we would even suggest this model of we should just predict the next word. If you can predict the word, the next word accurately enough, in principle, it could sound intelligent. Like, why didn't we see it? And he's like, we just didn't believe it. We didn't have machines that were, that had enough data, that had enough compute to actually play out the implications of that. And so we were just thinking too small.

• 57:57Geoff Nielson?

  Huh, that's so cool. So, yeah, it's like, it was waiting for the technology to catch up with the with the principle.

• 58:04Speaker 01

  A little bit, yeah.

• 58:06Geoff Nielson?

  Amazing. Michael, I want to say thanks so much for joining today. This has been a really, really enlightening conversation and I really, really appreciate your time.

• 58:13Speaker 01

  Now, thanks for all you do in helping all of us grapple with these questions and kind of figure out what it means to us. Thanks.