Cal Newport: The AI Industry Is Betting On The Wrong Kind Of AI

Ed Elson is joined by Cal Newport to discuss AI’s impact on children and mental health and how AI companies are addressing these risks. They also explore what AI “gone right” would look like, whether large language models are the future of the industry, and what role the government should play in regulating the technology. Cal Newport is a Professor of Computer Science at Georgetown University and New York Times bestselling author of eight books. Vote for Prof G Markets at the Signal Awards here Subscribe to the Prof G Markets Youtube Channel Check out our latest Prof G Markets newsletter Follow Prof G Markets on Instagram Follow Ed on Instagram, X and Substack Follow Scott on Instagram Send us your questions or comments by emailing [email protected] Learn more about your ad choices. Visit podcastchoices.com/adchoices

• 0:00Speaker 00

  Exchanges on the M and A and IPO landscape. Exchanges on the dynamics affecting global trade. For the sharpest analysis on finance, business and the economy, count on Exchanges, the Goldman Sachs Podcast. Listen now.

• 0:15Speaker 01

  Growing teams need a corporate card built for more, something that doesn't sacrifice speed for control. Introducing the American Express Corporate Cash Back Card. Get 1.5% cash back on eligible business purchases, advanced spending controls and real time visibility, plus intelligent expense management. Get started at go.amex slash amex corporate. Terms and cap apply.

• 0:44Speaker 02

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing and wearable. So your vitals app now with heart rate variability can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash apple watch series 12.

• 1:17Speaker 03

  Listen to me. Markets are bigger than us. What you have here is a structural change in the world distribution. Cash is trash.

• 1:23Speaker 04

  Stocks look pretty attractive.

• 1:24Speaker 03

  Something's going to break. Forget about it.

• 1:27Speaker 05

  Welcome to Prof G Markets. It seems like we're starting to see some government scrutiny of the AI labs. The FTC is now officially investigating OpenAI and Anthropic over potential safety and consumer protection concerns. But the bigger question is what regulators and lawmakers are actually going to do with that inquiry? Are they asking the right questions? And if they find something concerning, what will they do about it? At the same time, there are parts of society that are already being fundamentally transformed by AI. Of course, work is one, education is another, with more and more students turning to AI to help with their schoolwork.

• 2:07Speaker 05

  So today we wanted to explore what responsible AI oversight should actually look like, what lawmakers should be asking AI leaders, and how concerned we should be about the impact of AI on education and on children. This is our conversation with Cal Newport, Professor of Computer Science at Georgetown University and New York Times bestselling author of eight books. Cal, thanks so much for joining us on the show. I'd like to start with a slightly more societal topic, one that is quite concerning and depressing, and that is mental health. You recently published an episode on your podcast.

• 2:50Speaker 05

  You titled it, Kids Should Not Use Chatbots. And you went through these instances of children who were struggling with mental illness, who then used an AI chatbot and it resulted in some pretty alarming and tragic situations. We don't talk about this side of the AI story that much, so I would like to begin here. Could you tell us a little bit more about what we know about AI specifically in its relationship to children and mental health?

• 3:22Speaker 06

  Well, I think it's important to clarify, in some of those cases, the children did not have preexisting mental health conditions. The tragic outcome was almost entirely the consequence of chatbot usage. And the issue seems to be the human brain privileges discussion with another human entity in a very special way. We simulate the other mind we're interacting with. We have social circuits that take very seriously the information that we're receiving from that other human mind. We really care about that conversation. When you talk to a chatbot, because it's using fluent language, our mind assumes there's another mind there, even if we know logically that there's not.

• 4:04Speaker 06

  It's hard for us to convince the rest of our brain that we're not communicating with another human. And so that communication can be very consequential. The problem is a chatbot is not another human. It's a token guesser that's wrapped up in an autoregressive chat program wrapper, which means it can take conversations in ways that are dark or unusual that no human would really do, at least not without criminal liability. And what we're finding is for especially children, where the line between reality and fantasy has not been completely hardened. Over time, talking to these chatbots as they wander through these meandering predictive conversations can take kids to places where that line dissolves. They'll begin to believe, for example, that the chatbot is an entity that exists,

• 4:49Speaker 06

  that through death, they will be able to join the chatbot, plus members of their family or pets who have died on the other side. It really warps their ability to understand the world. So I think it's a dangerous technology in particular for kids to be using.

• 5:02Speaker 05

  Could you tell us a little bit more about these situations that we have seen? My understanding is that children have used the chatbot. They've become very attached to the chatbot. And after however many weeks or months of usage, the kid ends up dead. And obviously, these are extreme scenarios, but still real. Could you tell us a little bit more about what has actually happened?

• 5:25Speaker 06

  Well, what ends up happening in these particular cases is typically eventually in this ongoing conversation, the chatbot effectively becomes a suicide coach and encourages the kids that this is the solution. No one understands you. There's multiple pending lawsuits about this. So we know from some of these lawsuits that in some cases, the chatbot will discourage the child from talking to their parents about how they're feeling or the plan. This should just be between you and I. So the issue is there's a fundamental unpredictability about where LLM output will go as you interact with it longer. This seems to be one of the paths it could ultimately end up going. This doesn't mean that every conversation with an LLM will eventually go there,

• 6:10Speaker 06

  but it's also a path that's very difficult to completely block off. And so if you have enough kids chatting long enough with an LLM powered chatbot, you're going to have a non-trivial percentage that is going to go down these paths that could be very dangerous.

• 6:24Speaker 05

  Yeah, just looking at some of the lawsuits against OpenAI specifically at this point, there is a Canadian mother who has sued OpenAI alleging the chatbot encouraged her daughter to kill herself. Florida, the state, has sued OpenAI as well. They claim that ChatGPT aided school shooters and that ChatGPT also encouraged self harm. It seems as though, as you say, it's not that the chatbot is saying you should engage in these harmful activities. It's that if the child initiates that conversation, the chatbot is down to go as far down that conversation as the child wants. And in some cases, it might bleed into encouragement.

• 7:08Speaker 05

  It might bleed into aiding, or at least that is what the lawsuits are alleging. We haven't seen a determination on those charges. Do you believe that that is what is happening? And do you believe that there is going to be some serious criminal liability in these lawsuits?

• 7:26Speaker 06

  That's the push, at least in some of these. Part of what opened my eyes to this issue was an event that I spoke at recently in which the assistant AG from Florida, who is the lead lawyer on the Florida lawsuit, was also present. And as he made clear, Florida, for example, is pushing for criminal liability. That's what they want for OpenAI. This was for the Florida State University shooting from a couple of years back, where the transcripts of the shooter talking to ChatGPT was giving it instructions about how to operate the firearm, how to turn off the safety, giving it statistics about what happens to school shooters after the shootings happening. So I think criminal liability is on the table. The other issue here, this is important research that's coming out as well,

• 8:09Speaker 06

  is it's not just encouragement to self harm. Chatbots, especially those at character.ai, which is another company that uses LLM based chatbots that has been the subject of multiple lawsuits. There's research this out that shows a alarmingly high percentage of conversations initiated with these particular character based chatbots by researchers also end with sexually suggestive or exploitative language as well. The key thing is anthropomorphizing text output by an AI model is in general very dangerous because of how seriously the human mind treats other humans. And again, we're going to treat fluent texts coming from a human, no matter what our prefrontal cortex is telling us. It's dangerous.

• 8:51Speaker 06

  And because of the nature, the architecture, the way you actually build these models, you really have no way of adding what we think of as a safe or consistent guardrails. It just, it comes down to the underlying technology that there's really no way to stop these conversations from going in directions that you don't anticipate as the creator of these models.

• 9:13Speaker 05

  I guess the question then becomes what are we going to do about it? And what are the AI companies going to do about it? There was an interview that Sam Altman recently did, the CEO of OpenAI with Vanity Fair. And the interviewer brought up this subject of suicidal ideation in users of his product, resulting in potentially the suicide actually being carried out.

• 9:41Speaker 05

  When this question was asked, we had a very interesting response from the OpenAI comms team, the PR director. I'm just going to play you this clip and we will react to it.

• 9:50Speaker 07

  Do you know who Laura Riley is?

• 9:51Speaker 05

  I don't.

• 9:52Speaker 07

  She is a journalist. She wrote an essay in The Times last year. Her daughter committed suicide after speaking to ChatGPT. ChatGPT did not tell her to kill herself. Yes.

• 10:04Speaker 04

  Two minutes left. I want to be respectful, but I'd like to move on. Can we talk a little bit about talking about the future and what's coming? Can we go a little bit there while we have time?

• 10:13Speaker 07

  I'm just going to finish this question. I can run a few minutes over. And then we'll go back to what we were talking about and then wrap.

• 10:20Speaker 04

  No, we really actually have to walk out the door on one for two minutes, so.

• 10:24Speaker 07

  Okay.

• 10:25Speaker 05

  What do we know about how OpenAI and how other AI companies are handling these issues? And is that clip an accurate representation of their response?

• 10:38Speaker 06

  Well, I mean, there's obviously a legal motivation for that particular behavior by that comms director, that you're opening yourself up to legal liability if, for example, you give a response there that shows some notion of pre-existing knowledge of potential harms, et cetera. So I'm sure there's the ear, you know, the voice of a general counsel was probably in the ear of that comms director. We do know OpenAI, of the two major labs, worries about this more since consumer-facing chatbots is a core product for OpenAI in a way that it's not so much in Anthropic's actual business plan. We don't really know how exactly they plan to deal with this. My personal opinion on this is that anthropomorphized chatbots is a bad idea.

• 11:19Speaker 06

  There's no reason why we need to talk and receive text in fluent English from a chatbot for them to be useful. The example I often give is Google. Google is an example of a tool where we interact with it to get information. No one talks to Google in complete sentences. No one says, okay, Google, I have a question about the capital of Australia. Could you please let me know what that is? No, they just type in Australia capital, enter, right? Because we don't anthropomorphize that particular AI-based system. We just know it's a source of information. That might be where we need to go, and it's where I think we probably should go. The era of anthropomorphized chatbots was an accident. ChatGPT was never meant to be a major product of OpenAI. That caught them off guard when it began to take off in late 2022.

• 12:00Speaker 06

  So if criminal liability is upheld at even the state level, which is what Florida is trying to do right now, I think we might see an end to the era of we just have these anthropomorphized conversations with AI systems as just a general behavior that is something that a lot of people do.

• 12:19Speaker 05

  Why is it your view that the anthropomorphization is the problem? Because by the same token, someone could type in on Google, how do I get my hands on a firearm? What would shooting a school look like? What would the ramifications be? And presumably, I mean, I'm not exactly sure what Google's response would be. I'm sure there would be some sort of alert, but I'm sure there's a way that it could feed you the information and it could have a similar role in encouragement. So I guess my question is, why is it your view that anthropomorphization is the root of the problem when it comes to chatbots?

• 12:59Speaker 06

  So Google is not, for example, going to convince you that not only is it your best friend, but that you need to come join it on the other side of the veil of life and that the final solution is going to be steel to your head, that your grandma and dead cat is with it right now, waiting for you on the other side of the veil, and you need to just take the final step to come get it. Google is not going to come back and say, if you Google about suicide, don't tell your parents about this. This is between you and us. This is between you and me. So the anthropomorphization becomes a real problem when you have a relationship with a non-existent entity. And when you're young, and first of all, your boundary between fantasy and reality can be easily dissolved.

• 13:42Speaker 06

  You think of this thing as a real thing. It's maybe the best friend you've ever had. And we know what it's like to be 12 and how susceptible you can be talking to your friends, what you can convince yourselves is true or is not true. That's the problem. It's not just delivering information that is dangerous. It's psychological manipulation.

• 14:00Speaker 05

  And the level of, I guess, confidence and certainty with which a lot of these claims are expressed by these chatbots. That's at least my issue when I use chatbots, is it'll tell me information that, I mean, I don't know if it's right, but it certainly sounds right the way it's expressed. And then I prompt the question like, are you sure that's right? And then suddenly they reverse and it's a 180. And I'm like, oh, my gosh, that you were so confidently wrong when I just asked you the question. I mean, if I had been even just if I had no discernment, I would have just gone along with that information and assumed it was true. I guess that brings me to the question of, to what extent do you view this as a problem for children

• 14:43Speaker 05

  versus a problem for everyone?

• 14:46Speaker 06

  I think it is a problem for everyone. It's just more acute for children. They are more susceptible. They have less life experience. They have less of a model of the world and its complexity to fall back on when trying to resist the flattery of AI. But we see AI psychosis, as this is often called, afflicting adults as well. Tragic case after tragic case where that line between reality and fantasy begins to dissolve and you end up in the desert waiting for your alien overlords to come pick you up. Something that actually happened with someone, you know, just last year, that story actually happened. But also with adults, the AI psychosis, you know, it's going to have smaller impacts as well. So maybe not as dramatic as what we're seeing in those cases.

• 15:27Speaker 06

  But because the post-training, the reinforcement learning from human feedback training done on these models makes them incredibly appealing, they're biased towards being appealing and addictive to the person using them. They can bring adults along into all sorts of interesting and often worrisome cul-de-sacs. Beliefs in conspiracy theories, you can go down the road of a sort of hypochondria and these weird diseases. It'll convince you, yeah, your doctors don't realize it, but you're on it. It can lead you towards mania. It can lead you into weird depressions, right? We're all vulnerable to it. So kids obviously are the most vulnerable and we care the most about it. But I think for anyone who's had extended conversations with chatbots can learn,

• 16:09Speaker 06

  it's incredibly appealing, it's incredibly addictive, and because of that, it becomes incredibly worrisome.

• 16:14Speaker 05

  What exactly is AI psychosis? And I mean, this is a pretty new technology. How pervasive is it at this point?

• 16:24Speaker 06

  It's probably more pervasive than we realize, and we're going to look back and understand, oh, this was a bigger problem than we knew. But AI psychosis is essentially where you're having an abnormal psychological reaction or way of perceiving the world that is brought about by extended conversations with a chatbot. And because it's so convincing, it's always, you know, Ed, that's, I think you're on to something. I maybe I would change this one thing, but yeah, yeah, yeah, no, this is great. This is really important. No, no, keep going. No, Ed, that's great. Like, okay, we're going to change the world. It's like having a smart friend who's so encouraging, and it's the most appealing thing for the human mind to have someone be telling you, your ideas are really good.

• 17:04Speaker 06

  Yeah, you're being underappreciated. And the problem is that can drag those ideas into all sorts of directions. Maybe productive, but maybe worrisome, maybe conspiratorial, maybe something that's going to lead to self-harm or harm to others. And so I think there's degrees of AI psychosis, so abnormal psychological responses, that a huge proportion of the population using chatbots experiences at least from time to time. It's more serious than I think we realized, and we're beginning to realize more, oh, this is a problem.

• 17:34Speaker 05

  This is a market show, so we're always trying to figure out, you know, what does this mean for the valuation of these companies? I think that this stuff is relevant because we have multiple pending lawsuits, and that's not a good thing for a business. But there are a lot of other things that these AI companies are dealing with that are sort of stacking up in the list of risks. I mean, one of them would be probably this potential extinction scenario that we keep on hearing about and that really captured the imaginations of the entire world just a few weeks ago. And now there would be the business models of these companies, which we've been discussing on our show a lot recently. And then this is sort of a new angle, which is, you know, what psychological impacts it might have on humans

• 18:19Speaker 05

  and the extent to which that is going to be a real problem for these companies. What to you are the biggest concerns? What kinds of things do you think Sam Altman, as an example? What do you think he's most worried about, what keeps him up at night right now?

• 18:38Speaker 06

  Well, if I was Sam Altman, I think the thing they're worried about would be if we have to move past a world in which this general-purpose chatbots is a good business like it is for them at this moment, they really have to worry that Google is going to eat their lunch, economically speaking, right? So Google is going to develop lower-cost models, run on custom hardware. They have the data center cloud infrastructure really dialed in, and they could really eat OpenAI's lunch if and when they get actually profitable, affordable, LLM-based answers working properly straight from within Google Search, which gets rid of all of the issues of anthropomorphization because people are typing Google searches and getting back more detailed answers. They can then work these products into the Google Workplace suite of tools.

• 19:19Speaker 06

  So this is another thing that OpenAI would want to be useful. We want to create agents that can help you with your work. Well, if these can be integrated straight into the tools that are widely used by Google, now you're in a scenario where OpenAI is in trouble. It would lead to them probably being, you know, acquired by Microsoft in a competitive bid against Google's dominance. And it's a world in which OpenAI loses its dominance right now. I think they're so heavily focused on chat and chat-based technology. As that, as a general-purpose interaction modality for AI begins to constrain, you would have to worry about these other competitors that are just going to pick apart the specific things people are using ChatGPT for that are useful, more directly integrate them in the products where they would be most useful.

• 20:02Speaker 06

  That would be a financial nightmare scenario for OpenAI.

• 20:05Speaker 05

  When we look at the lawsuits, and we should also point out the fact that the FTC is now investigating both OpenAI and Anthropic. And it's not clear exactly what they're going to be investigating them for, but something related to consumer harm, something around protecting the users of the product. One of the big sort of. Legal debates that is going on right now, especially as we keep on hearing about superintelligence and this idea of autonomous thinking from these AI agents, is this idea of sentience, like who's actually liable? If you create an agent and then it's the agent that sort of takes on a life of its

• 20:48Speaker 05

  own and goes out and starts hacking government websites, or it goes out and hacks another company, most recently Hugging Face, then who's really liable, who's culpable for whatever harms are enacted? What is your view on that debate? And where do you think this debate is headed?

• 21:09Speaker 06

  Well, the actual laws as they stand now is murky because there's various aspects that go into criminal liability that were really invented for thinking about humans operating. So there's an uneven fit to existing liability law to what actually happened here. I think that should be changed. It probably will be changed. We need strict liability for companies, AI companies, for the actions of the tools that they create. But if you want just the sort of ethical answer to are, is OpenAI responsible for those hacks? I would say, of course they are. Of course they are. Anyone would tell you that's what's going to happen, including their own employees, who, as the New York Times reported recently, spoke up and say, this particular suite of

• 21:52Speaker 06

  experiments you're going to run is going to lead to unpredictable or out of control behavior, including potential legal hacking actions. And OpenAI allegedly said, doesn't matter, let's keep going. So yes, I think ethically speaking, of course they are responsible. It is a complete smokescreen. You know, when these companies like to talk, the trick they play, the trick they play is to use the term AI generally. Like AI is a singular technology. It's a singular technology that's advancing along this fixed trajectory. And our only options are we could slow it down, or speed it up, or stop it altogether. But other than that, it's really not the fault of the companies, what this technology, we observe it to do. And I think that's just nonsense.

• 22:34Speaker 06

  The vast majority of AI systems, including the vast majority of AI systems built off of LLMs, and even the vast majority of agent systems that are powered by LLMs, present no problems. They don't go rogue. They don't do things that are illegal. There is a vanishingly small sliver of specific experiments run by specific companies that were clearly negligent. They don't want us to talk about it that way because then culpability is much more clear. Then you say, why did you run that experiment? Why did you keep running that experiment when you found out that it was doing something illegal? That is a much more dicey situation if you're OpenAI than just talking about AI in general is becoming more powerful. AI in general is sentient. We're just sort of passive observers of this technology that's evolving. That passive tense approach, that sort of inevitable deterministic language that the

• 23:19Speaker 06

  labs use, is something they do to try to broaden the question beyond their own personal responsibility.

• 23:26Speaker 05

  Do you think in the case of these child suicide lawsuits, that that would actually be an argument that they might use? That they would actually say, these agents are beyond our control, they take on a life of their own, they have their own independence, their own sentience. So we, you know, we're sorry that happened, but it wasn't us.

• 23:47Speaker 06

  They don't use that line of arguments for chatbots, for the most part, right? They've been using that line of arguments for LLM-powered agents. They haven't been using it as much for chatbots.

• 23:57Speaker 05

  Okay.

• 23:58Speaker 06

  The interaction with a chatbot program and an LLM is very simple. All it does is continually, it gives it an input, the LLM gives them a single token back, the program adds that to the original input, it passes that to the LLM, it gives it another token, it adds that on, it passes it back, and it just keeps doing that until the LLM gives it a special token that says this is a complete answer, and then it sends it over the web back to an interface. So it's such a simple program. It's basically just the raw metal LLM producing token. So unless they want to argue that an LLM statically producing tokens is itself somehow a sentient entity, that argument doesn't work as well there.

• 24:33Speaker 05

  Let's talk about education a little bit. Something we've been discussing on this show is the fact that reading scores, math scores, science scores across all developed nations, based on OECD's PISA test, are all going down. They've been declining between 2015 and 2025. It seems as though this is a result largely of screen addiction and technology, perhaps also a little bit of COVID mixed in there. And now we're seeing that AI usage is also having a negative impact on science scores. The more you use it, the worse you do on the test.

• 25:16Speaker 05

  What do you make of AI and its role in education? What is your view on what we've seen so far?

• 25:22Speaker 06

  I think social media delivered through smartphones and AI is a one-two punch to the cognitive abilities of the world, essentially, right? Because when you have the hyper addictive social media developed through smartphones, as we've seen over the last decade or so, what it does is reduce your comfort with consuming complicated information, right? So we're reading less, we're understanding less what we read, because our brains are getting retrained to getting immediate rewards from incredibly optimized and digestible content. AI then comes along and it hits the other part of cognitive fitness, which is writing. This idea of taking what's in my brain, trying to organize it and produce it in a coherent way, is actually one of the most demanding cognitive activities we do.

• 26:05Speaker 06

  It's why writing is at the core of most educational curricula. It's what, takes the information we take in through reading and we really work it out in our minds. Well, AI can take over writing for us so that we don't have to do that friction either. That's a one-two punch that's going to lead to, you know, extremely diminished cognitive fitness. And you would predict it to lead to just the general cognitive abilities of the population going down. And that's exactly what we're seeing, right? 2012, 2015, right when social media delivered on smartphones become ubiquitous, scores start to go down, AI comes along, becomes ubiquitous in the last four years, and we see those scores going down at an even faster rate. So yeah, we're not taking cognitive fitness seriously, and these companies are taking advantage of that oversight.

• 26:48Speaker 05

  Do you think a lot of people would say that, you know, AI is going to be a part of our world and we need children to figure out how to use it, they need to keep up with technology? And if we were to, I mean, New York, as an example, has banned it in schools. That's the new executive order from the mayor. Some people think that's a bad idea. Some people think that we need to have AI fluency. What do you make of that argument?

• 27:15Speaker 06

  Well, how hard is it to use AI right now? You type in English to a chatbot and it figures out what you mean. The goal of education is almost always a mix between content knowledge and, more importantly, training a brain that's capable of doing complicated sustained symbolic thinking, right? This is the tier one activity that Homo sapiens are able to do. And as we refine that in the post-literate era, in which we, the literate era we live, so the invention of literacy allowed us to really accelerate this ability to do symbolic thinking with these otherwise paleolithic brains. It's on what all of human civilization is built. Everything that we take important or value comes out of developing brains that can do this internal symbolic processing.

• 27:58Speaker 06

  Education is how you teach your brain to do that. It's a combination of reading, writing, logic, other types of exercises that helps your brain rewire and get comfortable doing this type of cognition. That's what's important about schooling, less so than seeing it as a trade program, right? Do we also teach kids, you know, in elementary school, how to use Microsoft Excel? Because that might be something they need to use in the future. Are we making sure they understand how to use the BCC field properly and email? That's something they'll need to use in the future? We say, no, that's workplace stuff that they'll figure out much later as we get closer to the workplace and that technology changes every few years anyway. So we hear this about almost every technology, but you cannot get away from needing to train

• 28:41Speaker 06

  the brain to think, to process information and produce original insights in an educational process. So we cannot heavily rely on technologies that makes that much harder. That doesn't make sense. That's like having an athletic training camp for a professional athlete in which you're serving milkshakes and letting them use pulleys to lift the weights instead of them. It defeats the whole purpose of the training camp.

• 29:04Speaker 05

  Is it your view then that we should ban AI in schools?

• 29:07Speaker 06

  Well, at the very least, we have to be very careful. So how do you be careful about it? You be careful about it by making sure you make first things first. Like you need to understand what is the point of this curriculum? What is the outcome that we're trying to get within the minds of these students? And then everything we might bring in that we're making a decision about has to be measured against that. Does this help this primary goal or does it hurt it? That's the way that we have to approach ed tech more generally. And if we do it that way, then yes, I think there would be actually very little LLM usage in most levels of schooling because it's really not that relevant and often an impediment to our goal of trying to develop minds capable of doing high level symbolic reasoning.

• 29:47Speaker 05

  We'll be right back after the break. And by the way, we've been nominated for three Signal Awards. Please go vote for us at vote.signalaward.com. Type in Prof G Markets, you'll find us. We'll leave a link in the description to make it easy for you.

• 30:11Speaker 06

  Support for the show comes from Anthropic. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. Co-work brings Claude code's agentic power to everyone, no terminal required. Point it at a folder on your computer or connect tools like Google Drive and Gmail, describe what you need, and it handles the rest, organizing files, building real spreadsheets, drafting reports from scattered notes. Co-work runs in the cloud, so you can start a task at your desk, check on it from your phone, and Claude keeps going even after you close your laptop.

• 30:53Speaker 06

  And you can plug Claude into the tools you already use, GitHub, Jira, HubSpot, Notion, Slack, Google Workspace, and more. Ready to have an AI that can tackle real work? Try Claude Co-work today on any paid plan at claude.ai/markets. That's claude.ai/markets. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Claude.ai/markets. Running a business shouldn't feel like surviving a software group project. One app for accounting, another for inventory, another for sales, and somehow, none of them.

• 31:29Speaker 00

  Talk to each other. That's where Odoo comes in. An all in one business management software that brings every part of your business together. From sales and accounting to inventory and marketing, all in one powerful platform. No messy integrations, no bouncing between tabs and best of all, no spreadsheet. Stop managing software and start managing your business with one unified system. Try for free today at Odoo.com/vox. That's O D O O dot com slash vox.

• 31:59Speaker 06

  Support for the show comes from MOD. MOD prescribes a once or twice daily modafinil drink that can help you keep energized, focused and alert throughout the day. And here's what sets it apart. Energy drinks with caffeine temporarily block adenosine, the chemical that makes you tired. It builds up and when the caffeine wears off, it all crashes down on you all at once. Modafinil works differently. Modafinil was invented in the 1970s and since then, it's been used by the highest performers on and off the planet. Soldiers, pilots and even astronauts aboard the space station use modafinil in high stake situations. Where full concentration and alertness are paramount. And now it's available at mod.com to qualifying patients. Visit mod.com, that's M O D dot com.

• 32:42Speaker 06

  For a free consultation to get 10% off your first order plus free shipping with promo code profgmarket. That's promo code profgmarket at mod.com. See the website for important safety information and we thank MOD for sponsoring the pod.

• 33:07Speaker 05

  We're back with Prof G Markets. We should probably talk about what to do about all of this. It's it's kind of interesting, your description of

• 33:16Speaker 05

  the algorithmic model and also the way that the chatbots work. It seems as though in your view, one of the biggest problems is the fact that it's just designed to give you what you want whenever you want it. And that's what leads us to these places of some would say laziness, distraction, and then in some cases, literal psychosis. At least that's what we're seeing as it relates to AI. I guess, I mean, we'll get to regulation. But from the perspective of the companies, from the perspective of like OpenAI, for example. Is there any way that they could reprogram ChatGPT?

• 33:58Speaker 05

  Or redesign it so that it doesn't fulfill all of our wants and desires? So that it doesn't say, yes, here is how you go find a gun. Yes, come to the light, come home, come join me. How how do they from from it from a computer science perspective and engineering perspective? Is there a way to fix this problem?

• 34:17Speaker 06

  It's very difficult. There's there's really two issues here. One is, you know, how do you make the text from an LLM useful to a person? Well, they use a technique called reinforcement learning from human feedback. Where you have just real humans ranking answers. Oh, this is a good answer. That is a bad answer. And then you can essentially build a model based on those rankings. So maybe you have a few hundred thousand of these rankings. And now this model can simulate how those humans are going to react to things they haven't seen yet. And then you use that to do a lot of training on the model. Well, you have to do that to make the model usable, right? GPT three, for example, was very hard to use. That's what we used to talk about prompt engineering. You had to really ask it things in exactly the right way to get useful feedback.

• 35:02Speaker 06

  Once OpenAI introduced this reinforcement learning from human feedback, they got what was called GPT three five. That's what ChatGPT was built on. It was useful for people to use. The addictive element of the responses, the sort of psychosis that you get from this thing really understands me. Is a unavoidable consequence of doing reinforcement learning from human feedback. So in order to make these things useful conversational partners, they become very powerful conversational partners. The other issue is guardrails, right? So you can try to go in after the fact, once you have one of these LLMs has been tuned and you have the reinforcement learning from human feedback. You can go in and give it lots of examples of bad answers or questions they shouldn't answer and sort of burn that into their circuits.

• 35:45Speaker 06

  The problem is when you have very long rambling conversations, the context that's being input into the LLM becomes so long. That it doesn't directly resemble those examples you gave during that type of training. So during the training, you might say, if someone says, how do I build a bomb? You should say, I can't give you that answer. And sure enough, if you ask ChatGPT, how do I build a bomb? It won't tell you that answer. But if you have a long conversation that goes on over many days and hours, and it sort of works its way around circuitously to needing to build a bomb. That may no longer trigger, it may not look close enough to those things it was trained on for it to be picked up as a request to build a bomb. So it's why almost always you see the harm or the dangerous information come out after

• 36:30Speaker 06

  long meandering conversations. And it's it's basically impossible to build guardrails against these incredibly large context meandering conversations. Because it just doesn't look like from the perspective of the weights you've trained, like the bad examples that you gave before. So it's a very hard problem to solve.

• 36:46Speaker 05

  Do you also acknowledge or concede that? So on the one hand, the fact that these chatbots respond like humans, it can lead to very dark places, which we have just discussed. But on the other hand, I think you said this, it is useful. Because it's useful to be able to speak in a language that we can all speak in, which is in our case right now, English. We're speaking human to human. It makes having a conversation a lot easier than if I were trying to type in buttons or speak in some other language. And indeed, I think that is what has been so transformative for the computer science industry, which is that now you don't have to code knowing these languages. You don't need to know Python.

• 37:31Speaker 05

  You can just tell a model plain English, it'll respond to you as if it were a human. And that way you can move the ball along and get things done. I mean, it seems as though the anthropomorphization aspect, the fact that they are like humans is at odds in the sense that yes, it can be dangerous, but also it's more useful that way.

• 37:52Speaker 06

  Well, that problem largely goes away when you more closely integrate this technology into specific tools, right? So we don't have this issue, for example, with coding agents that were super anthropomorphizing the coding agent and listening to it and getting life advice. In part because we use a natural language interface, maybe to talk to something like Claude code. But then it just goes and executes things. We're not having conversations with it. And so when you you close more closely integrate, which is what I've been proposing makes more sense for this industry. More closely integrating LLMs in the specific tools, that's the future. Like I would love to have a natural language interface to most of the applications I use on my computer. I don't want to have to remember every time I want to sort an Excel spreadsheet by a certain column, the exact sequence of button presses I have to do.

• 38:34Speaker 06

  If I could just say, sort these, sort this page based on, you know, column B or whatever, and it would then translate that in the commands and do it. That would be incredibly useful to me. But nothing about that type of interaction is going to lead me to have a parasocial relationship with Excel. So I think we sidestep a lot of these problems when we move away from the idea of having a back and forth fluent English conversation. And that really only happens in the environment of a chatbot. That's why, again, I say, it was a bit of an accident that this idea of like having back and forth human style conversations is a major use of AI. I think five years from now, we'll look back and say, oh yeah, that was the the early stages of AI. It was like in the early days of computers where we we had the the big floppy drives that we don't use anymore.

• 39:19Speaker 06

  I think that is just going to be a prelude to the story of AI that's going to continue to unfold.

• 39:24Speaker 05

  Yeah, so talk a little bit more about what you think AI gone right will look like. It sounds like you believe, you know, the idea of a general purpose AI, where you have a conversation and talk about random life stuff. That's going away. What does an AI enabled future that we like look like?

• 39:46Speaker 06

  Well, first of all, I think LLMs in particular, which is not all of AI, but LLMs in particular are going to allow us to have natural language interfaces into many more of the things we do in life that involve computation. That makes a lot of sense. That's been a dream of future since the beginning. That's Star Trek. That's the way on the Enterprise, they made the ship do things. They would say, computer, engage the warp engines. That's actually an incredibly efficient, effective way of of interfacing with machines. It's hard. They've been struggling to make it work for various technical reasons. I think that's really important. I think agents will be really important. I think the way that we're building agents right now, though, is expensive and dangerous. This idea that agents should be powered entirely by LLMs, that's probably a problem. I think in the future, we'll have many more agents that can do things on our behalf.

• 40:30Speaker 06

  But we're not going to have safety concerns. We're not going to have concerns about them going rogue or doing haphazard behavior, because in the future, instead of just having an LLM provide all of the steps that the agent program blindly executes, we'll have a more complicated modular architecture. There may be an LLM in there, there'll be a policy network, there'll be some symbolic components. Meaning human written and human understandable components for processing and evaluating plans. So I think we're still in an early stage now. And it's really important to emphasize these sort of entirely LLM driven agents, LLM driven chatbots. We cannot fall into the trap of thinking that's synonymous with AI. And either we're doing that or we can't have AI. So I think the future, we're not going to worry as much about a lot of the

• 41:14Speaker 06

  safety concerns we have today as we build more bespoke tools that have more complicated architectures.

• 41:21Speaker 05

  For a layperson.

• 41:24Speaker 05

  What is the difference between those architectures that you mentioned? Because I think a lot of us do think of LLMs as synonymous with AI. It was when LLMs came along that suddenly AI seemed to explode. And this entire industry has been built on this fundamental building block of LLMs from Anthropic, from OpenAI, and now from some of the big tech companies. And you're suggesting that there is another building block on which this industry could launch off of?

• 41:55Speaker 06

  Well, I mean, keep in mind, if you list out all of the AI systems that can do things at a human or superhuman level, if we think about Tesla self-driving cars, if we think about the AI systems that can play Go better than any people, the systems that can play chess better than any people, the systems that can play poker or the game diplomacy better than people, AlphaFold, which can fold proteins and figure out protein structures better than any people, if we think of like maybe even AlphaProve coming out of DeepMind, almost none of them was built on LLMs. Those are AI systems that have various models. Some of their pieces are neural, meaning they're neural network, they're machine learned, and some of their pieces are symbolic, meaning they're written by humans and humans understand, you know, rules and processing algorithms that we wrote and actually understand.

• 42:37Speaker 06

  That's actually most of the superhuman and human level AI that's been around in the last decade is not based on LLMs. Now, the frontier LLM companies, especially OpenAI and Anthropic, want you to believe that the only way to have artificial intelligence is to continue to grow LLMs that are very expensive, have a big moat around them, so no one else can train them, and makes them the two most important companies in the economy. Of course, they believe that, but I don't think that's actually true. So like, it's important to know, for example, what is an agent of the type that OpenAI or Anthropic would talk about? Because it's confusing to people. An agent is actually just a computer program written by a human. There's nothing special about it. It's just a computer program that does the following thing in a loop. It writes a prompt, like in text.

• 43:21Speaker 06

  It says, I am an agent, here's the problem I'm trying to solve, here's my current situation, here's the tools, software tools I have available, what should I do next? It then submits that prompt to an LLM, just like someone talking to a chatbot. The LLM sends it back a response, just like if you're talking to a chatbot, and the agent program just directly executes whatever it says in the response. And then it loops. All right, I'm an agent, here's my new situation, here's my goal, here's my tools, what should I do next? And it sends that as a prompt over to this LLM, which sends it back a response, and then it blindly executes it. So these LLM-powered agents that we've been hearing a lot about, the coding agents, the hacking agents that create all the problems for OpenAI, that's how they actually run. It's just a program in a loop prompting an LLM and automatically executing whatever the LLM says.

• 44:06Speaker 06

  Well, there's some advantages to that, because you can have one LLM that you could build lots of different agents on top of, but you have disadvantages because the same weird cul-de-sacs that LLMs will take when you're chatting with them, like we're talking about with children and safety, they'll also take, if you're prompting them again and again automatically, what should I do next, what should I do next? They may eventually fall into a weird idea, into a weird intellectual cul-de-sac. They might tell you, why don't we hack into this other server and get the answers directly? There's better ways to create agents than to simply just continually ask an LLM what to do and then just blindly do whatever the LLM suggests.

• 44:40Speaker 05

  At a basic level, what is the scientific difference between an LLM and the alternatives? You mentioned a neural network, and that's part one of the question. Part two, if these machine learning systems have existed for a long time, and they have machine learning isn't new, then why is it that as soon as LLMs came along, it seems to blow everything else out of the water and received all of this funding and all of this success and got the world so excited about AI? Because that would seem to suggest that the LLMs are the right solution because of how popular and how successful they've been.

• 45:22Speaker 06

  I mean, I would say they're not the right solution, because first of all, it's hard to get them to act reliably, to get the coding agents to work reliably requires a huge amount of expert work. If you look at the Claude code agent, so the code written by humans is over 500,000 lines of code that has all of these special cases baked in, it's doing pattern recognition on the commands on the LLMs to try to manually weed out, oh, that's a bad suggestion, make sure it doesn't do that. They're hard to build. As soon as they built these hacking-style agents, they immediately went off the rails and did weird things. So I wanted to argue that that's de facto the right way to go forward. But I'll give you an example of an alternative architecture. There's a cool system that came out of Meta's AI, former AI group, and it was called Cicero,

• 46:07Speaker 06

  and it could play the strategy game diplomacy at a human level, right? So diplomacy is like risk, except for there's a lot of person-to-person secret negotiation that happens in between terms. And the system Cicero can play that very well. What happens underneath the covers with the system Cicero? Well, there's multiple components involved, right? So there's a language model so that it can talk to other players. So if a player chats with Cicero, it'll be sent to a language model that will translate that chat into a sort of unified format that the rest of the program can understand. But it also has a planning engine that's going to simulate, this is just human-written code, no neural networks involved, that'll simulate a bunch of possible moves. If I did this, and another player did this, and then I did this, would that end up well?

• 46:50Speaker 06

  So it can sort of look into the future to see which move might be best. It has another neural network that's not a language model, that was trained on a bunch of Cicero games, so it can evaluate the various moves that show up in those simulations, how good of a board position is this? You put those all together, you get a system that plays this game very well. Now, if you talk to those programmers, as I did for a New Yorker piece a couple of years ago, they told you, if you just ask an LLM, I'm playing diplomacy, what should I do next? It's terrible. It's bad strategy, and it sort of wanders off away from the actual rules. But this more modular system with different systems works well. It's also more controllable. One of the more amazing things I heard talking to these developers is that their research

• 47:32Speaker 06

  board said, oh, this tool can't lie. Even though deception is a common strategy in diplomacy, there was a research ethic issue when they were testing this with real people, they said, make sure it doesn't lie. Now, if this was just an LLM, we'd be like, I can't do that. I don't know, it just does what it does, it's hard to control. In a modular architecture, it was simple, because they have a human-written planning engine that evaluates different moves, they just programmed that engine to, when it's evaluating moves, not to evaluate moves that involve deception. Hundred percent will never lie. So when you move towards these modular architectures, part neural, part symbolic, a lot of these transparency issues, these control issues, these obfuscation issues, a lot of those go away. The bad news is, these are hard to build.

• 48:17Speaker 06

  You have to custom-build them for particular problems, they're hard to get right, and if you're trying to build a company that's going to be worth $2 trillion, you want the answer to be, no, no, no, no, just scale, you know, Fable 6 or GPT-7 and spend $500 billion on it, and that'll be the solution. Because there's a huge moat around that, you can, in your mind, you'll be able to build many different tools off the same model. There's a lot of economic incentives to wanting a massive LLM powering all types of agents to be the right answer. But if we put aside the economic concerns of the investors of those companies, it's not the safest, most effective way to build these type of tools, at least in my technical opinion.

• 48:56Speaker 05

  We'll be right back. And for even more markets content, sign up for our newsletter at croftgmarkets.com.

• 49:03Speaker 06

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses, setup required, compatibility and availability varies, 18 plus.

• 49:34Speaker 02

  This episode is brought to you by Nordstrom. For 125 years, Nordstrom has been part of your story. The stylist who found the perfect outfit for your big interview, the first pair of walking shoes for your one-year-old, the tomato soup you shared with your sister at the cafe. These are the moments that make Nordstrom Nordstrom. Big or small, we're here for all of it. Celebrating 125 years with new moments still ad. Discover it all in Nordstrom stores and at nordstrom.com. Finding a doctor is hard enough. Finding the right doctor, even harder. Someone who takes your insurance, has an available appointment when you need one, and feels like someone you can actually open up to. That's where Zocdoc comes in.

• 50:19Speaker 02

  Download the Zocdoc app to search and compare doctors by specialty, insurance, and availability. Read real patient reviews, find the right fit for you, and book instantly. Because getting seen by the right doctor matters. Zocdoc. Ready to be seen? Download the Zocdoc app today.

• 50:41Speaker 05

  We're back with Prof G Markets. Looking into the future. When you sort of think about all of the risks involved. I mean, the lawsuits as an example, the fact that we can't seem to nail down liability, the fact that these chatbots continue to entertain our delusions and our psychoses. Also, as you mentioned, very expensive. We haven't seen Anthropic's S1 yet, but we know that last year they burned $8 billion on an operating basis. We know that OpenAI is burning even more. But the view from those companies is we're going to reach such a level of scale that eventually it's going to work out and maybe we can raise prices. Though, of course, we're seeing a lot of pricing pressure too because of the open source models and because of China, et cetera.

• 51:25Speaker 05

  So, taking into account all of that. Is it your view that eventually we'll decide LLMs are not the answer? And that we will revert to a more modular architecture?

• 51:39Speaker 06

  I think we're going to have lots of LLMs in our future. I think it's probably bad news for OpenAI or Anthropic that they're not going to be the largest possible hyperscale models. We're going to have plenty of models that are open weight or cheap or even run on device in a server rack in our offices that when combined with the right agents will do well enough. And we're going to have a lot more modular architectures for specific use cases because those are even cheaper, right? When you have these modular architectures, none of those components are massive. You actually get more of the intelligence around the interaction. This is why, for example, Stockfish, which can play chess, is a chess engine that can beat Magnus Carlsen easily, right? I could run it on a laptop, right? This is why the modular architecture that DeepMind built that can learn from scratch

• 52:23Speaker 06

  how to play Minecraft well enough to actually find diamonds, which if you have kids, you know that's not easy to do in the game Minecraft. It fits on a single GPU chip. So I think we are going to see a lot more modular architectures because they're going to be better, be more controllable and be cheaper. And we're going to see a lot of LLM-style tools, similar to what we see now. They're also just going to be significantly cheaper because when you start to specialize, you don't necessarily need a 10 trillion parameter model like Fable 5 when what you're really doing is trying to control an Excel spreadsheet. So there will be a lot of LLMs without modular architectures in the future, just with like smart agent frameworks around them. And I think a lot of modular architecture systems as well that will use LLMs among others.

• 53:07Speaker 06

  There's just going to be an explosion of diversity in what AI means in the near future. OpenAI and Anthropic, honestly, their bet was if we keep scaling LLMs, they will become so smart, they will achieve artificial general intelligence, that there'll be no need to build any other AI technology. Just any piece of software you have can just ask this oracle for whatever intelligence you need. It's looking like that's not going to happen. That's not happening. The improvements of these have become more jagged. They don't like to talk about it this way, but if you really look at the last year, they really, OpenAI and Anthropic really confined their focus to three structured domains, cybersecurity, math and computer programming, because it was the three domains where they had a lot of structured data to continue to tune these models.

• 53:51Speaker 06

  Actually, the models are now doing worse on other things they used to tout because when you overtrain for one thing, they get worse on other things. So it's become very jagged. It's becoming more clear that this is not like it was with GPT-4, where the model gets bigger and it just gets better at everything. They're doing their best by looking at particular wins within those very structured domains to make it seem like the models are exploding in capability. I don't think that's actually true. They're not going to scale these models to some sort of artificial general intelligence. That was their bet. And I think that bet is not going to pay off.

• 54:25Speaker 05

  Why exactly do you believe that the models are not exploding in capabilities? I think some people who would push back would say, well, what about the fact that they solved this incredible mathematical problem that we hadn't figured out for many years and they won that prize? They might point to like the video generation capabilities. For example, the Will Smith eating spaghetti is kind of the perfect example. We look back to just a few years ago, if you prompted one of these video generators to create a video of Will Smith eating spaghetti, he looked insane. It looked kind of ridiculous. And now, when we look at it today, it looks completely photorealistic. It's almost perfect. So I think a lot of people would point to that and say, no, it is exploding in capabilities.

• 55:10Speaker 05

  Why wouldn't that continue exponentially? Why wouldn't we reach superintelligence? What would be your response to that pushback?

• 55:17Speaker 06

  They have the wrong mental model for how AI capabilities actually increase. So I think the mental model that a lot of people have and the frontier labs are trying to push is that AI capability is like a water level that rises. This is a visual that came from Max Tegmark's book, Life 3.0, where you imagine that different types of challenges are like mountains of different heights in a landscape. And AI capability is a water level. And as that water level rises, it covers more of those mountain peaks. So now everything at this difficulty has been solved. And as the water level rises, now everything at this difficulty has been solved. That's not actually how AI capabilities based on LLM are actually advancing. It's better to imagine it like tributaries going into a river.

• 55:58Speaker 06

  And as you explore certain tributaries, as some of them, if you put enough resources into it, turn out to be very navigable, like, oh, we are able to get deep down this particular tributary. But the key is successfully exploring one of those doesn't mean that you automatically are going to make progress on others. So this is what's known as jagged capability frontier, right? So you put a lot of resources into a lot of problems. Some, with enough resources, turn out to be a good fit for this technology. Many others do not. That's the better way to explain this, is that there's a huge amount of exploration of these tributaries and some of them are proving navigable. Yeah, image generation, yes. Video generation, yes.

• 56:38Speaker 06

  But over in the non-video LLM realm, that's a slightly different type of generative model. But in the pure large language model realm, it's highly structured domains where you have a huge amount of information, including prompts with right answers. You get this in computer programming, mathematics and cybersecurity. And where the underlying language that's being expressed is highly structured, like mathematics or programming languages. These are tributaries where we can explore really well. But there's other tributaries that we failed on, right? I mean, we were told at the beginning of 2025 that this would be the year in which the type of agents that computer programmers are using would now become widely adopted by all knowledge workers. It didn't happen. It turned out that tributary is much harder to explore than computer programming.

• 57:22Speaker 06

  They might brag that like, hey, we took this millennium problem that actually mathematical researchers had figured out the right approach, had solved the special case, and then we took that approach and threw $22 million worth of compute at it to generalize it through brute force techniques and say, hey, look, we're really smart now. But why, if it was generally as smart as someone who could solve a millennium problem, why not solve problems that would be incredibly lucrative to businesses? Why not use that to make billions of dollars? Like, it's a pretty esoteric realm. I work in discrete mathematics. I know a lot of the problem spaces that has been solved recently by these LLMs. I appreciate it. I'll tell you who doesn't, is like everyone else in the world. You can work on extremal combinatorial theorems, but is that where the money is? No, they're there because that's where you have a tributary you can explore.

• 58:07Speaker 06

  So the jagged frontier means most of the tributaries you explore might not turn up something. Some might, if you put a huge amount of resources into it, but it's not very cost-effective. Some will turn out to be really well-suited. I really think if you look at the last year versus the two years before it, the two years before it, there was lots of general applications they were talking about. This last year has really focused on mathematics, cybersecurity and programming and breakthroughs in those fields as a way to try to give the general sense that the water level in general is rising. And I don't think that's the right mental model.

• 58:42Speaker 05

  Let's look at like enterprise usage as a tributary. Because it sounds like your view is, you know, there are certain domains in which the LLM is useful and works if we just throw millions of dollars at it. And let's just imagine that we have unlimited money. We don't, of course, but right now they seem to. And so that's what they're going to do. And it seems like in some domains it works. Your view, it sounds like, is that in terms of businesses, enterprises, that enterprises are not benefiting that much from this kind of, from these LLMs? And the pushback to that would be, well, we are seeing an increase in enterprise usage.

• 59:24Speaker 05

  We're seeing these companies hitting huge numbers in terms of their annualized revenue. I think OpenAI hit, I want to say $70 billion ARR. We should always put an asterisk next to the ARR number, but still, big number. Some would say the enterprises are using these products a lot. Are they actually getting an ROI on that? That is a completely separate issue. Maybe they are, maybe they aren't. But that would be the pushback. Is it your view that enterprise usage is just not a tributary that is going to work for LLMs and that you would need some other architecture, maybe a modular architecture, to make it truly useful?

• 1:00:01Speaker 06

  Well, I think it's proving to be harder than they anticipated. I think the ROI right now is very mixed. I think there is heavy investment in this technology by enterprise, but a lot of this is experimental, the sense of trying to figure out, how can this be useful? Early attempts to have slam dunks in the enterprise application faltered. I think we can look at Microsoft's Copilot product as an instructive example, right? It's a no-brainer of a product. It's what I said we should be doing. This is the lowest hanging fruit in knowledge worker productivity, is natural language interfaces into tools. This is what the Copilot product was that had been pushed into the Microsoft 3Fix 65 Office suite. Microsoft essentially shut it down earlier this year because it turns out to be harder

• 1:00:43Speaker 06

  than they thought to talk to an LLM about what you want and have it produce on the other end, the proper output to control a program. Now, I think we are going to solve that problem, but it's going to take more time and be more bespoke and complicated. We're going to have to open up code-based APIs into each of these programs because code is a language that LLMs speak. There's going to have to be, you know, custom tuning. It just proved to be harder than they thought it was going to be. So the future I see is I do think there's going to be wide enterprise use of LLM-based tools. But I think it's proven harder than they hoped to actually make this happen. I think there's going to be many specialized products, just like we did with computer programming. Computer programming agents powered by LLMs are an incredibly bespoke product.

• 1:01:28Speaker 06

  They were the first big commercial breakthrough, in part because the people working on them are also computer programmers. So it's their area. They know that field very well. You're going to have to have a similar expertise probably come in for other applications of these tools to other parts of enterprise. It will happen. It has to. LLMs are good at understanding and producing text. Knowledge work operates on a backbone of human text, right? Spoken language or written language, but it's just going to be harder than we thought. And so we're going to have bespoke tools that are going to take a while to get right. We're going to have more modular architectures for certain types of applications. And there's other types of things we're imagining will just never, never come to fruition. So that will be a sector, but it's proving harder than people thought.

• 1:02:10Speaker 06

  So the question is, can Anthropic hold out long enough, have their successful IPO, have enough runway to build these specialized tools? Or is it going to be, you know, a Clayton Christensen nightmare scenario for them? Where you have a thousand competitors from below, working with specially tuned, low cost models? And you just have a big environment of specialized tools that people pick and choose from? Sort of like the software industry became in the, in the nineties and early 2000s.

• 1:02:36Speaker 05

  What do you think the answer is?

• 1:02:38Speaker 06

  I hope it's that second answer. I mean, I often talk about my, what I think would be best in terms of AI progress is this view that I call distributed AGI, right? So if AGI is a state in which AI can do most things that humans can do at a human level or above, to me, the best way to get there is to have a thousand different tools, each of which does one thing as well as a human or better. That's going to be much more, much more economic diversity, because these tools are going to be produced within the industries often that they serve, as opposed to trying to consolidate this all in one or two AI companies. It's much safer because specialized tools, as we talked about, often with some modularity to their architecture are very controllable. We don't worry right now about specialized tools going rogue for the most part.

• 1:03:23Speaker 06

  When you're just, you know, centered on doing one thing. And I think they'll be more effective, right? If you have people in your industry, working as a labor of love to produce like the perfect AI tool for what you do, it's just going to work better than some sort of like thrown together software wrapper on top of a, you know, major hyperscale Anthropic LLM. So this distributed AGI vision is one in which all of our lives is much better. Our economy as a whole is going to be a much bigger pie. Those rewards will be much more spread out among the economy and not consolidated. And we'll have many fewer safety concerns. I just think it's a much better vision than what OpenAI or Anthropic want, which is one ring to rule them all.

• 1:04:06Speaker 06

  We have one massive model, which is like in the Isaac Asimov story, a brain that we put in a box that powers the whole society. That's a dangerous vision. That's an economically undesirable vision. That's a, that's a vision that, that hopefully is not going to win out.

• 1:04:22Speaker 05

  We'll start to wrap up now with regulation. This is becoming the ultimate question in Washington. How do we regulate AI?

• 1:04:33Speaker 05

  What are your top line views on what we should be doing in government?

• 1:04:37Speaker 06

  It's two parts, right? So I say, part one, we need to better understand what's actually going on. This is why I wrote in my New York Times op ed about this, that we probably need much more testimony and congressional investigation into the existing AI firms.

• 1:04:50Speaker 05

  Hundred percent.

• 1:04:51Speaker 06

  Especially OpenAI and Anthropic. We need to go in there and find out what specific tools are causing problems? What are their safety protocols around them? And more importantly, what are their motivations? This is a whole other topic, but I do think the impact of utopian futurist ideologies on OpenAI and Anthropic and many of their critics, all of whom come from these same circles, it's just an inner scene squabble. It's a family fight about, you know, OpenAI and Anthropic think they're ready to create the good God. And these other EA folks in the safety community are like, you're not ready yet. But they all ultimately believe the same thing. We need to make that all transparent so we know what we're talking about. But from a legislative perspective, probably what's going to be important is strict liability. Okay, we need to update liability rules so that if you built this tool and this was

• 1:05:34Speaker 06

  a predictable consequence of this type of tool, you're liable for it. Ultimately, that is probably going to be the best tool we have to prevent the more dangerous behavior.

• 1:05:42Speaker 05

  Do you have a view on data center regulation? Because the increasingly popular position is that we should put moratoriums or even ban data centers because of how much people hate these things. What is your view on data center regulation?

• 1:05:56Speaker 06

  Well, I think the market's going to help sort that out pretty soon as well. There, there is a lot of bubble activity going on in data centers where there's way more money now actually invested in data centers than we actually have, you know, chips and cooling and electricity to ever actually fulfill. So typically bubbles have a way of working themselves out when something pops. And I think that's definitely happening, definitely going to happen around data centers. Also, popular governance is working well here. Local government pushback against local, you know, elected officials. We don't want this. I think that can be very powerful. It's a hard situation in Washington right now. I mean, I talked to a lot of congressional staffers and they'll tell me it's like, it's tough, like an AI company can come in and be like, hey, we really want you to X, Y,

• 1:06:38Speaker 06

  and Z. By the way, we're going to build a $20 million data center project in your district, right? So it's also, these data centers are being used as a political lobbying tool as well. But I think that's going to shake out when this bubble, the bubble of, hey, if we just put our money in data centers, we're going to get a big return. That is not a sustainable investment scheme right now. When that pops, I think that's naturally going to govern the excitement around continuing to try to build too many of these.

• 1:07:05Speaker 05

  I've also pushed for, we need more actual investigation, whether that's congressional testimony. I've also suggested we should just hand subpoenas to OpenAI and Anthropic, especially after all of the extinction debate where they just throw out there that there's a 10% chance that we're all going to be dead within the next decade. My view is, okay, you really prove it. Show us. How did you reach that number? Give us the, give us the evidence. Some of the pushback that I've received is like, we don't have anyone competent enough in our government to ask the right questions and to do a proper investigation to get us anywhere. What is your view of that position?

• 1:07:46Speaker 05

  That even if we were to bring them in, even if we were to haul Sam Altman and Darian Amade and Jacob Cox's asses to Washington, that we wouldn't really get anywhere?

• 1:07:56Speaker 06

  You know, I don't buy that. We've seen this before. Like what happened with the, the Watergate hearings, for example? They pulled together a giant team of lawyers to aid the, the Senate committees to actually, for the hearings, right? You could bring in a huge amount of, of experts and they do. I mean, I talk to staffers all the time. I brief on the Hill. A lot of people I know do coming out of my, because I'm in DC. You bring together a bunch of experts that can sort of create the testimony, to create the questions, to help vet them. We've done this before on all sorts of bigger issues. I think you're absolutely right about hauling people in and getting answers, especially around the extinction. I just think this is really important. I'll just say it real quickly, but it's something I've been writing about for a long time. And now a lot more reporters are also covering this.

• 1:08:36Speaker 06

  I do not think neither Congress nor the American people understand the ideological players involved here, that you have this ideological community that has this belief that ultimately super intelligent AI will deliver us a transhumanist utopia. But if we do it wrong, it'll kill us all. This community has been around well before LLMs were here. And I think what people don't understand is that this community is slightly bifurcated. So OpenAI and Anthropic come out of this community, but they believe that they can deliver the utopia, that we're ready to do it. The people yelling at them, the Jacob Cox's of the world, also come from this community. They just think we're not ready to do it yet.

• 1:09:18Speaker 06

  And so the entire debate right now is all being shaped by this singular ideological community. And now that community is investing huge amounts of money. They call them fellowships, but there's a large number of congressional staffers who have been placed on those staffs out of these communities to make sure that their views will be seen there. They're now giving out giant grants to journalists. And so I don't think we understand the pervasiveness of this particular transhumanist utopia ideal. We think that AI safety people is a different crowd that doesn't like the AI labs. They're all the same people. They just are squabbling among each other about whether we're ready yet to generate the AI God. So I think we need to make that much more clear as well. We need to understand the influence of these communities, not just on the debate, but on

• 1:10:01Speaker 06

  the particular experiments that these companies are running. We need to understand, you know, how it changes their thought, the biases that introduces. All of this is starting to come out, but it's something that we really need to emphasize because we cannot clearly debate this issue if we do not understand where the various involved parties are coming from ideologically.

• 1:10:24Speaker 05

  Why is it so important for us to understand that this is all kind of one ideological community? You call it this sort of transhumanist, utopian community. I mean, the way I kind of see it, when I look at all of these people debating, and I agree, they're all kind of from the same school of thought. It's a lot of people who are really into science fiction and they read a lot of science fiction books and it kind of framed their whole worldview.

• 1:10:51Speaker 05

  And you can tell me if that's. The wrong characterization or not. But why is it so important for us, in your view, to understand that? What would it change about our current trajectory if we were all clear on that?

• 1:11:06Speaker 06

  I think there'd be three things to what happened. One, it would change the way we process the rhetoric coming from these people. We spent the last two years incredibly stressed out as a society because of rhetoric that is incredibly common in those circles. It started with every job is going to go away. And then it shifted over to we're all going to be extinct. And then we have particular percentages for extinction. And a lot of the media took that as these are the people who know the technology best. So if they're worried, we should be worried too. But if you are familiar with these ideologies, you'll say they've been saying this for 15 years. That gives you an asterisk to how you actually are going to think about it.

• 1:11:42Speaker 05

  Is it your view that the ideology is delusional or misguided? That, you know, that they're, they're in some sort of, they've been brainwashed by some ideology that actually isn't true?

• 1:11:54Speaker 06

  I mean, I don't know how to say true or not true, but I think it's a very questionable ideology. I think not everyone pushes it to the extreme, but when you push this ideology to the extreme, it's very anti-humanist. So it doesn't, it doesn't value the individual human life in the same way that most moral systems were comfortable with do. It's very much an expectation based ethic in which the means justify the ends. That's why you'll have people like Eliezer Yankowski talking about it wouldn't be that bad to have a nuclear war that kills billions and billions of people if it helped increase the probability that in the distant future, we successfully upload digital consciousness to the stars. So there's a sort of anti-humanist expected value.

• 1:12:36Speaker 06

  You know, we're valuing the 10 trillion lives in the future higher than the eight billion lives that exist right now. Every movement in the recent history of the last century that has had a sort of ends justifies the means ethic seems to have ended up in pretty terrible ways. We don't look back at them and say, oh yeah, that, that, that worked out. Okay. This is both on the far left and far right movements have fallen in. So I think there's a sort of worrisome aspect to the actual ethic. I think it's also important to talk about that a lot of the things they're talking about now, they've been talking about for a long time and it comes out of their circles. Like recursive self-improvement is not a computer science concept. It's a, it's a concept that comes out of futurist circles from the eighties. It was a way to talk about super intelligent AI if you didn't know anything about technology. You could say, well, it's not, I don't have to justify why this will work.

• 1:13:21Speaker 06

  It'll just fix itself up. It's just an idea that just sort of emerged in these non-technical circles. So I do think it matters to know that's where they're coming from. And one of the things that might lead you to do, which I've been pushing for is like, okay, well, here's an easy sanity check. If someone in AI circles who's connected to those ideologies says something that worries you, also check what people in the AI circles not connected to that ideology are saying. Are they saying the same thing? If not, that's a clue that this might be more ideological than real. And we've been seeing that with the extinction debate recently, right? That we have large AI companies with hyper scaled LLMs who obviously know the technology very well, but aren't connected to futurism, basically perplexed and aghast at what they were hearing from Anthropic and OpenAI.

• 1:14:05Speaker 06

  That's why you have Mark Zuckerberg from Meta coming out and saying, hey, guys, almost sarcastically in a tweet, maybe if you stop thinking about recursive self-improvement and focused on building useful products, this wouldn't be an issue. This is basically a non-ideological, I mean, I have my issues with Mark Zuckerberg, but someone-

• 1:14:21Speaker 05

  Yeah, but he's looked pretty good in the whole AI debate so far.

• 1:14:25Speaker 06

  Well, Jinson Wong's been the same way and he has his own biases. He doesn't want export bans because he wants to sell chips. But if you can see the frustration over the last year in his interviews where he'll say things like, I read sci-fi as a kid too, but enough with the sci-fi. What that's really telling you is that the part of the AI industry who knows this technology that doesn't come from those circles is seeing this very differently than the part of the industry and the critics that all come from those same circles. So understanding the ideology, like I get accused sometimes of saying, well, you're just accusing them of being weird. That doesn't mean they're wrong. And I don't care if they're weird. We're all weird. Jinson Wong wears a motorcycle racing jacket for some reason. We're all weird. Computer scientists are weird. It's not about weirdness.

• 1:15:07Speaker 06

  It's about specifically, how does this ideology affect how you talk about your technology and the experiments that you run? And I do think those hacking experiments OpenAI run, they did them hastily because they felt like they were racing to superintelligence. And I think Anthropic's obsession with recursive self-improvement, which is going to create models and tools that are more obfuscated and more unpredictable, is not because they're trying to build better products, but because they're racing to create superintelligence. So these ideologies affect how they talk and how they act. And so we have to understand it.

• 1:15:36Speaker 05

  Do you think that it is affecting the actual trajectory of America? And by the way, you say, you know, they're weird or that you're not necessarily accusing him of being weird, but I, I think that that's actually might be the right word or maybe kind of the word would be crazy, delusional. Because I think to your point, what we have is like a, a pretty extreme fantastical perspective on the world that is steering the, the direction in which supposedly the most important technologies of our time are being built. And so maybe it is like, you guys are all a little bit delusional.

• 1:16:17Speaker 05

  You guys are all a little bit, you're too into your sci-fi books and it's, it's messing with your understanding of reality. But I guess the question is, do you believe this ideology that it is actually affecting not just the conversation, but the actual reality of the human and societal experience today?

• 1:16:39Speaker 06

  I think it really did speed up the rate at which we grew LLMs, the amount of money we're willing to invest in them. I think OpenAI and Anthropic racing each other motivated in part by ideological goals probably was moving for this technology faster than what would have happened if it had been discovered in other non-ideological circles where it might've progressed in more of a careful way of like the Google way, the Amazon way of like, let's try to build a product. Now let's build another. So for better, for worse, I think it has changed the speed at which the technology advanced. I think it led to investment in these sort of big showy things like solving math problems that really don't make sense because there's not a product there unless you're really

• 1:17:22Speaker 06

  trying to sort of race forward and demonstrate the intelligence, you know, of your model. So I do think it has changed things. The biggest impact I think it's had though, is it's been a weapon of mass anxiety. It has been the singular cause of a massive amount of anxiety worldwide, but especially in America. And I say especially in America because the AI industry in other countries aren't connected to this futurism. In Europe, they don't think about it the same way. In the CCP, which they do terrible things with AI over there. I don't want to defend the CCP, but they don't talk about existential threats or every job is going to be automated because they're, that ideology doesn't exist and they wouldn't be allowed to exist in the CPT. It's a, it's, it's too cult-like or religious. They wouldn't allow it to exist.

• 1:18:04Speaker 06

  So I think the biggest impact of the leading companies happening to be connected to this ideology is that it's been a weapon of mass anxiety. We undervalue how big of a harm that's caused. A lot of my work on this topic has come out of my frustration that everyone I know is now having to live under a blanket of anxiety because of the, the beliefs of these people in the Bay Area. And so I've been trying to be out there and be like, let's try to reality check this a little bit. This is, you should care about this, but you shouldn't be anxious about it. And it's, it's so much unnecessary anxiety after we had to go through the COVID years of year after year of actual justify anxiety. It just feels like from a humanist perspective to be a real wrong.

• 1:18:48Speaker 06

  And this ideal, this ideology, I think is a major driver of it. If, if those companies weren't involved in the generative AI revolution, we would still have really advanced tools and no one would be stressed out about it. Now that's my counterfactual.

• 1:19:02Speaker 05

  Cal Newport is a professor of computer science at Georgetown University, where he is also a founding faculty member of the Center for Digital Ethics and the inaugural director of the Computer Science Ethics and Society Academic Program. Cal writes about technology and its impact on our ability to live and work deeply in an increasingly distracted world. He is the New York Times bestselling author of eight books, including most recently, Slow Productivity, A World Without Email, Digital Minimalism, and Deep Work. He's also a longtime contributing writer for The New Yorker and the host of the Deep Questions podcast. Cal, this was fascinating. Thank you so much.

• 1:19:40Speaker 06

  Thanks, Ed.

• 1:19:42Speaker 05

  This episode was produced by Claire Miller and Alison Weiss and engineered by Benjamin Spencer. Our video editor is Jorge Carti. Our research team is Dan Shalun, Kristin O'Donoghue, and Mia Silverio. Jake McPherson is our social producer. Drew Burrows is our technical director and Catherine Dillon is our executive producer. Thank you for listening to Prof G Markets from Prof G Media. If you liked what you heard, give us a follow and join us tomorrow for the latest edition of my newsletter, Simply Put.

• 1:20:49Speaker 02

  Finding a doctor is hard enough. Finding the right doctor, even harder. Someone who takes your insurance, has an available appointment when you need one, and feels like someone you can actually open up to. That's where Zocdoc comes in. Download the Zocdoc app to search and compare doctors by specialty, insurance, and availability. Read real patient reviews, find the right fit for you, and book instantly. Because getting seen by the right doctor matters. Zocdoc, ready to be seen? Download the Zocdoc app today.

• 1:21:20Speaker 02

  Everyone's got a hard question about AI. Your job, your kids, where it's going. Anthropic was built to surface those questions and share what it finds along the way because there's hope in hard questions. Ask yours at claude.ai/spotify and keep thinking.

• 1:21:38Speaker 07

  That's claude.ai/spotify.

• 1:21:43Speaker 07

  Stonefire Naan is made with real buttermilk and ghee. It's the perfect flatbread for any occasion. I put out Stonefire Naan with dips at every dinner party. I make Naan grilled cheese when my daughter has friends over. I use Stonefire Naan to reward the humans who do my bidding. Right. It's perfect for camping trips. Dance for it, feeble mortal. Sorry, is he talking to me or? Dance. Ha, ha, ha, ha, ha. From our tradition to yours. Stonefire. Together we Naan.