20VC: Is More Compute the Answer to Model Performance | Why OpenAI Abandons Products, The Biggest Opportunities They Have Not Taken & Analysing Their Race for AGI | What Companies, AI Labs and Startups Get Wrong About AI with Ethan Mollick

Ethan Mollick is the Co-Director of the Generative AI Lab at Wharton, which builds prototypes and conducts research to discover how AI can help humans thrive while mitigating risks. Ethan is also an Associate Professor at the Wharton School of the University of Pennsylvania, where he studies and teaches innovation and entrepreneurship, and also examines the effects of artificial intelligence on work and education. His papers have been published in top journals and his book on AI, Co-Intelligence, is a New York Times bestseller. In Today's Episode with Ethan Mollick We Discuss: 1. Models: Is More Compute the Answer: How has Ethan changed his mind on whether we have a lot of room to run in adding more compute to increase model performance? What will happen with models in the next 12 months that no one expects? Why will open models immediately be used by bad actors, what should happen as a result? Data, algorithms, compute, what is the biggest bottleneck and how will this change with time? 2. OpenAI: The Missed Opportunity, Product Roadmap and AGI: Why does Ethan believe that OpenAI is completely out of touch with creating products that consumers want to use? Which product did OpenAI shelve that will prove to be a massive mistake? How does Ethan analyse OpenAI's pursuit of AGI? Why did Ethan think Brad, COO @ OpenAI's heuristic of "startups should be threatened if they are not excited by a 100x improvement in model" is total BS? 3. VCs, Startups and AI Labs: What the World Does Not Understand: What do Big AI labs not understand about big companies? What are the biggest mistakes companies are making when implementing AI? Why are startups not being ambitious enough with AI today? What are the single biggest ways consumers can and should be using AI today?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  OpenAI abandons products like crazy. They want to build the machine god. If you have any talented people, you're going to have them building the next technology for AGI. If you have compute, that's what you throw it at. I mean, they're incidentally making $3 billion run rate this year, I think, by like just accident. But there isn't really a product there right now. It's the chatbot and the API. I think a lot of people in this space are just assuming scale solves issues. The real problem right now is every startup in the world is betting against AGI, which I find really funny because all the funders are like, yeah, AGI is coming in next five years. If it is, why are you funding these startup companies? None of them survive in an AGI world.

• 0:37Harry Stebbings?

  Welcome to 20 VC with me, Harry Stebbings, and I'm so excited to welcome our guest to the hot seat today. Joining us, Ethan Mollick. Now Ethan is one of my favorite writers on AI and his blog, One Useful Thing, is an absolute must read for me. For those that do not know, Ethan is a professor and co-director of the Generative AI Lab at Wharton. Now, there's a lot in this show. Time to get the notebooks out, maybe take down the playback speed to a 0.8x. It is quite fast, but it is an incredible discussion today. But before we dive in, when a promising startup files for an IPO or a venture capital firm loses its marquee partner, being the first to know gives you an advantage and time to plan your strategic response. Chances are, the information reported it first.

• 1:20Harry Stebbings?

  The information is the trusted source for that important first look at actionable news across technology and finance, driving decisions with breaking stories, proprietary data tools, and a spotlight on industry trends. With a subscription, you will join an elite community that includes leaders from the top VC firms, CEOs from Fortune 500 companies, and esteemed banking and investment professionals. In addition to must read journalism in your inbox every day, you'll engage with fellow leaders in their active discussions or in person at exclusive events. Learn more and access a special offer for 20VC's listeners at www.theinformation.com/deals/20vc. And speaking of incredible products that allows your team to do more,

• 2:04Harry Stebbings?

  we need to talk about SecureFrame. SecureFrame provides incredible levels of trust to your customers. Through automation, Secureframe empowers businesses to build trust with customers by simplifying information security and compliance through AI and automation. Thousands of fast-growing businesses, including NASDAQ, Angel List, Doodle and Coder, trust Secureframe to expedite their compliance journey for global security and privacy standards such as SOC 2, ISO 2701, HIPAA, GDPR and more. Backed by top-tier investors and corporations such as Google, Kleiner Perkins, the company is among the Forbes list of top 100 startup employers for 2023 and Business Insider's list of the 34 most promising AI startups of 2023.

• 2:49Harry Stebbings?

  Learn more today at Secureframe.com. It really is a must. And finally, a company is nothing without its people. And so I want to talk about Cooley, the global law firm built around startups and venture capital. Since forming the first venture fund in Silicon Valley, Cooley has formed more venture capital funds than any other law firm in the world with 60-plus years working with VCs. They help VCs form and manage funds, make investments and handle the myriad issues that arise through a fund's lifetime. We use them at 20 VC and have loved working with their teams in the US, London and Asia over the last few years. So to learn more about the number one most active law firm representing VC-backed companies going public,

• 3:30Harry Stebbings?

  head over to Cooley.com and also Cooleygo.com, Cooley's award-winning free legal resource for entrepreneurs. You have now arrived at your destination. Ethan, I am so excited for this dude. I told you just now, I am like your biggest fan from afar. So first, thank you so much for joining me today.

• 3:48Speaker 00

  Oh, I'm thrilled. It's uh, it's great. And I've been an entrepreneurship professor for a very long time before anyone knew about my AI work. So it's always great to be connecting to the VC entrepreneurship world.

• 3:57Harry Stebbings?

  Now, for anyone that doesn't know your work, can you just give a 60-second intro on your work and how you've become much more well-known in the last years?

• 4:06Speaker 00

  I'm a former entrepreneur myself. So the startup company I helped co-found invented the paywall. So I still feel like I'm trying to make up for that in the late 90s. So just trying to pay back after that. Then I've been a professor of entrepreneurship. I got trained in MIT and then I've been at work ever since. You know, I do a lot of work on teaching and research on how entrepreneurs become successful. But I also had this side gig of thinking about AI and teaching for a long time. So I worked at the media lab with a guy named Marvin Minsky, who is one of the founders of AI. And I was like the non-technical person there who was like trying to translate what the lab was doing for the world. And then I've been building tools for how do we teach entrepreneurship at scale? Because it turns out it really matters. Little bits of entrepreneur training make a huge difference in people's lives. And we've been playing with AI and other tools.

• 4:50Speaker 00

  So when AI sort of came out, I was in the weird place of actually practically using these tools for a long time beforehand. It turns out everybody else who was taking this stuff seriously was computer scientists. So I sort of was there at the early days of like, oh, I know business stuff and entrepreneurship stuff and education. And these things are actually quite useful. And I already have a Twitter following, so I just sort of became the go-to person. And then there is this Matthew effect of like, all the labs started talking to me and I get insider information and everything. It becomes a sort of self-reinforcing prophecy.

• 5:17Harry Stebbings?

  By the way, your Twitter game is fantastic. So like, don't change that at all. I love it. Um, I want to start though, and it's pretty perfect timing. I said we were pretty casual in how we did this. You know, we saw the new Lama 3.1 model come out yesterday. I'm just really intrigued to hear your thoughts, Ethan. What did you think? Is it what you expected?

• 5:36Speaker 00

  So there's like four or five dimensions that the Lama model is super interesting in. We could talk about open weights and open source being one model. I'm not surprised that they caught up with the leading edge state-of-the-art models. I think people are underestimating how much ammunition the closed-source labs have and are going to release in the near future. I think it's great. We now have an open-source GPT-4 capable model and it's going to be everywhere. And, you know, it's just interesting because how much of that gap gets closed by that model. So every national government had worse AI than every kid in Mozambique had access to through GPT-4O. So now there's a openly available, fine-tunable model. We're going to see a lot of weird effects from AI that were delayed.

• 6:16Speaker 00

  It happened as a result. Actually using it, it's pretty good. I mean, I don't think it stands out compared to Claude at this point or something else, but it will soon because people will be working on it. And it is a downloadable open weights model, which is kind of a big deal.

• 6:29Harry Stebbings?

  How much of that chasm do you think will be closed by the closed source providers with their next releases?

• 6:35Speaker 00

  I think we don't know a lot, and even the people training the models don't know a lot. I mean, part of the weird bit here, right, is the people training the models are all computer scientists, basically, I mean, doing computer science. And they don't have a huge idea of the implications of their systems. When OpenAI released GPT three point five, they didn't expect to destroy higher educator, you know, education and they would have to rebuild it because everyone's cheating all of a sudden, right? I mean, they're already cheating, now they're just cheating really well. But we weren't expecting like a large scale revision of like how the world works, right? Everything I'm hearing from everybody is that the next generation of models is going to be smarter, right? The exponential continues. Whether or not that translates to real world implications is a different kind of concern.

• 7:13Harry Stebbings?

  Do you know what I find challenging, Ethan? Is every week you go on Twitter? And there is this transience of dominance between the different providers, you know, OpenAI do something and it's like, wow, that's incredible. And then Claude do something and it's like, wow, that's incredible. Lama, what? And every week it seems like this one's the winner and the rest are losing. And there's just such transience and speed. I almost don't know where to go. It's just not understandable.

• 7:37Speaker 00

  It makes complete sense. I mean, it doesn't help that social media likes buzz. For normal people sitting back, everyone's just gonna keep using chat GPT because that's what they're using, right? They might gradually switch to Claude, like the enthusiast community is very different than when I talk to the outside world about this stuff. And I think that on the grand sweep of things, what really matters is when these models top out and how long that takes. And I think worrying about who's in the lead at one moment is probably less of an issue. Then the big labs are all gonna keep building. There's no tricks in Lama that they really told us that were unusual or indicated some sort of secret breakthrough. We still don't know if there's secret sauce in some of the other labs that are very different, like it's a very early days in some ways.

• 8:14Speaker 00

  So I think trying to get, you know, if you're enthusiastic like me about this technology, great, follow along and keep track of the MML, you know, MMLA ratings. But otherwise, I do think there's a little bit of like unnecessary. Necessary to get every detail at this stage.

• 8:28Harry Stebbings?

  I totally get you. And you mentioned there about topping out. Before we discuss kind of the potential topping out and what happens when that does. I do just want to start on actually the four potential outcomes. You highlight this in your book, which I loved. And I just thought it'd be helpful to start there as a framing. What are those four potential outcomes first?

• 8:48Speaker 00

  Okay, the other four outcomes, Twitter, it only talks about, and you know, the press only talks really about one and four. So let me go through one and four first. I'm gonna give you the boring middle, right? So option one is this is it. Models don't get much better or, you know, and it's sort of this whole thing sort of fizzles out. I think this is unlikely because I think not only will models get better, but also we haven't even started integrating them into work yet. Right? Like the way you work with these things is an insane process of actually using a chatbot and having a conversation with with it with a chatbot is how people are using it for work at this stage. So we're not even at the stage of integrating, but it's possible that things kind of that, you know, in this, in which case we have 10 years or so of integrating the system slowly into human systems.

• 9:27Speaker 00

  I think everything sort of it stabilizes out where it is. We're not going to see, we'll, we'll see an economic improvement, but we probably don't see kind of a massive large scale shift, except some industries change more than others. Right? I think it's very likely that photography changes a lot and that there's other field, like customer service changes a lot with our current systems. But that's one option, right? Nothing much happens. Then option four is the machine God, right? We sort of achieved this AGI plus super intelligence thing. Machines are smarter than humans. We have a intelligence explosion and God knows what happens next. We've had a good couple hundred thousand year run as a species. You know, we'll figure out what our successes are. And there's a lot of obsession with this. Because I think that's where everybody, both boosters and people feel negative about this. This is where their minds go first, right? Is like super intelligence.

• 10:10Speaker 00

  I think the more common scenarios when we see a technology, right, are either continued exponential growth or else linear growth and ability. And I think that's what we're under preparing for. So if you look at the trends lines of this stuff, everyone's like trying to anticipate that every model is either going exponential from here. Or is, you know, everything's going to top out. I think much more likely, you know, our study, this AI was as good at the 80% level of consultants. Next year is the Eighty-fifth percentile, ninetieth percentile, eighty-first percentile, one hundred and eightieth percentile, we don't know. So I think a large part of this is that kind of world. And a linear growth world where the models get a little better every year. I think that's much more adjustable to one where it could use exponentially. Um, and we sort of get close to that AGI world.

• 10:54Harry Stebbings?

  Maybe rightly or wrongly, I always think about actually iPhone releases. And you know, the first iPhones, there were big differences between the early releases of like the three and then the four. And then slowly it just became kind of a little bit better camera and a little bit better battery. And maybe, you know, the calculator slightly bigger buttons or whatever it is. And I'm like, what is it that AI has or people believe AI has that believes it will have escape velocity of development? And it will never achieve that plateauing.

• 11:23Speaker 00

  You're right. Now we're at the classic sort of top end of a technology where it's all about like the calculator was a major factor in the release of the new iOS. And you're like, this is where we are right now is much better calculator. I think hilarious. So there is a topping out. Now, if you look at a process like Moore's law, it's a sustained exponential curve for years. The difference is that it's a bunch of underlying technology that get swapped out for each other. So the real question is, what is this sort of top line intelligence? What does that max out at for what an AI can do? Are its limitations exceedable? Right now, AI is jagged, so it's really good at some stuff, really bad at other things. So, and as a result, it can't sub in for all of human work because on one hand it'll do a great job.

• 12:05Speaker 00

  And some of the job stuff, so it will do a bad job at other things, just like any machine does. The question is, can that jaggedness get overcome? We don't know the answers to these questions yet.

• 12:14Harry Stebbings?

  Kevin Scott always says that, you know, compute will solve all problems. And many have always believed that, that performance will be answered by compute and just more brute compute. I have other people on the show, your Alex Wang's at Scales, who say that data is the core bottleneck. When we think about compute, data or algorithms, what do we think is the core bottleneck to performance now and in the next twelve to twenty-four months?

• 12:39Speaker 00

  I'll try and answer that, but I want to give the contrarian view first that I always want to indicate first, which is for most people, they just don't care. Let's say that LLMs top out and it turns out we have to switch to, you know, Mamba or some other like, who cares? Nobody cares. They're using these systems. There's a lot of like in the weeds that you get when you're watching this like, uh, like a sports game of like, who's winning and what situation? The top line capabilities matter and there's a lot of room left there. Like to me, the thing that gets left on is computer science discussions are often the system, the human systems that these things have to interact with. The organizational system they have to interact with, and that's where we need to see kind of more growth, right? That being said, we don't know what the bottleneck is, right? There's, there's this idea in the history of science called the reverse salient, which is that technology sort of moves forward, but there's always something that's kind of lagging.

• 13:24Speaker 00

  And all the effort goes into fixing the lag. So in the early days of electricity, we had generators, but transmission was a problem. So there's a huge amount of work to make transition better. In our current electrical sort of new economy, it's been batteries. So there's huge amounts of work going into batteries because solar panels are good, but batteries aren't good. I kind of feel like we're just going to hit a whole bunch of reverse salience, right? So like, oh, the data pipeline isn't good enough. Great. Is it going to be real world data, synthetic data, or maybe this is the end? But once every, all of science concentrates on one thing, we tend to find ways forward. So I think it's going to be a bunch of debates over what the trailing indicator is, and then everyone forgets about that because it gets solved. It's not a bad approach. It also is just kind of how technology works, right? Because the money is all to be made in the reverse salient.

• 14:07Speaker 00

  If you can make a billion dollars as a data company and, but you know, because that's the area everyone's stuck on, you become a data company, right? This is capitalism and science at work. The heart problems are the ones where all the money and prestige comes from.

• 14:19Harry Stebbings?

  You said where the money is. I loved an analogy that you've said before. And it's, you said, a lot of people use the analogy of picks and shovels and the gold rush. You said that's maybe not such a good analogy and that the steam train was more apt. Why do you believe that it's not a good analogy? And why is the steam train more apt, Ethan?

• 14:38Speaker 00

  It's a great question. I just analogies are really powerful and we have very bad ones in AI. VC people get taken in by this, right? So it's like, I hear you want to sell picks and shovels. And first of all, I don't a hundred percent know, like, everyone defines that slightly differently. They're like, oh, no, no, you want to sell compute, you want to sell the tools that help people scale up and pick their, you know. First of all, it's unclear what the analogy is, but the second deeper problem of this is that that isn't actually how a new technology spreads across our organization. You don't want to sell picks and shovels to the people trying to mine gold. What you want to do is figure out how to get them to use this new technology, which doesn't have a gold rush analogy at all, right? Instead, the steam power, the secret was not James Watts's steam engine, which was important, right?

• 15:20Speaker 00

  Huge breakthrough. Two interesting things, by the way, things didn't really take off until Watts patents expired and it can be openly adapted, but the real value of the steam engine came from having skilled artisans in your factory who said, I've got this thing that can make power go back and forth. How do I create the gearing to connect that to my, you know, my spinning Jenny, my ammunition manufacturing machine, my bottle shaping tool, and it was the skilled artisans that made all of this work and made the manufacturers capture all the money. So you want to be a skilled artisan right now. You want to figure out how to take the back and forth power of an LLM and convert that into usable work inside your organization.

• 15:58Harry Stebbings?

  What floors me is the lack of human descriptions around how to use these tools effectively. It's like no one's written using LLMs for dummies, using AI for dummies, which everyone needs. Why are these providers not doing what is so obviously required?

• 16:16Speaker 00

  I think that if you talk to Silicon Valley people, they are very obsessed with the race for superintelligence. And I totally get it, right? If you can build a machine, God, you win. So that's kind of the secret story behind what's going on here. So the real belief that if scale solves everything, the biggest thing you could do to waste your time is do anything that isn't scaling. Your smartest people have to be scaling. All of your compute has to be scaling. And the bigger models will solve all problems, as you were saying, you know, that's a sort of view in Silicon Valley. So they're going to come back and figure this out later because why would you bother, you know? And there's some truth to that, right? I spoke to a very large financial institution, spent a huge amount of money building a GPT three powered sales assistant tool, but as soon as chat G P T came out was instantly obsolete, right? So like,

• 16:58Speaker 00

  why bother with this? I mean, it was a smart idea at the time. They were way ahead of the curve, right? But I think that the real issue is, is that as a result, all use of this stuff has kind of been dropped, right? There is no manual out there for this stuff. There's not even a dis- like, there's not even a set of points about what A I is good at and what it's bad at. As a result, like it's, I would call it documentation by rumor. It's a bunch of people on Twitter. There's like seventeen people posting about how they're figuring out how L M's work. And then everybody else is just kind of using it like a chatbot. It is a very weird situation.

• 17:29Harry Stebbings?

  Can I ask you, you said that kind of about Silicon Valley and how they think about where the true value lies? You know, we have Vinod Khosla on the one hand, that says we cannot have such powerful models open source. We have Marc Andreessen and others say that they have to be. What do you believe is best?

• 17:48Speaker 00

  I am generally in favor of technological progress, and I think that openness frees people to do lots of really interesting things. There's a very obvious low hanging fruit with A I and healthcare and education that I think are going to be very helpful in large parts of the world don't have access to good doctors or good tutors, right? For places that do have access to that, there's a lot more nuanced discussion about when do you turn to A I for some of these things? So I think open models will make a big difference. They'll spark entrepreneurship. We know that people who get advice from A I do better as founders in Kenya if they're already doing well. There's a lot of really exciting stuff here about openness, but there's also downside risks and it feels very weird for people to say like, it's all one thing or another. I do think that open models will immediately have their guardrails breached. And we already know three or four low hanging threats. I think we are overly worried about science fiction threats, right?

• 18:33Speaker 00

  It's not good enough to help you build a virus at this point, but it could be in the future. We just don't know. But what I am worried about is our entire computer security system depends on it being very expensive to spearfish somebody. This does spearfishing at scale. What do we feel about that? Like that you could do this? These systems are going to be harnessed for very good catfishing campaigns. How do we feel about that stuff? I just feel like there's not this conversation. So I think the open models both carry risk and reward. I don't think there's a lot of thought going into this stuff. I think it's all corporate strategy at this point, right? So Meta doesn't really want to make money from models, so they're going to spoil, you know, their rivals. Microsoft has a chance to go after Google, so it adds AI into Bing. Like, there's a lot of like back and forth among a few firms. And I don't think we actually know the full meaning of open source AI.

• 19:18Speaker 00

  And it's a little weird to both say it's super powerful, it can do everything, and therefore it's high risk, and also it's not that big a deal.

• 19:25Harry Stebbings?

  You said there's a lot, not a lot of thought going into it. What thought would you like to see going into it? Like, what do you think would be a commensurate level of thought and analysis?

• 19:34Speaker 00

  I think that we need to be built for fast reaction to these models. I, what I'm worried, so there's Josh Rigan, who's a professor at University of Toronto. I think has a really nice model for AI regulation that I think is probably right. Which is when you have a new technology, you don't know what the problems and issues are going to be, you do fast follow up regulation. You don't try and pre-regulate because you don't know what it's good or bad at. But you do watch what's happening and have rules that you put into place and policies and fast reaction. Now, we can talk all about how government's not built to do that, how it's not cooperating well with industry. That's the same way I'd be thinking about open source right now. So we've just released a very powerful model, open source. Who is setting up to learn for what the implications of this are going to be? And do they have a pipeline back to the open source makers of these models?

• 20:17Speaker 00

  Is there something that would stop Meta? Is there an event that would stop Meta from outsourcing, from open sourcing its models? I don't know. Who's watching that stuff? Are we have, is there any kind of monitoring system out there to find out? How this is disrupting the world one way or another? There doesn't seem to be. To me, a really responsible view would be, sure, let's release open source. But then let's be watching over the next six months to get a sense of what this is good or bad at and, you know, react to it. And that's what's worrying me a little bit.

• 20:42Harry Stebbings?

  I'm sitting in Europe where we have the EU AI Act, which is incredibly stringent. EU is also particularly talented when it comes to regulation. I'm very worried that actually we will have such constraining regulation that it will actually cause the plateauing effect of AI, both in development and in adoption. Do you think intense regulatory scrutiny is a cause for concern in the path to much more developed AI systems?

• 21:08Speaker 00

  Yeah, I mean, I think that not being fast and reactive is a problem, right? You want to have people develop this new technology. You want it developed by the societies that you want to develop these technologies in. Like, you want to be used in democratic ways. All of that stuff indicates, like, we want to see continued growth. It just feels like it's either or for so many conversations. Like, either there's no regulation and no scrutiny whatsoever, and technology always benefits everybody. And I'm a technology optimist. Like, it does benefit people, but like, it's weird to have no downside risk. On the other hand, you have the, we must regulate in advance to stop a bunch of harms that haven't occurred yet, and that the current levels of models clearly will not cause, right? We're not going to get a runaway superintelligence from a Llama, a Llama 3.1. So, we have to have some sort of balance here.

• 21:50Speaker 00

  I think the EU has, EU has definitely put in a lot of stringent things in place. I don't know whether Europe would be leading in AI anyway. I mean, there's a weird ecosystem problem. Talk about, you know, this is 20 B.C. VC has always been a U.S. thing. London did okay for a while there. But aside from that, you know, more money went to graduates from Penn, from the school I teach at, than everybody in France and Germany put together. We already have a whole bunch of innovation ecosystem problems in place. Regulation is one of them. I think a lot of people are pointing at EU regulation, being like, this is the cause. There's a multi-causal problem here in terms of Europe versus the U.S. on technology development. Everyone moves to Silicon Valley because you kind of have to, and all the stats show that's actually a really good idea for almost every venture.

• 22:35Speaker 00

  Like, there's a machine here that keeps working, right? And so, to go back to the bigger issue, I think a lot of, putting a lot of tight regulation on AI at the beginning is definitely an issue, because Llama breaches the high-security risk level in terms of number of flops with the EU.

• 22:51Harry Stebbings?

  Would you tell your students today that they have to move to the Valley if they want to increase their chances of winning?

• 22:56Speaker 00

  That's an empirical result from a bunch of studies. Companies that, you know, there's been a study of Israeli companies moving to the Valley, New York companies. The issue is, is that that's where the connections are. And it turns out Zoom only gets you so far. The average distance between a VC and a company you invest in is about 40 miles. That's because, when you look at where VCs spend their time, it's networking and it's monitoring. It's networking with other, with, you know... And learning about companies and then it's monitoring the portfolio companies. And that's much easier when you're local. Zoom doesn't let you do monitoring in the same way. In fact, when direct flight is added between SFO and another city, VC investment in that city goes up because it's just getting, it's easier to fly there and help do and do monitoring there. It's a local business, right? Everyone's like, oh, it's global, it's connected, it's a local business.

• 23:40Harry Stebbings?

  Can I dive into a couple of the different market participants? We've already touched on some of them, but I want to start on AI labs. We mentioned Llama and model progression earlier. In terms of the AI labs, what do the big AI labs not understand about companies themselves, do you think?

• 23:55Speaker 00

  I think that's such an important question. There is just, the products being released are just super weird, right? Like, I think there's very little consideration of use cases. I mean, you look at the number of people inside these organizations who've worked at large companies. I often joke, when I go to the West Coast, there's like, you know, it's all like cold plunges and how do you live forever? And really, you know, and then in the East Coast, it's like, we, you know, we're drinking coffee till we die. And the goal is like, get our work done, get home, like, you know. In a large company, it's very different. And there's a lot of like contempt, I think, for large companies. That's where most smart people are, right? Are in large organizations doing, you know, other work that is not Silicon Valley work. For every coder, there are 16 managers. I, I think that there's not a sense of what this stuff does for them. And as a result, there's a lot of half-built products that are brilliant and then get walked away from.

• 24:39Speaker 00

  Code Interpreter is a huge world-changing product for data analysts that got partially abandoned by OpenAI. They haven't, they haven't moved the needle in that sense. Chatbots and APIs remain the main area. Almost all the documentation is technical documentation. And almost all the interesting use cases are not being discovered by technologists, who are actually quite bad at using AI often because it doesn't work like a normal technology. They're being discovered by end users, managers, and the system's not built for those things. So there's just this huge gap between technology and use.

• 25:08Harry Stebbings?

  I'm so sorry to be so naive. Why, if Code Interpreter is such a generational-defining product for analysts, why would they walk away from it or not walk away from it, but, you know, not progress in the same manner as they started?

• 25:20Speaker 00

  I think OpenAI abandons products like crazy. I think these products are passion projects from various people. Again, they want to build the machine god. If you have any talented people, you're going to have them building the next technology for AGI. If you have staff, that's what you throw it. If you have compute, that's what you throw it at. I mean, they're incidentally making three billion dollar run rate this year, I think, by like just accident. But there isn't like a, there isn't really a product there right now. It's, it's the chatbot and the API and the system gets smarter, solves more problems. I think a lot of people in this space are just assuming scale solves issues. So why would I buy, or, or development solves issues? So why would I bother spending some time thinking about, you know, how to prioritize this when the product will be obsolete in a year anyway?

• 26:02Harry Stebbings?

  Can I ask you, on the flip side, we have the companies themselves. What are companies getting wrong about AI that they should know more about?

• 26:10Speaker 00

  I mean, I speak to organizations all the time. And first of all, almost nobody uses these systems. I mean, they all tried ChatGPT. Every, when I ask my hand, everybody's tried ChatGPT. Five to 10% of people in any room, whether, and that's by the way Silicon Valley, actual people, right, who aren't in a lab, whether that's at a large bank, whether that's at a conference of innovation professionals, maybe five to 10% have used those models and maybe two or three percent have used 10 hours, which has been my guideline, you know, minimum number. Again, there's no onboarding. You're faced with a chatbot and when people are faced with the tyranny of the blank page, they panic. What do you talk to the system about? There's no information, there's no instructions, and so people aren't really using it. So the issue is that partially it's that they need to adopt. Because when people start using it, they find

• 26:52Speaker 00

  uses, right? So a new study just came out of Denmark of people who are using ChatGPT and, you know, in knowledge intensive work environments. And, you know, they're estimating that over 30% of their tasks, they're saving 50% of their time. So once people use it, they find productive uses. So then the question becomes, how are you harnessing those uses? What policies do you have? I mean, there's so much we could talk about, about what companies are getting wrong.

• 27:13Harry Stebbings?

  In terms of how we harnessing the uses, what would you like to see changed there? Because that's where we can fundamentally drive productivity, which is arguably the most important thing.

• 27:21Speaker 00

  I mean, so first of all, it just starts with policies. When you look at companies, a lot of them don't even allow access to GPT-4 because the regulatory environment's unclear. So one thing that'd be great for a regulatory Perspective is not just, we've talked about the negative side of regulation. There's a reason why banks are regulated or pharma companies are regulated. It would be useful for clear guidance about how to positively use AI. And I think there's been some movement towards that. That that will be so that I would want the EU to be doing a lot more of too. It's like, okay, what are the ethical use cases that we should be pushing and opening regulation for? So that but that extends to the company policy side. Company policies are often very vague. You know, don't use this or use it, but don't get in a way that doesn't get you fired. And then there's a whole bunch of like uncertainty over how you get rewarded. What happens if you figure out a solution to your work?

• 28:05Speaker 00

  So what I find is inside organizations, when I finish with the talk, all these people come up to me and reveal that they were secret cyborgs all along. So they've been using this for all of their work. But they're not telling anyone. They're not telling anyone because they're worried they get fired. They're worried that people will stop respecting their work because they realize it's AI written. Right now Reddit's full of people saying, I'm a people think I'm a wizard at work. They're worried that people will realize you don't need as many staff members. So you fire them or you fire their their colleagues. Or you won't reward them for it. So everyone's hiding AI use. I spoke to a woman who banned ChatGPT at a major bank. She used ChatGPT on her phone to write the ban. Because like, why do it by hand? So once people start using it, they're all using it secretly. And so we need clarity around how do you get rewarded for this? So like, what happens if I automate my job? And to go back to our industrial revolution analogy.

• 28:48Speaker 00

  If you are a brewery in the early seventeen hundreds and you are serving your local community, because everything was kind of local and you got steam power. You kind of have a choice. Do I want to fire a lot of people and make the same amount of beer for less money and have a higher margin? Or do I want to be Guinness and expand my production around the world and hire another hundred thousand people? And we're used to IT solutions being a cost saving measure, right? If I get thirty percent productivity boost, I fire thirty percent of people. Your people are never going to show you how to use AI at that rate. And you're never going to win in a world if we really believe there's an industrial revolution happening. So policies are really at the heart of the problem.

• 29:25Harry Stebbings?

  And I always feel that in the majority of cases, we just redistribute talent, as you said there, more effectively for new projects, for new initiatives, for expansion. What I find worrying is here, especially in the early days, it is killing the lower classes if you're being horrible and Blunt, which is like Klarna had like seventy percent improvements with AI in terms of customer service and cut so many of their workforce. You're seeing especially customer service be the core kind of Trojan horse, which is replacing ninety percent now of customer service teams. To what extent do you think I'm over worrying and actually we'll see the continuing redistribution of talent, not the removal of talent?

• 30:04Speaker 00

  So this is a case where I think we're being a little sanguine about this. I mean, I, I think every technological revolution, people lose jobs and then new jobs are created. But there's, you know, and we talk about this all the time. There are two big caveats to that. Caveat number one is not always, right? When the telephone switchboards went from sort of manual to digital and starting, not digital at that point, but mechanical in the nineteen, starting in nineteen thirties. At that point, I think one out of every sixteen women had spent time as a telephone operator. It was like a job. And then if you got fired from that, if you were young, you found other jobs. If you were older, you never found another job as good, right? Because you were really good at telephone as a telephone operator. So not every job ends up with a new category replacing it.

• 30:45Speaker 00

  And the other thing is living through the industrial revolution kind of sucks, right? Like you can look backwards and say, like, oh, great. Everyone got better jobs. They were much richer now. But there were also people, you know, smashing machines because they didn't want their jobs replaced. There was a lot of unrest. There's a reason why that there was, that's when the great debates between capitalism and Marxism arose because there was unrest during this period that was serious. So I think part of what I worry about is a little bit of like, even if you have a sanguine view that everything is going to be fine, that doesn't just happen automatically. Like you can't say the market makes everything great. So let's not worry about it or else everyone's going to lose their job and we need UBI. There has to be something much more specific about, yes, there's going to be waves of disruption heading through the economy. How do we do things we are very bad at like retraining? It's a problem that actually be solved. It's not something that it has to be made as science fiction.

• 31:30Harry Stebbings?

  Well, what I worry more about is actually the distribution of knowledge and productivity versus the distribution of wealth. And what I mean by that is there is this like one percent of like Silicon Valley and tech elite, I think, who are using AI and the surrounding products incredibly well to do ten X the work that we used to do and to be way more efficient and way more cost effective. And then there's the rest of the world. You know, we joked about Europe, but I live in the UK. You go to a place in the UK, they've got no idea what ChatGPT is, let alone how to use it to create marketing campaigns that are ninety nine percent cheaper in ten percent of the time. I think it's just creating even more knowledge and productivity to one percent, and the world could get left behind. Am I right to fear that too?

• 32:14Speaker 00

  Yes, I'll say yes and. Okay, so Denmark study I told, I was talking about, did find that the people using this skewed mostly male and mostly wealthy, right? There were people finding use cases because that tends to be a fairly common tech adoption curve. The thing that is unusual about AI though is first it's ubiquity. Normally, getting your new tech installed means I've got to have, you know, know how to use a computer really well and be really in with like, you know, how do I get the, you know, distro from GitHub and like, you know, there's work involved that is, that is a narrow set of work that requires time, effort, money. That isn't the case here, right? The chatbot is accessible through a phone in, you know, a hundred sixty nine countries around the world have access to the world's best AI systems. That's one thing, right? And chat is a fairly normal interface, especially when you have voice.

• 32:57Speaker 00

  The second is early evidence is that coders are not particularly good at working with AI, right? Because it doesn't do the things you expect it to do. My favorite example is Simon Williamson, if you don't follow, is terrific and really great at this stuff, but he works on data journalism. And he built, he was using Claude for OCR, um, political campaign donations. And when he checked back, uh, he just found that Claude refused to do the work because there were names and addresses and even though they were public, he was, Claude was like, I don't want to violate anyone's privacy. Like, we're not used to systems that object to the tasks that they're given or like sometimes argue with you or give you different answers every time. So coders are often not the best users. Often the best users are people who are actually really good at working with humans. I mean, my, my wife is probably one of the best prompt engineers on the planet. She's got a doctorate, we worked together, we're co-directors of the AI lab.

• 33:42Speaker 00

  She's never coded a day in her life, but regularly does stuff that OpenAI and Anthropic are like, wow, that's a really amazing prompt. We didn't know, like, Google user prompt is the gold standard to measure their fine-tuned models against, right? But what she has is a, you know, doctrine education, building educational teaching games for a long time, and she has good theory in mind for other people. If you can write instructions, if you can manage, you can use this. So that's what I'm hopeful for. It looks like a tech adoption curve, but tech people shouldn't have the advantage they have in other spaces, and word just has to get out.

• 34:11Harry Stebbings?

  I do have to ask, you mentioned that kind of about the quality of prompts and how amazing your wife is with her quality of prompts. You also mentioned kind of the white screen of death, and when you have kind of that blank template, not knowing what to do with it. You've said before about bluntly the kind of challenges of the chatbot interface and what a weird interface it is. What do you think will be the interface of the consumer between the power of AI and consumers?

• 34:34Speaker 00

  I think multimodal is really the answer here. All the pieces are in play. Some of the most interesting people, you know, people I talk to are really using AI or just having conversations with it. Like, I think about Ally Miller, who is a really great sort of person who has been thinking about a lot about AI, ex-Amazon person, and she has conversations with the AI every morning while she's doing her hair, right? Just the limited chat interface. Once these things have full visual, which they do, right? They have a lot of latent capabilities that people haven't recognized yet in multimodal, and you could chat with them, then it starts being more like having a human on call. I think once you start adding agency into that, where they can take action in the world, I wonder if we just sort of skip the step of, you know, how do you use these things? So like, oh yeah, you talk to your phone and your assistant does the thing that you want it to do. There is this narrow window, I think, where prompting style really matters,

• 35:19Speaker 00

  where being really up to date in these systems matter. But then they come to your phone. And also, by the way, if they save you time and work, if they really do do that, humans are exquisitely designed to figure out how to minimize the effort they put into things. There's a reason why adoption rates are over 70% in universities for ChatGPT, and while they're like at a few percent elsewhere in the world. We figure stuff out like this. And I think that that's the other piece that's missing.

• 35:44Harry Stebbings?

  One element that we haven't discussed is startups themselves, actually. And so on that, like, you've said before that you don't think startups are being ambitious enough in the face of AI. What should they be doing, Ethan, and why are they not doing it?

• 35:58Speaker 00

  I think we're, I think the problems of the lean method are coming home to roost. What every VC wants to see is they want to see product market fit. There's a method we have, right? You come up with like, you know, rough business model canvas and then you go out and you do, talk to people and then you test in the world. That is not a good model for breakthrough innovation. That's a really good model for incremental innovation where you find market need. So part of this is that we're incentivizing startups to find solutions right now for a moving technology and they're just going to get lapped and they're not trained to be imaginative. They're not like, they're trained to think money first and you know, how do I get a market, product market fit? Which is fine in normal technological regimes, not a great idea in radical regimes.

• 36:42Harry Stebbings?

  What is a good model for a radical regime then? Cause I've been brought up quite rightly, as you mentioned there, in the incremental innovation kind of economy where it's like test, iterate, find product market fit, someone pays for it, good, well done. So what is the right model in this new age of kind of radical innovation shift?

• 36:59Speaker 00

  So, I mean, we, VCs have funded this model, right? And it's like deep tech, medical, things where you're making larger bets in the future, where there, you know, where there's payoff is where, where, when it's revealed to the world, they're going to succeed or not, right? And where you're making a bet on technology itself, that's where VC got its start. It sort of became, you know, perverted a little bit to this like, how do I get, you know, make money fast machine? I mean, not that fast, right? It's still years till exit, but there's the idea of like, it's all about pirata rights and the idea of like, I make a lot of small bets initially and then I could double down on the people doing well and not do double down on others. And it's about finding the diamond in the rough. Like all of that stuff is like a great model for funding incremental innovation. If the market's changing, like, and we're used to market changing slowly enough that like, that's not a problem. I think it's an issue here.

• 37:43Speaker 00

  I think you need to be imaginative. I think you need to be subject specific. I need, I think you need to assume model. I mean, it is very strange from one hand, for all these people at Silicon Valley to be like, yeah, AGI is coming. And then the applications they're building are like these very narrow, like, hey, I slapped something on top of llama. That's not going to do it.

• 38:01Harry Stebbings?

  What should I and what should my fellow venture investors change then about the way that we invest, do you think?

• 38:06Speaker 00

  I think that what you should be thinking about is have a position on the future and the startups you talk to have a position on the future of AI. How good does it get and how does your model work? The second thing I think people need to be thinking about is how actual adoption happens again. It used to be that if you have a large enough market to play with, we just go after all of it and you know, some part of it starts to respond and we double down in that section. You're going to be much more opinionated about how you imagine your technology being spread or adopted. How does it spread throughout our organization? Is it fit? How does it fit with organizational structure and approach? Requires people to have more plan and strategy than they did before, rather than just letting the market tell them the answer.

• 38:46Harry Stebbings?

  Do they not go in contradiction? You said there about, hey, you know, people work on small kind of minute things on top of llama, say, and then it's like, well, you need to be opinionated about who you're going after and who you're not going after. You need to be more targeted. Is that not kind of one and the same, which is like the verticalization of approach and the targeted approach being the core?

• 39:05Speaker 00

  Well, I think it's not about verticalization as much as opinionated, right? I think you need to have a strong opinion of what the future looks like and where the gaps are going to remain. This is a jagged technology. Figure out where you think there's going to be jaggedness and not going to be organizational jaggedness, interface jaggedness. But I mean, you're also basically, the real problem right now is every startup in the world is betting against AGI, which I find really funny because all the funders are like, yeah, AGI is coming in the next five years. If it is, why are you funding these startup companies? None of them survive in an AGI world.

• 39:35Harry Stebbings?

  For those that don't understand, why will none of them survive in an AGI world, Ethan?

• 39:38Speaker 00

  So the common definition of AGI is a machine that's smarter than humans at every task. The machine will decide what to do. You're not like, who cares about your stupid product, right? Like you've been making this for humans and getting product market fit and the but the humans will say, you know, optimize my trading strategy or the AI will just decide of optimizing your trade strategy. I mean, no one knows what AGI looks like. So I'm not going to try and paint a science picture of future, but I will say there's a huge contradiction between a market embrace and saying AGI soon and like, we're funding a bunch of companies that are helping you. Like already, I don't know if you've played with them, not that we're naming your AGI with this, but you know, You could tell Claude, come up with thirty ideas for a product to serve market X. Then rate them all on quality and feasibility level. Then create, this is one prompt, by the way.

• 40:19Speaker 00

  Then create a playable prototype of the interface for the application. Then interview me as a user about how to change it and adapt it as we go. And it does it. I get a little playable interface for a game and I can then audit the game and, you know, say like, oh, I wish it was more. It's just not fun enough in some way. And it's like, okay, great. I'll make it more fun for you. If that cycle is really there, what is your stance and what an AGI world looks like becomes very relevant is all I'm saying.

• 40:43Harry Stebbings?

  Do you find it interesting to see the different people's opinions on a, especially on the founder's side? Different people's opinions on the time to AGI and their requirements for funding. And so what I mean by that is like Demis and Zuck are very long term minded in terms of how long it will actually take to achieve AGI. And they also don't need any money. And then there are other founders who I will remain nameless, who are pumping it as being much sooner. But they need to present that future because they need the money.

• 41:13Speaker 00

  I don't trust anything. People are very self motivated, right? I think the signal you should pay attention to is that people are betting their careers to a large extent, right? On this being possible. And I think there are people who care about their reputations. That is a signal to me. They don't have to be right. I mean, look, I work with Marvin Minsky, like I said. He was there in, in the fifty seven conference where they out, in Dartmouth, where they outlined the concept of AI. I mean, we're in a world where like, you know, AGI is always soon. So I think you would take everything with a grain of salt. But I think you need some coherence about your own viewpoint on this set of stuff. Now, the large companies, I mean, we're seeing a lot of, you know, people warning that this is coming soon. I mean, in the, in the meta paper, the, the paper outlining the release of Lama three point one that came out yesterday. It says we see exponentials continuing for the, we don't see any reason why exponentials are going to stop.

• 41:58Speaker 00

  What does that mean for you as a startup? Feels like a relevant question. You're betting for a future world. So what does that future world look like? And you can't both say everything is changing, but also I'm doing this minor thing. I also think crypto did us dirty in this kind of front, which is like, it made all technology feel like hype. And it emphasized, again, short buck return. If you just believe something will happen and that's a really great way to think.

• 42:19Harry Stebbings?

  I actually liked Sam Altman when he said on the show, the simple kind of heuristic of like, whether you're going to get steamrolled by OpenAI is, would you be excited or scared by a hundred X improvement in our model? If yes, then you're going to get, you know, steamrolled. If no, then great. But I liked it as a heuristic.

• 42:35Speaker 00

  But I don't think it's useful as a heuristic. What does that mean? What is a hundred times better G P T? Like, it is, it's a baffling heuristic to say better. Like, what does that mean, right? It's an uneven system. It has gaps in the world. That mean a hundred times better reason? Like, how are you supposed to? Like, this is what I mean. When you start looking at these things, it's like, what the heck am I supposed to do with that? It's a hundred times better. It's a machine god, like, what? And so I don't like it as a heuristic because I like, I don't have any way to operate within that. A hundred times better, does that mean it will be able to process an entire legal document and do a very good legal review of a document on its own? Great, that's disrupts a huge industry. But that is a actual question about hallucination rates. It's ability to handle words, you know, to think about words instead of tokens,

• 43:19Speaker 00

  to understand precedent, to be okay across different languages, to hold a huge amount in its context window. That feels like a useful question to ask. Could it write an academic paper on its own, right? Where you give it a data set, it generates hypotheses, tests them, writes a really good paper, formats in latex, writes a letter to the editor and handles reviewer responses. We're getting close, but there's a lot of gaps there. Give me a concrete example of what this thing does. And then we could talk about a heuristic. But like a hundred times better is a really hard one.

• 43:47Harry Stebbings?

  No, those use cases are twenty eight times better. They're not a hundred times better, specifically twenty eight times. I'm joking.

• 43:53Speaker 00

  But maybe, but I mean, I think that's a valid question. Some of those things are huge gaps. Some of those things are small gaps. If you ask me about that, reviewing the legal document, not really a problem. Review, like, we're close to that. But like, if it does that out of the box, that also implies a lot lowering of hallucination rates below a threshold that they're not currently at. There's no benchmarks on hallucination. So we have no idea how good we're getting on the hallucination rate side. It also, though, implies the ability to seamlessly move between different perspectives. We could do that with agents today. Is it an agent based model taking action? Like, there's so many questions. I would love some specificity, and that's why I'm saying field specific is great. If you are a lawyer who knows law field really well, you probably might have some interesting things to think about and where the real gaps are or not.

• 44:38Speaker 00

  And I don't think a lot of the A I firms know that. I know this because we're deep working with all of them on things like education. And like, they don't really understand education. There's no educators there. So they don't really understand what teachers do. And they don't really understand what classrooms are for. And so it's all the A I will replace everyone. And I think we're a lot, and we're not there.

• 44:55Harry Stebbings?

  That was a trigger, wasn't it? Just give you, give you a Sam Altman heuristic. Ethan, I was like, ooh.

• 45:02Harry Stebbings?

  Tell me, you said there about kind of the importance of being opinionated and for startups to have strong opinions about where AGI will be, how they fit into it, organisational design. If I were to ask you and flip that on you, where are you most opinionated in your views around it? Where would you suggest or point to first?

• 45:18Speaker 00

  So I think education is a good starting point, but an education tutoring is the gold standard for interventions based on the research we have. And AI is incredible one-on-one tutor, like it's transformative. But what I find Silicon Valley people and AI and education people often think is like, well, once we have a really good tutor, we don't need teachers. Or like, I hated this subject in school. Or people will be self motivated to learn. Absolutely untrue. People are not self motivated to learn. And even all the computer scientists out there were like, I was like, yeah, you're autodidacted some narrow area, but you would have learned nothing about very important topics because you only cared about one topic, right? People need extrinsic motivation to learn. It turns out that there's value in having an instructor guiding the direction of a class, that there's value in putting

• 46:00Speaker 00

  things into practice. So even an incredible AI tutor that knows you and loves you really well, doesn't sub in for teachers. And also, forget all that, let's talk about systems. Schools are in a complex system of society and where they are for providing daycare services to how they fit into education networks, how we do credentialing, to the teacher unions, to like, there's a billion things about schools that don't get replaced by AI by having a magical button you push to make stuff happen. There's gaps and opportunities that are very different than a naive view of how education changes.

• 46:32Harry Stebbings?

  So one of the biggest problems in UK education today, I'm not sure if it's the same in the US, so you can tell me, but it's the exponential increase in class sizes that we've seen, particularly in public schools, which is the schools provided by the state. And the quality of education has gone way down. When we look at AI's ability to increase education standards, will we see the ability to maintain high education standards with increasing class sizes? How do you think about that?

• 47:00Speaker 00

  I mean, I hope so, but let's just talk the first randomized control trial we have, some of my colleagues at Warden was giving G P D four people for math tutoring in Turkey. Now, they didn't do a huge amount of like, you know, it was a assigned class and they use the system. But it turns out that everybody who used it used G P D four without any special prompting or anything else had much higher homework scores, and then did much worse in the test because basically the A I just did the work for them. And once you have better, once you have better prompts, that that effect disappeared, that we didn't see educational gains from it. But I think that's an early sign of like not being naive about how these systems operate, right? Like, we need to put the work into building scaffolding. I absolutely believe that we can't be naive about the work that needs to be done

• 47:41Speaker 00

  here to make this stuff operate. So you can't just drop these systems in. But a good tutor will make a difference. I think in the long term, we'll have flipped classrooms where that twenty percent, where that giant classroom is actually fine, because a lot of your learning is done outside of class, you know, A I tutor help. And then inside of class will be activities, exercises, application, where large class size doesn't matter as much. But there is a road to get from here to there.

• 48:05Harry Stebbings?

  I think the show's done well because I'm not scared to admit my own flaws and stupidity. Everyone talks about kind of the incredible optimism that A I brings for education and talks about tutoring. Great. But I don't really understand what that picture of the future of education looks like then. Does that look like when kids come home from school, they just have perplexity or open A I up and they have another tutor with them? Where, as you said, in most cases, they end up doing the work for them and so they don't learn? Is it a crutch that and they, I don't understand actually intangible reality. What does the future of education look like with A I? And why is it optimistic?

• 48:43Speaker 00

  So we actually have a lot of research on this. It turns out that, you know, first of all, there's a couple of things you need to know about learning that people don't tend to think about, which is learning is hard and sucks. What makes you feel like you're learning isn't what's learning. Like, you have to do grinding work. There's no solution to it. It's just like any other thing, like exercise or anything else. You have to be pushed to desirable difficulties where you're having trouble, you're not failing at a thing, you're not working hard enough. Like, that's, which is why you often need to intrinsic motivation. And the second thing is, We actually have some research. We know things like active learning, where you're in a classroom doing activities beats the idea of passive just receiving a lecture. When we have those sets of pieces, there's been a move that kind of fizzled called flipped classrooms that has some early evidence in its favor,

• 49:23Speaker 00

  which suggests this idea of like classrooms should be about doing stuff and outside of class should be about getting the basics. Because we can get you to do stuff in the classroom setting. So that would mean that outside of class, that what that practically meant is you watch videos outside of class, your teacher talking. So the lecture stuff is all outside of class. That's your homework. Read the book, do that. Then your homework is in class, where you can mess up in front of people and work in teams and you learn by kind of watching other people and how you're doing it. The teacher can help you solve problems. I think flipped classrooms are a very natural fit and active learning with AI based approaches. So instead of having a passive video you watch, you'll have an AI tutor outside of class. You'll log into the school's website and that tutor will be amazing. It'll be adapted to you.

• 50:04Speaker 00

  And then it'll pass that information on into the classroom setting where you actually, the teacher gets advanced stuff. And by the way, we've actually built a version of this already at the Journal of AI Lab at Wharton. We'll be open sourcing all of that, like that does this kind of stuff. It's not that hard to imagine. We just have a ways to go still.

• 50:18Harry Stebbings?

  Is that really an order of magnitude improvement if we compare that post classroom? You could give me incredible, high quality videos of you talking, lecturing, giving examples that you give to your students now, very easy to do versus that AI tutor. Is it, is it 20% better? Sure, maybe it's personalized, but is it really an order of magnitude better?

• 50:39Speaker 00

  Education is a complex system. So I think order of magnitude is a very weird thing to talk about because every student has their own talents, abilities, interests and gaps. The early work on in one-on-one tutoring, we don't talk about order of magnitude improvement because that doesn't really work in the education world. It's very hard to say what order of magnitude is, but we could talk about grades a lot. The classic study that is probably would not be replicable, but it sets up our model is that one-on-one tutoring creates a two sigma increase in classroom outcomes. That's two standard deviations, which is a fairly huge improvement. You go from the 50th percentile to the 97th percentile of class. We have no idea if that's going to hold up with, you know, AI tutoring, but if we could do that, that is as amazing an improvement as you could possibly.

• 51:19Speaker 00

  They ask for. I mean, a 10% improvement is amazing. I kind of feel like we aiming for order magnitude education. If we can get improvement in a system, we're in great shape.

• 51:28Harry Stebbings?

  I also think that doesn't include a lot of different elements that you mentioned. Extrinsic motivation being a big part of it. I think a big part of like having a tutor means you actually have a bond with them. You want to impress them. You want them to feel proud of you. Does that extend to an AI tutor where you don't have that human?

• 51:43Speaker 00

  Maybe. It's not clear that that is the key to tutoring is the bond with the human being. Forcing people to confront what they don't know turns out to be a lot of the value of tutoring. So tutoring is often reflective back. So when we built a tutor chatbot, right? What that tutor chatbot, like the way we test, by the way, education, technology, chatbots, our rule of thumb is that if it asks you if you understand a topic or you're ready to move on, it's a bad tutor. Because humans don't know when they're ready to move on or not. What the AI should be doing is asking you questions, probing what you know, and making you expand on what you don't understand, then helping you fill those gaps. It's not the one on one bond. There is a, there are like methods to teaching that we actually know make a difference. Self-reflection makes a difference. Repeated practice makes a difference. Low stakes testing makes a difference.

• 52:27Speaker 00

  Like to zoom back out to what we were talking about before. Subject matter expertise is going to be absolutely critical in making AI work. It's a system that experts, I can look at a prompt in entrepreneurship and education and instantly tell you about whether that's going to work or not, or whether it's a stupid idea or a good idea, or whether the subtleties that the system is missing are a problem or not. Because I am an expert. And if you're not an expert, you're going to be like, that looks really good. So like, expertise actually matters. I'm sure in the same way, you know, it's one of the things I actually, when I talk to my students in, you know, and teach them how to pitch, right? One of the things I talk about is there's this really interesting research that shows that venture capitalists are not swayed at all by the quality of the speaker. Their ability to be a good speaker or not is absolutely irrelevant.

• 53:10Speaker 00

  Amateur and angel investors are swayed by that. Why? Because you're an expert. You've seen so many pitches that you instantly see through all of that stuff. And you're like, you know what the core issues are right away because you've seen 10,000 pitches. You've seen how they play out in the world. You know, you have to be a really amazing speaker to pull off. I'm persuasive on top of that. And so in the same way, I think expertise is going to matter. A lot here.

• 53:30Harry Stebbings?

  When you look at the pervasiveness of AI and specifically ChatGPT in homework and in coursework and in the answers that many students give today, is there any point in university or educational facilities doing homework or coursework when it's largely done by AI today?

• 53:46Speaker 00

  Of course there is. We like, everybody was already cheating. Like, there's this great study at a repeating university that found that homework improved, when you did the homework, improved like eighty percent of people's test scores in two thousand and eight, and by two thousand and twenty, it only helped twenty percent of people. And that's not because homework stopped helping, it's because everyone was cheating. And so we have ways around this. There's really two options in how to use AI in education. One of them is to ban it cautiously, right? People are still going to use this explainers and stuff like that, but you you have in-class tests and blue book writing, like we've solved this problem in math. And like you make people do exercises and do work, nobody likes it, but there's no shortcut to learning. It sounds dumb, it's like what your teacher said, oh, turns out it's true. You need to do a grinding amount of work to understand something. You need to do interleaved practice, you need to like, there's

• 54:31Speaker 00

  a lot of stuff you need to do to learn something. And so we absolutely can make you do blue book work in class. We absolutely can install terrible monitoring systems. I don't like this approach, but like, a couple of companies already have this, that watch what you're typing and make sure you're not pasting stuff in from AI. Again, I don't necessarily recommend it, but like, these are possibilities. I think you're underestimating how much you can do those kind of things. Homework is valuable. Cheating is bad. What is AI cheating? We have to define that. I'm a big, but the other option is transformation. My classes are a hundred percent AI based at this point. Students have AI mentors and tutors they talk to. They have AI based assignments. When they learn how to do hiring, I build it, we build a simulator that actually makes them have to fake hire somebody, and the AI plays the person they're interviewing and gives them multiple choice answers, and they

• 55:14Speaker 00

  have to reflect on the assignment. There's one of the other assignments is they have to teach the AI to do something. You could do really exciting stuff, it's just not going to happen right away.

• 55:22Harry Stebbings?

  All of the different avenues, functionalities that we've spoken about require a lot of intense compute, considering there was such a trigger when I gave you a last quote from Sam, thought I'd give you another one. Compute is the currency of the future, what Sam Altman said. And energy is a concern when looking at the energy requirements that this next generation of AI will bring. How do you think about the energy requirements required for this next generation of AI usage in society and whether Sam is right that compute is the currency of the future?

• 55:51Speaker 00

  That's what Sam believes. I mean, Sam believes in AGI and he believes that it's going to be achievable in the near term, right? And when you talk to OpenAI insiders, they feel the same way. If that's the case, if intelligence of demand is the case, intelligence of demand is power hungry and there's infinite demand for intelligence on demand because there will be, right? If you have an AGI, I want that to be looking over all of my medical records and monitoring our airspace and, you know, finding scientific ideas and helping me with a project I have to do and also booking tickets for the ultimate trip. Like, there is infinite demand for intelligence, right? So then compute becomes the currency and energy becomes the big deal. That that will make a big deal in that case and we're going to build a lot of nuclear power plants, I guess, and, you know, in relatively short

• 56:33Speaker 00

  order. It seems like that's a problem, or AGI figures out how to do fusion and it doesn't matter or we'll get turned into batteries, all our matrix, although we don't produce enough wattage, you know, so I don't think that's really the issue. Training data, that's what the AI is well used for. But anyway, mostly joking. But right now, I think the energy debate is an interesting one because again, it's one where doomers and optimists sort of like to talk about because on the downside risk, when I meet people who are skeptical of AI, the first thing they talk about is energy use. And the truth is that AI uses a lot more energy per query. We don't know exactly, probably two orders of magnitude than a Google search, but a lot less orders of magnitude of energy than a human doing the same amount of work, right, with a laptop. You know, how do we balance those kind of things becomes an issue. Right now, 1% of US power goes to data centers and 10% of

• 57:16Speaker 00

  that goes to AI at most. So we have a lot of room left at the top before this becomes an issue. So again, we're assuming AGI is available, instantly useful, and in which case absolutely compute becomes and energy becomes the issue. But then that becomes the reverse salient. And, you know, there's a lot of money to be made that if if if the currency of the future is compute and computer is energy, then there's a hell of a lot of money to be made in building your own nuclear power plants.

• 57:40Harry Stebbings?

  The final one before we do a quick fire. A friend of mine, who's also quite a well-known venture capitalist, Jeff Lewis, said that when it comes to democracy, in the future, we will vote for algorithms, not for people. To what extent do you think AI pervades into electoral systems, electoral voting, the political fabric of our society?

• 57:59Speaker 00

  When something feels like a dystopia to most people, it probably is something that's not going to happen very quickly. Human systems are complicated. I just keep seeing this technological view, which is like, in a rational world, the machines will rule us all. It's just people don't want that, right? So like, we already have algorithms ruling lots of what we do. You know, your FICO score determines a huge amount of, you know, things that happen in your life, and that's an algorithm. Like, we have these kind of systems in place, but the idea of an overall all-seeing kind of approach, it's hard. Like, now, on the other hand, we do find that AI is hyper persuasive already, right? In a controlled experiment where you do, where your apps to be, talk to a normal person versus the AI, you're eighty-one point seven percent more likely to change your views to the AI's view than to a human's view. That is going to change marketing in very big ways, which is going to change politics, right?

• 58:44Speaker 00

  Deep fakes are going to be our big deal already, although it's been funny how little a big deal they are, because it just turns out all you need to do is show a video of politician X talking and say, I can't believe he said he's going to eat babies in minute three, and everybody shares it online, who should know better? And without actually watching the video at all. Like, when I post have a viral tweet, nobody clicks the link. I feel like we way overestimate people and therefore how much this stuff was going to matter. But in a world where AI is hyper persuasive, this does change things. In a world where AI gives really good advice on everything, people should have an AI second advisor happening in every role, including in politics, right? That would make things better. But people aren't going to listen to it. Politics changes much more slowly, is much more human than people think. By the way, it plugs into larger issues of like, when we can produce all this stuff on demand, what is actually valuable or not?

• 59:29Speaker 00

  I mean, everything is going to change. It's very hard to make predictions how a general purpose technology rolls out. But I do think people overestimate how quickly the short-term change is going to be. And as usual from Mara's Law, underestimate the long term.

• 59:41Harry Stebbings?

  I completely agree with you there. My biggest concern, actually, you know, as a content creator in many respects, is with the infinite supply of content, the value goes down and discovery becomes much more challenging. That is a big concern.

• 59:54Speaker 00

  I mean, that problem has already happened, right? I mean, you know, to me, the really interesting thing is like Suno and Odeo and like they're getting pretty good. At what point does having an AI-generated song playlist, you know, that has a couple of real musicians, but also makes up songs based like that doesn't feel as far off for in terms of people enjoying it like what happens to content creation is a very big deal, right? I mean, like I'm an author, my book's a New York Times bestseller. That's amazing. I don't think people realize how few copies you need to be to be a New York Times bestseller. Like you're if you're like, you're selling like 6,000 hardcover copies in a week. That's getting on the New York Times bestseller list. Attention's already scattered across a huge amount of content. The one thing you'd hope for is maybe AI creates better connections, right? And, you know, in some ways.

• 1:00:36Harry Stebbings?

  I'm not being rude, but could you not just buy? I know you haven't, but like, could I not just do it? Do a book and spend $75,000 and be a New York Times bestseller then?

• 1:00:45Speaker 00

  So people do that all the time. The New York Times has a small cabal of people who refuse to talk about how they do this. So they use the number ranking, but then they also try and exclude bulk buys. They actually try and cut that out. So those you'll notice there's a little dagger next to the name of of companies of the bestseller list that they think that they're including, but they still had potential bulk buys. I actually got the little dagger on mine because a company bought 500 copies, which wasn't the main reason for the list, but they were found that suspicious. They're trying to filter that out by hand. But yes, you can often buy your way into the list and people do do that all the time.

• 1:01:17Harry Stebbings?

  Yeah, no, I've seen many of my friends who are VCs who have books and I'm like, really?

• 1:01:22Speaker 00

  There are ways of doing this. You scatter buyers across multiple locations and they all do. But like, it is very much true that a lot of your unnamed VCs do seem to have asked a lot of friends to buy bulk copies of their book.

• 1:01:33Harry Stebbings?

  Amazing. I love that. Uh, listen, Ethan, I could talk to you all day. I want to do a quick fire round. So I say a short statement, you give me your immediate thoughts. Does that sound okay?

• 1:01:41Speaker 00

  Sounds great.

• 1:01:42Harry Stebbings?

  What do you believe that most around you disbelieve?

• 1:01:45Speaker 00

  I feel like the very simple idea that AI is very profoundly is much better than people think and is a heap betting better is something that I think most people don't actually believe.

• 1:01:53Harry Stebbings?

  What's the most concerning future that AI could bring?

• 1:01:56Speaker 00

  The most concerning future, I think, is one where we lose agency and not necessarily to the AI systems, but to the systems that incorporate AI. What I mean by that is we have a chance to make AI be used for human thriving. That's not an automatic process, right? That means not firing people when you have AI in your company, but it means figuring out other uses for them that are valuable. It means building systems that help people feel like they're accomplishing more as a result of using these things. And I worry we're not seeing enough people modeling that kind of behavior. It's all about just the technology itself and then how do we get cost savings?

• 1:02:29Harry Stebbings?

  What have you changed your mind on most in the last 12 months?

• 1:02:33Speaker 00

  I have gone back and forth on how much juice the technology has left and now I'm back to the it has lots of juice left, like the exponential continues for a while. And I think I was not clear on that for a long time.

• 1:02:43Harry Stebbings?

  What caused that shift backwards?

• 1:02:45Speaker 00

  Accumulation of evidence, right? So we talked about Kevin Scott saying scale. There's a bunch of people who weren't talking about scaling solving everything six months ago or eight months ago, who are now more competent, which indicates to me another generation of models came out and everyone at all the labs are getting that hounded look in their eye again. I don't know when we'll see these models, but they're clearly, people are seeing things that indicate to me that there's more, more left in the curve and they're all talking about it.

• 1:03:07Harry Stebbings?

  Do we see all large players and incumbents move into the chip layer? We've seen Apple move into the chip layer, internalize margins, remove reliance away from Nvidia. Do we see that as a large shift in all providers?

• 1:03:21Speaker 00

  Your requirement in any supply chain pipeline is to eat the value. If you're, you know, like, that's the whole idea of like how, you know, how those things work. So like, if you're spending a lot of money on chips, you go into chip making, just like if you're spending lots of money on, you know, warehousing, you figure out a way to, you know, reduce your warehouse costs, like, you're gonna figure something out.

• 1:03:41Harry Stebbings?

  What elements, sorry, of AI development has most positively surprised you?

• 1:03:45Speaker 00

  How clever these things are. Like, I'm having, like, if you haven't seen my Twitter feed where I asked the AI to remove references about squid from the novel All Quiet on the Western Front, I just look for that because like, these systems are really clever. They're, they're kind of joyful to use and I, I think that that's kind of surprising.

• 1:04:03Harry Stebbings?

  Final one for you, Ethan. What question are you not? Ever asked that you think you should be asked more.

• 1:04:09Speaker 00

  The question that I think people should be asking and that I don't have an answer to yet is, why are so many people bouncing off these systems? Like, why are they not, you know, why are so many people using them a little bit and not forever? Because I don't think it's just simple, like it didn't work. People are getting kind of freaked out by these things in ways that we don't really understand how humans are relating to these tools. Like, we always talk about the systems, the technology, how industry is going to change. And I don't mean just like the dating thing, which tends to be like, well, you have a relationship with AI. But like, how are we relating to these kind of tools? That's one of the questions. Let me do a second take on this. The thing that I would be thinking about, because it's related, is meaning. People don't ask me enough about meaning of work. That matters a lot. You know, Graber's bullshit jobs, I think, was mostly not correct based on survey data and other stuff we saw. But it's real. People do feel eliminated from work.

• 1:04:54Speaker 00

  People do, like, most employees say they're bored at least 10% of the time at work. But they're doing work that they feel is meaningful. When you survey people, most people think their jobs matter in the world. And what's going to happen that I'm very worried about is when you realize as a middle manager that AI does your work and nobody cares. What does that mean for the nature of work? How does that matter if people don't care? Like, if AI subs in and does stuff? If your boss is responding with an AI answer to the emails you send them? And I think that that meaning crisis is one we're not talking about enough. It's one thing to be replaced in a job, it's another to semi-replace yourself and realize, why am I doing this? And I think that's going to be a bigger issue that we're not talking about.

• 1:05:31Harry Stebbings?

  Ethan, listen, I apologize for going in so many different directions. I so appreciate you putting up with some of my base questions, but this has been fantastic. And for me, as a lover of your writing, it's been a huge pleasure, my friend.

• 1:05:42Speaker 00

  This has been wonderful, too. Please don't tell Sam Altman I disagreed with him because he's building a machine god and I don't want him to be angry with me.

• 1:05:51Harry Stebbings?

  As I said at the beginning, Ethan's writing is some of my favorite writing on AI. So if you haven't checked that out, you can check that out by checking out Ethan Mollick on Substack. And if you want to watch the full interview with video, you can find it on YouTube by searching for 20VC. We always love to see you there. But before we leave you today, when a promising startup files for an IPO or a venture capital firm loses its marquee partner, being the first to know gives you an advantage and time to plan your Strategic response. Chances are, The Information reported it first. The Information is the trusted source for that important first look at actionable news across technology and finance, driving decisions with breaking stories, proprietary data tools, and a spotlight on industry trends.

• 1:06:32Harry Stebbings?

  With a subscription, you will join an elite community that includes leaders from the top VC firms, CEOs from Fortune 500 companies, and esteemed banking and investment professionals. In addition to must-read journalism in your inbox every day, you'll engage with fellow leaders in their active discussions or in person at exclusive events. Learn more and access a special offer for 20VC's listeners at www.theinformation.com/deals/20vc. And speaking of incredible products that allows your team to do more, we need to talk about SecureFrame. SecureFrame provides incredible levels of trust to your customers through automation. SecureFrame empowers businesses to build trust with customers

• 1:07:14Harry Stebbings?

  by simplifying information security and compliance through AI and automation. Thousands of fast-growing businesses, including NASDAQ, Angel List, Doodle and Coder, trust SecureFrame to expedite their compliance journey for global security and privacy standards such as SOC 2, ISO 2701, HIPAA, GDPR and more. Backed by top-tier investors and corporations such as Google, Kleiner Perkins, the company is among the Forbes List of Top 100 Startup Employers for 2023 and Business Insider's List of the 34 Most Promising AI Startups of 2023. Learn more today at SecureFrame.com. It really is a must. And finally, a company is nothing without its people. A global law firm built around startups and venture capital.

• 1:07:59Harry Stebbings?

  Since forming the first venture fund in Silicon Valley, Cooley has formed more venture capital funds than any other law firm in the world. With 60-plus years working with VCs, they help VCs form and manage funds, make investments, and handle the myriad issues that arise through a fund's lifetime. We use them at 20VC and have loved working with their teams in the US, London and Asia over the last few years. So to learn more about the number one most active law firm representing VC Backed companies going public. Head over to coolie.com and also cooliego.com. Coolie's award-winning free legal resource for entrepreneurs. As always, I so appreciate all your support and stay tuned for an incredible episode coming Friday.