Ep 63: Khan Academy Founder/CEO on Salman Khan on Classrooms in 20 years, Rolling out to 1.4M Users & Sal’s Hopes for AI Education

When Khan Academy launched Khanmigo, Salman Khan thought they might reach 100k users by 2025. Today, they’re at 1.4 million. 🎧 Sal joined us on Unsupervised Learning to talk about AI's role in education— from the vantage point of someone deploying it at scale. As founder of Khan Academy, he’s overseen the rollout of AI tools to over a million teachers and students, giving him a front-row seat to what’s actually working in classrooms.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Salman Khan is the founder and CEO of Khan Academy, one of the most influential education platforms in the world. They serve over 150 million learners across 190 countries. And they've also been doing some really interesting stuff in AI with Khan Miga, their AI-powered tutoring assistant. It's already been deployed to over 1.4 million students and teachers. And Salman and I had a really interesting conversation around how AI is transforming education for students, teachers, and school systems. We talked about where these models work, where they don't, and what it takes to build AI that actually makes a difference in the classrooms. Some of my favorite parts included why proactive AI is the next evolution of tutoring, some surprising ways students and teachers are using AI, what it takes to reduce hallucinations and math errors to near zero, and how AI is reshaping engagement and assessment. This was an awesome conversation for someone who's really doing

• 0:44Jacob Effron?

  one of the widest deployments of AI today in the real world for a really good mission. Before we get to the episode, I just want to say a huge thank you to all those that rated the show on Spotify and Apple last week. If you're enjoying this on either of those platforms and haven't yet left a rating, please consider doing it. Now here's Salman Khan.

• 1:03Jacob Effron?

  Well, thanks so much for coming on the podcast. Really appreciate it.

• 1:06Speaker 01

  Thanks for having me.

• 1:07Jacob Effron?

  Yeah, this is going to be a fun one. And I figured I would start with the most overly broad question I could, which I'm sure our listeners are curious about. Given all the rapid advances in these models and what you've been building on top of them, what's your current thinking on what a classroom looks like 20 years from now?

• 1:25Speaker 01

  Yeah, it's a good question. I think if we had this conversation five years ago, I would have thought that's a boring question. And now I think that's too far out because things are changing so fast.

• 1:34Jacob Effron?

  Always hard to find the right time period in AI worlds.

• 1:37Speaker 01

  I'll tell you what I'd like to think a classroom looks like in 20 years or even in five years. But let's go with your 20. I'd like to think that more classrooms are actually going to look like what great classrooms already look like. A great classroom today is one where students are not just passively listening to a lecture. They're engaged, they're interacting with each other. They're actually doing things, whether it's problem solving, working in groups, giving presentations. In a great classroom today, a teacher is walking around, working with the students, getting them to do interesting things. Now, the reason why I think AI is potentially going to play a role there

• 2:21Speaker 01

  is right now those great teachers. They have to spend a lot of time making great lesson plans, being very creative about it. These great teachers are able to do that even though they still have hours of grading and lesson planning and progress report writing. These great teachers are the ones that can really observe a classroom and almost intuitively understand what students might need. And those are superpowers. And I think there's a world where artificial intelligence can give more teachers, all teachers some time back on their planning side of things, giving them better insights on how do they

• 3:02Speaker 01

  well, some ideas and insights on how they can better manage their classroom, better insights on where the students are at any point in time, better interactivity with their students. So you know, when these students are all doing breakouts, and I can imagine in 20 years for sure, it's not going to necessarily be on your laptop. I think one of the powerful things about where generative AI is going is there's no reason why it can't be ambient. And it's just observing the classroom and seeing what's going on. And so I think you'll have, right now, people try to think very binary, okay, technology is you're just staring at a screen, non-technology is you're running around the real world. I don't think that necessarily has to be a trade off in 20 years.

• 3:43Speaker 01

  I think in 20 years, you're also going to have a whole other aspect, which is, which sounds science fiction. But I think that's a point at which virtual reality, augmented reality will become very mainstream. And that with generative AI and superintelligence of that, who knows? You're going to be able to immersively go into simulations, be part of virtual worlds, go back to ancient Rome and try to stop or maybe help hasten the assassination of Julius Caesar. Well, whatever it might be, but you know, that it's literally like a magic school bus ride.

• 4:16Jacob Effron?

  Yeah, no, I mean, a lot of really compelling threads to pull on there. I mean, I think especially as you talk about kind of giving teachers superpowers and kind of augmenting what the best ones already are able to do. Probably a good segue just to talk a little bit about, you know, what you've been doing with Khan Migo and the efforts, you know, at Khan Academy, Khan Academy building AI products. Could you just give a little bit of context for our listeners on, you know, what you guys have built to date? How you've been thinking about this?

• 4:38Speaker 01

  Yeah, it's interesting because I think in this time of rapid change, it's always important to have like, what are we in this to begin with for? Like, what's our true north? And Khan Academy's true north, I've been articulating it this much more clearly, actually, ever since generative AI became a thing. But if you go back to the early days of Khan Academy and it was just a hobby, I was tutoring family members, then I started writing software, I started making videos. Obviously Khan Academy became much more than me, but everything we've done over the last almost 20 years is trying to approximate or replicate some of that personalization that a good tutor would do. You know, not only make hopefully high quality materials available, but make it so that it can be, you know,

• 5:20Speaker 01

  personalized to the student, they can get practice. Teachers are an important part of it. How does a teacher with 30 kids in their classroom personalize it more? So we've always been doing that. And even in the early days of Khan Academy, I used to cite Diamond Age, the Neal Stephenson book on Young Ladies Illustrated Primer. I was like, we're going to build that wonder. This is what we are building. It's just we're going to be doing it incrementally as the technology gets better. So when we saw what was possible with the latest generation of, you know, in particular, GPT-4, which OpenAI gave us access to months before even ChatGPT existed. It had issues, but we said, okay, this can, this is going to be able to approximate tutoring, it's going to be able to approximate teaching assistants and other things that we probably hadn't conceptualized before.

• 6:03Speaker 01

  So that's how we launched Khan Migo, which is our AI assistant as a tutor and a teaching assistant, putting guardrails so that teachers can see what students are up to. It won't cheat. It, you know, safety, privacy, et cetera. It's much more Socratic. So we're really trying to lean into the, let's make this really good pedagogy. Let's make this safe. But let's also, let's also make it useful. I think what we've We've had a lot of successes. When we launched, I thought maybe by 2025, we'd have maybe 100,000 folks using it as a pilot. It's now pushing about 1.3, 1.4 million teachers and students. And these are, these are, you know, our mission is free world-class education for anyone, anywhere.

• 6:45Speaker 01

  But we've had to charge these districts for it because of the compute cost and the support and the training that we've been giving. We charge them about $15 a year. So this is, you know, to have 1.3, 1.4 million districts paying within a year and a half, and the interest continues to be very, very strong. So one thing that we are doing now is realizing that the next phase is making the AI much more proactive. If, even if I walked into any math classroom or any classroom and I said, hey, I'm a great tutor. I'm here in the back of the room. If you have any questions, come ask me. There's probably going to be about 10 or 15% of kids who do it. And we're seeing that with the AI. So this next version of Khan

• 7:29Speaker 01

  Academy you're going to see, we're going to start piloting it in back to school. We're calling it Khan Academy Classroom. But it's from the student point of view, a much more proactive AI that every time you go to Khan Academy, it's like, oh, welcome back, Jacob. Hey, it's been a little while. Hey, here's what your teacher wants you to do. Hey, how can I help you here? And from the teacher point of view, same thing, much more like a concierge, front and center, as opposed to being just something to be asked.

• 7:51Jacob Effron?

  Yeah, it's so interesting. I feel like, you know, across AI products, there's like this blank screen problem where you get, you can, you get to a place where you can prompt anything, but it's like, how do you figure out actually what, what the right thing to prompt is? How have you kind of thought about, you know, just like starting to teach students and teachers as you've, as you've kind of rolled out? I think Newark was one of the places where you did a huge rollout. Like, what have you learned, I guess, about, you know, teaching folks how to start using these products and, you know, kind of getting started from a cold start?

• 8:16Speaker 01

  Yeah, it's a very real thing. And it is where, you know, we've done, we've done things like little dynamic action bubbles where we're suggesting things that people might want to try next. Obviously, there's some training, things like that. But I think the core is, is making the AI much more proactive about things. The other thing I'll say, because it's important, right now, there's probably 500 people who are claiming to make some version of an AI tutor in some, in some way, shape or form. I don't think the AI is quite ready yet. To be by itself and drive learning for most people. I think if you're a curious person, you could go to chat GPT.

• 8:58Speaker 01

  And if you spend half an hour every night with chat GPT and just you're really good at prompting it and asking it questions, you can learn a ton. But for most people, that isn't most students. And so, and, and the AIs aren't great yet at creating a high quality questions that are not going to have errors that can give you. They're getting better every day, but they're still not where they need to be. So, you know, a lot of what we see in places like Newark is the efficacy, and we really are seeing some amazing efficacy numbers coming out of there. It really happens from doing the traditional practice on Khan Academy. And the AI there is a, is a support. It's something to help drive engagement. And a lot of people in education and ed tech always think about, oh, can I come, come up with a more efficacious intervention?

• 9:43Speaker 01

  And that's obviously matters. But it turns out if you engage with anything reasonably, that seems reasonably healthy, it's probably efficacious. The hard part's the engagement. And this is where we are, you know, we're, we're trying to look at every dimension of that pipeline, all the way from how do you get a teacher activated faster? And where's the AI there? How do you get the district administrators in a polite way, holding the classrooms accountable? That like, okay, you're really engaged in this tool we're using. The students are really using the AI. And then of course, how does the, the district leaders and the teachers hold the students accountable? These types of human systems and how the AI can help the humans hold other humans accountable is actually how you get engagement.

• 10:25Jacob Effron?

  Yeah, it's such an interesting point that engagement is actually like really the thing you need to solve for. You know, and on the student side, I'm curious, like, obviously you've kind of released this now to a ton of people using it. Any like surprises in the way that it's been used? I thought, I mean, I've heard you speak on another podcast about, you know, kind of, obviously there's some of the shortcomings of these models is they aren't always right. And you found that like students were explaining their reasoning to the models and then the models were like iterating on that reasoning. And it actually was like a really interesting way for like students to learn and the models to actually correct themselves with the mistakes they're making. Like anything else you've noticed in, in the way people use this that might have surprised you?

• 11:00Speaker 01

  There's definitely cases, I know, these aren't mainstream, but I said, if you're a really motivated person and you engage properly with these models, there's some amazing magical things that can happen. I gave an example, this was a TED Talk two years ago, I gave an example, but it's still a fun story of this young, young woman at Khan, Khan World School, which is an online charter. School we have and she was in India at the time and she um was reading about the Great Gatsby and we have an activity where you can talk to AI simulations of literary characters and she had like a lengthy conversation with AI Jay Gatsby about every and you know our um simulations they don't just try to answer questions they actually try to drive the conversation so what have you thought are there things like in your life that you

• 11:43Speaker 01

  wanna and um I I remember her telling me about that interaction and and thinking okay this this is beautiful this is the kind of thing that we want to see more of I I have I've I've talked to my son a lot my oldest who's uh just turned sixteen and you know he's he's he's he's doing some pretty advanced math now he's he's probably at my level or maybe even a little ahead of me at this point um and and he's actually using it all the time to really explore ideas and I'm like is it a hundred percent right all the time he's like no but neither are you

• 12:18Speaker 01

  and yeah so um and and you know we one of the things that we are putting a lot of resources and I don't think a lot of other people are are really trying to not just improve the AI's accuracy but measure how good it is and as far as we can tell right now Khan Migo when it's anchored on Khan Academy content you're at about a two percent error rate and the two percent error rates about split evenly between just a straight one percent of the time it's a math error and one percent of the time it's an evaluation error so maybe the answer is one third you put zero point three three and it said great job it should have said well close are you sure that you know do the threes keep going or can you represent that as a fraction um and we we obviously would love to get it to a zero percent error

• 13:02Speaker 01

  but I actually think my son is right it's actually already when I'm tutoring my own kids now it's probably one out of every ten or fifteen times that I'm I'm like wait that's not what I got wait let's do that again and I'm like oh yeah you're right you're right you're right you know and so um I actually think the error rate is actually already better than a lot of human tutors.

• 13:23Jacob Effron?

  What about on the teacher's side? Like, you know, how have you seen, you know, the best teachers leverage these tools? What's kind of the overall reaction been to some of these things, you know, entering the classroom? I mean, obviously, I guess, in some senses, through ChatGPT's broad release, they've entered the classrooms regardless of whether it's through Khanmigo or not. And so what have you kind of noticed on the ground there?

• 13:39Speaker 01

  Yeah, the ideal, and this is maybe not quite as sexy. I mean, there's some sexy use cases from teachers too. But the ideal is they just use it regularly. They create a habit around it where they're working with the AI, helping tweak their lesson plans, making it a little bit more entertaining, right size for the classroom, then delivering it in the classroom. We have a partnership with BlueKit for those, you know, BlueKit's, I believe, as popular or more popular than Kahoot. It's like an in-class gaming based on question sets. So our partnership with BlueKit is Khanmigo will generate the BlueKit questions for the teacher. So what used to take a teacher maybe half an hour, hour to write 15, 20 questions or more,

• 14:21Speaker 01

  they can now do with like, in about like two minutes. And we're seeing a lot of great things there. So we're seeing the planning, the delivery inside of the classroom, and then the getting insights from it, and then that's for another wave of planning. And there's a lot of things that we're building for this coming year that will make that much more streamlined and integrated. But a teacher who has that kind of habit for me, it's always been the case, who's using Khan Academy to make assignments, hold the students accountable, look at the data, and then keep doing that, they seem to get very, very good results. Once again, it's all about engagement. In terms of sexier use cases, I mean, we've definitely seen teachers talking about, you know, AI simulations. They're definitely opening up a class and saying, all right, everyone, we're gonna, we're gonna talk to AI simulation of Harriet Tubman

• 15:06Speaker 01

  or George Washington, ask your hardest questions, which is really, which is really engaging. We are seeing, we have another tool called Writing Coach, which is our answer to the fears around cheating, et cetera, where the teacher creates the assignment with the AI, assigns through the AI, students do it with the AI, but the AI really acts as an ethical writing coach. And then when the student submits, the teacher doesn't just get the final output, they get the process, and they can talk to the AI about what went on. And if, if I copy and paste something from ChatGPT, the AI says, I don't know where this came from. So it actually undermines not just AI cheating, we believe all forms of cheating. But teachers are starting to use that reasonably regularly for writing assignments.

• 15:51Speaker 01

  So that one's I'm pretty excited about.

• 15:53Jacob Effron?

  I love that approach for just like rather than, you know, say, hey, we're going to just ban these tools completely. It's a unrealistic and B not how people are actually going to interact in the world. It's like, let's find a way to teach people to use them in a way that still develops their own thinking. I think it's really, it's really clever. Do you imagine that being true across other subjects as well? Or like, obviously in writing, it makes sense, you know, given that in the workforce, people are going to use these tools going forward. Do you think like most of schooling should involve using leveraging them to some extent? Or are there still places where they should be, you know, you should be able to do math and writing completely without them?

• 16:25Speaker 01

  I think it's got to be both. I definitely think if you're going to be managing an AI to do some of your work, you need to be able to write well yourself. So I think especially if we're pre-high school or even early phases of high school, yes, you should do some more writing inside of the classroom with the teacher there. It could be a little short form writing, et cetera, or maybe you do it over multiple class periods. That I think is healthy. At the same time, especially once you get into high school and college, yes, you should have more opportunities to use the tools to get more productive. I have to give a few commencement addresses later in a couple of months. And I was like, I better work on this.

• 17:07Speaker 01

  And I was in an airport and all I did is I, I, I recorded my, my thoughts, my like life advice, you know, like, and then I, I had it transcribed by an AI and then I had it, and then I had it turned into a, a first draft. And that first draft, I mean, it speaks to both things. That saved me a ton of time that I would have otherwise done. But if I didn't know how to write and if I just took that first draft that just came out of my like, my, my random musings, it's a horrible speech. But it had, it had the essence of some of the ideas. Like I was just looking at it last night and I was like, wow, you know, this one way of phrasing it was actually quite beautiful and I, I'm going to tweak it a little bit because it's not exactly how it sounds like me.

• 17:51Speaker 01

  But it, it going from zero to one, and really it went from, it went from point three to like point seven, right? Because I gave it, these are my thoughts. But these things are massive accelerants and people should learn how to, how to do this. I, yeah, just speaking out loud, I can imagine when you and I were in school, the term paper, they say you have two weeks. What if a teacher says?

• 18:16Speaker 01

  You know, your opinion on this matter, but by the end of the class period, I want to, I want to see some output.

• 18:21Jacob Effron?

  What about on the, on the district level? I feel like in other categories, obviously, we're seeing tremendous pressure from boards, from, you know, CEOs like, hey, let's, let's adopt AI as fast as we can. You know, obviously, the, the schooling system is, is its own kind of unique universe. Does it feel like there's that same kind of pressure for uptake or, or momentum? Or what have you kind of observed at like the, the policy and kind of district level?

• 18:43Speaker 01

  I think generally speaking, this is one of the cases where schools might be one of the first places where you see mainstream adoption of AI for productivity and, and learning and, and just, you know, doing the day to day work, which is really trying to help kids learn.

• 19:01Speaker 01

  As you already mentioned, teachers have already leaned into this because so much of what teachers do can be streamlined with AI.

• 19:10Speaker 01

  Especially on the planning side of things and the grading side of things. And as the models get better and can support students better and get more proactive, I think everyone sees it. And yes, there is compute costs, et cetera, but it's dramatically cheaper than anything else that's come before. After the pandemic, there was $86 billion that was spent on S or these, these funds to help kids remediate. And that, you know, that's like 2,000 something dollars per American student. And a lot of districts plowed it into fairly expensive paid tutoring, like live tutoring. And there are some exceptions, but for the most part, there's not much to be shown for it.

• 19:50Speaker 01

  You know, so instead of something that was costing 25, $50 an hour, you're now looking at something that costs 10, $15 a year. And, and you get much more dosage if you want to. So, yes, I, I, I think I'm actually seeing more in, in school districts than I'm seeing as a leader of Khan Academy. And, you know, I've been pushing the Khan Academy team. I was like, you know, what, what are we going to be able to get, you know, automate some of our bookkeeping or when can we do theirs on this or when can we do, you know, I'm constantly pushing the engineers on how much more productive are you getting with the coding? I heard that company X is, you know, 100%. Why can't we be 100%?

• 20:30Speaker 01

  But um, yeah, I think schools are, schools, there's, there's school districts we've talked about with, they are, they're saying it's saving their teachers at least five hours a week, if not more. Um, they're using it as a recruiting tool, retention tool.

• 20:42Jacob Effron?

  Yeah, it's really powerful. I mean, I guess from a, from a society perspective, I'm glad that schools have been one of the fast adopters, if, if any place was gonna, was gonna be. Um, you know, I guess I'm curious, in the process of building Khan Migo, I mean, you talked about, obviously, you had to add guardrails in. It sounds like you've, you know, tied the models to your own content that you've already had on Khan Academy. Like, anything else you needed to build on top of the OpenAI models to like, make this work in the, in the classroom setting?

• 21:06Speaker 01

  Oh, yeah, I will. I mean, I could go down the list, even, you know, it, it is surprising. On one level, a lot of these AI applications, and unfortunately, I think many of them are, are just kind of thin prompting layers on top of a, on top of a model. But um, yes, you want to do the safety, you want to do the moderation. Moderation is something that frankly, we probably were overly conservative to begin with.

• 21:29Jacob Effron?

  Makes sense, given your, your, your audience.

• 21:31Speaker 01

  Given our audience, uh, but then, you know, there were a lot of false positives. So I think we, we now have the handle on, have the handle on that one. The math, math accuracy in particular, this is where there's a lot of work. I mean, just to get the error rate low. And once again, the, the hardest errors aren't, can this thing figure out what five to the eighth power is? The hardest errors are evaluation errors with the students, especially when there are certain students who are just hammering it, trying to get an answer. And they're, they keep switching context, et cetera, et cetera. And so how do you um, handle, handle all of that? A lot of work, just making the user interface something that feels more natural.

• 22:13Speaker 01

  Obviously, as I mentioned earlier, we're doing this whole reengineering of the front end of Khan Academy to imagine a more proactive AI-first um, approach. And I think when people see that, they're gonna see like, okay, this is, this is not just a chatbot. This is the AI integrating in every aspect of what, the writing coach we've been doing, you know, that's where, once again, it's not just a chatbot. It's, it's.

• 22:37Speaker 01

  There's a brainstorming on your thesis statement. There's an outlining tool that the AI can see and has context. And then when you're drafting, it really is kind of like you're on a Google Doc and it's highlighting parts of it based on the dimensions. So, yes, there's a lot to be done. That's well above and beyond just a thin prompting layer. There's a lot of, you know, I think the world now doesn't want 20 different apps on an AI tool. They want an app that's smart enough to figure out and behind the scenes, do a little bit of prompt chaining and swap prompts out. So there's a lot of work there and making sure that that's robust memory. I could keep going.

• 23:19Jacob Effron?

  Yeah. What are like the capabilities that are meaningful to you that are still on the come? Like things that you, you know, breakthroughs, I guess, you know, either in the core LMS or some of these multimodal capabilities that would be, you know, kind of game changing in what you're able to do in the product.

• 23:32Speaker 01

  I mean, memory is a big one. I know right now you could go to chat GPT and ask it about, you know, apparently it remembers everything you've done. But yeah, for people like us to have access to that so that we can give them the models much more context. It has all the memory, but at the same time, maybe there's ways to reset some aspects of that memory.

• 23:54Speaker 01

  Know what it's remembering or inferring about you. But I think memory is a big one. The advanced voice, which is out there, you can use it already in chat GPT and Gemini and all of that. I think integrating that with our platform is going to be pretty cool. We are building for a world where over time, the Khan Academy written content might become less and less relevant. You know, that breaks my heart as someone who's made 7,000 videos, but it's just a reality. And so.

• 24:27Jacob Effron?

  You know, they're all being used to train the models anyway, right?

• 24:29Speaker 01

  I'm sure.

• 24:29Jacob Effron?

  They're still in there.

• 24:30Speaker 01

  They've had some utility. Yeah. But the ability for the models to create higher quality questions, I think is big. You know, I think there's some very cool capabilities of models, like the ability to make videos and certain types of images and all of that. Those are fun. I don't think the world has come up with really good pedagogical use cases for them just yet. I mean, there could be some fun project based learning, things like that. But yeah, the pedagogical side hasn't been. Well, one on image, we do hope this is probably about a year and a half out. For students to be able to, especially if they're on a tablet device, be able to show their work and the AI being able to see their work.

• 25:13Jacob Effron?

  I think you had your son demo that, right? With the

• 25:16Speaker 01

  Yeah, yeah. With that OpenAI demo from last year, which, you know, in fairness to the demo gods, that was a fifth take. It was making a ton of errors, but it eventually did work. But that idea of the AI very naturally being able to talk while seeing your work and giving you feedback. Yeah, I think that's, that feels almost, if it's robust, it feels indistinguishable from when I was tutoring my cousins or now I'm tutoring my kids. So I'm hoping if in the next year and a half, two years, we can make that a mainstream thing, that'd be a big deal.

• 25:53Jacob Effron?

  Yeah, one thing I'm struck by, and you alluded, obviously, to like, there's kind of like fun use cases one could explore. And, you know, to your earlier point that like so much of this is around engagement and like, you know, a lot of the techniques, there's a bunch of different techniques and a lot of them are effective and it's really about just getting people to engage with any of them. I'm curious to see whether we see some, you know, interesting, kind of just like fun, new ways that we haven't conceptualized that actually drive, you know, drive people to engage. And then obviously you can layer some of this other stuff on top of it.

• 26:16Speaker 01

  Yeah, and you know, I'm always, we've been gamifying a bit over the years. And obviously there's people in ed tech who've done very good jobs, people like Duolingo. You know, Duolingo's have the benefit of, they kind of make up the standards, right? And they can, they can, they can, they can, they can make those dopamine hits as long as, as much as you want. When we started, we were on that same journey, but then as soon as we became more mainstream in schools, we're like, okay, we have to align to the standards. And some of these standards are not the most pleasant things for some students. And so there's, there's always going to be, you know, if you really want to do, and learning isn't always easy. So there is always that like productive struggle you have to do.

• 26:54Speaker 01

  Now, I do think if we can be really creative at about, about like quest based learning and when people go on scavenger hunts or escape rooms, they are willing to solve pretty hard, cognitively challenging problems because it's part of a broader game. So I think there might be something there, especially if the AIs get good enough. My 10 year old's, his last birthday, when he turned 10, I made a scavenger hunt for him and I used, you know, the reasoning model to come up with most of it. I didn't do it out of the box. I had to tweak it a lot, but, you know, in a few years, you could imagine it being able to create games out of these types of things.

• 27:30Jacob Effron?

  Totally. Do you have a go-to thing you use to test the new models? I remember, I think famously when you got showed GPT-4, you were using, I think, the AP BioExam to figure out whether these models were any good. Like, what do you use now as like a new model comes out from, you know, Google or OpenAI or Anthropic to figure out, you know, how good it is?

• 27:46Speaker 01

  You know, some of the things that we were worried about a year or two ago, I don't, I don't worry about anymore. My gut sense is, and you know, the data seems to back this up, that all of the Gemini, the Clods, the Groks and the, you know, GPT-4.5, that they're all in the same plus or minus. Some would say they're, you know, one's better than another. But even that's kind of unprompted. I think if you prompt, you could get almost identical behavior out of all of them.

• 28:20Speaker 01

  So now it really is, there's obviously a cost performance, but it, it, the differences between them are subtle enough that I, I have to lean more on our own evaluation framework to saying, okay, out of, out of our tough test cases, how many of this is succeeding or failing, et cetera, versus just, you know, me playing around with it.

• 28:41Jacob Effron?

  Yeah. How, how do you guys do model evaluation today? And like any, any learnings on that since getting started?

• 28:46Speaker 01

  Oh, well, there's a bunch of, um, we, we have, we have a series of tough test cases that from the beginning we saw, and it's now several hundred cases that we know these models have had trouble with. And, you know, they are, some of these classic, point three, three is what the student says, one-third's the answer. Distributive property straight is, is one of these things that sometimes the models, back in the day at least, had troubles. Well, we have the whole list of it. When we started this of the, let's call it 200 test cases, it, it was actually failing on about 70% of those test cases. Now, these are very niche. It's not like it fails on 70% of interactions, but it's 70% of the hard test cases.

• 29:29Speaker 01

  Now those numbers, I haven't looked at it in the last month or two, but it's like, it's like sub 10% of the very hardest, um, uh, test cases is my understanding. So we, we, we do some work like that. We also, you know, we have machine labeling of interactions. So, you know, can we use an AI to look at the AI interactions and saying, okay, it looks like there might have been an error here. The AI had said some point, it said, oh, my bad. Hey, you got it right when I, you know, wrong or things like that. But that, but then we also do human labeling because the AI isn't, the AI labeling isn't perfect. And so it's that human labeling work. And we did about 2,000 sample conversations. We did this about six months ago.

• 30:12Speaker 01

  That, that helped us get a much, you know, when these numbers, I'm telling you like two, three percent of errors is based on that human labeling. We're also doing some labeling on, I would, not just errors, but the productivity of the conversation, like what percentage of conversations are healthy and the student is engaged with it versus what percentage of the students just saying, I don't know, I don't know, I don't know.

• 30:35Jacob Effron?

  Have you been able to train a model to evaluate that? Or is it all, all human labeled today of like what's a?

• 30:39Speaker 01

  Right now it's all human labeled. In theory, we might be able to train a model to, to evaluate. In fact, we, we should be able to over time, but yeah, we're doing human labeling. It's also, it would be interesting because I think in a, in a traditional classroom, you'd probably see similar ratios of the number of kids who are just like checked out and saying, I don't know, I don't know, they're not even engaged. But, um, anyway, we're, we're trying to label all of the above.

• 31:00Jacob Effron?

  Do you have like an early, you know, any early thoughts on, uh, you know, obviously I feel like people are studying the impact of these models in lots of different, um, you know, uh, contexts. And I feel like in the workplace, there's been some studies that are like, actually it's like, uh, for folks that maybe were in the bottom half of the workforce that are like super impactful. And, you know, I think it's everyone's still trying to figure this out. Like any early inklings on like, you know, at least to date where, you know, for the types of students or where these models have been like, uh, or these products have been most impactful or still too early to tell.

• 31:25Speaker 01

  On the student side, I think it's, it's, um, I, I think for, for students who are curious and are already engaged, I, this is their dream come true. I mean, they can ask any question, et cetera. So I, I, you know, but there's, there's the efficacy. We're starting to run some and we are seeing some correlation with engagement, um, and potentially learning. Uh, but I, I think the models have to get much more. Part of the learning experience than just being there to answer questions before we start seeing real movement there.

• 32:00Jacob Effron?

  Yeah, makes that makes total sense. I mean, obviously, you talked about evals and I imagine in building all of this, you're one of the larger users of LLMs out there. And so I'm sure there's all sorts of things you had to build and things that were required on the infrastructure side. You know, what do you kind of wish from a tooling perspective was was available that you guys didn't have to build that ideally you could just go get from from somewhere else to make these things, you know, more easier to use or kind of like easier to operate at the scale you guys operate?

• 32:26Speaker 01

  Yeah, especially the things I've been talking about, you know, some memory architecture, some robust eval architecture, automated eval architecture would be pretty, pretty good.

• 32:43Speaker 01

  Yeah, those are, those are the, those are the, the two. And then, you know, if there's some architecture for some people who've really like nailed it on, on things like math and things. So that it's, it is evaluating well as opposed to just, we know, you know, yeah.

• 33:00Jacob Effron?

  You know, obviously, it seems like you're making a big shift to these proactive, you know, making the models proactive. And, you know, to what extent does that end up being like customizable by an individual teacher versus like, hey, there's just general best practices for we're teaching algebra. And so we know the kind of like pedagogy or prompts to do like, how have you thought about, thought about that?

• 33:19Speaker 01

  Yeah, the teacher in the loop is essential. I, like, one of the issues is you can, you can say, hey, I'm an AI here. Ask me any questions. You know, some kids will engage. You can say, hey, I'm the AI. I show up right when you come to the website. Come ask me some questions or I'm here to, or you might have this question and maybe that's a little bit better. But actually, the best is, and this is what we're building for next year. If the AI notices that a student, let's say it notices Jacob is having trouble with the distributed property. It tells Mr. Khan, Jacob's teacher, hey, Jacob's having trouble with the distributive property. Click here and you will assign a tutoring session with Khan Mego on the distributive

• 34:03Speaker 01

  property and Jacob has to do it by tomorrow night. That's what will make, it's his assignment. Jacob has to do it. So when you have teachers assigning AI tutoring interventions and then the teacher being able to hold the student accountable to that, that's when you're going to start finally seeing kids engage.

• 34:25Jacob Effron?

  It's so exciting. I mean, I thought in hearing that, obviously, we're, you know, in San Francisco, there's all this, you know, techno optimism and you kind of hear that and you're like, God, like, it almost feels, you know, it almost feels like a disservice to the world that this stuff isn't more widely available. We have these capabilities today, I mean, especially, you know, even if you look at emerging markets where like, maybe there isn't as high quality of education today, like, I'm sure there's a lot of techno optimists that are like, God, we should just be throwing these models at students like completely today and let them run. Like, what are those people missing? And I guess, you know, how do you see the next few years playing out like globally? Because when you, when you describe that use case, it seems, it seems unbelievably compelling.

• 35:03Speaker 01

  Yeah, it's, look, there are some percentage of, of, of the, of students in the world that can just run with things. There was this gentleman, Sugata Mitra, who did the school hole in a wall, school in a wall, something like that. And he put like tablets in rural India, poor kids and I, I, you know, they claimed that the kids just started going up to the tablets and started learning, et cetera, et cetera. I think there was some, I mean, there's definitely curiosity there, but I think that's limited. And I mean, there probably would be some benefit of just giving kids in villages in India access to chat GPT or something. They, some, some subset will probably start making really good use of it. But once again, if you don't know what you don't know, I mean, you talked about the blank, the blank, you don't know how to even structure your own journey.

• 35:48Speaker 01

  You wouldn't even know to prompt it to say, hey, can you work me through the Indian national standards in algebra? And, you know, you wouldn't even know to do that. So I do think. You've got to structure it more. I think once you and that's what Khan Academy, one of our values is that we've always had that structured content. And there are kids, you know, I just met a young girl, freshman at MIT from Afghanistan. She couldn't go to school. Khan Academy was her education. She used another platform we have called schoolhouse.world to prove what she knew. MIT accepted that and admitted her. So there are people out there who are, who are running with the things that already exist. But um, yeah, I think you just start throwing stuff out on, you know, you're not going to. But I, I do think in another year or two, we will have stuff that you could.

• 36:33Speaker 01

  Ideally, you do have access to a teacher who knows the algebra or the physics or whatever. But if you don't and you're in a container with, you know, Starlink access and some thirty, forty, fifty dollar tablets that are shared by five or six kids over the course of a day. I actually think you could get pretty far.

• 36:53Jacob Effron?

  Yeah. Do you think we will see that?

• 36:55Speaker 01

  Yeah, I mean, there's already people who are trying it out, but I don't see any reason why we wouldn't see it. One of the things that I hope to do in the next year is for Khan Academy. Let's call it two year. It really, we need some funding on the transcripting side to automate the transcripting. But I hope to give high school, high school credits and high school diplomas. And I think once you offer high school credits, high school diplomas, and eventually college credits and college diplomas. I mean, I don't know when we'll do college diplomas, but college credits for sure. Then all of a sudden, if you're this kid in India, it becomes very compelling for you to have access to one of these devices. And, you know, at least get a internationally recognized high school diploma goes a long way.

• 37:40Jacob Effron?

  Yeah, I guess I'd love your commentary on just like the broader AI and education market, like, you know, what do you, what have you kind of seen happening? You know, what are some other spaces or areas you pay attention to in that? Obviously, it seems like you're really going for the heart of it. But curious like what else you've seen or what else you find interesting going on?

• 37:56Speaker 01

  You know, there's, I think we're just, we're, there's a lot of noise right now. There's a lot of startups.

• 38:01Jacob Effron?

  Sounds like AI.

• 38:02Speaker 01

  What I, what we see is a lot of people who've done some a layer over chat GPT and some prompting and, you know, some of them maybe get a little bit of traction, but. It's it is. It is actually a hard environment. I mean, you're an investor here. It's a it's a it's in some ways, it's a dream that you can make apps so much faster than you could before. Apps of substance. At the same time, the switching costs and the barriers to entry are so low. I do think some of not to pick on you guys, some of the incentives of the investor community don't necessarily allow you to take a slightly longer point of view. You know, that's what I tell our team like Khan Academy doesn't.

• 38:45Speaker 01

  If the market fully solved the problems, Khan Academy doesn't need to exist. We're a nonprofit.

• 38:53Speaker 01

  We exist. I would say in this moment, our value is we actually do have permission to take a slightly longer view. We still have a, you know, we put a lot of pressure on ourselves to still be very relevant in the short run and compete with anyone, but we we we have funders who will give us, you know, a five year grant to think through assessment, to think through writing, knowing that it's going to evolve and change. And honestly, some startups don't have that luxury. They just have to, you know, the next nine months, get some product market fit, otherwise they're going to disappear.

• 39:25Speaker 01

  And I think our other value is the trust that, okay. You know, this people are more sensitive around AI. Okay. Are they really not about, you know, somehow changing undermining teachers? Are they really about the pedagogy? Or do they really have efficacy studies here? Is this something that I can feel good about? And, you know, over the years, hopefully, we've been building a lot of that trust. So that helps in this AI world.

• 39:47Jacob Effron?

  Yeah. Are there any corners of like the education market that maybe you guys aren't focused on, but you're like, oh, I wish there was startups that were trying to build, you know, this area or help out with these kind of products?

• 39:56Speaker 01

  I'm sure there are startups that are doing it. I'm not aware of them and none of them have gotten mainstream as far as I can tell. But, um, oh, I mean, there's just so much, the whole like interviewing piece is so resource intensive and broken and interviewing is assessment. And it's just, it just doesn't feel like assessment because it's so non-standardized and it's so expensive. And, um, you know, I used to do even before AI, like if I get to a final round with Google and Google tells me, hey, Sal, you're a great guy. It was either you or Jacob. And we really had to flip a coin. So we gave the job to Jacob, you know, good luck with your life. And

• 40:41Speaker 01

  And then I go to Microsoft and have to start over again. And Microsoft has to start over again and spend all that time. They shouldn't have to like the fact that I got all the way to that end. Google should give me some type of certificate saying like Sal is a really good. You know, and then all Microsoft needs to do is do a final round to make sure that I'm not, you know, that I'm I fit in and and that I'm aligned with the job and things like that. So there's always been huge, huge, huge inefficiencies. You know, just even who gets access to an interview. It's a small subset. We just had, we have a product manager job out. We get hundreds and hundreds of applicants for the job. We're probably going to be able to interview phone screen, maybe 20, 25 people.

• 41:25Speaker 01

  I'm sure there's some false negative in. I'm sure we're going to miss in that 180, 280 people that we're not interviewing. Not even phone screening. I'm sure there was some talent there, but maybe the best applicant is in that 280 that we don't even call back. So anyway, I think there's, there's interesting things there. I think there's interesting things in just corporate training, development. You know, all those, those things that we all have to do these, you know, what's appropriate at work type of cybersecurity training or sexual harassment training and all of that. I got to believe a simulation with an AI would be way more interesting. Like the following has just played out. What do you do next?

• 42:04Jacob Effron?

  Totally. Well, I mean, you, you, you mentioned obviously this, you know, the application of these in the, in the corporate world. And I guess, you know, maybe zooming out, I feel like one broader question I'm sure you think a lot about is like the skills that will actually matter, just how fast the world's changing. Like the skills that will actually matter for students that we're training for like the future workforce as, as these models keep getting better. Like, how do you think about that? And like the jobs we're, we're training people for and to what extent, if at all, like what we teach has to, has to adjust or change to adapt to that?

• 42:31Speaker 01

  Yeah, I'm, I'm, I believe that a smart, cogent, strong, critical thinker is always like, that's always really great baseline skills to have. So I, I think kids should continue to learn to write and read and do their math well. And, you know, have good general knowledge of social studies, history, et cetera, civics. But I think the thing that's really going to be, I think, I think.

• 43:01Speaker 01

  Economists talk about entrepreneurship as a factor of production, right? This ability to put resources that already exist, but put them in new permutations to create value that didn't exist in the world before. And most people always assume entrepreneurship is kind of like starting Khan Academy or what you'd invest in. And it is that. But I actually think that's going to be more and more of table stakes in almost any career, especially over this next 20 years when there's just like super rapid change. The people who have these, they're otherwise have solid skills, but they're just always saying, hey, wow, I just heard about this one thing. Let me try to use that and put it together with this thing. And wow, if I do this and then I take that output and I put it here and I can make this,

• 43:45Speaker 01

  I might be able to make a pretty good. And it might not be the final output, but it might get me 80% there. And then I have to use my skills to tweak it to be, those people I think are going to be, that's, those are the skills. Those are the skills. And that's, those are who I'm looking to highlight in our, in our organization. And, uh, you know, organizations that are not doing a lot of that are also going to suffer. If they don't have a lot of that entrepreneurship inside, they're not going to be able to innovate. Their cost structures are going to be way higher than everyone else.

• 44:15Jacob Effron?

  I love that. Um, well, we always like to end our interviews with a quick fire round where we get your, your take on a, on a set of questions. Um, you know, maybe to start, uh, what's one thing you've changed your mind on with regards to AI in the last year?

• 44:26Speaker 01

  It's obvious in hindsight, but AI on its own, and I don't think I would have said AI on its own is going to solve all the problems. I wouldn't have said that even two years ago, but I've definitely shifted even more towards a lot of what we talked about, which is like, it's more about how does AI really empower the teacher to hold the students accountable and engage students in other productive things.

• 44:51Jacob Effron?

  What's your favorite way that you use AI today within Khan Academy? And then anything on the wish list where you hope that you'll be using it, you know, a year or two from now?

• 44:58Speaker 01

  When I'm preparing for videos, uh, to make a video, uh, you know, one of my, I guess you could call it, if you want to say my superpowers is I ask all of the dumb questions in my head that sometimes people are afraid to ask. And I think sometimes teachers glaze over them because they, they were afraid to ask them too. But I'm like, wait, how does this make sense? Even if I'm doing like a fourth grade concept, I'm like, wait, wait, that doesn't make sense. Okay. And sometimes I can figure it out by myself, but a lot of times I, you know, in the past, I would, I would do web searches, I'd call people up. I have found that for someone who's asking the right questions, AI can dramatically accelerate that process. I'm always able to make fun images now for videos that can visualize what I'm trying to show.

• 45:39Speaker 01

  So that's been valuable. Every now and then I want to get, I tend to, most of my speeches I give off the cuff, but I've had to give a few formal speeches, things like commencement addresses and things. And that's where just doing a verbal dump and then having the AI transcribe it and then at least doing the first pass at the speech and then tweaking it saves hours.

• 46:01Speaker 01

  I want to try doing some, you know, I've been talking to some of these people who have these platforms. They say that you can now like prompt the AI, it'll make the whole app and they'll host it for you and everything. I want to try that out because I've

• 46:13Jacob Effron?

  Yeah, vibe coding on the docket.

• 46:15Speaker 01

  Vibe coding, there's thunkable, which is, I know the founder pretty well. And he was telling me the other day that, no, you can make the whole app. And I even described an app that I wanted to make and he's like, yeah, that would probably work. And I'm like, wow, okay. So I need to try that out. I keep telling my son, my 16 year old, who's really into programming and making games, like, well, you should be doing this vibe, Kobe. And it's so funny because he's like an old man. He's like, no, that's cheating, you know? And I'm like, no, but that's the future of coding. You got to do a little bit of both. He's like, no, I need to learn it properly myself.

• 46:44Jacob Effron?

  And I guess, you know, any, what's been like the biggest surprise in building these features? I guess maybe something that you thought would work really well that didn't, or something that you weren't expecting much from, and it's actually been hugely successful in the adoption of Conmigo.

• 46:56Speaker 01

  I mean, it's not surprising in that almost everything turns out this way when you try to do something at scale and enterprise. But yeah, it's a lot more work. Like when you, when I first saw these models and I'm like, wow, these are magical. And look, I still do worry that, you know, will, will these models leapfrog the apps that are using them in some way, or will, you know, the general apps like the chat GPTs leapfrog the specialized apps?

• 47:25Speaker 01

  It's still a concern, but yeah, I mean, the amount of work to like make it really work well for a certain use case is is a lot. Things that have been better than expected. I, I, you know, I was afraid two years ago whether our team was ready for this type of a pivot. And I was, I was ready whether, I was worried whether the education community was ready for this type of a pivot. I actually thought we might get a lot of flack for leaning so hard into AI. But um, it's, you know, it is pretty interesting now. Our, our team does view itself as an AI first organization. And it's easy for me to say, but when I really think about ourselves two, three years ago, that was not, that was not an obvious thing. And there was some pushback and there were people who didn't like that direction.

• 48:09Speaker 01

  And I would also say the education community has moved faster than expected.

• 48:13Jacob Effron?

  What were the weeks like? I mean, obviously you got the GPT four demo, you did your AP bio tests and then you came back and you're trying to reorient. What, what was that like first month like post, uh, post that demo?

• 48:22Speaker 01

  Well, almost every day we were talking to the OpenAI lawyers. We're like, we have five more people we want to get under NDA or 10 more people. And we had a little, uh, we had an onsite, I remember that first, uh, and, and we had a little office. And every time we got people, we said, come with me. And I wanted to show people because it was such a like, they would, they, they thought they were getting punked. This was before chat GPT or anything. I'm like, I have something to show you. And they would walk in. And just, I, I just enjoyed doing the demo. They're like, what's going on? What's, what is this? What is this? What are you showing me? And I was like, this is the future.

• 48:56Speaker 01

  But you know, a lot of those people, um, I think initially about half the company was like, this is everything. We have to stop what we're doing. And then half of them like, hold on, makes errors, hallucinates, bias. It's going to freak people out, cheating. And we said, look, we've, both sides are right, but we can't, we, we've got to take all those risks and turn them into futures.

• 49:20Jacob Effron?

  Well, I love that. Well, look, this has been a fascinating conversation. I want to make sure I leave the last word to you. Anywhere you want to point our listeners, uh, where they can go to learn more about Khan Academy, the AI work you're doing, uh, anything else, the, the floor is yours.

• 49:31Speaker 01

  Yeah, you know, people who want to try it out, you go to Khan Academy, Khanmigo, could try it, Writing Coach, if you're, you know, homeschooler teacher. I also, I, people should know about Schoolhouse.world, which is a sister nonprofit that I helped start, which is around free tutoring. We do it through volunteers, so if people want to volunteer and give tutoring or get tutoring. We're about to launch something called the Dialogues Initiative, where you're going to have conversations about tough topics, but, and then rate each other on how well the other side listened. So, um, yeah, take, take a look and remember we're a nonprofit, so donate.

• 50:04Jacob Effron?

  Yeah, awesome. Well, thanks so much, Tru. So this is a fascinating conversation.

• 50:07Speaker 01

  Great, thank you.

• 50:09Jacob Effron?

  Hey guys, this is Jacob. Just one more thing before you take off. If you enjoyed that conversation, please consider leaving a five star rating on the show. Doing so helps the podcast reach more listeners and helps us bring on the best guests. This has been an episode of Unsupervised Learning, an AI podcast by Redpoint Ventures, where we probe the sharpest minds in AI about what's real today, what's going to be real in the future, and what it means for businesses in the world. With the fast-moving pace of AI, we aim to help you deconstruct and understand the most important breakthroughs and see a clearer picture of reality. Thank you for listening and see you next episode.