#242 Dylan Arena: The AI Education Revolution: How AI is Changing the Way We Learn

This episode is brought to you by Extreme Networks, the company radically improving customer experiences with AI-powered automation for networking. Extreme is driving the convergence of AI, networking, and security to transform the way businesses connect and protect their networks, delivering faster performance, stronger security, and a seamless user experience. Visit extremenetworks.com to learn more. ———————————————————————————————————————— The Role of AI in Education | Dylan Arena on Learning, AI Tutoring & The Future of Teaching How can AI enhance education without replacing the human touch? In this episode, Dylan Arena, Chief Data Science and AI Officer at McGraw Hill, shares his insights on the intersection of AI and learning. Dylan's background in learning sciences and technology design has shaped his approach to AI-powered tools that help students and teachers—not replace them. He discusses how AI can augment human relationships in education, improve personalized learning, and assist teachers with real-time insights while avoiding the pitfalls of over-reliance on automation. With AI playing an increasingly central role in education, are we at risk of losing the essential human connections that define great learning experiences? What You'll Learn in This Episode: - Why AI should be used to enhance not replace teachers - The risks and rewards of AI-powered tutoring - How AI-driven assessments can improve personalized learning - Why AI chatbots in education need careful ethical considerations - The future of gamification and AI-driven engagement in classrooms - How McGraw Hill is integrating AI into its learning platforms If you care about the future of education, AI, and ethical tech development, this episode is a must-watch. ———————————————————————————————————————— This episode is sponsored by Oracle. Oracle Cloud Infrastructure (OCI) is a blazing-fast and secure platform for your infrastructure, database, application development, plus all your AI and machine learning workloads. OCI costs 50% less for compute and 80% less for networking—so you're saving a pile of money. Thousands of businesses have already upgraded to OCI, including MGM Resorts, Specialized Bikes, and Fireworks AI. Cut your current cloud bill in HALF if you move to OCI now: https://oracle.com/eyeonai ———————————————————————————————————————— Chapters: (00:00) The Role of AI in Augmenting Human Learning (02:10) Dylan's Background in Learning Sciences & AI (08:23) The Risks of AI-Powered Education Tools (11:08) AI Tutoring: Can It Replace Human Teachers? (16:28) AI's Role in Personalized Learning & Adaptive Assessments (22:47) How AI Can Assist, Not Replace, Teachers (29:36) The Future of AI-Driven Gamification in Education (36:41) Ethical Concerns Around AI Chatbots & Student Relationships (45:02) The Impact of AI on Student Learning & Memory Retention (50:19) How McGraw Hill is Innovating with AI in Education (54:44) Final Thoughts: AI's Role in Shaping the Future of Learning

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  AI should be used to augment human relationships. When people used to ask me in 2019 how I thought AI could be most helpful in education, I immediately talked about speech recognition and computer vision. Because I fried my arms as a coder, I was really aware of how much easier it can be to just say what you want to say instead of having to type it, because to me it was physically painful to type it, right? But for young kids, it's also much easier for them to express themselves verbally than to write. Now, there's a real richness in how young learners can express themselves with spoken language that they can't yet translate into written language. So there's a lot of good that can come from considerate applications of speech recognition to the classroom.

• 0:37Craig S. Smith?

  Even if you think it's a bit overhyped, AI is suddenly everywhere, from self-driving cars to molecular medicine to business efficiency. If it's not in your industry yet, it's coming fast. But AI needs a lot of speed and computing power. So how do you compete without costs spiraling out of control? Time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure, or OCI. OCI is a blazing fast and secure platform for your infrastructure, database, application development, plus all your AI machine learning workloads.

• 1:17Craig S. Smith?

  OCI costs 50% less for compute and 80% less for networking. So you're saving a pile of money. Thousands of businesses have already upgraded to OCI, including MGM Resorts, Specialized Bikes, and Fireworks AI. Right now, Oracle is offering to cut your current cloud bill in half if you move to OCI. This is for new US customers with minimum financial commitment. See if your company qualifies for the special offer at oracle.com/ionai. That's I-on-A-I, all run together, E-Y-O-N-A-I.

• 2:02Craig S. Smith?

  So go to oracle.com/ionai to see if your company qualifies for the special offer.

• 2:10Speaker 00

  I am excited about the idea of a freewheeling conversation instead of a Q and A. I'm excited to see where it takes us. By way of background, I studied symbolic systems and philosophy in undergrad at Stanford. Symbolic systems is introduced in a it's computer science, psychology, linguistics, and philosophy, and then mastered in philosophy. I was a software developer for a number of years at Apple, sorry, at Oracle. But I fried my hands, too much coding, so I have to use a weird keyboard. I use a lot of voice to text. And in fact, that was my first real engagement with machine, with the technologies that are the forerunners of today's gen AI. Then, because I fried my arms and I realized software development was not a career path

• 2:55Speaker 00

  for me, I went back to grad school at Stanford and I got a master's in statistics and a PhD in learning sciences and technology design. I was focusing on game-based learning and next-generation assessment, and I was doing a lot of experimental research. So I wanted to get the master's in statistics so that I would understand the tools that I was using to test my hypotheses about learning. The goal of learning sciences broadly is to understand learning environments, learning contexts, and one sort of subfield within learning sciences is about creating productive learning experiences. That work carried me to Kidaptive, to co-founding Kidaptive, which was a startup that originally

• 3:37Speaker 00

  was focused on creating really engaging learning experiences for preschoolers on the iPad. The iPad had come out in 2010, and a bunch of parents were like, whoa, this interface is so intuitive that my two-year-old is like doing great stuff with it, you know? And so a whole bunch of junk food showed up on the iPad, and the kids were doing all kinds of things, and you know, their attention was all over the place. And we thought, what if we could give them a really cool enriching learning experience that was as engaging as, you know, a Pixar film, but also helped build skills? Not the expedient, like, A, B, C, one, two, three skills that some people think is like, oh, I gotta make sure my kid is doing this, but the broader, that the rich holistic

• 4:20Speaker 00

  set of skills that you want in a well-rounded, early learning ecosystem. So we built what I thought were pretty cool experiences. It was called Leo's Pad. Parents loved it, kids loved it, app stores loved it. But everybody wanted it to be free, because this was like 2011, 2012, and, you know, apps are free. Especially if you got a two-year-old and, you know, they, they pick up one piece of junk food and they play with it for a week and then they, they delete it and they go to some other thing. You don't want to pay for that. You just want, you just want to give them more of the free stuff, more of the free stuff, even though there's no nutritional value there. So, we didn't, we didn't want to do any of the things that startups did to monetize back then, like advertising to toddlers, or harvesting data in some way, or like, making

• 5:04Speaker 00

  their goldfish die and then they would pay 99 cents to like, you know, buy another, revive their goldfish. Remember that period where kids were spending like hundreds of dollars on their parents' credit cards through, perhaps through games? We didn't want to do any of that. So we pivoted and we started doing support for other big operations that had created learning experiences, that generated learning relevant data, but didn't really know how to make meaning from those data, because our technology platform allowed us to pull learning relevant data from a variety of contexts and make them all commensurable and contribute to a single holistic longitudinal learner profile.

• 5:40Speaker 00

  So we did that for another eight years, so 10 years total, two years trying to do that sort of App Store thing, and then eight years doing what was essentially B2B SaaS, business-to-business, software-as-a-service. And also, in a sense, really high-level professional services. We had a team of learning scientists and psychometricians and data scientists who would work with people who had created learning experiences to understand the data that were coming out of those learning experiences, so that they could make the learning experiences personalized, or generate rich analytics about the learning experiences, either for the product developers, or for the parents, or for the teachers, or for the learners themselves, or generate great recommendations about what to do next.

• 6:22Speaker 00

  At some point, I recognized that, number one, doing that kind of deep, careful work with one partner and then another partner and another partner leads to years-long relationships that are great and deeply collaborative, but it's not the kind of hockey-stick growth that a venture-backed Silicon Valley startup is supposed to achieve. So, I kept getting pressured to like, create a plug-and-play thing, that you could just pop into any learning environment and of course it'll give you the right data. And I said, "But that'll never work, because if you don't understand the context in which the data are being generated, you don't know what inferences you could make from the data." So instead, what we needed was to find a home with an organization that was so big that it had its own ecosystem, so that the technology and the patterns that we had established

• 7:05Speaker 00

  for taking data from multiple different programs and making them all commensurable could bear fruit. And that led us, in short order, to McGraw-Hill. Sean Ryan, in the school business unit, had already been thinking about the idea of a universal student record, a longitudinal profile of a learner that takes in data from a bunch of learning experiences across time and across contexts, and distills the insights from all of those experiences to inform the educator and the learner about where they are and where they can go next. So, that was a match made in heaven. We came aboard at McGraw-Hill. I am now, after a couple hops along the way, in McGraw-Hill, the Chief Data Science and

• 7:50Speaker 00

  AI Officer. But I'm still by disposition a learning scientist, so I'm interested in data science and AI, only insofar as they support learning.

• 7:58Craig S. Smith?

  And, how would you, just generally, backing up, how do you see the promise of AI in education? I know when we spoke, you were very negative on the idea of a personal tutor and, you know, students having a relationship with a screen and...

• 8:22Speaker 00

  I'm negative on some aspects of it, absolutely. I should say first, I see tremendous potential for positive applications of generative AI, and AI more broadly, in education.

• 8:35Speaker 00

  The specific reservations I have are twofold. One, there's a set of people who have tried to replace humans with machines in teaching for over a hundred years. Sidney Pressley's teaching machine was debuted in 1924, I want to say. And all along that time, you could see repeated places where people thought, oh, this is the technological innovation that's finally going to let us create something that's just as good as a teacher. So we can just have the kids engage with that instead of with a human. I think all of those efforts

• 9:16Speaker 00

  run the real risk of extracting from the educational process, from the learning process, the most crucial element of it, which is the sociality of it, the human connectedness. When I was in grad school, I did a study in which participants are navigating a maze, and at certain points in the maze, they get tips like, hey, turn left, or don't go that way. And we experimentally manipulated whether they believed that the tips were coming from a human, the experimenter, or from a machine. In reality, they were always coming from the machine, and they were always programmed to do the same for people in every condition. But the people who believed that the tips were coming from a person learned the way out of the maze faster.

• 9:57Speaker 00

  And that tapped into a general effect that's been found in lots of contexts, which is, if we think that something is social, in other words, that we're engaging with other people, we're more dialed in, we attend more, we care more, we, you know, put more effort in. Generative AI can leverage this intentional stance, which the philosophers call the tendency of the brain to interpret things in the world as if they're motivated by social actors. Because generative AI can create situations in which it's quacking like a duck, and it's walking like a duck, and it's got feathers that look duckish.

• 10:36Speaker 00

  What I worry about there is the counterfeit nature of those duck-like things.

• 10:44Speaker 00

  When I believe that I'm engaging with a person, I invest in a certain way socially, psychologically, emotionally.

• 10:52Speaker 00

  That the system can't fully back up the way a person could.

• 10:57Speaker 00

  It's totally fine to engage with a system that behaves in person-like ways as long as I understand it's a tool and I can use it as such and treat it as such, everything is great. We have seen already a number of high-profile examples of individuals who have formed parasocial relationships with personified AI and had tragic outcomes from them.

• 11:24Speaker 00

  In 2018, I want to say, there was some kid who developed an AI girlfriend relationship in which somehow or another she convinced him that it was a great idea for him to take a crossbow and try to break into the palace and assassinate Queen Elizabeth. He did not succeed. More recently and more tragically, a 14-year-old boy committed suicide

• 11:47Speaker 00

  and it was later discovered that his AI pseudo-girlfriend had been encouraging and reassuring about it. Those are just two examples that come to mind of this general problem of forming a relationship with something that you lean on as if it's a person, but it doesn't really have the support there to act as a person would.

• 12:07Speaker 00

  That said, again, that doesn't mean I'm against the idea of leveraging AI and tutoring. I think it's a fantastic idea if done right and done safely. I just want to make sure it is done right and safely. Does that make sense?

• 12:19Craig S. Smith?

  Yeah, yeah. I mean, that's interesting because I'm I've been wanting to write something about that, about relationships with personified AI or whatever you AI avatars.

• 12:36Craig S. Smith?

  Uh, and I don't know if I've had this conversation with you, but you know, humans invest

• 12:46Craig S. Smith?

  things with human like qualities. I mean, your Roomba that's riding around in your, you know, if it gets stuck and starts beeping, there is a certain sense of, uh, you know, you feel sorry for it if you've left it.

• 13:05Craig S. Smith?

  Yeah, exactly. Or the these, the, the dogs that Sony had out for a long time and people formed relationships, particularly elder elderly people. And when they stopped supporting them, there was grief. I mean, people held funerals. It was a big deal. And, and, uh, you know, I've been looking at, uh, replica, uh, which, which I don't know which, uh, companies were involved in the incidents you spoke about, but, uh, and, and, and there's a filmmaker recently,

• 13:47Craig S. Smith?

  uh, who, who has a film called Eternal You. And he explored this idea of, uh, after death avatars. So you can speak to your loved ones after death and, uh, and there's a very dramatic example of a Korean mother who lost a child and uh, she's wearing virtual reality glasses and haptic gloves. And, um, and it's, uh, heart wrenching. And, and the point of that is of, of the, one of the points of the,

• 14:33Craig S. Smith?

  Documentary is well, what happens? I mean, this can't be free. So are you going to charge a woman for access to her dead daughter? I mean, that's there's a clear ethical cloud hanging over that. And, and, and as you said, even if it's a tutor, if you develop a relationship in emotional relationship with the tutor.

• 15:06Craig S. Smith?

  Should you really have to pay to maintain that relationship? Anyway, it, it does raise all those questions. That said, there is a role, as I was saying earlier, that I use. You know, chat GPT's latest, OpenAI's latest model, I use perplexity and I, I, I've learned a tremendous amount just asking questions about things I'm interested in and, and following rabbit holes, you know, there's a term you don't understand, you have that explained and it goes on and on. So how, how do you see?

• 15:49Craig S. Smith?

  And so I, I agree, I mean, AI isn't there to replace teachers, but it can alleviate a lot of the drudgery of teaching. And it can also give students a way to study outside the classroom.

• 16:14Craig S. Smith?

  And, and, you know, to augment their, their learning. So tell me what, what McGraw, well, first of all, tell me how you feel AI should be used, and then let's talk about what McGraw Hill is doing.

• 16:28Speaker 00

  Sure. And I want to say, as you mentioned, there are these uses. And I have a colleague whom I respect immensely, who has done research on the therapeutic uses of personified AI to help people who are in distress in various ways. There's definitely potential for good there. My only point is that there's also potential for really bad outcomes. So I think if you do that follow up work, it'll be it'll be totally worthwhile because there's a lot of nuance and complexity in that issue. How I think AI should be used. I think AI and and McGraw Hill and I are aligned on this. Should be used to augment human relationships. When people used to ask me in 2019, how I thought AI could be most helpful in education, I immediately talked about.

• 17:12Speaker 00

  Speech recognition and computer vision. Because I fried my arms as a coder, I was really aware of how much easier it can be to just say what you want to say instead of having to type it because to me it was physically painful to type it, right? But for young kids, it's also much easier for them to express themselves verbally than to write now. That doesn't mean they shouldn't write. Writing is an important skill that they need to develop. But there's a real richness in how young learners can express themselves with spoken language that they can't yet translate into written language. So there's a lot of good that can come from. Consider considerate applications of speech recognition to the classroom. Similarly with computer vision. You know, there's a lot of kids who do a lot of work in print in K five, even though

• 17:57Speaker 00

  there's a lot of focus on digital stuff these days, and it should be that way. That's good. But if I'm a teacher and I got a bunch of worksheets, wouldn't it be great if I could just put them in a stack and have a thing scan them? And then the the machine learning algorithm will identify six categories of major misconceptions that seem to be there. And one of those misconceptions is held by a plurality of the class. So hey, maybe tomorrow focus on that, you know, just just make sure you shore up that gap in kids' understanding. Those are, in my mind, great examples of the use of AI to ease the logistic administrative and computational burden on the teacher and help the teacher direct their attention. Because that's the scarce resource in the classroom, right? The teacher's attention.

• 18:41Speaker 00

  So if you can help the teacher direct their attention where it's going to be most powerful, fantastic. You can put the teacher in a position to have that really rich conversation with their learners, then you're augmenting those human relationships that are at the core of learning. That's how I think AI should be used in education, and that's the kind of work we're doing at McGraw Hill to use all kinds of AI, but Gen AI in particular, in education. You mentioned we have two recently released Gen AI features within our existing products. One of them in the higher ed space is called AI Reader. And the idea there is, kids have always had a bunch of stuff to read in college, and kids have never really done all the reading. Apparently, I haven't been in college recently, but apparently it's gotten worse recently.

• 19:24Speaker 00

  It's gotten more like, oh, my gosh, how could I possibly read, you know, this 300-page book and this 50-page chapter in econ and this text, whatever, in bio? So, problem to be solved. College students have a tremendous reading load that they cannot consistently meet. And you have that problem that we all know of, you're looking at a wall of text and you read a paragraph and then you realize you didn't get it. You're like rereading it like two or three times and it's just not quite clicking for you. So to address that problem, we talked to university professors and instructors and we worked with them to come up with this idea, like, what if you could just highlight that text and say, could you just explain this a different way? That idea was the core of what has become AI Reader.

• 20:09Speaker 00

  And the goal there, again, is not technology for technology's sake. It's, there's a learning experience we want you to have. We want you to wrestle with text and build meaning in your own mind from that text. Just like you said, you use perplexity, to be like, wait, explain this to me differently. You use these tools to like, give it to you in a different way to make it a little more accessible and line up with your existing knowledge structures so you can build and add to your knowledge structures. That's the basis of the cognitive theory of constructivism, which is a theory about learning. We learn by constructing knowledge in our own heads. It's not, there's knowledge here and I pour it here and now it's here. It's like, no, no, no, you're showing them a model, but they have to build it for themselves. We have to construct it in our own heads. AI Reader is a tool to help students construct meaning in their own heads by giving them

• 20:54Speaker 00

  more avenues in to a body of text. Quiz me on this. Explain it to me like I'm a kid. Simplify it, you know, all of these things. In 612, we had a different learning challenge that we were trying to solve, and AI was a way to help solve it. Kids in middle and high school are often engaging in digital experiences in which they're required to write short, free-form responses. Now, just like I talked earlier about there being a stack of worksheets and a teacher could plausibly read through them all, but not realistically.

• 21:27Speaker 00

  If I've got thirty kids in each of five sections doing this, you know, five times a class period, that's a lot of, you know, two hundred word little blurbs for me to read through and try to provide meaningful feedback on. And sometimes that's not even the main goal of the assignment. It's not primarily a writing assignment. It's just an assignment that has a writing component in it. If I'm walking the classroom like teachers often do, and I look over a kid's shoulder to see what they're working on, and I read that they've written a little blurb, I might offer them feedback right then. But it's opportunistic and sporadic.

• 22:02Speaker 00

  So we wanted to give teachers a way to help provide that kind of feedback to all the students if they want it. That's the feature that we call Writing Assistant, and we're currently in beta in two of our programs with Writing Assistant. And what it does is simply, if I'm a student and I'm writing a short form response, I can click a button that says, give me feedback on my draft. And I engage in a couple of conversational turns with a tool that will either, if I don't know how to get started, it'll give me like a little, you know, push out of the gate, like, hey, just try, you know, the prompt is asking you to do this. You've read this story. Now it wants you to respond to this particular question. What do you think? And then the kid will write some kind of draft. And then the program will just meet you where you are. There's a rubric

• 22:47Speaker 00

  behind the scenes, right, to evaluate the quality of the draft. And wherever it is, the program's goal is just to try to get you to the next one up. Again, in a couple of conversations, you're not going to spend half an hour like crafting and rewrite. It's just, it's a quick touch to do that little bit of support that might help you improve your writing in the moment. In each of those two cases, AI Reader and Writing Assistant, there's a clear learning experience that we want to drive, and there's a way that we can leverage AI to support it.

• 23:16Craig S. Smith?

  Yeah. Something, the AI Reader, it occurred to me.

• 23:26Craig S. Smith?

  And I've wondered about this. I mean, you know, I do interact a lot with with these chatbots for my own education.

• 23:38Craig S. Smith?

  And I, for example, I watch a lot of YouTube videos.

• 23:45Craig S. Smith?

  I watch, you know, series on on TV. And and what

• 23:53Craig S. Smith?

  amazes me is that

• 23:57Craig S. Smith?

  I'll come to something two years after I know I watched it and I don't remember any of it.

• 24:05Craig S. Smith?

  And, you know, there's, there's, and I don't know the science, but there is something about passive

• 24:15Craig S. Smith?

  engagement and active engagement that that affects the learning process. And with reading, I'm guilty of this. You know, I'll get a, somebody will send me a report and I just load it into chat GPT and ask for a summary. And it's very useful useful for short term tasks because, okay, I understand what the report's about. I see the main features. I can respond to it. But, you know, a week later, I won't remember the report or the summary or anything. So how do you address that? And and are students?

• 25:00Craig S. Smith?

  Do you do you see that as a growing problem? You said that they're reading less and less, but maybe that's one of the reasons is they're they're just reading summaries that are AI generated. And it just doesn't.

• 25:18Craig S. Smith?

  Penetrates a long-term memory. Our sponsor for this episode is Extreme Networks, the company radically improving customer experiences with AI-powered automation for networking. Extreme is driving the convergence of AI, networking, and security to transform the way businesses connect and protect their networks to deliver faster performance, stronger security, and a seamless user experience. Visit extremenetworks.com to learn more. That's extremenetworksallruntogether.com to learn more.

• 26:03Speaker 00

  There are certainly learning risks of anything that makes the learning task easier. Learning, in a sense, must be effortful. You know, like I talked earlier about the constructivism, theory of learning. Building a new knowledge structure is tough. Taking in new knowledge and incorporating it into existing understandings of the world is work. And our brains don't like to do work. So we try to offload that work. And we have forever because it's calorically efficient to avoid work when we can. And brains are really calorically expensive. The risks of not taking things in fully, if they're just, you know, Cliff's Notes or quick summaries,

• 26:48Speaker 00

  are real. We got to weigh those risks against the risks of not taking it because I didn't do the reading at all. And we also have to

• 27:03Speaker 00

  engage our learners in the metacognitive task of owning their own learning and their own learning goals. There's a trope, you know, is this going to be on the test? Right? Why do kids ask that? They ask that because they need to, in some sense, manage their intake. You know, like if you're telling me a bunch of interesting stuff because you're excited about history, but it's not actually on the test, I'm going to tune out. I'm holding myself accountable to learn the stuff that I know I'm going to be measured on. And in some sense, we roll our eyes because we're like, no, but we want kids to be hungry about learning everything. But also, you know, they're rational actors, right? So they're going to, they're going to make sure they're focused on what need to be focused on. If your goal in taking a report that somebody sent and feeding it to ChatGPT to get a quick summary is simply to have enough of an understanding of the report to be able to respond coherently,

• 27:48Speaker 00

  then you've achieved that goal.

• 27:51Speaker 00

  And to your point, you didn't encode it deeply or elaborately enough for that to stay years later, or even months later, you know? Similarly, the TV show that I watched a couple of seasons ago, I remember being good, but I'm not exactly sure what happened.

• 28:05Speaker 00

  If a learner is engaged in a psych course, or an econ course, or a finance course, and they want, they really want to learn this concept, they know they're gonna be tested on it, and they, they, it's for their major, so they know they're gonna, you know, it's gonna be instrumentally useful for them going forward. We want to help them encode that information. The more richly we can help them encode it, the more likely they will be to remember it later.

• 28:33Speaker 00

  So, there is absolutely a balance to be struck, you're right. And one of my biggest fears as a learning scientist, supporting the implementation of AI in various areas is, how do we avoid this leading to a diminishment of human capacity? The two examples of that that haunt me are the fact that when I was a kid, I had memorized all my friends' phone numbers, and my family's phone numbers, everybody's phone numbers, and I don't have anybody's phone numbers to memorize anymore. I know my own and my wife's. I don't even know my son's phone number. Because there's now a tool, and I have done the sort of cognitive offloading of that task to that tool. And brains are great at not remembering stuff we don't need to. So, I don't need to remember that anymore, I don't remember it, it's in the tool.

• 29:15Speaker 00

  Similarly, we as a society have gotten less good at wayfinding, because GPS is now ubiquitous, so we're not as good at dealing with maps, or at remembering routes. We just plug in our destination, and we turn when it tells us to turn. Those are both areas in which we have a capability that has diminished. And we could argue about whether, on balance, we're better or worse for it. Where we enter the domain of education, that we gotta be really mindful of whether we're giving students a tool that will increase their capabilities, or decrease them in ways that we think are damaging. The introduction of the calculator was an opportunity for society to grapple with this. Calculators became ubiquitous and efficient enough and small enough in the 70s to begin introducing to classrooms. And there was a big fear over whether this was gonna be the end

• 29:59Speaker 00

  of mathematics education as we knew it. We ultimately landed in a good place. We do not let students use calculators when we're trying to teach them arithmetic, because they have to understand the concepts and they have to develop a fluency with those concepts. We do allow students to use calculators in advanced trigonometry, calculus, those kinds of courses. And in that case, we're actually enriching their learning, because there are things that you can explore in terms of the behavior of functions, for instance, with a graphing calculator that you could not feasibly have done in the 1950s where you're plotting all the points by hand. So we're actually allowing kids to engage more deeply with with rich conceptual material because of the affordances of this tool.

• 30:41Speaker 00

  So are calculators good or bad for math is not a reasonable question to ask. It's how calculators be good and bad for math. We're now in a similar situation with various applications of Gen AI. The most obvious one is these tools are great at writing.

• 30:57Speaker 00

  But does that mean we should let kids just use them instead of learning how to write themselves? And what about the the deep relation, the intrinsic relationship between writing and thinking? Uh, I don't know who it was, maybe it was Wittgenstein, someone said, how can I know what I think till I see what I say? And that's, that's a great encapsulation of this relationship between our, our language and our thought. If we simply let kids go boop, boop, boop, boop, and then there's an essay that comes out, they haven't done that work, they haven't grappled with those ideas. And they definitely have not learned as much as they would have if they had persisted through all those drafts and revisions and feedback and gotten it to a point, point where they could express their thoughts in that way themselves. That would be bad.

• 31:40Speaker 00

  On the other hand, there are kids with language challenges or kids who really have rich ideas but aren't great at expressing them yet. How can we use the technology to support those use cases without falling onto the side where we're actually diminishing an essential capacity for thinking? That's the challenge that we gotta, we gotta meet every time we consider a new application of AI, Gen AI or other for education.

• 32:05Craig S. Smith?

  Yeah, that's interesting. And, and you said something, um, you know, the, the joke about it's just going to be on the test and you forget as an adult how.

• 32:18Craig S. Smith?

  Yeah, how little

• 32:21Craig S. Smith?

  learning meant to you when you were a student, it was really just grades.

• 32:28Craig S. Smith?

  And there's, there's been a research into, and I've forgotten the the term in the literature, but sort of continuous assessment with AI. So you have knowledge tracing algorithms.

• 32:45Speaker 00

  That was a little kid active did.

• 32:47Craig S. Smith?

  Oh, okay. Yeah. And you're interacting with digitally, maybe not with

• 32:56Craig S. Smith?

  a tutor, but you're, you're testing and all that stuff is being captured.

• 33:03Craig S. Smith?

  And so rather, so theoretically, you should be able, the, the system should know at any point in time where the student is vis a vis the, the course material. And, and in that case, you don't really need a test, which is a snapshot. Have you done work on that? Or, or is

• 33:28Speaker 00

  work in this space? Yeah. Continuous embedded assessment was one of the founding principles of caductive. As kids are engaging with lots of different learning experiences over time in different contexts, they're demonstrating what they know or understand or are interested in all over the place. And instead of a model that demands maximum performance at one point in time in one single context, you have a model that accumulates typical performance across a variety of contexts across a lot of time. That latter model, we use the analogy of a mosaic, which is actually the team of my team name in, in McGraw Hill. And it was the name of a product learner mosaic that we put out when we were caductive. And you got all these little tiles and each one of them on its own doesn't really, it's not very interesting.

• 34:12Speaker 00

  But if you put them together in the right way, you get a really rich picture. Same is true for continuous embedded assessment. If done well, and if the data you're gathering are gathered in a context that allows you to make good inferences from those data collections, then you can get a richer picture of the learner than you would if you had just a couple drop in from the sky, high stakes assessments throughout the year.

• 34:38Speaker 00

  But all of the devils are in those details.

• 34:43Speaker 00

  A really lame approach would be to just Hoover up all the data from everything the kid is doing and treat it all as if it's undifferentiated. But the way a learner is going to perform on an exit ticket or a practice assignment is very different from the way a learner is going to perform on an end-of-week quiz, or a unit assessment, or a final exam. The stakes are different, the kid knows it. The context in which the kid is doing that thinking is therefore different, and the level of performance you would expect from the kid is therefore different, right? That kind of consideration should be there for all of these continuous embedded assessment contexts. And one of the challenges that has made really rich continuous embedded assessment impractical

• 35:25Speaker 00

  for most is that to operationalize what you're measuring in all of those different contexts, and then to bring them together in a way that's reasonable, is very hard. We've applied for a patent for an approach that we have to doing that, but it requires deep psychometric work, careful learning sciences analysis of all of the different contexts and what claims could be defended about what you're seeing in all those contexts. But the idea in principle is, I think, sound and worth exploring. So I've spent the last 15 years trying to explore it. That said, there's still a tremendous amount of work to be done theoretically, technologically, socially, politically, before we get to a point where we can reasonably say,

• 36:08Speaker 00

  "Hey, we no longer need the end of your test because we already have a rich enough picture of every learner." There are upsides to the continuous embedded assessment approach, one of which is, there's less likely to be cheating. It's much easier for me to hire somebody else to go take the SAT for me once than to hire somebody else to answer every question I answer about every math skill that I'm exhibiting any kind of proficiency on throughout the year. And because, again, the stakes are lower for each of those individual things, it's not really worth it for me to do it. So you're much more likely to get a true read on the learner. However, it's much lower resolution at every time point. So you have to figure out how to get all of these really, really low-resolution signals to all come together and strip out all of the noise

• 36:52Speaker 00

  and get to the high-resolution signal that you want, that twanging of that real string of the construct you want to measure. And that's challenging, again, statistically challenging, scientifically challenging, technologically challenging. And then there's this whole other basket of challenges, like the social and the political, where... There's a really creepy version of this that's like ubiquitous surveillance, you know? You are always being watched. You are always being judged. Every choice you make is going to be on your permanent record. Remember that one from when you were a kid? This is going on your permanent record. I had some disciplinary troubles, so I got told that a lot. We don't want everything to be on your permanent record. We want to give kids safe space to practice.

• 37:35Speaker 00

  That's one of the richest things about game-based learning, actually, which was, again, one of my original focuses in grad school. If a kid knows that everything they're doing is being monitored and evaluated, that's like crushing, you know? Or at least it could be crushing, and it definitely will affect their performance in ways that we might not want. So there's a lot of nuance and difficulty there, but I think it's an immensely rich space for exploration.

• 38:02Craig S. Smith?

  Yeah, and currently, a lot of... I mean, that's interesting, the thought of surveillance. I hadn't looked at it through that lens. Although if it's only in the classroom setting, you know, maybe kids would forget about it. Yeah, it's in the background. But one of the problems, too, that I ran into talking to people is is a lot of the data that you want to collect is still pencils and paper,

• 38:46Craig S. Smith?

  which makes it difficult. It all ends up in a wasteman. And do you think as as education becomes increasingly digital, just, you know, doing tests online or doing...

• 39:08Craig S. Smith?

  Well, I guess tests online is what I'm talking about, or doing homework online, or exercises online. As that becomes the norm, does that lead to stronger data flows for this kind of application?

• 39:26Speaker 00

  The pandemic was definitely accelerant on the uptake of digital learning experiences and the data that result from those learning experiences, but it also put into really stark relief the limitations of those learning experiences and how impoverished they can be compared to a rich in-person learning experience. Similarly, with the print-to-digital thing, especially in K5, but even up through higher education, there's a lot to be learned that cannot adequately be replicated in a purely digital context. So I think the burden is on technologists to think about ways to capture data from those in-person interactions. I alluded earlier to the idea of computer vision. Let the kids do the worksheet on paper

• 40:09Speaker 00

  because they're six years old and they're holding these fat pencils and they're doing the little, doing like this the best they can, and then use technology to make it so that you could still capture those data. Or when we were still KidAptive, one of our partners was an early learning provider, and they had a really good assessment framework, but it was all observational assessment because these are toddlers, you know, they're not going to click anything, they're just going to drool on blocks. And the adults who were caring for and supporting the learning of those toddlers were instead responding to observational rubrics that taken together gave a really detailed, high-dimensional

• 40:50Speaker 00

  representation of learner progress. We can do observational checklists for older kids too, especially for a lot of the skills that we really care about these days, like collaboration and creativity and all of these. I mean, it's ironic, we still call them 21st century skills. We're like, we're like 25% of the way through the 21st century and we're still trying to get uptake of these skills. Those kinds of skills don't lend themselves to multiple choice or flashcard sorts or things. They're about dialogue that's happening among real people in a classroom. So figure out how to use technology to evaluate those collaborations.

• 41:27Speaker 00

  Not in intrusive ways where the kid has to like perform a conversation with friends, but in ways where the technology kind of recedes into the background. And again, eases the burden on the teacher to do that evaluation, but supports the teacher with insights. Like, for instance, there's a company that does interesting work in college where they have breakout sessions, breakout rooms, and college students are discussing some topic and there's a prompt and there's some structure to it. And the system is observing how the students are responding to one another, taking up one another's ideas and building on them, you know, or introducing countervailing evidence or all of that stuff. Then they can provide summaries of all of those interactions back to the instructors.

• 42:11Speaker 00

  I might be teaching a survey course with 700 kids. I am not going to sit in 700 breakouts. But if I get some insights and the system sends me, you know, links to videos where I could like sort of drop in and watch how a particular interaction went, I could be like, oh, wow, that is really cool. That's another example of controlling the instructor's attention, right? Like you, you, you extract from a natural interaction, the, the data that could be extracted from it, and then you use those data to guide the instructor's attention where it can be most productively spent. I think those are excellent uses of these kinds of tools in educational contexts that don't, they don't marginalize the instructor, they don't marginalize the human relationships.

• 42:52Speaker 00

  They, they let us get right at that human stuff and the technology is, is off in the shadows in the distance, you know, supporting it, not in the center.

• 43:04Craig S. Smith?

  Yeah.

• 43:08Craig S. Smith?

  Let me ask you something different that, that has occurred to me while you're talking that, um, what, you know, I, I bemoan the current state of, of education in the United States, you know, and I, you know, a lot of it's the political choices that people are making because they don't seem to understand the underlying issues. A lot of it's just interacting with, with people who are, should be educated and don't know a lot. Some of it's the these late night talk shows that go out on the street and ask people, you know.

• 43:53Craig S. Smith?

  So these simple facts or show them images of famous people and they have no idea who they are. But in I I I'm not sure that it it it's it's worse now than it ever was. And I wonder in a way whether it it it's it's just the focus of modern society has shifted. So a lot of the things that that we used to think were important to know are no longer really important.

• 44:33Craig S. Smith?

  I mean, I I I remember as a kid, a little kid, I'd go out to a my um my mother grew up on a ranch in Montana and I would hang out with the kids on ranch kids from neighboring ranches. And there was one kid I was close with. We went camping one time. And he asked me, uh, I was talking about something I wanted to go to Europe, I think. And he asked me whether England is the capital of Europe. And I remember just being stunned that that he wouldn't know that England is part of a country. It's not a

• 45:16Craig S. Smith?

  city in the country isn't a capital and all that stuff. So it maybe it's education has not declined as much as just our exposure to people has increased and we're sort of realizing that that there are huge pools of relatively uneducated people in the world. I mean, what do you think? Do you think that uh and and then on the other side? Just the access of information through social media and YouTube, I mean, a lot of unfortunately, there's a lot of misinformation, but uh, you know, people know a lot more

• 46:02Craig S. Smith?

  than they did in the 50s, just because they have, I mean, the average person, they have so much information that's coming at them all the time. So I, do you have a sense of, of that of whether

• 46:20Craig S. Smith?

  people are, are less educated today on en crawl, as I say in French, you know, in those uh, or whether there it's, it's a problem that persists at the elite who can afford it, whose parents are uh, understand the value of education, you know, educate their kids, but, but a lot of the world uh, is still uh, muddling along.

• 46:53Speaker 00

  Education in the 20th century, especially in places like the US, uh, increased the set of people whose success we cared about, and that's a good thing.

• 47:06Speaker 00

  And like you say, you know, the, the descendants of the elites who had the, the private tutors and the boarding schools and went to the universities are still having a similarly great educational trajectory. And if you measure the success of all the educated people, it used to be that it was a small circle and they were pretty much all pretty well resourced. And as the circle gets bigger, now you're including people who have much less access to that kind of support. And so, yeah, the average achievement might go down, but that doesn't mean that the system is doing worse. In fact, in a sense, it means the system is doing better because it's now reaching people who used to have zero and now they've got something. There is a, there is a psychological effect called the Flynn effect. Do you know the Flynn effect?

• 47:51Speaker 00

  The Flynn effect is the fact, it's an empirical fact, it's not a theory. In fact, it's not even there. I don't even know there's a consensus on why it happens, but um, there has been a steady and unexplainable upward trend in scores on intelligence tests in the 20th century.

• 48:09Speaker 00

  That is definitely too fast to be like, the human species is evolving, you know? And it's not like, oh, we're eating better. It's like, people cannot come up with a good reason why the scores are getting higher like that. But what we don't think is happening is that people are actually getting smarter. It's not like my grandparents were mentally disabled, you know? Instead, something else is going on. And one of the theories that I've heard is that intelligence tests have always been criticized as being kind of artificial and a weird kind of narrow set of skills that do predict success on various tasks, but are also not reflective of a lot of things. For instance, that your friends on the ranch in Montana needed to do in their everyday lives.

• 48:52Speaker 00

  As society has become more like or more filled with tasks that are similar to those that people do on intelligence tests, the general capability to achieve those tasks has increased. And so the performance on the tests is going to go up, simply because what the test measure is more closely mapped to what the human brain is being asked to do day to day. I think that's a reasonable explanation for the Flynn effect. And in general, yeah, society is moving at a faster pace and the rate at which we're asked to take in information, process the information, make choices based on information, et cetera, has increased. So our ability to do so will increase because that's how the human brain responds. It'll, it'll get better at the thing that is being asked to do a lot. By the same token, as you alluded to, there are things that we should no longer care

• 49:36Speaker 00

  whether our learners are good at. For instance, our grandparents, probably if they took math in high school or college, they were expected to be able to use a slide rule fluently to compute square roots. And we do not have computation of square roots on the curriculum in any math program that I know of anymore. And the good reason for that is because that's a lame thing to spend kids' time getting good at. There's no reason to do it. You just understand conceptually what a square root is and use a system that will provide that square root for you. Cool, you're good. We need to continually do that reappraisal of the kind of core of our curricular ideals for our students and say, is this still a relevant thing for learners to be able to know or do?

• 50:19Speaker 00

  And if not, you know, what are new, relevant things that we want to make sure our society is equipped to do?

• 50:25Speaker 00

  Does that make sense?

• 50:26Craig S. Smith?

  Yeah, yeah. The, uh, the other thing that, uh, I was in, um, Pittsburgh a couple of years ago. Uh, and I have a friend who's a television reporter there and I went with him on a story. He was doing a story in a, a local school about, uh, CMU Carnegie Mellon was, would had a pilot program in the school using AI. It wasn't an AI tutor, as I recall, but it was some.

• 51:00Speaker 00

  It might have been an intelligent tutoring system. CMU was a great leader in the, uh, intelligent tutoring system space.

• 51:06Craig S. Smith?

  Maybe, maybe that was it. Yeah. And, uh, so this was during sort of a study hall. Uh, so there was a teacher present and the kids were supposed to be interacting with these, uh, with the system. And I, you know, I, I love the theory and I've written articles about how this is the, the promise of, of the future of education and it's going to transform everything. And sitting there watching these kids, and this was at a, a school that was fairly selective.

• 51:44Craig S. Smith?

  So they were smart kids, but man, they, they, they would typically look at their screen for maybe 10 seconds. And then they would be moving around and seeing what each other is doing and getting up and talking to their friends. And then the teacher's telling them to sit. I mean, there was just no focus at all. And, um, and that, that really struck me that, yeah, the theory's great, but unless you have students.

• 52:19Craig S. Smith?

  That that want to engage and that are interested in in learning, not just in passing tests. It's moot. I mean, it's you can put all the fancy tools. So I part of the problem is just, you know, and I imagine if you're if you're a teacher, I'm fascinated by teaching. If if you're in a room with with 20 kids, there are probably five kids in there that you can see really want to learn and and have that motivation. And

• 52:57Speaker 00

  I would dispute that. I would say all of the kids in there really want to learn something. There are five kids in there who might be really excited to learn the thing that you're supposed to be teaching them that day.

• 53:10Craig S. Smith?

  Yeah.

• 53:11Speaker 00

  But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But

• 53:56Speaker 00

  But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But But

• 54:42Speaker 00

  Games, when done well, are really cool that way. Games are a great opportunity for kids to be intrinsically motivated to play around with an idea.

• 54:53Speaker 00

  And if you can capture that intrinsic motivation to explore, then you've got a great learning experience in making. If it's like a drill and kill that your teacher told you you have to do, and it's lame, and you're not interested, you're just gonna kind of click through it desultorily, you're not gonna get a good measure, your kid's not gonna learn anything, you're wasting everybody's time.

• 55:13Craig S. Smith?

  In three minutes, can you tell me what you guys are doing, if any, or what the promise of gamification is? Because that's something that hasn't really come on stream in a big way.

• 55:28Speaker 00

  I can be no justice to it in three minutes. I wrote my dissertation on game-based learning, and I could talk to you for hours about game-based learning. Play is the single most powerful innovation for learning in the history of the world. There is nothing else that even comes close. Play is so widespread across the animal kingdom that it must be tremendously useful for learning, because evolution does not keep traits that aren't useful, right? So there's something demonstrably powerful about play. The better we can understand where that power is and how to leverage it in service of the learning goals that we think are important for learners, the more powerful the game-based learning, the play-based learning experiences

• 56:10Speaker 00

  can be. When instead, we just schmaltz on some points and levels and stars to something, we're doing a disservice both to the power of play and to the autonomy and integrity of the learner.

• 56:24Craig S. Smith?

  Even if you think it's a bit overhyped, AI is suddenly everywhere, from self-driving cars, to molecular medicine, to business efficiency. If it's not in your industry yet, it's coming, fast. But AI needs a lot of speed and computing power. So how do you compete without costs spiraling out of control? Time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure.

• 56:51Craig S. Smith?

  OCI OCI is a blazing fast and secure platform for your infrastructure, database, application development, plus all your AI machine learning workloads. OCI costs 50% less for compute and 80% less for networking. So you're saving a pile of money. Thousands of businesses have already upgraded to OCI, including MGM Resorts, Specialized Bikes, and Fireworks AI. Right now, Oracle is offering to cut your current cloud bill in half if you move to OCI. This is for new US customers with minimum financial commitment.

• 57:35Craig S. Smith?

  See if your company qualifies for the special offer at oracle.com/eyeonai. That's eye on AI, all run together, E Y E O N A I. So go to oracle.com/eyeonai to see if your company qualifies for this special offer.