Beyond screen time: What makes educational technology effective, with Nicole Barnes, PhD

Digital tools now help teach students everything from spelling to math to science -- but flashy features don’t always translate to learning. Nicole Barnes, PhD, executive lead psychologist for education at the American Psychological Association and lead author of a new report on children’s and adolescents’ learning with educational technology, discusses how to tell the difference between engagement and real learning; why generative AI tools call for extra caution; and what teachers offer that technology can’t replace. Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Are your ad campaigns lighting up the dashboard? But not the pipeline? That's bullspend, and marketers are calling it out in. Dashboard Confessions.

• 0:09Speaker 01

  My boss asked for results, so I opened my dashboard for the only positive sounding metric I had: impressions.

• 0:17Speaker 00

  Cut the bullspend. See revenue, not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 0:31Speaker 02

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies, 18+.

• 0:58Kim Mills?

  If you're the parent of a K through 12 student, there's a good chance your child is using educational technology or ed tech. Digital tools and platforms are being used to teach everything from spelling to math to science in classrooms around the country. And in the right context, well-designed ed tech can help students learn. But how can you and your kids' teachers tell the difference between an effective tool that builds real skills and one with flashy bells and whistles that keeps kids entertained, but doesn't help them master new skills and knowledge? In September, the American Psychological Association released a report on children's and adolescents' learning with educational technology,

• 1:41Kim Mills?

  with 10 research-backed recommendations to help educators, parents, and school leaders decide when and how to use ed tech. Today, we're going to talk with the report's lead author about what constitutes effective educational technology. How can you tell the difference between engagement and true learning? How should students' relationship with ed tech change as they move from elementary to middle to high school? And what special cautions should schools and parents consider when evaluating AI-based tools? Welcome to Speaking of Psychology, the flagship podcast of the American Psychological Association that examines the links between psychological science and everyday life. I'm Kim Mills.

• 2:29Kim Mills?

  My guest today is Dr. Nicole Barnes. She is APA's Executive Lead Psychologist for Education and the lead author of the EdTech Report. Dr. Barnes earned her PhD in Educational Psychology from Rutgers University. Before coming to APA, she was a tenured professor at Montclair State University, where she studied the psychological foundations of teaching and learning. Earlier in her career, she spent several years as a fifth-grade teacher, an experience that she says continues to ground her commitment to educational equity and innovation. Dr. Barnes, or should I call you Nicole, my colleague? Thank you for joining me.

• 3:06Speaker 04

  Happy to be here, Kim. And yes, please call me Nicole.

• 3:09Kim Mills?

  Let me start by asking, why did APA decide to put out this report now? What was it about this moment that made you and your colleagues feel it was needed?

• 3:18Speaker 04

  Great question, Kim. And I just want to start by recognizing those amazing colleagues that were part of this effort. I couldn't have done it without them. And so, all the success that is coming from this report is really because of their collective expertise. So, big thank you to all the co-authors. You know, for me, the inspiration for this report actually dates back to COVID. So, at the time, my son Hudson was in kindergarten, and like millions of children across the United States, he was learning from home, you know, at the desk right next to mine. And one day, I looked over, and you know what, Kim? I thought he was playing a video game.

• 4:01Speaker 04

  There were just colors flying everywhere. You know, there was characters that he was dressing. There were all these rewards that were popping up on the screen. And so, like any parent during the COVID era, watching their child, you know, I, of course, under my breath, so that the people who I was Zooming with couldn't tell, you know, turned to Hudson and say, "Stop playing that video game and get back to your math homework." And do you know what? He did not even skip a beat. And he said, "Mom, this is my math homework."

• 4:39Speaker 04

  And wow, did I pause. Like, I can remember like it was yesterday. Because here I am, you know, a university professor. I had been a teacher myself, and I could not tell whether what I was looking at was a game or a learning experience. And that moment has stayed with me for years. And I have been wanting to do a report on educational technology and the psychological foundations of learning with educational technology since that moment.

• 5:09Kim Mills?

  Well, how do you define ed tech in the report? What do you include? What do you exclude?

• 5:15Speaker 04

  Sure. And I'll even go a level up here, because I think it's important to first separate educational technology from technologies that children use more broadly. Because not every use of technology is educational. So, what I mean by that is, a child, let's say, watching TikTok on their phone is certainly using a technology. But that is not the kind of technology that we're talking about in this report. We're talking about those digital products, those services that are being used or are being promoted as intended to be used to support learning. And then I'll actually, like, answer your question.

• 5:59Speaker 04

  That, within the ed tech space, there are important differences. You know, some tools are meant specifically for teaching. So, if you think of like a tutoring platform or a digital reading program, those tools were designed with instruction in mind and with the outcome of that instruction being learning. Others, we would classify as having a broader purpose, but they're used to support learning. So, think about like a video meeting platform, or even, you know, an app that helps you design your own videos. Those weren't created for schools. They weren't created for instruction.

• 6:42Speaker 04

  But that doesn't mean that they aren't used by educators

• 6:48Speaker 04

  as part of the instructional experience. So, I think a lot of times, you know, people will ask me, "Does ed tech work?" And to me, that seems a little bit like asking, "Does transportation work?" You know, we have bicycles, we have school buses, we have airplanes, and they all work to get people somewhere. But they all serve very different purposes. And we wouldn't talk about a bicycle and an airplane as doing the same thing. And ed tech is exactly the same way. So, important to all you great listeners out there is that, you know, what Kim and I are going to be talking about today, you know, are those technologies that are intended and marketed for learning.

• 7:29Kim Mills?

  Now, that application that your son was using during COVID, and you mentioned, and you thought it looked like a game, that was a kind of ed tech, but was it, was it good?

• 7:39Speaker 04

  Apparently not. And no, I will not name the app that he was using. But I will tell you this, it was intended for learning, and it was being, you know, assigned as a homework assignment, you know, by his teachers. And so, that is, is really what, you know, for years, just stuck with me as so important to help educate, kind of like, the public and the education sector on, is like, this differentiation between what a tool is marketed as versus what we actually have the evidence, you know, to support its intended outcomes.

• 8:19Kim Mills?

  Well, a lot of the debate right now around kids and tech has centered around screen time. But one of the report's recommendations is to move beyond just screen time and talk instead about what screens are replacing. So, why is it important to do this, to move beyond screen time? And what should we be thinking about instead?

• 8:38Speaker 04

  I'm so glad that you asked this question, because I feel anybody turning on the news right now, you can't get through a half hour of it without hearing about kind of this screen time debate. And let me just say, I'm a mom. You know, I have two middle school kids. I completely understand why people start with screen time. It's, you can see it easily. It's easy to count. And I will say that screen time, it might be an appropriate metric for recreational uses of technology, like Snapchat or a video game. But when we're talking about technology for learning, then screen time by itself, it tells us very little about whether a child is learning

• 9:23Speaker 04

  or whether that experience is helping the child to learn. So, just let's take an example to play with. You know, if I tell you that a child spends 20 minutes on a screen, how much do you really know about that experience? The child could have been working very deeply through a math problem, or, like in Hudson's case, spend 20 minutes dressing a character. So, the amount of time is the same, but the experience is not. And so, when we're talking about educational technologies, whether they're in schools or whether you're using them at home to help supplement learning, the more useful place to start is, you know, what is the child actually doing?

• 10:08Speaker 04

  And what that time is replacing? Those two questions tell us much more than the number of minutes alone. Now, I won't say that time doesn't matter, especially when technology is getting in the way or replacing things like sleep or relationships or other things that we know children need for healthy development. But you cannot, as a responsible parent or educator, treat every minute on a screen as if it has the same value. You need to look at the quality of that experience and the trade offs that it creates. You know, what looks productive on the screen

• 10:52Speaker 04

  might not always tell us what the child has learned.

• 10:56Kim Mills?

  So can you talk a little more about what parents, educators, and even school districts are looking for when they're evaluating ed tech to tell if the tool is really going to help children learn?

• 11:08Speaker 04

  Well, first off, I will say, please do not start with how impressive the tool looks or how many features it has. I would start with the learning. You know, questions like, what is the tool helping my child or our students understand or do that matters? And then I would watch for who is doing the thinking? You know, is that tool helping the student work through an idea? Or is it doing so much that the student can move forward without really understanding? I mean, that's where the real risk is, that we can, it can look like learning has occurred

• 11:52Speaker 04

  and we really have very little evidence that learning has occurred. So if it was me, and I, I, I promise you, I'm not only talking from, you know, executive lead psychologist hat, I'm really thinking about what would be important to me as a mom. I would start to ask myself two questions when, you know, looking at my, my child kind of engaging in ed tech. I would ask, you know, does the feedback that they're getting through this app prompt my child to think more deeply than simply pointing them towards the right answer? And then if I had some evidence that that was the case, I would ask if there is credible evidence that the tool works for, you know, especially if I'm thinking

• 12:37Speaker 04

  from an educator's point of view or from the point of view of a teacher, that that tool works for students similar to those who will actually use it. Because a lot of times, the tool is marketed on evidence for one population and that doesn't mean that it transfers well to another population. And so that claim on a website, you know, is more about marketing than it is about evidence. I would add just one more thing, that it's going to be important for us to start separating activity from impact, especially when we start talking about AI. But this relates to engagement as well. You know, students can do a lot of clicking.

• 13:21Speaker 04

  They can look like they finished a lot and they can still learn very little. So we have to make sure these tools are vetted against a clear instructional goal. And we, we have to start listening really closely to teachers and students so that we can figure out whether we have evidence of learning beyond that app. And if the benefit is not there, then schools need to be willing and parents need to be willing to let that tool go.

• 13:53Kim Mills?

  How can educators test students to determine if these tools are working as they should? I mean, do they have to design new tests? How will testing need to change in the face of new technologies?

• 14:05Speaker 04

  This isn't really as difficult to uncover as we might think. I, I really, I would urge any parent out there to just look over the shoulder of what your child is doing for no more than two minutes. And I think you will be able to tell whether that app is really resulting in learning or not. But the true test is going to be not what happens when the child is using the app, it's what happens next. So shut down the computer, close the laptop and ask the child, the student, to remember the idea, to explain it in their own words or to use that knowledge in a new situation once those prompts are gone. Do you remember those keyboards, Kim,

• 14:50Speaker 04

  that used to, I think they still have them. They used to light up, like the keys would light up to tell you, like what button to press? This was marketed for years as a way to teach folks how to play piano. It's a little like that. You may, you may play the piano perfectly when those lights are lighting up for you and telling you which buttons to press. But we find out what you actually learned when those lights go away. And that's the same thing that we're going to do now. It's not going to be so much about like, evaluating what's happening in the learning experience itself. It's going to be evaluating what happens after that learning experience and then using that information to kind of then make judgments

• 15:35Speaker 04

  about how effective that educational experience was.

• 15:41Kim Mills?

  Are there certain types of learning or even academic topics that ed tech is particularly well suited for and others that it's not?

• 15:50Speaker 04

  In general, ed tech is most useful when it offers a real advantage for the learning task. It should not be the default simply because a digital option exists. So for example, technology can be super helpful when a student needs immediate feedback. Technology can be super helpful when they need practice at the right level or to even give them access to an experience like an AP course or access to a foreign language that would be hard to create in that particular classroom or a school that they're in. I could even think of like in a, in a science class, a simulation that, you know,

• 16:34Speaker 04

  might let a student see a scientific process that's normally invisible or a, a practice tool that adjusts as the student improves. But technology is a much weaker substitute for things like a rich discussion or a hands-on discovery or even encouragement from another person. So I would say that the rule of thumb that I generally use is use the technology when it adds something meaningful to the learning. For learning, that depends on complex reasoning, when it depends on relationships, when it depends on working with other people, technology can certainly have a supporting role, but it shouldn't become the whole experience.

• 17:18Speaker 04

  And that balance is, is going to even look different when we talk about, you know, to complicate it more, when we talk about, you know, different ages of children.

• 17:28Kim Mills?

  We're going to take a short break. When we return, we'll talk about how effective ed tech looks different for students at different ages.

• 17:40Speaker 00

  Are your ad campaigns lighting up the dashboard? But not the pipeline? That's bull spend and marketers are calling it out in. Dashboard confessions.

• 17:49Speaker 01

  My boss asked for results, so I opened my dashboard for the only positive sounding metric I had. Impressions.

• 17:57Speaker 00

  Cut the bull spend. See revenue, not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 18:11Speaker 02

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration blog or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up. Required compatibility and availability varies. 18 plus.

• 18:39Kim Mills?

  Does effective ed tech look the same for a kindergartner versus a high schooler? And is there an age that's just too young for ed tech?

• 18:48Speaker 04

  Absolutely, yes. Effective ed tech does look different at different ages and we shouldn't treat it all as the same. It's like a six-year-old, a 12-year-old and a 17-year-old are of course going to bring different levels of judgment and independence to the same tool. And we know, you know, from the science that as children get older, they are generally better able to manage their attention. They're better able to question what they see and they're better able to use technology more independently. So the tool, it might not need to change completely, but certainly the support around it should.

• 19:32Speaker 04

  You can think of it a little bit like driving. We would, we wouldn't hand like a six-year-old and a 16-year-old the same set of car keys and expect the same outcome, right? I mean, no one would do that. That would just be like mind boggling to us. And we, we can't do the same with ed tech either. The tool may be the same, but the learner's judgment and need for support are very different. In the second part of your question, you asked about young children, you know, in particular. And I, I do, I think, want to take a moment to, to call out some of the risks for young children while also saying that our report focuses on school age children. So we, we did not get into the research on what would be, you know,

• 20:17Speaker 04

  early, early childhood experiences. But I would say, what the report hints at is that for young children in particular, we want that digital experience to be brief. We want that experience to be in collaboration or shared with an adult and it should be connected to something in the real world. Older students, high school students can use technology more independently. They still need to, to question what's, what it gives them and they need to question like how it shows what they know. And we can teach them literacy, digital literacy strategies to be able to do that. But for young children especially,

• 21:01Speaker 04

  I would say that interaction with caring adults and to the physical world needs to come first. And my advice to parents would say that the younger the child, the harder the tool should have to work to justify its place.

• 21:16Kim Mills?

  I want to talk for a minute about artificial intelligence. Explain for us the role of AI within ed tech. And I'm just wondering, because one of the report's recommendations is to approach generative AI tools with extra caution. Why is that?

• 21:32Speaker 04

  Definitely. And I will say that although we call out generative AI in one recommendation in particular, an assumption of the report is that certainly as these educational tools continue to develop, we know plenty of instances where AI is either operating in the background or student facing. So we don't see the other recommendations as being separate from that. The reason why we call out generative AI in particular and add that caution around it is because it can make a final product look better while making it harder to tell

• 22:13Speaker 04

  how much thinking the student actually did. Another thing that we don't talk about too much around generative AI is that generative AI can sound pretty darn confident even when it's wrong. And that can be especially tricky for children who are still learning how to judge what they see. So I keep coming back to this question, you know, is the learner getting better? Or is only the assignment getting better? You know, though the work, it can look stronger, but the student, you know, might be less able or less prepared to be able to start, reason, or revise without help. If that's the case, then we have a

• 22:56Speaker 04

  problem. You know, I don't know if you do this, but my mom has just started working with a personal trainer. And, you know, like, we were joking the other day that, you know, the point of working with a personal trainer is not to see how much weight the trainer can lift for you. The point is to build what you can do when the trainer steps away. And I think the analogy is, you know, it's quite similar here. AI, it can have a useful role. It can certainly respond to a student's thinking after they have made the first attempt. I think that that is super important, that the child is making the first attempt. The

• 23:36Speaker 04

  AIs can offer feedback. They can help the student test an idea. But I want the student to begin the thinking, and I want the student to remain responsible for deciding what to accept, what to question, and what to revise. I think if AI is acting in that capacity, then it's making humans stronger, not replacing humans.

• 23:59Kim Mills?

  Is there any central clearinghouse, someplace or some organization that's reviewing these tools and telling educators, this is a good one, this is one you should stay away from? Is that something that APA is contemplating?

• 24:14Speaker 04

  So there is what's called an ed tech index. It was an initiative that was funded by philanthropy and supported from a bunch of educational technology associations, not APA, though we do remain a very close partner with a lot of the organizations that were involved. And that is like a clearinghouse where you can go onto the website, you can see, you know, a myriad of educational technology tools, and you can see the badges that they have earned. And there are different badges for different qualities, like a badge for

• 24:55Speaker 04

  privacy versus a badge for learner outcomes. So there is a resource available right now that is, could be very, very helpful for families and educators who are making these kinds of decisions.

• 25:09Kim Mills?

  So you just mentioned privacy. What should families know about how these apps are using their children's data? How can they determine whether an app is safe from a privacy perspective?

• 25:23Speaker 04

  Well, I will just tell you, from being a mom, that I was so surprised by how much information an educational app can collect. I mean, it's quite frightening, actually. You know, it could be not just capturing your child's personal data, but it's capturing their work. It's capturing their thinking and how they move through that tool. And it's giving them a lot of information about the, you know, about the device and about the learning app. And in some cases, it might even be drawing conclusions about the learner and more broadly, you know, to other populations of learners from those patterns. So for me,

• 26:05Speaker 04

  it's much more than simply whether a privacy policy exists. It's whether the amount of information being collected makes sense for the learning task, and whether the company is using it for anything beyond those purposes. So if I was speaking to families, or if I was speaking to schools, I would suggest perhaps that they ask, you know, a couple set of questions, like, why does the app need this information? That seems like important to know. Like, is the information that they're collecting actually being used to help the learner with that goal of advancing their learning? A second question I might ask is, will it be used for anything besides my children's learning? And then third, and I would not

• 26:50Speaker 04

  forget this question, my friends, can we delete it? You know, can we, do we have control over deleting that information? And if those answers are hard to find, or if those answers are hard to understand, I would take that very seriously.

• 27:05Kim Mills?

  There are some school districts that are approaching all of this with a lot of caution. Districts, including New York City and Los Angeles, recently put a pause on using AI-based tools in their classrooms, at least for younger grades. Do you think that's the right approach, or are they just deferring the inevitable?

• 27:25Speaker 04

  I actually think it's a very reasonable approach, the pause. And I actually respect quite a lot. I've read deeply into New York, less about the others, but what they're doing to try to approach this in a responsible way. I will say that in the report, we do not call for a blanket ban. We call for age-appropriate policies. We call for paying really, really close attention to what AI is doing to learning. And importantly, though, we haven't really talked about this yet, is we talk a lot about the educator oversight and input that will be needed when the decisions that schools are making could affect a student's future.

• 28:08Speaker 04

  I think schools need clear boundaries, but teachers and educators still need to be the ones who are exercising that judgment.

• 28:16Kim Mills?

  Tell me more. What about the role of teachers versus technology? I mean, what are teachers adding to learning that technology just cannot replace now and maybe ever?

• 28:25Speaker 04

  Oh, I do not think that these technologies should replace a teacher. I, in fact, I think quite the opposite, that the more sophisticated the technologies become, the more important teachers become. And I'll give you some examples that can hopefully help to paint the picture. So the technology, you know, gives us information about a student, but a teacher or an educator knows the student, and that's a very different thing. You know, teachers see the context around a student's performance. They notice things like when that child is hesitating. They notice things like when that child is losing confidence

• 29:04Speaker 04

  or becoming curious or quietly just plain checking out. And I don't think that a score or a dashboard is going to be, maybe I'll be wrong, and we can play this back at me 20 years from now, but I'll be in an old lady's home with eating ice cream. But I don't think that that score, that dashboard, is going to fully explain those moments in the way that the teacher can respond to them in real time. I have yet to meet a technology tool that can noticeably tell whether that child is quietly giving up before that child clicked the wrong answer. But as a teacher, I have noticed that almost every day in the

• 29:47Speaker 04

  classroom. And I think that even things like a teacher believing in a student and knowing that they're more capable than what they thought they were. I don't see technology being able to do that. And great educators do that every single day of their life. So is technology needed? Absolutely. Does it, does it play its role if used well intentionally and with evidence that it's doing what it claims it's doing? Absolutely. And can it extend what a teacher's, you know, work and maybe even impact might be by providing that practice and that feedback or another way into the material? Absolutely. But I

• 30:31Speaker 04

  don't see it yet deciding what this particular student needs in this particular moment. That is a whole child question, not just an academic, where are we on this particular learning objective question? And that's where I think that teacher judgment and those relationships matter.

• 30:51Kim Mills?

  Well, just to wrap up, are there any other recommendations from the report you want to highlight or advice you want to give to our listeners?

• 30:59Speaker 04

  I guess I would sum up by saying that for us, this wasn't a report about technology. This is a report about learning. And that is why psychology plays a role here, because we are the discipline that understands how children learn, you know, what motivates children. We understand behaviors. We understand the relationship behaviors and emotion and learning. And so for us, this was really about giving the community, the field, society, you know, the science that helps to make them, to help make better decisions. And hopefully, you all find this report to be very useful with your children and with your students. And

• 31:44Speaker 04

  please reach out and let us know if it was. We'd love to hear from you.

• 31:47Kim Mills?

  Well, Nicole, I want to thank you for joining me today, and thank you for spearheading this important report. I'm sure this is just the beginning of our learning how best to use ed tech going forward.

• 31:58Speaker 04

  I hope it is. And again, thank you to all of my co-authors. Could not have done this without you.

• 32:03Kim Mills?

  For our listeners, you can find a copy of APA's expert report on children's and adolescents' learning with educational psychology on our website at apa.org. You can also find previous episodes of Speaking of Psychology there, as well as on Apple, Spotify, YouTube, or wherever you get your podcasts. And if you like what you've heard, please subscribe and leave us a review. If you have comments or ideas for future episodes, you can email us at [email protected]. Speaking of Psychology is produced by Lee Weinerman. Thank you for listening. For the American Psychological Association, I'm Kim Mills. I'm Kim Mills.

• 33:21Speaker 05

  This episode is brought to you by State Farm. Football has a play for just about everything. You can go deep, keep it on the ground, or draw up something nobody sees coming. And whether you want to run a flea flicker or the tush push, you need a plan to fit your unique situation. That's why State Farm takes the time to get to know you, so they can help personalize coverage for your life and budget. Now that's a solid game plan. Like a good neighbor, State Farm is there.