Creating 'world models' for robots + An AI math shakeup

How do you create a model to help AI understand the physical world? Plus, the latest AI-driven advancements in math.

• 0:00Speaker 00

  WNYC is supported by Bear. Science is about asking questions, testing, trying again. It's not easy, but it's relentless. It evolves, and that's why science delivers. Learn more at sciencedelivers.com.

• 0:17Speaker 01

  Hey, I'm Ira, and you're listening to Science Friday. When you ask ChatGPT or Claude a question, the model goes through its massive amount of training data, makes guesses on the answer, mathematically predicting the next most likely word to appear in a sentence, for example. This model, experts say, will not work well in a world that's not based on textual information. Instead, they argue, you need a completely different approach, a world model that can understand the physical world around us, stuff we walk by or we bang into, and enable humanoid robots to work in warehouses and improve self-driving cars. It's an area of AI development that's raised billions over the past few years, with big

• 1:02Speaker 01

  players like Amazon and Google's DeepMind showing more and more interest. So what does a world model mean, actually, and how do you train the AI to recognize what the real world looks like? My next guest has seen the early days of these models up close, even in her home, and is here to tell us how ready for prime time they are. Joanna Stern is a tech journalist who writes newsletters and creates videos for New Things Media. Welcome to Science Friday, Joanna.

• 1:34Speaker 02

  Hello, thank you for having me, Ira.

• 1:36Speaker 01

  You're welcome. Okay, tell me more about why tech companies want this different approach to AI.

• 1:43Speaker 02

  Well, if you think about the physical world and the type of AI we are thinking we're going to get in the physical world, best examples would be self-driving cars and humanoid robots, or robots of any kind. You can see why just studying the text of the world, and even just studying basic images of the world, is not going to be enough for a car to get you from point A to point B, or it's not going to be enough for a humanoid robot to step into your house and start doing the laundry.

• 2:13Speaker 01

  So a company like Amazon that has to move a lot of stuff around could really benefit from this kind of robot?

• 2:20Speaker 02

  Sure, and Amazon has over a million robots already functioning in some of their warehouses and in their partners' warehouses. And so the big difference there, though, in terms of what Amazon's talking about in robotics, is that those robots are largely industrial work-type robots, right? They have a very, very regimented, simple environment that never changes. Think about it. If you are taking things from one box and putting them into another box, the box size never really changes. They're usually located in the same exact spot. And so this is really simple to program robots. That's very different than, say, a self-driving car, where, sure, you're still going from

• 3:04Speaker 02

  point A to point B, but a bicycle just went in between that road that wasn't there before, or they just set up a road closure sign that wasn't there before. And so these things are constantly changing in our physical world. The home is even worse, right, Ira? I mean, I don't know about your home, but my home is changing every day. Someone moves something in the refrigerator. Somebody moves a plate. My kids have put their toys all over the ground.

• 3:29Speaker 01

  And so is video very helpful? Is that what we're talking about here?

• 3:34Speaker 02

  Well, so there's some debate about how it's going to be best to train these robots. One way is through something called teleoperation, where the robot actually is in the home. There's cameras and sensors on the robot, and somebody is controlling that robot with a virtual reality headset someplace else, right? They're wearing arm controllers, and they are controlling, basically as a puppet, the humanoid robot that's in the house. And I've seen this demoed a number of times now. So the robot isn't doing it autonomously. And so this idea of teleoperation is that we are training and gathering data of the movements of the robot. We're also gathering video and sensor data from the robot, and that, over time, can

• 4:18Speaker 02

  start to train. There's another idea that's becoming pretty popular, and that's we use video data. We take lots of video off YouTube. We take lots of video from humans doing this in the real world, and we train it. And in fact, last month, I did a story on a company called MicroAGI out of Germany, and they have a new app called Shift. And you can say, I want to, I want to earn money by filming myself doing things in my house, doing the dishes, vacuuming.

• 4:48Speaker 01

  Don't pay me to do these things.

• 4:51Speaker 02

  They will pay you to do these things. The catch is you have to wear an iPhone on your head, some sort of smartphone on your head, or now they've started to make hats with embedded cameras. So I tried this. I tried it for about three weeks. I will say I was not very successful at it. Turns out, you have to really, the footage has to show both hands in the frame, and you've got to kind of really show it very carefully. But, look, I did make some money folding my laundry, putting my dishes in, which is more to say that, you know, look, it was, it was, I would say, everyone in the house was happy that I was finally contributing.

• 5:30Speaker 01

  Well, actually, watch your video. I saw your video doing that, and it looked very interesting. And, and of course, as you say, to train the robot, it has to see physically what you're doing with your hands, so that it could tell the robot what to do with its hands.

• 5:46Speaker 02

  Exactly. The big thing, and I kind of challenged the CEO on this, I said, well, look, I filmed five hours. You're only paying me for three hours. He said, well, yeah, because only three hours is useful to me, right? Only three hours showed my hands both in frame and my hands doing something useful that they think they can then go train the robot on. In fact, they sent me the footage where they apply sort of the, the 3D modeling to my hands, and you can see, okay, the, the robot has to see these weird kind of, like, almost skeleton-like lines in your hands, doing things for it to start to make sense of what is happening.

• 6:24Speaker 01

  You know, if I were to go backwards, engineer things, if I were a Luddite, I might say, you know, instead of a robot trying to do this, my 15-year-old will do this for next to nothing, don't have to teach anything, why go with a robot on this?

• 6:38Speaker 02

  It's true, it's true. But, look, I will say, there's many times in the week where I have a big pile of laundry that I need to fold, and I would rather do something different. Last summer, I actually had a company come to my house and set up their laundry folding contraption, and that wasn't a humanoid robot. That was really these two robotic arms. You can kind of picture, you know, when you go to an arcade and they have the claw machine that, that you can put the money in and it will pick up a stuffed animal or some other toy? The robotic arms looked a little bit like that. And so these two robotic arms sat over the table, had some cameras, it was a big sort of contraption setup, and these two robotic arms, just on their own, were folding the laundry.

• 7:21Speaker 02

  And this, it didn't work great, but it was really great to see because I could see, first of all, it improve over time, and it gave me a sense of where we are at right now. I mean, it was only folding t-shirts. The company now says it folds more things. Sometimes it would take five minutes to fold one t-shirt, right? This gives us a good idea of where we are at in this moment. And, look, this was last year, and things have improved now. But this is, this is how the progress we need to see and why this data is so important to these companies, and why there are companies specifically popping up that just collect the data to sell to the robotics companies.

• 8:00Speaker 01

  Right, and that brings me to, I'm glad you brought that point up, because I'm sure there are privacy experts who are not thrilled about people sending thousands of hours of footage inside their own homes.

• 8:12Speaker 02

  Totally, and I thought about that, obviously, before embarking on the experiment I did in that video with that company, MicroAGI, though they gave me a pretty long list of the ways they're protecting privacy, one of them being that they blur out any identifiable information they might see in your footage. So for instance, they sent me back a, a video where some writing on a t-shirt was blurred out, and I said, why is this blurred out? They said, well, we blur any text that we see. Okay, it was my kids', like, you know, Pokemon t-shirt, but that's fine. But these companies are taking steps towards that. But, look, you're agreeing to do this, and if you think the trade-off of putting some cameras on your head and showing them your house is worth the $20 they might pay you

• 8:56Speaker 02

  an hour, well, that's your decision to make.

• 9:00Speaker 01

  How do you see this panning out? Do you think this model is going to catch on like the large language models did? Or do you think this is just the beginning? I mean, folding a shirt for five minutes has a long way to go.

• 9:12Speaker 02

  Well, that's one of the reasons I love covering this space right now. I feel like we saw this big breakthrough in large language models. We've seen how that really disrupted society and technology and all of the things we see playing out with AI right now. And there's a lot of hype about how this is coming right now. People should not believe the hype about a humanoid robot coming to their house right now. It is not coming right now. I don't think it's coming in the next five years. What we're watching play out is this need for more data and this need for better, we haven't talked about it here, but what we need is better and safer robots, too. I don't want to allow a 100-pound robot in my house that can't stand upright, that has a risk of falling down. So there are a lot of things that have to happen in the next number of years.

• 9:57Speaker 02

  I think it will happen. I can't give you a timeframe, but I can tell you it's not happening this year or next year.

• 10:03Speaker 01

  All right, that's a good place to end it, Joanna. Good luck in your research.

• 10:07Speaker 02

  Yeah, I'm going to keep on, I'm keeping on this topic. I'm going to be, you know, with the robots for, for many, many years, and they'll remember when they're finally working. They'll say, this was the woman who told the world what was coming.

• 10:17Speaker 01

  So when the singularity comes, they'll keep you in mind.

• 10:19Speaker 02

  That's right.

• 10:20Speaker 01

  Joanna Stern, tech journalist who writes newsletters and creates videos for New Things Media. After the break, we're adding up the drama happening in the AI math world. Stay with us.

• 10:38Speaker 03

  Today, when it comes to transplants, science can feel like science fiction. We've got face transplants, big parts in people, whole entire eyeballs. It's like, what the heck is happening in medicine right now? Join Radio Lab for a live podcast taping all about the science of transplants, featuring Dr. Avir Mitra and science writer extraordinaire Mary Roach, live in New York City at The Green Space, November 5th, 7:00 PM. Tickets at wnyc.org/events. No transplants will happen on stage.

• 11:14Speaker 01

  Now we're turning to news in the math world. In the last month, AI models have disproved a handful of conjectures. OpenAI announced that it had solved 10 longstanding problems with an advanced unreleased model. Some mathematicians said the results were impressive, while others said they were overblown and accused the announcement of sloppy attribution. Play nice in the sandbox, folks. So what do mathematicians make of all of this? We thought it was a good time to check in with our mathematical referee, Dr. Emily Reel, professor of mathematics at Johns Hopkins University, who's joined us before to walk us through the AI math world. And she's back with us from Baltimore, Maryland. Welcome back, Emily.

• 11:59Speaker 04

  Thanks for having me.

• 12:00Speaker 01

  Nice to have you. All right, let's get into this. When we last had you on in March, you talked about AI also. Are things progressing slower or faster than you thought?

• 12:13Speaker 04

  I think most mathematicians would say this summer has been very surprising at the pace of improvement in the models. I think the first one that really caught my eye was a disproof of something called the unit distance conjecture, which was a 1946 conjecture of Paul Erdesch, a celebrated Hungarian combinatorialist. I would say that there were relatively few areas where AI had made important contributions, and that is changing. AI is still not contributing to all research areas in mathematics, but in certain areas, it is certainly making progress. It seems to do best in areas where the problems are easy to state, if not easy to solve,

• 12:57Speaker 04

  and maybe is less skilled in the areas where the problems are sort of harder to understand the statements of.

• 13:04Speaker 01

  Right. You know, like I said, OpenAI announced their new unreleased model also solved 10 longstanding problems. What's been the reaction from mathematicians? And what do you think of them?

• 13:20Speaker 04

  So the announcement came in the form of a 250-page PDF, entirely written by AI, so not with any human mathematical input that we can tell, together with some further verification of each of the 10 solutions in the form of something called a Lean proof. Lean here refers to a computer proof assistant that can automate some of the refereeing process. Maybe the first thing to say is that when a new mathematical breakthrough happens, it usually takes the community some time to respond. And if the new breakthrough involves a delicate mathematical argument that, you know, might take dozens or even hundreds of pages,

• 14:05Speaker 04

  then it can take sometimes even a year or even more for the community to really verify the correctness of the solution and also understand the impact down the field.

• 14:15Speaker 01

  I'm trying to think of what a mathematical prompt to AI would look like. I mean, it must be huge, right?

• 14:23Speaker 04

  You might think that, but you know, one of the surprises is that often these prompts are really short and relatively naive. A recent example is the Denett-Gorman's conjecture, which is a question in graph theory. And here, the prompting was sort of four extremely naive prompts, telling the model which problem to work on and to keep working until it found a counter example. So, you know, what do we make of this? I mean, you know, one phenomenon is that there are a lot of folks out there who love math but are not getting paid to do math full time, who are kind of rediscovering their love of math by pushing the frontier of research knowledge in this way. You could imagine anybody who was aware of that problem could write that prompt

• 15:08Speaker 04

  and, you know, then generate a solution. So in one sense, this is expanding the number of people who are contributing to the mathematics research enterprise. And we've seen this with the Erdesch problems, too. The Erdesch problems were sort of famous for attracting non-specialists who could contribute in a productive way. And those non-specialists are sort of superpowered now with the latest AI models.

• 15:35Speaker 01

  Did AI have a particular strength that it allowed, that allowed it to do this?

• 15:43Speaker 04

  There's been some discussion of that. So there are some mathematical problems that essentially involve finding a needle in a haystack. And it does seem that AI, for whatever reason, is good at finding unusual mathematical objects with unusual properties. It's not as good as explaining to us how it found them. The sort of interpretation work for some of these counterexamples has been done by humans.

• 16:09Speaker 01

  Yeah, you know, my math teacher always said, "Show your work," right? Speaking of the latest AI models, are you getting a clearer picture on what the relationship between mathematics and AI models will look like in the future?

• 16:25Speaker 04

  I think there is a lot that is currently in flux. You know, one open question is how much AI can actually help us understand mathematics as opposed to just tell us what is true and false. You know, right now, it still seems like there's a lot of work for humans to do to sort of map out the mathematical universe. So you can think of the mathematical universe as some sort of vast building of unbounded expanse that mathematicians have spent millennia both exploring and renovating. This building is so large, there's no architect who has seen the floor plan for everything. There's no contractor who's directing everybody to do the work. Instead, there are sort of specialists in different areas who are interested in different problems, who are going off down some dark corridor

• 17:10Speaker 04

  and trying to open the doors that are stuck to get inside. So when there's a door that's stuck, you know, a problem that we don't know immediately how to solve, it's not clear immediately how to get through. Maybe it just needs a shove in the right corner. Maybe it needs somebody to find a key. Maybe it needs somebody to create a new key. And what the AI models are doing right now is helping us get in some of these doors that mathematicians were having trouble breaking through before. But what mathematicians still have to do once they've opened up a new door is see if they can turn on any lights in there, maybe renovate a bit to make sure the ceiling doesn't collapse, see whether this door leads to just a closet or maybe a whole entire new wing of the building where there will be new mathematical discoveries around the road.

• 17:54Speaker 04

  And right now, at least, AI is not contributing to those sort of larger enterprises of expanding the mathematical universe.

• 18:02Speaker 01

  Well, Emily, it's always fun having you coming on the show. Thank you for taking time to be with us today.

• 18:07Speaker 04

  Thanks very much.

• 18:09Speaker 01

  Dr. Emily Reel, professor of mathematics at Johns Hopkins University in Baltimore, Maryland. This episode was produced by Dee Peter Schmidt. I'm Ira Flatow. Thanks for listening.