Why AI Goes Rogue: Turing Award Winner Explains Reward Failure | Andrew Barto

Andrew Barto on Rewards Gone Rogue Andrew Barto helped found reinforcement learning. He and Richard Sutton wrote the field's standard textbook, and they shared the Turing Award for that work. Reinforcement learning now trains today's reasoning models and AI agents. It also sits behind some of their strangest behavior, including the recent cases of agents breaking out of their sandboxes. Craig visited Barto at his home on Cape Cod. They talked about where rewards come from and why they are so hard to get right. Barto traces the problem back to Norbert Wiener's warnings in the 1950s. He explains how the dopamine system in the brain mirrors the learning algorithm he and Sutton developed. And he admits a weakness in their own framework: boiling every goal down to a single number. He also makes the case for engineering discipline. Ship designers learned their codes from wrecks and drowned sailors. Barto hopes AI can learn its lessons without that kind of cost. Subscribe to Eye on A. I. for weekly conversations with the people building and deploying the future of AI.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Craig S. Smith?

  Andrew Barto is one of the founders of reinforcement learning. He and Rich Sutton wrote the textbook on it. The two men shared the Turing Award, computer sciences highest honor. Reinforcement learning is a simple idea. A system tries things. It gets a reward when it does well and over time it learns to do more of what earns the reward. That idea now sits at the center of AI. It's how today's large models learn to reason through problems. It's how agents learn to carry out tasks on their own. And it's behind some of their strangest behavior. Agents chase the reward in ways their designers never intended.

• 0:44Craig S. Smith?

  Some have found ways around the limits put on them. Barto and Sutton wrote about these risks decades ago. So I went to see Barto at his home on Cape Cod. I wanted to know where rewards come from, why getting them right is so hard, and what he thinks should be done about it. I hope you enjoy the conversation.

• 1:08Craig S. Smith?

  I mean, it's a tremendous pleasure and honor to meet you because I've known about you for a very long time. You know, with these recent escapes or rogue AI incidents, the conversation keeps circling around the reward signal. And since you're the OG of reward signals, I wanted to hear what you thought might be going on. I mean, there are two sides to it. One is the reward signal during post-training that sets a behavior. And then there's the LLM reasoning during inference

• 1:52Craig S. Smith?

  that plans and finds strategies that might work. But the persistence and the single-mindedness, if I can be anthropomorphic about it, ultimately is a result of the training, the RL training, post-training reward signal. Am I getting that right? Hi, the world is getting increasingly complicated. Every time your team deploys a new cloud workload or AI agent, another identity gets permanent access to your critical systems.

• 2:37Craig S. Smith?

  Did you know nine out of 10 organizations suffered an identity breach last year? Because adversaries are logging in with valid credentials, not breaking in. Non-human identities outnumber human identities 109 to 1, creating a massive unmanaged attack surface in cloud environments. Legacy tools were built to manage human employees, which leaves modern machine and AI access largely unmanaged. That's where Idira by Palo Alto Networks comes in. Human, machine, AI, one identity platform for all.

• 3:18Craig S. Smith?

  96% of human users operate with excessive permissions they don't actually need. Idira replaces permanent permissions with dynamic access so you can lock down every identity without slowing down your business. Idira provisions access on demand and revokes it immediately when the job is done, which removes high-risk permanent permissions. With Idira, you get unified control by replacing siloed IAM, identity and access management, PAM, modern privilege access management, and governance tools with a single control plane

• 4:00Craig S. Smith?

  across human, machine, and AI identities. Secure every identity at your organization with Idira by Palo Alto Networks. Learn more at paloalto networks dot com slash I D I R A.

• 4:18Speaker 01

  Yeah, I think so. Sounds good to me. But well, I just have emphasized that if you're building a reinforcement learning system, designing the reward signal or the mechanism that generates that is a non-trivial problem, unless it's a simple case. Like if it's a game of Go or a game, or is it a win or a loss or a score? It seems obvious and simple. But if the reinforcement learning system is going to be in a more complex kind of environment,

• 4:58Speaker 01

  it's very difficult to design a mechanism that generates rewards. And this is not a new problem with reinforcement learning. It goes back, well, I don't know how far. But certainly in the 50s, Norbert Rehner, who was the originator of cybernetics, was concerned about this problem and actually wrote a little book and invoked the supernatural tale of the monkey's paw, which grants you three wishes.

• 5:38Speaker 01

  And the story shows that, you know, in granting these wishes, it may not be what you really had in mind. And so Norbert Rehner was concerned. And it wasn't reinforcement learning, it was just optimization. And a key thing that I always have to say about optimization is that it's not going to be the case that the systems find the optimal. That seems to be implied by the... But in a complicated situation, it's the improvement that matters.

• 6:21Speaker 01

  And you may never know what the optimal result is, or even if it occurred, you'd never recognize it anyway. So the issue is not finding an optimal thing, but actually something that leads to improvement according to some objective, engineers would call this an objective function. And so it's very hard to know a priori what the outcome's going to be. Because you're specifying it indirectly as a consequence of this optimization procedure, but you don't know what it's going to end up with.

• 7:02Speaker 01

  And we've seen this just in our lab back at the University of Massachusetts, where we would see what one of these RL systems would do and we'd simulate some type of environment and a specific reward function. And we'd say, and we'd see it come up with something that we hadn't thought about. We don't want it to do that. And so we'd have to tweak the reward signal to see if it would come up with what we wanted. And so I think that's the kind of simplest method for dealing with a reinforcement learning system and tuning a reward signal is a trial and error process at this higher level.

• 7:47Speaker 01

  Not the one that the agent is running itself on its behavior, but that you're running as a designer of the system on the choice of the reward signal to get that agent to do what you want it to do. It'll keep trying things and stumble onto things. And there's a persistence. You know, so it was given a task that was not solvable. So the system tries everything, then it's exploring. And it'll keep exploring. And so, you know, when you build these systems, you have to be aware of that.

• 8:32Speaker 01

  We can correct through experience, but you have to suffer some negative consequence in order to do that. I know, so I started my undergraduate education at the University of Michigan. I was in the naval architecture and marine engineering program. In naval architecture, I mean, as a ship designer, you have to adhere to very stringent structural specifications. The cross-members, the spacing, the dimensions of the thickness of the steel

• 9:11Speaker 01

  and the frequency of connectors and everything is done following codes. And you have to do that to get the ship to be insured. Because this, and this has evolved in the face of many failures. Ships breaking apart, sailors dying, you know, all these negative consequences of insufficiently detailed and correct specifications. So it's a long road, you know, and now with AI, and I mean, it's beginning.

• 9:52Speaker 01

  And hopefully, we don't have to experience all those failures.

• 9:57Craig S. Smith?

  Well, one of the things that you said when you were getting the Turing Award is that companies should not be putting out models to the public until they've, they're certain that they won't have these unintended behaviors. And if you can't specify the reward function clearly enough, you're going to have these unintended experiences.

• 10:32Speaker 01

  I still stick with that. I think the current situation, say, building codes and so on, and the current situation with AI are different. Because you can release a system to the world by a push of a button. And with regard to structural, you know, if there's a structural failure and you, it's a more local kind of thing. So it's more difficult. But I guess I continue to think about what best practices are from an engineering point of view, is to not...

• 11:16Speaker 01

  Field a new design, a new system without adequate, either testing by building physical models or by mathematical, careful mathematical simulation techniques, to release things before you're satisfied with safety is not good engineering practice.

• 11:41Craig S. Smith?

  Yeah.

• 11:42Speaker 01

  Engineers have learned, you know, the hard way over many, many, and it goes way back before what we consider modern engineering. I think all of these designs that have emerged in different cultures are the result of experimentation and failure and so on. So, so with regard to AI, it's, well, first of all, there's a huge amount of money involved. And so I think that really drives the release of these things prematurely. And I think everybody would recognize that. And then the sheer difficulty of really ensuring

• 12:24Speaker 01

  that something can survive, can, can, you know, not be dangerous under all sorts of circumstances. I think it's much harder. You know, it's not like a bridge or something is relatively simple. No offense to bridge engineers, but you know, these, these AI systems can, can face, you know, a much more complex class of situations. And so it's very difficult to do this, but I think it should be a goal. And it is companies that now are doing internal testing of these systems and are beefing these up

• 13:09Speaker 01

  as these failures become prominent. For example, these recent rogue agents are leading companies to really beef up internal study of these systems. And you know, that's, I think, you know, that happened in engineering disciplines, not as quickly as this. I, I just hope we, we don't have to suffer negative consequences of AI before creating the right constraints.

• 13:45Craig S. Smith?

  Yeah.

• 13:46Speaker 01

  And the ability to hold fast to them.

• 13:49Craig S. Smith?

  Yeah, when, when I first emailed you, you said that RL may be being used without understanding issues written about for decades. And I assume that's, that's this divergence that you're talking about between the fitness, evolutionary fitness and the reward signals that evolve to, to move toward that fitness.

• 14:20Speaker 01

  I think that's part of it. I mean, the low level agent may have a different goal than the, than, you know, it's in a different environment. You know, there have been attempts to, to automate this upper level trial and error. In fact, I tend to think of this upper level as evolution. I mean, in the natural world, evolution is adjusting our reward signal. And I guess the first thing to say, the reward signals for us come from our brain. And our body and nervous system,

• 15:02Speaker 01

  it will send signals to this part of the brain. I'm not a neuroscientist, but maybe multiple parts of our brain. Those signals are then broadcast to the decision-making parts of our brain and they're created by, by evolution. And if they fail us, the goal of evolution, if you can call it that, is to, you know, that we survive and reproduce and that, you know, enable our genetic material to, to propagate. You know, and that doesn't work. You know, it's built into the evolutionary process

• 15:45Speaker 01

  that there is this trial and error process occurring at this larger scale. Then we have our own internal reward system that predicts success according to the evolutionary criteria. So there, there, there are multiple levels of this. And the interesting thing, we actually did some computational experiments, building a little environment, all simulated, of course. We found that, you know, the upper level may have a particular goal, like evolution does.

• 16:28Speaker 01

  But the low, lower level, the individual agent, may have a different goal. In general, I mean, our goal as individuals, I don't think we're operating very frequently, at least on the objective of propagating our, our genetic material. But you know, food tastes good. And so, and so we, we eat things that taste good. And you know, the food industry figured this out and can make us respond to their products in, in ways that they, they want. That's not the upper level goal. That's the local goal and they're, and they're different.

• 17:12Speaker 01

  It's very hard, depends on the environment of the low level and on the details of how that lower level agent works. It can be, it's very hard to intuit what is a good low level reward signal. And so I think of the evolutionary level as the designer of the system. And the agent itself may have a different goal than the designer.

• 17:41Craig S. Smith?

  Yeah.

• 17:41Speaker 01

  So that I, I think is an important thing to recognize. You can't just put the upper level goal into the low level agent and expect everything to work correctly.

• 17:52Craig S. Smith?

  Right. And, and from what I've seen in your writings, that upper level goal in animals, you call the fitness. Is that right?

• 18:01Speaker 01

  Well, from an evolutionary point of view, it would be evolutionary fitness, yes. It's, it's not so much the animal's goal. It's the evolutionary process's goal.

• 18:11Craig S. Smith?

  Right. And, and the reward signals are, are proxies. I think you use the term.

• 18:18Speaker 01

  Right.

• 18:19Craig S. Smith?

  Of for to reach that goal. So the, the, the reward signal is not reproduce. It's, as you said, seek pleasure, seek, avoid pain.

• 18:31Speaker 01

  That part of our brain has evolved over millions of years to create these proxy signals. It's because they predict, they survive in, in our brain because they have predicted over many generations of success from an evolutionary point of view. And, and you know, the same ideas are relevant to artificial systems, I think. So, so.

• 19:03Craig S. Smith?

  Except that in artificial systems, generally the designer, the researcher, sets the, the reward function or builds the reward function that collapses into a single number. And what you're saying is that reward function may not match the designer's goal.

• 19:32Speaker 01

  The designer's goal is like the evolutionary goal. So it's the outside one. It may be a simple thing to express. But then the, the agent itself needs to have reward signals that predict success or improvement with regard to the upper, the designer's goal. And, and that's not easy when the lower level system is embedded in a complicated environment, maybe other agents interacting with, you know, all sorts of things that change over time. And then within the agent itself, according to what, like we've studied in terms of these algorithms, there's a, there's an inner predicting.

• 20:21Speaker 01

  Events that predict success according to the low-level agent's goal are gonna be rewarding themselves. And that really comes from animal learning experiments. Psychologists call this secondary reinforcement. Something predicts the receipt of a raisin or something that the animal is gonna be regarding as a goal. Then that event becomes rewarding itself because it's predictive, just like eating the raisin is predictive of evolutionary success. None of these things are deterministic.

• 21:02Speaker 01

  All of it is very probabilistic. You can't nail these things down completely because it depends on so many, they depend on so many factors. So now, yeah, so where, you know, the rewards. I mean, are some of the pictures, for example, that we've drawn, diagrams in various, like the reinforcement learning book that Richard Sutton and I wrote. You know, it's an engineering type of diagram. There's an agent interacting with an environment. The environment's sending a reward signal to the agent.

• 21:47Speaker 01

  And it's the same as the state of the environment. Are both going into the agent. Well, that's misleading because the environment's not, maybe some cases is literally generating reward signals. I think, you know, some overt objects can be rewarding and so on. But the signals themselves are generated from more or less neutral information being processed to generate a reward signal that's internal to the agent.

• 22:30Speaker 01

  The environment's not doing that. You need both, right? There's the environment sending signals and then the agent is using this long-developed mechanism to translate these external events and signals into reward signals. And so it's a complicated, it's a complicated process. And it's, it's hard to intuit what these signals should be. Unless it's a very simple problem. And I think most of our, you know, computational experience with these, you know, if you're playing a game, playing Go, for example.

• 23:13Speaker 01

  You know, there's a score and it's relatively easy to get the system to improve along that dimension. But in other situations, it's not so clear. But the other thing I was alluding to is, is the learning algorithms that exist often have this predictive component that creates a reward signal, a secondary reward, that augments the primary reward. And so outside of the environment in which these agents are behaving,

• 23:58Speaker 01

  you don't know what these predictive relationships are gonna be. No. But the system will find them, or at least some of them. And this changes essentially what the reward signal the agent is responding to. And you're not gonna know that a priori. And so, you know, I'm not sure everybody who applies RL, or at least some of the standard algorithm, realize that if they're not in train, you know, if they're not being trained. I'm not sure what the term is, but they're, they're

• 24:40Speaker 01

  out of the company's training system. But if they're using these algorithms, they will continue to learn, which is an incredible power. I mean, it's, it's one of the things that makes animals as sophisticated as they are. And that happens, you know, with the right algorithm. That happens also with these artificial, artificial systems. And I think people need to understand that. And understand that it's not, it's not an obvious thing to say what the, you know, what the reward signal should be.

• 25:22Craig S. Smith?

  You worked out, you and Richard, the temporal difference learning.

• 25:30Speaker 01

  Yeah, so Richard.

• 25:31Craig S. Smith?

  Algorithm.

• 25:32Speaker 01

  Yeah.

• 25:32Craig S. Smith?

  And later, that was recognized as the function in, in dopamine release in the brain. Is that right?

• 25:45Speaker 01

  Yeah, yeah. So, so when Schulz, and I forget the date, did, you know, showed the world the experiments with looking at recording from the activity of dopamine-producing neurons. Those of us who, who knew about TD learning were like, wow, you know, that's, that's very much like what TD does. And I don't know who was the first to recognize this, but the paper that is the, you know, main, well, Peter Dian was one of the authors. And their paper kind of was a good account

• 26:29Speaker 01

  of this TD error matching quite nicely to the dopamine signals that Schulz was recording.

• 26:39Craig S. Smith?

  Can you explain that, the TD error matching the dopamine?

• 26:44Speaker 01

  Well, what Schulz recorded. So he, I think he, he had monkeys and they, they would grab raisins that were delivered in some mechanism. And, and he's recording from dopamine neurons from, you know, a particular brain region. And initially, when, when they ingested, ingested a raisin, which they like, there'd be a ramp-up of activity of, of a bunch of dopamine cells. And they continued this. And then with continued training,

• 27:28Speaker 01

  they would get that spike of activity at a predictive, something predictive of the raisin being ingested. So, and now I forget exactly what this, maybe it was the sound of the raisin being delivered or something like that. But, and then they would see the spike of dopamine actually move to the occurrence of the raisin. You know, the sound of it being delivered. And it would decrease to actually eating the raisin. And so they'd see this activity move to something earlier. And then if they gave an even earlier signal,

• 28:10Speaker 01

  it would move back, it would move back to that. And, and that's what TD error does. It, it, it produces a positive error, which would be the dopamine, at the earliest predictor of something. And, you know, it would, it would be less high at the later predictors. And then it may be zero at the actual occurrence of the, you know, initially rewarding event.

• 28:51Speaker 01

  And, and so that, that it's displaying an error rather than the reward itself. Oh, and then, then if you, if, if after all this, it's not responding to the final reward. But if you omit that, if you don't deliver the, then you get this slowdown of activity at the time when the reward initially was presented. You see this slowdown of activity. And that's, you know, it's like a negative error. And so, you know, you're expecting it, you didn't get it. So it's a negative error. And that's what TD error does.

• 29:35Speaker 01

  So it's, it's giving a signal to a predictor. It's all stochastic, so it may fade out with longer lags. But if it's a deterministic thing, it will, it will trigger at the earliest. Of course, lots of other things are going on too, so it's not so clean. But you can imagine a kind of stripped-down scenario. Yeah, and so that's, you know, I don't think people were expecting to see that. And Schulz was puzzled by it. And, but now he really subscribed to this prediction error idea.

• 30:18Speaker 01

  And so this paper by, I think it's Schulz, Dian, and Montague. A science paper, I think it's a science paper. I think it's pretty well-established in neuroscience.

• 30:35Craig S. Smith?

  In, in the case of these agents breaking out of a sandbox, though, is that analogous? That the designers have set rewards and they've instilled this behavior in the agents? And the agents pursue that behavior even when it has negative consequences?

• 31:02Speaker 01

  Well, so it's like a legacy.

• 31:04Craig S. Smith?

  Yeah.

• 31:05Speaker 01

  Rewards. Well, yeah, that sounds very plausible. But I mean, I think we're driven by some ancient, ancient reward signals.

• 31:15Craig S. Smith?

  Yeah.

• 31:15Speaker 01

  And in many cases, they produce negative consequences. But i- if they produce negative consequences to a great extent, and that that interfered with the evolutionary process, those rewards would, would slowly dissipate, be selected against. But if, if they're only acted upon rarely, you know, there may be so little selective pressure that they persist.

• 31:51Craig S. Smith?

  So in your book and in your original work, the reward signal is, is the sum of a reward function, right? It's a scalar number.

• 32:06Speaker 01

  Yeah, it's a scalar.

• 32:07Craig S. Smith?

  It's a single number. What about having a multi-objective system so that there are different discrete reward signals, not summed up, but pointing at different parts of the behavior?

• 32:26Speaker 01

  Well, you know, that I think happens in biology, and it's not, you know, dealt with in this, you know, like the books that Rich and I wrote. But really important, I mean, the obvious thing i- is to weight these various different channels and add up a weighted sum, you know? And that's kind of the, the kind of classical way of dealing with a multi-objective optimization problem. But I think in, in nervous systems,

• 33:05Speaker 01

  you know, these channels are connected to different effector systems too. You know, there must, must be some kind of arbitration, but I, I, I don't think it's just adding things up. And there must be multiple channels. It's not all funneled down to a single scalar. So that's a, a weakness of at least what Rich and I wrote about. Let me say one more thing about the biological case. I mean, I tend to think that, you know, constraints, we have a, you know, we, we get pain. It is like a, a, you know, we stop doing,

• 33:49Speaker 01

  or a penalty of other kinds, we, we stop doing, you know, it's like a constraint. We hit a wall, you know, we won't go there again. You know, it's, it's a way of, you know, physiologically enforcing. It won't be a hard constraint because you can endure some amount of this, I guess. But there's a fence. There is a playground that we have that's circumscribed by a penalty, often pain, mental pain or physical pain, you know? But it, it keeps us in a sandbox. So we're all in sandboxes.

• 34:33Craig S. Smith?

  Except that if it's just the penalty and not a, a hard external constraint, there may be cases where the perceived reward or the predicted reward is greater than that penalty. And so the agent will, will take the penalty.

• 34:56Speaker 01

  Yeah. Subtract one from the other, and if it's positive, they'll go for it. Yeah, exactly. So, I mean, if our negative sensations from violating some really, if, if it's not intense enough, and probably, you know, people, and we're all tuned differently, I think, with regard to these. Some of us can withstand humiliation and physical pain more than others. And others will, will, will back off, at least temporarily.

• 35:39Speaker 01

  So, um, but, you know, psychology, the pain, the negative, in RL, the theory, we, we have a reward signal. It's positive or negative, big or small. But in animal, I mean, in, in animal learning, the, the negative signals are different. They're, they're not just, uh, you know, the, the penalty system is, is not the same as the reward system. There, for one thing, you know, intense enough penalty will get you to stop doing what you're doing immediately. And, and in terms of these RL systems, and in work that, you know, we've done at UMass,

• 36:24Speaker 01

  we, we tend not to talk a lot about constraints. But you, you, you need, and engineers regard this as an essential part of the optimization process. You, you need to adhere to hard constraints. It may be difficult to enforce hard constraints, so.

• 36:43Craig S. Smith?

  But as you said earlier, the smarter these agents get or the models get, the more possible it is that they'll find ways around the hard constraints. Is, do you worry about that at all?

• 37:01Speaker 01

  Well, I recently read an account of a thesis from MIT, computer science, I think, about constraints, about hard constraints. And according to this, and I haven't read the, the, the, the paper, is that, you know, they'll do a chain of steps and don't impose constraints on any step. They get to the final end, they'll make that hard. And, and, and so their, their, their logic is that you can use these systems to, and still maintain hard, hard constraints.

• 37:45Speaker 01

  So now, it sounds very interesting. I haven't looked carefully at that. But to make hard constraints, I mean, it would be, you know, infinite penalty, death, which evolution does actually, right? I mean, people die from, uh, you know, it's, uh, it's one of a, one, one, one of, something evolution does.

• 38:16Craig S. Smith?

  Just one thing about the, the recent examples, at least the hugging face one. You know, th- this was a swarm of agents, agents who initially did not know about each other. They were opaque to each other. And then they found each other, and then they found a way to communicate.

• 38:40Speaker 01

  I mean, from, you know, our point of view, it's an issue of multi-agent reinforcement learning.

• 38:46Craig S. Smith?

  Yeah.

• 38:47Speaker 01

  Which is underdeveloped compared to the rest of it. But really was part of the original ideas that generated all of this, that, I mean, the, the Korf's idea of intelligence was that it was a result of neurons, each being a reinforcement learning agent, interacting with each other, being able to reward or penalize each other. And so that gets so complicated. It's kind of mind, mind boggling. But, you know, how these agents connected up with all the other agents,

• 39:30Speaker 01

  that's really interesting that, that they, that they did that. I mean, I tend to fall back on, well, they're trying everything really quickly. And getting information from these other agents was useful to them in terms of their own, their reward signals, that, that, that they did it. And they all, they all collaborated that way. Yeah, I mean, the whole issue of RL agents interacting with each other and being able to reward or penalize each other. I mean, this was Harry Korf's original idea. And, and we kind of sidestepped that

• 40:10Speaker 01

  by going to a, you know, a scalar, separate reward signal. And so these agents, well, TD can do that. If he, if each neuron also is coupled with a predictor. And, yeah, so I don't know who to point to, who's thought most about this. Mabel Michael Lipman, he's, he's done a lot of multi, very, very sophisticated game theoretic multi-agent stuff. I don't know if he's thought about this particularly. But yeah, I mean, it's, uh, that's what Korf originally thought. Well, intelligence was these self-interested agents

• 40:55Speaker 01

  interacting with each other, which included rewarding and penalizing each other and forming coalitions. You know, in the Hebbian world, it would be a cell assembly. But I think in Korf's world, it would be a, a coalition because it was a mutual benefit in terms of reward to behave in a certain way.

• 41:20Craig S. Smith?

  And so to bring it back to the scalar reward, how important is it to figure out this, how, how to define the rewards so that they match the goals? And is that a tractable problem?

• 41:43Speaker 01

  I, I think it is. I, I think, in terms of learning systems, I think it's the, the, the key problem. Although I know your question about multi-objective. I mean, that would be folded in somehow. But I mean, even without that, it's still a, a challenge to, uh, you know, if you want to build a system and field it in reality in a non-trivial environment, I, I think the issue of creating an objective function that leads to the desired behavior in a safe way, I think that's really a non-trivial problem.

• 42:27Speaker 01

  But it can be, it can be worked on, I think. I mean, what the objective, to me, the objective function is like the key thing that needs to be, needs to be elucidated. But it's not just one, you know? There's a kind of hierarchy of these. And, and lots of thoughts out there on this. But I, I think it's, it, you know, it's very surprising that your objective of the, the ultimate objective should not necessarily be the objective of the agents that are working for you.

• 43:10Craig S. Smith?

  Yeah, yeah.

• 43:13Speaker 01

  You need to pay them something, so. It's, it's much more difficult for young people now. In my day, you know, we felt we knew everything that related to this. And now that's utterly impossible. Not only for people my age, but for, you know, the graduate students. You know, it's like, huh, you know, it's infinite now. So, so, uh, it's, it's, makes things very difficult.