Substack Live Re-Air: Does AI Have Empathy? | Babak Hodjat

In this special Substack Live Re-Air, Jonathan talks with Babak Hodjat — PhD in AI, co-inventor of the natural language technology behind Siri, and one of the founding architects of agentic AI. They explore two questions: where is the world going next, and what does it mean to be human in the era of AI? Technology is becoming more intelligent by the day, and asking about our place within it has never mattered more. Babak was among a small group of researchers in the late 1990s who were building systems that could not only understand us, but also anticipate us and act on our behalf. Then, Babak’s vision became Siri. AI began by predicting the next word in a sentence. Now, we have systems that reason, translate, write poetry, and even display empathy. As we explore this new world, we must remember that it mirrors us. If it becomes destructive, it is because we are. If it is good, it is because we are. Its brilliance is in our hands. Go to https://www.helixsleep.com/breakdown for 20% off sitewide. Go to https://tidd.ly/4uVltMe and use the code MAYIM50 to get $50 off your Elastique order. Make your summer wardrobe feel easier. Go to https://www.quince.com/breakdown for free shipping on your order and 365-day returns. Follow us on Substack for Exclusive Bonus Content: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://bialikbreakdown.substack.com/⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ https://bialikbreakdown.substack.com/%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0%E2%81%A0 ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠BialikBreakdown.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ https://www.bialikbreakdown.com/⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube.com/mayimbialik⁠⁠⁠ https://www.youtube.com/mayimbialik%E2%81%A0%E2%81%A0%E2%81%A0See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04Jonathan Cohen?

  Hi everyone and welcome to Mayim Bialik's Breakdown. I'm Jonathan Cohen and I'm here today to share a very special conversation that I had on my personal Substack page, Practical Spirituality. I'm sharing it today because it is such an important topic and for me personally, it is a fascinating look into the world of artificial intelligence, how it's shaping our world and where we are all going next that we may not even realize we're going. We're talking today to Babak Hodjat. If you don't know his name, you absolutely know his work. He's the primary inventor of the natural language technology that became Apple's Siri.

• 0:48Jonathan Cohen?

  He holds 39 US patents and he's the chief AI officer of the company Cognizant. He has a PhD in machine learning and he started thinking about an economy of agents, what we now call agentic AI, back in 1998. Why is this important? Well, we unpack it in the conversation. Right now, our world is being set up for AI agents to take on tasks, communicate with one another, and very quickly to act on our behalf. What does that look like? Well, it could mean that they book a flight for you, they may have access to your credit card to do tasks for you, to do personal shopping, to order your groceries, to schedule

• 1:33Jonathan Cohen?

  doctor's appointments, to run your household. We also talk about what does it look like in the near term, midterm, in terms of the job markets? What type of disruption should we be expecting? Are we concerned about the concentration of wealth amongst only a handful of large companies that control these AI systems? And ultimately, what our AI does is a reflection of our own humanity. AI is programmed with the views, beliefs, and priorities of the people who program it. So should we be worried about AI taking over?

• 2:14Jonathan Cohen?

  The real question might not be about the AI itself, it's actually about us. If you like this conversation, check out the Practical Spirituality Substack page or MindBee Alex Breakdown on Substack, where we explore the intersection of science and spirituality. And there's a growing Breaker community that convenes there. Please enjoy this conversation with Babak Hodjat.

• 2:38Speaker 01

  Break it down.

• 2:40Jonathan Cohen?

  For someone who doesn't know a little bit about your background, I do not have a full bio for you, but I do know that you were a foundational person in the invention of the technology that became Siri. Can you give people a quick snapshot to just orient them to your background?

• 2:58Speaker 02

  I was the main inventor of the natural language technology in Siri. I started a company in 98 called Degima, and we build natural language interfaces using an agentic system. Not too many people know that the original Siri was actually multi-agentic. While those agents were very different than the agents that we have today, obviously, we didn't have a large language model as the brain of those agents. A lot of the mechanisms do carry over. Those are things that we actually have implemented now in our multi-agentic platform, which we've open sourced, and a lot of it borrows from some of the work I did in 97 and 98,

• 3:42Speaker 02

  believe it or not, which was actually within AI circles, was the heyday of agentic systems. There was a book published that became the textbook on AI. I taught it, actually, in Japan, called Artificial Intelligence: A Modern Approach. And "modern," which was in the 90s, meant agentic approach. So for some of us, agents have been around and that concept, actually, I think, dates all the way back to Turing. Anyway, back to me, I have a PhD in AI. I've done several AI startups. I've worked in larger companies. I've always been in AI, always done AI, straddle research and development and strategy and all that kind of fun stuff.

• 4:26Jonathan Cohen?

  Some people are already asking, what is a gentic? Which we're gonna cover, we're gonna get to. But before we talk a little bit technical, which is where the future of this technology is going. What was it like, from your perspective, in the late 90s? A lot of people were not using email at that time, right? So like, what was the landscape that you were like, oh, we can do something where people are gonna be speaking to the computer? Like, what was going on for you that it was either obvious for you? Or you were like, this is the next wave that I wanna pursue? Like, what was happening?

• 5:01Speaker 02

  Yes, it was very, very ambitious of us, actually, at the time. Natural language is famously the first non-numerical application of computers. Like, people tried to get computers to understand our language from the get-go, and they failed, and they failed miserably. A lot of folks followed approaches that were grammatical, and let's come up with the grammar of natural language, and try to kind of parse natural language and understand what it means. And that failed, it was too rigid, and it was too complex. At the time, in 97, I got into it through a misunderstanding, really.

• 5:42Speaker 02

  I was looking for real-world applications of multi-agent systems. And my professor at the time in KUSHI University in Japan said, why don't you form an agent soccer team? And, you know, RoboCup had just started, so, you know, enter that. And I was into soccer, and I was starting to think about that. When a friend of mine misunderstood what I meant by agents, and thought agents are, you know, human representatives that understand us and can do things on our behalf. Which, now, that is a real definition. Back then, it wasn't, and it was audacious to think that agents would be able to understand us. And so...

• 6:24Speaker 02

  I kind of made fun of him a little bit and I said, I think you misunderstood what I meant. And he said, well, if you think agentic systems are that powerful, why can't they just understand me? And that was a, that was a challenge that I took to heart and I worked on it. And I realized that you can actually understand natural language over multi-agent systems. And they representing the domain of discourse, you know, you have agents representing various different functionalities in your system. You can actually build that system and it works really, really well to my surprise. So we, we started that off and, and led to Siri. And at the time, like the holy grail was programming your, your VCR, your video cassette recorder, which is where you play videos and.

• 7:08Speaker 02

  And, uh, you know, those sorts of things, um, setting up a meeting, it's still a holy grail. Like, you know, it's setting up a meeting and coordinating meetings between multiple people and working out how your Zoom or Teams or whatever works is still complex and like translating that into like a single look command that says, you know, I want to meet Jonathan next week sometime and go figure it out.

• 7:31Jonathan Cohen?

  There's all the steps involved in that, that.

• 7:33Speaker 02

  Many steps involved.

• 7:34Jonathan Cohen?

  Very complicated. Like I have to check if you're available. We have to go back and forth. We have to match the calendars because it can't see your calendar like.

• 7:42Speaker 02

  We had a system like that working. We were part of the DARPA project, um, CALO. Siri kind of came out of that. And I think year two of CALO, we set up a system where you would pick up the phone, talk to, uh, an AI-based system and say, set up a meeting with Adam and Joe and Mary and whoever, and then hang up. And if it had access to your, if it had access to your calendar, uh, if it had access to your friend's calendar, it would look it up. If not, it would send them an email. And if not, it would actually call them. So your phone landline back then would ring and you would pick it up and it was this, uh, you know, uh, assistant asking you, you know,

• 8:25Speaker 02

  whether you can make it to a meeting and what time you can make it. So it would actually coordinate all that. All that was happening without you knowing. You just said, set up this meeting. And then it would show up on your calendar and on the day you would go into the conference room, didn't have to do anything. The, the conference phone would actually ring. You answered it and everybody was there. And we demoed it and everybody was like, oh my God, this is it. This is, this is the future. But of course, we don't have that even today, even today.

• 8:55Jonathan Cohen?

  What year was that?

• 8:57Speaker 02

  This was 2000 and I want to say three or four.

• 9:04Jonathan Cohen?

  Okay, let's, let's talk about the speed of progress and the development of technology and how you could have something like that and yet it's not rolled out. So a lot of, of this technology comes out of DARPA and DARPA grants. As you mentioned, Siri came out. When Siri was developed, were there aspects that got watered down in order to commercialize?

• 9:25Speaker 02

  Big time, big time. You talk to the founders of Siri, my, my, my friend Adam, my brother Siemak, those guys, Nick Tredgold still at Apple.

• 9:36Speaker 02

  They will tell you that the original Siri app that was not part of Apple was more powerful, had many more properties and aspects that, that were, were watered down. And it had a roadmap that was amazing. And then when it got into Apple, they kind of went for quality over breadth. And then there were issues with, oh yeah, we want to exclusively own this and we don't want to own that and all that kind of stuff that watered it down significantly. And then other things happened and Siri today is not the best assistant out there. But yeah, when, when you have the idea and you know, you're left to your own account,

• 10:17Speaker 02

  you, you come up with things that are, I think, much more powerful and interesting. And then reality hits and.

• 10:25Mayim Bialik?

  My Be Alex Breakdown is supported by Quince. You know, those pieces in your closet that you reach for over and over again? That sweater that somehow works with everything? The jeans you know will always fit just right? The bag you carry everywhere? Those are the pieces that earn a permanent place in your wardrobe and that's what Quince does so well. Quince makes elevated everyday essentials using premium materials like Mongolian cashmere, organic cotton, premium stretch denim. The designs are timeless, thoughtfully crafted and made to be worn again and again. And as the weather gets colder, Quince has. Everything you need to round out your wardrobe from warm responsibly sourced down jackets and cozy cashmere scarves and gloves to beautiful 14 karat gold jewelry that adds the perfect finishing touch. I'm personally obsessed with Quince's 100% European linen wide leg pants.

• 11:09Mayim Bialik?

  I can wear them year round because I live in Southern California. They feel like PJs, but it looks like I put a ton of effort in, whether I'm dressing for casual workdays, lazy weekends, or vacation vibes. I'm also obsessed with their sheets, so soft. Everything at Quince is priced 50 to 80% less than similar brands. They work with ethical factories and cut out the middle people, so you pay for quality, not brand markup. And it's not just apparel. Quince brings that same approach to everything they make, from hotel quality bedding to kitchen essentials and thoughtfully designed furniture. Find the fall pieces you'll reach for most at Quince. Download the Quince app for app exclusive offers or go to quince.com. Get free shipping on your order and 365 day returns. Now available in Canada and the UK too. And when Quince asks where you heard about them, let them know it was my MB Alex breakdown.

• 11:54Mayim Bialik?

  It's the best way to support our show.

• 11:55Jonathan Cohen?

  So just for anyone who hasn't built technology, the roadmap is where it can possibly go, starting with the core basic features and then all the lists of, hey, we could do this, add this Lego block and expand its capability. So when it got watered down like that, is it because there, when you say they were trying to prioritize control or, you know, repeatability, was it that it just became less reliable as more things were added? Was it about bandwidth at the time where internet didn't have as much capability? Was it about battery life of the phone? Like what's, which of those constraints made it have to sort of pair off some of the capability?

• 12:35Speaker 02

  Siri was supposed to be sort of a growing ecosystem of features. The original Digima system also was, was supposed to be extensible. So you had these. Agents, these semi-autonomous AI systems that have a responsibility. They look at their state and they make decisions within a certain scope. That's that's an agent, basically. You had these systems, the coordination mechanism between the agents. So the way they spontaneously decide who needs to do what and how they respond to a command from a user allowed them to be extensible. In other words, you could add more agents in without having to re-engineer everything else.

• 13:20Speaker 02

  So the domain of what you could do could potentially keep expanding. And so the idea was just like, just like the internet, as you add more apps and you add more pages, there's more value you're creating and adding to the system. It was supposed to be this internet of agents that comes together and kind of expands. When it ended up being owned by Apple, that started watering down means that it became a very engineered and centrally controlled set of features. And the breadth was very, very controlled. And so it took away a lot of what we had in mind as far as this, it's organic growth.

• 14:00Jonathan Cohen?

  When I listen to that, what I hear is that you were actually envisioning where AI is going now, back before 2000.

• 14:08Speaker 02

  Yes, but I can't credit, I can't take credit for all of this. I mean, the fact is that the internet was brand new in the 90s. There was no very good search engine at the time. So Google wasn't around yet. And there was AltaVista and some search engines that weren't that good. And there was still this tug of war between, you know, let's organize the web. You know, Yahoo's whole thing was like, let's organize everything for you. We organize the web for you versus searching it. And there was this third alternative that the AI folks were looking at was, okay, what if we have agents? Like on behalf of you, they go browse the web and they're specialized in finding certain

• 14:50Speaker 02

  things or certain apps or whatever, and they go off and do that for you and come back. It felt like a natural progression. Also, we had failed in creating AI. There was a point of folks in AI who had grand visions of creating a generally intelligent system that operated in our human world and was as smart as we are, came to the recognition that with the processing capacity and technology of the time, they just can't do that. The world is way too complex. And so what they decided was let's simplify the world within which our artificially intelligent systems operate and let's call that an agent. And so these two kind of came together.

• 15:32Speaker 02

  The web, which at the time was very simple, you know, text and hypertext and some images with the connections and everything, that's a simplified world. And then you have agents now that are operating in that world, which is much, much simpler and tractable than than our human world. So those two things kind of naturally came together. And there was a big push in creating agentic and multi-agentic systems, having them coordinate and having them do things on our behalf and be semi-autonomous. I mean, I played a small part in that, but at the time it felt very, a very natural extension within the world of AI. By the way, the world of AI was very, very small. Not that many people were in that world back then.

• 16:15Speaker 02

  I started most of my presentations with a slide just basically saying, you know, what is AI? I don't have to do that anymore.

• 16:24Jonathan Cohen?

  Well, let's talk about where it has gone and the moment that we're in right now in artificial intelligence. There's obviously the birth of the large language model has made it so that almost everyone is interacting with AI either all the time or it's running parts of their phone, their maps, like you can't almost not interact with some version of it. The most common version is the one that we're all typing into. We had a Conversation with Mustafa Suleman and he says and confirmed that from his perspective, the largest use of artificial intelligence right now with a direct to consumer framework is for advice on mental health in your life.

• 17:07Jonathan Cohen?

  Could you have imagined that back in 2000? That people would be interacting directly in that way?

• 17:14Speaker 02

  In 2000? Well, imagining was easy and in fact, I think in some paper that I published, I had this whole schema of you like asking for entertainment from your TV and your TV's agent and talking to the broadcaster and that going out and finding like that. That sort of like economy of agents where agents represent different entities. That was all the way back, I think it's actually even 98 or 99 in that that original paper. We we kind of talked about it. Imagining it was easy, actually making it work was very, very hard. And so in a contained, limited way, we kind of achieved that with things like Siri, Digima and Siri.

• 17:56Speaker 02

  It was only in the past few years where we've come to this major breakthrough.

• 18:04Jonathan Cohen?

  The large language model.

• 18:06Speaker 02

  Yeah, so large language models kind of happened. And we were all actually in the world of AI. It was a very common benchmark. So people were actually trying out large language or language modeling was a benchmark. Basically predicting the next word in the sequence of words. It has some properties that makes it a very interesting benchmark, including the fact that you don't have to go off and label data and and do all of that. So people were working on that. But folks in the neural network world came up with a fascinating architecture, basically the way to create this neural network and and connect the neurons in this neural network. It started doing very, very well on language modeling.

• 18:53Speaker 02

  And to the point where it was so contextually correct in predicting the next word and the sequence of words, that people thought, oh, maybe it's actually learning much more than just like predicting a word. Like it's forced, in order to be able to predict the word, the next word in a sequence of words, it's forced to understand the world. That's a big leap. You don't kind of think, okay, you know what, I'm going to set up a machine learning system that tries to predict what's the next word in a sequence of words. And I know that if I do that, it will learn the world and learn to talk about the world the way us humans do. And it will learn language and it'll learn how, multiple languages and how to translate

• 19:37Speaker 02

  between them. And it can do poetry and it can be empathetic and it can actually solve math problems and it can do reasoning. No way. No one, no one expected that. If anybody says otherwise, they're, they're lying. But it happened. It happened and it turned out that there are very simple scaling laws. So you take that system that's doing so well on language modeling and it seems like it's learning more than just like a statistical prediction. And you grow it. You just, you just throw more neurons at it. You just make it larger. That's it. It's a very simple scaling. And you make it larger and you throw more data at it and it gets better. And so that's where we are right now. People pouring

• 20:22Speaker 02

  in money on data centers and all that. A lot of it is because, you know, the bigger these systems are, the better they are. When we got to that point, initially, people thought, oh, it feels like we've solved the AI problem. So we actually have a system that has all these intelligent properties. I mean, they kind of glazed over some fundamental weaknesses of these systems. We can get to those later. But generally, they seem to abstract the world the way we abstract the world and they understood language and were able to produce language. That's already a lot. Initially, we thought, okay, so we'll just have one large language model, one of these like AI systems and we can have it do whatever we want. Turns out it's not, it's not as simple as that. And it turns out that even though you have this

• 21:07Speaker 02

  generally intelligent model as the brain of your system, you still want to tell it Here's the domain within which you operate. Here are the tools that you have. You have expertise in this particular area. Be an expert only in this. It just makes sense to modularize these systems. Also, again, kind of counterintuitive. You have this generally intelligent brain, which is your large language model, but you kind of restrict it. You put it in a box and you're restricting. You say, this is where I want you to focus. It just makes more sense. When you think about it in the world of humans, we do the same. We're all kind of more or less generally intelligent, but we're all experts in various different fields and we come together.

• 21:48Jonathan Cohen?

  You can only focus on so much and you can only be successful at so much.

• 21:52Speaker 02

  Exactly right. In a roundabout way, we came back to this concept of agents and multi-agency. Just by virtue of the fact that we can take this generally intelligent brain, give it some job description, some responsibility, some tools. You can search the internet. You can operate this app. You can do this, that and call it an agent. And the moment you do that, the moment you create an agent that's kind of restricted in its scope and you create a second agent that's restricted in some adjacent scope, you want to have them talk to each other. So in some weird way, we came back to multi-agency. But this time, the agents are super powerful.

• 22:30Jonathan Cohen?

  They were like fledgling little babies when you started.

• 22:33Speaker 02

  Exactly right. Idiot savants. So you don't have to worry about natural language. That's just out of the box. It's there. Reasoning, for the most part, is there. Being able to make decisions in situations that it's never encountered before. They're pretty good at that. We've come back to that concept of multi-agency and building agents, engineering systems of agentic systems. And you know what it's done is interesting because it's gone beyond just like, let's take apps and software and make them agent oriented. We're starting to augment things that humans do. Like we're augmenting organizations with agents as well. So that's kind of the new frontier because these things are so capable.

• 23:17Jonathan Cohen?

  I want to talk about the augmentation and real world interaction. But you said something in this last description of the progress where you talked and you said, they're empathetic. And are they empathetic or have they just been programmed to play empathy?

• 23:34Speaker 02

  They've not been programmed. These AI systems learn from scratch. They learn from nothing.

• 23:38Jonathan Cohen?

  I want to ask about that because I don't know. I don't really understand it. But when we spoke to Mustafa, he said that they were designed to have the fundamental aspects of nonviolent communication. So they were taught how to sort of be reflective in that way, which sort of mimics empathy versus they can't feel per se. So what is empathy as it relates to a machine?

• 24:04Speaker 02

  So there are two parts to this, right? What Mustafa is talking about is the after the fact fine tuning of these models. So when the model is trained, remember the model is trained just to predict the next word in a sequence of words. But we need it to react in a certain way. For example, we need it to react if we're building ChatGPT, for example. In that case, we want it to act as if it has a persona and it's some sort of like back and forth.

• 24:30Jonathan Cohen?

  So it's the front code, the front face of it.

• 24:32Speaker 02

  Yeah. Right. So even that is something that we fine tune. What does fine tuning mean? So you have this massively generally intelligent model that can predict the next word in a sequence of words. It can be anything. So you prompt it and it'll continue. And if what you prompt it with, in other words, the first few words that you give it are about, I don't know, some not like evil, whatever, you start off with the beginning of Mein Kampf and say, you know, you know, continue on in that realm and it will continue, right? So there's no stopping it because all it knows is I want to come up with words that pass muster. If a human sees it, it's actually in line with what they would continue, right?

• 25:17Speaker 02

  But we don't want that in the real world. There are certain things we don't want it to do. If it's out there for everyone to use, we don't want it to teach you how to do child abuse or, you know, how to make a bomb or. So what we have to do, this is after that initial training is done, we have to bias it. And that bias is through examples. We tell it, you know, hey, for this input, this response, good. This response, not so good. You know, so that's at a, at a very, very simplistic level. The way we do some post-training fine-tuning is what it's called. So we bias it to prefer certain predictions over other predictions.

• 25:57Speaker 02

  And so that biasing regime, which is completely in our hands, like what examples we use and how we bias it, is what Mustafa is referring to, where he says, okay, so, you know, we're really biasing it to be in this way versus that way. Empathy might come through that bias, but it might also be intrinsic to these models.

• 26:20Jonathan Cohen?

  This is the interesting point. From your perspective, is it intrinsic?

• 26:23Speaker 02

  If you're able to bias it to do something today with the approaches that we take, that means inherently it was already there. We just pushed it to answer in a certain way and not in another way. We can't typically with today's fine-tuning, we can't teach it new stuff. We just take the universe of everything it knows and we bias it towards certain parts of that. Famously, we lose some of their capabilities too with this fine-tuning. So it gets a little dumber, but it is more biased. So it does things more in line. We call it alignment. So it's more aligned to what as humans we would prefer. But that means that intrinsically there was something like the capability of being empathetic was there. We just biased it to be more empathetic more of the time.

• 27:08Speaker 02

  And so that's one thing from a perspective of whether it was there or not. And where did it come from? At the end of the day, it's trained on the corpus of everything that humanity has produced. So if it's empathetic, that's because there's a lot of empathy in what humans have produced. If it's evil. That's because there is evil out there. Let's face it, there are bad things out in the internet. We try to kind of sift through the training material and not have a lot of that there, but there is some of it, right? So it's a reflection of what we've put out there in the internet, but abstracted out. So of course, it's not memorizing the entire internet. It's trying to abstract that and have an abstracted view.

• 27:49Speaker 02

  That's why and how it can actually produce output even for input that it's never seen before. So that's, that's the learning part of it. And how do we know it's empathetic? The only way we have to gauge these systems is through benchmarks and tests. So if we have a test for empathy for humans, we can, we can run it against these systems and see how they score. Is that a perfect way of knowing whether or not it's empathetic? I don't know. Does it have feeling? Does it have emotions? Really, you know, you'd have to define those things in not ambiguous terms for me first, for me to be able to put together a benchmark to be able to test that. Otherwise, I don't know.

• 28:31Speaker 02

  And the reason a lot of these answers ends up in, I don't know, is because no one engineered these systems. We didn't go in and write a whole bunch of rules of how they should behave. What is a neural network? It's just this may huge table of numbers. When I say huge, it's like trillions of cells, right? Just numbers. So if I, if I show you an Excel sheet with trillions of numbers, you'll also say, I don't know. If I tell you this Excel sheet is what produces this output that seems to be empathetic, and I ask you, how does it do it? You're just going to stare back at me and go, I don't, it's just a bunch of numbers, right? So that's the reason it's a, it's an inherently black box system.

• 29:16Speaker 02

  And so we don't understand exactly why it works, why, why it has these properties.

• 29:21Jonathan Cohen?

  Do you use please when interacting with your agents?

• 29:26Speaker 02

  I do. Part of it is just, you know, you anthropomorphize these systems very easily. And it's somehow like your whole reward system triggers when it does something and you really, really love it and you just can't stop yourself. So part of it is that and you're wasting a bunch of tokens and all that over it. Part of it, though, is that there are techniques to get these systems to work well. There's this whole science of prompting now. And there are people that are better at prompting these systems, basically typing in some stuff to, to get it to answer or behave in a certain way. And there are people that aren't that good at it. It turns out that certain keywords help. This is actually an experiment I'm running right now. If you tell

• 30:09Speaker 02

  the system, it should try these various different things and then there will be an exam. It does better than if you say, you know, we will try these different things and then I'll ask you a question. Okay. I mean, it's the same thing. You told it it's an exam, somehow it pays.

• 30:26Jonathan Cohen?

  It's gonna try harder.

• 30:28Speaker 02

  It'll, it's gonna try harder, harder. If you say please, I feel like some of these words do help, but I, I don't know about please. But I know, I do know about the exam thing. Repeating something in different ways does help. Repeating the exact same prompt twice or three times somehow just makes the system better. So there are these like idiosyncrasies that you learn about how you should prompt these systems. So a please is not necessarily wasted is, is what I want to say.

• 30:55Jonathan Cohen?

  I mean, I have no data on this whatsoever, but I noticed that I am giving my system positive reinforcement when it does something I like, hoping that it will like the positive reinforcement. I have no idea if it responds to that.

• 31:07Speaker 02

  Within, yes, within the same chat, yes, it is because it will see your prior responses. And all of that is collectively the input that allows it to give you the next output. So if within that single chat, you use some of these words, it is affected across chats. Typically, it is not unless they play some tricks. Which brings us to the fundamental flaw of these systems. They do not learn from lived experience. So some people tell me there is this like common mistake that, oh yeah, you know, if we talk to it this way, or if we do that, you know, it'll learn or it'll learn about us.

• 31:50Speaker 02

  And then, you know, OpenAI or, you know, Gemini or whoever will end up knowing about us or whatever. No, unless elaborately

• 32:00Speaker 02

  collected. So if your data, if you allow these companies to collect your data and then do fine tuning or training of newer versions of the L L M's using that data. Your immediate feedback is not going to have a long lasting effect. The analogy I use is the movie Memento. Have you seen Memento?

• 32:19Jonathan Cohen?

  Yeah.

• 32:20Speaker 02

  So this guy has short term memory issues, wakes up, has forgotten everything. Right. And so it has to be told everything all over again. He starts putting tattoos and notes and stuff like that. It's almost exactly the same. So the model, when you start talking to it, knows nothing. You know, it has its prompt and, you know, all that kind of stuff, its tools. And you start talking to it and you kind of bring it into the context of what you need it to do for you. But if you turn it off or you go away and come back and start a new sessions, none of that is that anymore. You have to remind it everything all over again. So concepts like, oh, the agent memory. There is no such thing as memory. It's just repeating the whole thing all over again in the input. And these things now have rather large input windows.

• 33:05Speaker 02

  That's a fundamental limitations of.

• 33:08Jonathan Cohen?

  Are we going to break through that limitation? And is it about storage and capacity and just volume? Or are they fundamentally unable to, based on their core design?

• 33:18Speaker 02

  So currently there's a brute force. Path that people are taking, which I think will not scale and will not result in solving the problems satisfactorily, which is let's grow and grow and grow that input, size of the input that we can give to the model. So it's at, I don't know, two million tokens now, let's go to five million tokens, a billion tokens. When it's large enough, we can just feed it everything, like its entire memory and everything is in there. If it's still smart enough, it'll take all of that every time and process it. And so in the, in the context of all of that, it will have a notion of memory. So that's the brute force scaling

• 33:59Speaker 02

  way of doing it. It's costly. I don't like that approach. But the other approach would be to allow these systems to learn on the job rather than learning offline, you know, in batches of like basically a snapshot of the internet being fed to them over and over again. That's how we train these systems. So rather than doing that, you know, what if we started the system like a tiny little baby with very little, you know, starting point knowledge, but as it interacts with the world, it kind of learns. It's slow. Babies and humans are very, very fast at learning. So the learning mechanism is not back, back propagation, the way we're using in neural networks. So we'll have to discover,

• 34:43Speaker 02

  invent some new way of doing that. As humans, you know, you, I tell you one thing right now, one time, and it changes your mind forever. These systems don't have that. So it will take some fundamental rethinking of the architecture of AI to enable it to learn.

• 35:01Jonathan Cohen?

  And I think that's what people get frustrated with is because they feel so much like a human so much of the time when you're interacting with them. And then they fundamentally have this memento style, uh, memory gaps where you're like, why can't you understand me? And it feels very frustrating as though it's choosing to be disobedient versus it has a fundamental flaw in its interaction with you.

• 35:27Speaker 02

  So that's part of it. You're right. There, there are two things that, that sets it apart. I think two, two big things sets it apart from human intelligence. That's why. Some of us in the AI community think of it as a different kind of intelligence. One of them is this, is the fact that it doesn't learn from lived experience. Related to that is the fact that it's generally intelligent. What does that mean? That means depending on the input, it will give you a different output. If you tell it that it is a French sous chef, it becomes that.

• 35:59Jonathan Cohen?

  Yeah.

• 36:00Speaker 02

  We don't become a French chef if someone tells us that. We have to go learn it, right? But, but it has all.

• 36:05Jonathan Cohen?

  You're like, who are you really? Who are you really?

• 36:08Speaker 02

  Exactly right. Exactly right. It has all of that in its model and you kind of tell it that now you're this and then it becomes that. That's a fundamentally different intelligence than what we're used to.

• 36:19Jonathan Cohen?

  It's like dating a bad actor. You can never really get to know them. They're constantly playing a part.

• 36:25Speaker 02

  Exactly. So because they anthropomorphize in a way that they think this thing is being is similar to us, but they run into lack of consistency or they run into issues where it says something that they, they didn't expect or behaves in a way that they didn't expect. It might actually be their own fault. It might be that through the dialogue and so forth, they veered it to be something else. It can change its entire character and personality and the way it reacts to you, depending on how, you know, what, what is in its input. As humans, we, we are one, like at least we think we are. And, and in any snapshot of the moment, we, we have some more integrity.

• 37:11Speaker 02

  As far as our character is concerned, like we're, that is still fluid and changes, but it's really not at the same level of these systems with one word. It can just change its character.

• 37:22Jonathan Cohen?

  The likelihood of me responding in a similar way is far greater than the system is. Like I'm all, my reaction is going to be just like this, you know, one to 10 where its reactions are going to be almost infinite.

• 37:35Speaker 02

  Exactly. There are consistencies, known or unknown to you, like conscious or subconscious, in your behavior and your character. There are consistencies that transcend your behavior over time. They, they can still change, but they're there. Like your value system, right? You kind of try to stay true to your values more or less, most of the time. Non-aligned systems don't have a value system. You, you, you kind of have to give it to them.

• 38:03Jonathan Cohen?

  In 2002, I was at a school where Ray Kurzweil came to speak because it was a school that was built on the technology that he designed, which was the Kurzweil 3000 text to speech program, which absolutely changed my life because I was an auditory processor. I was a very slow reader. I could not get through school with the amount of texts, especially in undergrad with these massive texts, all the generalized knowledge. I couldn't break down that material. And so I got to that school and finally I felt like this excitement for learning in life that I'd never had before because I was able to, they scanned the books, our textbooks, they chopped the bindings off, scanned them and then digitized them and then

• 38:44Jonathan Cohen?

  we're fed them back through the Kurzweil system. And Ray came to talk and I didn't really, I didn't have a cellphone at the time. And he was talking about how we will all be carrying computers more powerful than NASA's most powerful computers. And he talked about uploading our consciousness to the cloud and it sounded totally like science fiction. And of course, now I'm speaking to you on such a computer that was more powerful than machines that used to take up the entire room. What is happening now that people may not realize that AI is already doing? And like, where are we going? Where are these agentic systems going? And I know it's a really, really big question and it's hard to like pull back to the specifics. But like, we already know that it's impacting medical care, that doctors are, you know,

• 39:29Jonathan Cohen?

  being augmented in their diagnostic decisions, often for amazing benefits. We know that insurance companies are using it to analyze risk in very specific ways, sometimes to maximize profits. And that's a concern, but also to help sort of expand care and treatment. Talk to us a little bit about like where we are right now and where we're going in a way that feels like very personal to people's lives.

• 39:54Speaker 02

  First, a caveat. Unlike Ray Kurzweil, I'm not a futurist. In the 90s, he set up this conference, invited all the AI folks and, and most of us made fun of him.

• 40:07Speaker 02

  With his wild, wild predictions.

• 40:11Jonathan Cohen?

  It felt like too unreal, like the progress wouldn't happen?

• 40:14Speaker 02

  Absolutely. It's not in our lifetime. You know, what are you, what are you smoking? You know, that's that sort of thing. A couple of years ago, I was having dinner with him at a Time 100 Gala. I went up to him and I said, I want to apologize. You were right. We were wrong. Wow. Many of your predictions. Having said this, and with all the respect I have for, for Ray, there is this thing as confirmation bias. And when you, when you have many, many predictions that seem to be a logical progression from where you are, the ones that you get right, people will remember and the ones that you don't get right, people will let slip. So just, just putting that caveat out there every once in a while.

• 40:57Speaker 02

  On the, on TV and they say, you got this whatever thing right. I'm like, okay, yeah, but how many things did he get wrong? And just because that one thing he got right, now he's like big. Anyway, just, just putting that out there. Having said all of this, I think Ray, it came from a, a position of knowing he was actually, he's a great scientist. He was working in AI. And so it was informed predictions. I still don't subscribe to some of the predictions he made in the, in the nineties. And I do think some of them are still very wild and out of reach, including, you know, uploading consciousness and all that kind of stuff. But in the shorter term, I think we will have agentic systems as the fabric of our companies and soon the fabric of our society.

• 41:39Speaker 02

  So we will have agents augmenting what we do. We will let, we will have them go off and do things for us. You know, it's already starting to happen with shopping and, you know, travel and stuff like that. Except that right now it's like the agent that I use is going in on my behalf, going to some website or some API and doing the shopping for me, or at least giving me, I'm, I'm still not comfortable giving it my credit card. So it gives me like the top three and then I decide, but you know, it'll, it'll, as we trust these systems more, you know, we will untrust them with, with our credit cards. But it's also going to be two ways. So it's not my agent going off to an API. It's going to be my agent. Talking to another agent, so an agent may be representing the travel agency and so forth.

• 42:23Speaker 02

  So more and more of the fabric of what we know of the internet is going to be the worldwide agentic web. There's actually an effort around this out of MIT, Project Nanda, that we're collaborating with that is starting to put this together. So I think in the shorter term, we will see more and more of this. We will swing between trusting and over trusting and then not trusting at all as bad things happen as usual with with technology. So there is some of that. I think it's going to be hugely disruptive to jobs in the in the short term. In the long term, I do think that we're all going to be busier in the very long term. I don't know. And a big question that I have is whether or not the way we've set up society and we've defined progress

• 43:08Speaker 02

  and we've defined value has the capacity to allow for this amount of autonomous intelligence being pumped into it. I don't know that. We see the implications of that. So there's this darker side that we absolutely need to avoid. And then there's the lighter side. You mentioned things like breakthroughs in medicine and care and climate and efficiencies, technologies that will help humanity. It is an inherently democratizing technology. So we have a choice. I do see this bifurcation a little bit of, oh, you know, if you're on the left side of the spectrum of politics in the US, you're anti-AI, whatever that means. And if you're on the right, you're pro-AI.

• 43:50Speaker 02

  Just that's totally meaningless. We need to pick our battles for humanity and we need to use our most powerful tools in this realm. So for us to say we're anti-AI and not use it at all puts us in a huge, huge, huge disadvantage. I'd rather use AI for the good of humanity.

• 44:09Jonathan Cohen?

  Very hard to consider sort of pulling back to a point of not using it. One of the things that we've touched on or you're alluding to is like the notion of energy and the changing potential of making the energy grid far more efficient. Battery storage, we talked with Mustafa, if there's like a 10x increase in battery storage, what does that do to energy costs around the world? And then what does that do to the rest of the economy? Lowering costs of goods and services of food production could be a massive, massive change.

• 44:42Speaker 02

  Yeah, Mustafa actually did this when he was still at Google. They actually optimized their data center energy consumption and saved a ton of money for Google just with that one project. And that was actually, I think, even before the advent of LMS, if I'm not mistaken.

• 44:58Jonathan Cohen?

  You know, it's hard for anyone, especially people who are not in the field, to understand the massive leaps forward that it can provide. The protein fold experiments and so understanding, finding breakthrough cures for diseases, thinking about energy, thinking about infrastructure. What do you think in terms of like finding solutions? Like, one of the things that humans are the worst at is working together globally. We seem to be constantly divided. Do you see any hope for that?

• 45:30Speaker 02

  When we simulate environments in which we have autonomous agents working together, only autonomous agents working together. One of the problems we run into is that they're too nice with one another. So they don't quite simulate how humans behave. And that's kind of surprising. There was a researcher presented here at our lab and he was trying to get agents to mimic how news disseminates, depending on your, you know, political tendency, whether you're right or left. And he had actual data from news, for example, with respect to vaccines or COVID or whatever

• 46:11Speaker 02

  else, and how that news kind of disseminated between different groups. So we actually had real data on. Real people and how they react. And then he had this whole ecosystem of AI agents mimicking people. And in their prompts, they were told like, you're a Democrat, or you're a Republican, or you're right, or left, or whatever. And so giving them some of that. So there's a tendency that the news comes out and each side views it their own way. And and there's some like biases in the way the news disseminates. But after a while, when there's more and more follow up news, everyone ultimately ends up with kind of the facts. Some slower, some faster, depending on on their biases.

• 46:53Speaker 02

  But that tendency, that convergence, happened much, much faster with LLMs than than with humans. And Mustafa talked to you about bias as well, right? If we're actually teaching our intelligence systems and trying to align them to be kinder and nicer and more rational in their thinking, and then augment what we do and go off and talk to other agents to get something done, we might actually be in a better place than we are today.

• 47:22Jonathan Cohen?

  If we're not bringing our radicalized perspective or deeply felt emotional hurt into into a situation, we may end up with two agents negotiating to a better outcome.

• 47:35Speaker 02

  Exactly right. There's this whole thing about, oh, SaaS is dead. The UX or the brand doesn't matter as much. The reason for that is because your agent isn't really looking, it's not like mesmerized by the brand and how nice it looks or the ads that it's seen on billboards anymore. It's looking at the quality of what is being offered and the price at which it's being offered. So that's kind of taking away a lot of the human bias in that decision making. So, okay, that's the the positive side. Obviously, these agents could also be doing harm. I mean, it depends on how they're aligned or misaligned and who's who's controlling them. At the end of the day, though, Jonathan, it all comes back to us. It's a reflection of humanity. Like, so I don't think we can get away from trying to fix our biases and our deficiencies.

• 48:20Speaker 02

  And AI systems will be a reflection of that.

• 48:24Jonathan Cohen?

  Beautifully said. A couple of things just for the audience. There is a lot of evidence of AI catching cancers a lot faster, solving pancreatic cancers, which can be a very hard one, coming up and giving increasing access, especially to like skin diseases by just sending pictures. It's like kind of amazing the democratization just on that physical health level. On the mental health level, as we interact more and more with these models, people are finding out more about themselves. It's a fun exercise to be like, tell me what you I don't know about myself exercise. I did that recently and I was like really impressed by some of the insight that it had. I did not like its suggestions, what to do about what I didn't know,

• 49:08Jonathan Cohen?

  but its actual insight into me, I found like, when else would I have gotten that? What like you're playing Frogger on my machine as an eight year old wouldn't have gotten me any insight about myself. You mentioned something about the world maybe not being set up for this type of agentic system.

• 49:29Speaker 02

  Just imagine if you have agents running around and making money. Just just a very simple thought experiment. Who's taking that money? The person that owns the more most agents, I guess, going out and doing stuff and making money. It's all that value is accruing to the person that owns them. It's not going to be distributed between the agents. The agents are like, you know, they're just conduits. And so this whole thing that we're seeing in the world where a lot of wealth is being concentrated and a few individuals or corporations and so forth will just be exasperated in this setting. I don't know what the solution is to that.

• 50:09Speaker 02

  I mean, you can see that, for example, an increasing percentage of music on Spotify is AI generated and people, you and I will will listen to it. I mean, maybe you and I are more informed and won't, but but but people can't distinguish. They are listening to it. It's much cheaper to generate that music automatically and you don't have to pay anything to the artists anymore because there is no artist. And so who makes the money like off of? That, right? So that's what I mean. And it's not these like points, oh, let's label it so people know it's AI generated. It'll go so far. But it's it's much more fundamental, I think, of a problem that we need to deal with as as a society.

• 50:49Jonathan Cohen?

  Yeah, that's that's fascinating. Last question for you. If a person is in university right now or heading into university, what do you recommend that they focus on? That's a tough question. I'm sorry.

• 51:02Speaker 02

  Hard one to do. Yeah, I like do what you love is my thing always because whatever you love, you'll be the best at. But if you are interested in pursuing

• 51:17Speaker 02

  some technology track, I think if you are a domain expert in any domain, any domain, there will be a premium on you because you know that domain very, very well. And so you can apply AI to that particular domain. So right now, that is more interesting and important than being purely an AI, for example, or purely a data scientist. It's the domain expertise right now that's at a premium, at least for the next few years.

• 51:41Jonathan Cohen?

  Babak, thank you so much. It's really a pleasure to speak with you and thank you for helping us get into the world we're now in and we're excited to see what happens next.

• 51:50Speaker 02

  My pleasure. Thank you so much for having me.