#248 Pedro Domingos: How Connectionism Is Reshaping the Future of Machine Learning

This episode is sponsored by Indeed. Stop struggling to get your job post seen on other job sites. Indeed's Sponsored Jobs help you stand out and hire fast. With Sponsored Jobs your post jumps to the top of the page for your relevant candidates, so you can reach the people you want faster. Get a $75 Sponsored Job Credit to boost your job's visibility! Claim your offer now: https://www.indeed.com/EYEONAI In this episode, renowned AI researcher Pedro Domingos, author of The Master Algorithm, takes us deep into the world of Connectionism—the AI tribe behind neural networks and the deep learning revolution. From the birth of neural networks in the 1940s to the explosive rise of transformers and ChatGPT, Pedro unpacks the history, breakthroughs, and limitations of connectionist AI. Along the way, he explores how supervised learning continues to quietly power today's most impressive AI systems—and why reinforcement learning and unsupervised learning are still lagging behind. We also dive into: The tribal war between Connectionists and Symbolists The surprising origins of Backpropagation How transformers redefined machine translation Why GANs and generative models exploded (and then faded) The myth of modern reinforcement learning (DeepSeek, RLHF, etc.) The danger of AI research narrowing too soon around one dominant approach Whether you're an AI enthusiast, a machine learning practitioner, or just curious about where intelligence is headed, this episode offers a rare deep dive into the ideological foundations of AI—and what's coming next. Don't forget to subscribe for more episodes on AI, data, and the future of tech. Stay Updated: Craig Smith on X:https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) What Are Generative Models? (03:02) AI Progress and the Local Optimum Trap (06:30) The Five Tribes of AI and Why They Matter (09:07) The Rise of Connectionism (11:14) Rosenblatt's Perceptron and the First AI Hype Cycle (13:35) Backpropagation: The Algorithm That Changed Everything (19:39) How Backpropagation Actually Works (21:22) AlexNet and the Deep Learning Boom (23:22) Why the Vision Community Resisted Neural Nets (25:39) The Expansion of Deep Learning (28:48) NetTalk and the Baby Steps of Neural Speech (31:24) How Transformers (and Attention) Transformed AI (34:36) Why Attention Solved the Bottleneck in Translation (35:24) The Untold Story of Transformer Invention (38:35) LSTMs vs. Attention: Solving the Vanishing Gradient Problem (42:29) GANs: The Evolutionary Arms Race in AI (48:53) Reinforcement Learning Explained (52:46) Why RL Is Mostly Just Supervised Learning in Disguise (54:35) Where AI Research Should Go Next

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I see a bunch of data. I'm going to postulate that they were generated by some process. And I'm going to try to identify what that process was. This is actually what generative learning is. So you come up with this model like a visionary. This is like, well, this is how you generate patients that their symptoms. First, you decide which disease they have. And then you decide which symptoms they have. What fever are they going to have based on the fact that they have COVID? Et cetera, et cetera. You generate the data. And this is why it's called the generative model. In opposition to discriminative models, which is what most of machine learning is about, which is to just go say like, okay, I'm going to go forward from this data and try to decide, you know, what, what you suffer from or whether it's a cat or a dog. As opposed to like, how do I generate, you know, a cat or a dog? Right? So there's always been this thing in machine learning between generative and discriminative.

• 0:43Speaker 00

  Discriminative has always dominated and, and, and really still does. Because it just work, it's, it's an easier problem and it works better.

• 0:51Craig S. Smith?

  When you need somebody, you need somebody. You want to hire people fast. The longer you wait, the more the market drifts, the more your needs drift. For example, you just realized your business needed to hire somebody yesterday. How can you find the right candidate fast? Just use Indeed. When it comes to hiring, Indeed is all you need. Stop struggling to get your job post seen on other job sites. Indeed's sponsored jobs help you stand out and hire fast. With sponsored jobs, your job postings jump to the top of the page for your relevant candidates. So you can reach the people you want faster and it makes a huge difference.

• 1:33Craig S. Smith?

  According to Indeed data, sponsored jobs posted directly on Indeed have forty five percent more applications than non-sponsored jobs. One of the things I love about Indeed is that it makes hiring so fast. I've searched for candidates before and it can take months. Particularly if you're just working through word of mouth. In those days, I didn't have Indeed. Plus with Indeed sponsored jobs, there's no monthly subscriptions, no long-term contracts, and you only pay for results. How fast is Indeed? In the minute I've been talking to Twenty three hires were made on Indeed, according to Indeed data worldwide. So there's no need to wait any longer.

• 2:18Craig S. Smith?

  Speed up your hiring right now with Indeed. And listeners on this show will get a $75 sponsored job credit to get their jobs more visibility at indeed.com/eyeonai. Just go to indeed.com/eyeonai. That's indeed, I N D E D dot com slash eye on AI. All run together E Y E O N A I. Go right now and support our show by saying you heard about Indeed on this podcast. Indeed.com/eyeonai. Terms and conditions apply. Hiring indeed is all you need.

• 3:01Speaker 00

  Um, the biggest surprise about the XIP for me was how surprised people were. There's really nothing particularly surprising about anything they did. Same with what the others are doing. It's the natural progress. And a lot of it honestly is, um,

• 3:17Speaker 00

  it's adding more capabilities and tweaks and variations that are all, you know, useful and understandable, but on top of a core that isn't changing that much. And honestly, I think we're converging, as is often the case, to a local optimum. I personally am more interested in what the global optimum is than the local one. But in the short term, that's where we are.

• 3:42Craig S. Smith?

  Yeah, uh, it's kind of what happened with supervised learning. It was, everyone was amazed and then a lot of people started focusing on optimizing and tweaking. And, and it became stronger and stronger. Is that right?

• 3:59Speaker 00

  Uh, when you say supervised learning, uh, what do you mean exactly?

• 4:02Craig S. Smith?

  Yeah, I, I mean, I mean, the, the, uh, yeah, I mean, the, uh, labeled data pattern recognition systems that came, that dominated neural nets before generative AI.

• 4:19Speaker 00

  Oh, I see. Interesting. Well, I'll tell you a very short story. When I was in grad school, this was in the early 90s, one of my colleagues of the same advisor, he left because he said, there was no future in machine learning, because it was all, you know, the only thing that worked was supervised learning and supervised learning. There was nothing new left to happen.

• 4:47Speaker 00

  Well, we can see how that turned out, right? I don't think, I mean, I see what you're saying, but I was, supervised learning is such a broad paradigm in machine learning, that it's, there are, there are, there are certainly and have been local optimal within supervised learning. It's hard to see supervised learning. I mean, I guess in the current context, people talk about like, oh, we're running out of data and so on and so forth, and we need to do other things. And that's where the lot of work is. But the problem is that they're not, they're not getting the most out of the data, is that the supervised learning algorithms are not that

• 5:28Speaker 00

  good. And in fact, the, the, the key idea in some ways of L L M's, maybe we'll talk about this, is, is to take advantage of state that is to, of, of data. The, the, the problem with supervi-, it continues to be the case that the only thing that really works in machine learning is supervised learning, you know, it's, yeah, that reinforcement learning and even unsupervised learning, blah, blah, blah. The key innovation in, in, in L L M's in many ways is just, and it's not a thing that's new to L L M's, but it's what drives it, is how to turn a massive amount of unsupervised data into supervised.

• 6:05Speaker 00

  Yeah. So all, you know, under all of this stuff, there's still supervised learning going on, and moreover, and related to deep seek, people are always talking about, these days I've become, you know, honestly, a little skeptical, and just like, oh, we have our latest, greatest reinforcement learning, blah, blah, blah. And when you, when you look at it up close, it's really just doing supervised learning. But it's sexy to call it reinforcement learning. So there you go. Yeah.

• 6:29Craig S. Smith?

  And so anyway, let's let's get on with the connectionists. One thing that interests me is how, I mean, and I think I mentioned this in talking this whole series, is that there are a lot of strategies or schools or tribes, as you call them, that remain viable for for progress and continued research. I had on the podcast a couple of weeks ago, sup Hulk Rider. I'm not sure how to pronounce it. When he pronounce it, it doesn't sound

• 7:09Craig S. Smith?

  like that. The the guy who came up with the long short long short term memory of networks, LSTM, and he is continuing research and has now created a company to use his research in industrial applications. And I thought that was really interesting. So, yeah, let's talk about connectionists, where they came from, where they're going, and whether there are schools of of connectionist theory that are being neglected because the bright shiny

• 7:54Craig S. Smith?

  object is capturing everyone's imagination.

• 7:59Speaker 00

  Yeah, sure.

• 8:02Craig S. Smith?

  So go ahead. And Pedro, reintroduce yourself. I apologize for that. But yeah, go ahead.

• 8:10Speaker 00

  I'm Pedro Domingos. I'm a professor of computer science at the University of Washington. I have been an AI researcher since the 90s. My specialty is machine learning. I've worked in a lot of different areas of AI along the different paradigms and also in a lot on unifying them. I'm probably best known as the as the author of a book called The Master Algorithm, which is an introduction to machine learning for a broad audience, and more recently of 2040, Silicon Valley Satire, which is basically a spoof of AI hype and fear and the tech world.

• 8:47Craig S. Smith?

  Yeah, uh, so tell us about connectionists, how they emerged uh and how they operate alongside the other uh schools of thought that we've talked about uh and how they relate them to what's going on now.

• 9:07Speaker 00

  Connectionism is one of the oldest paradigms in AI. You could say it's the oldest after symbolic AI. Uh, it started in the forties, um, which is a long time ago in in in AI terms. And you could say it's really one of the two big ones along with symbolic AI. Sometimes people think of, I think of like these two big opposing groups and they are the symbolists and the connectionists. And in many ways, the connectionists historically have been find themselves in opposition to the symbolists now. One interesting thing that has happened is, so the idea in connectionism is uh actually a very appealing one. And that appeal, I think, is very important, which is, okay, we want to build intelligent machines.

• 9:50Speaker 00

  Uh, what do we know that's intelligent? It's humans, uh, and in particular, the human brain. So like good engineers, let's, when we're be, when you're behind the competition, what you do is reverse engineer it. You open it up, right? You open up the car, right? Whatever. You're Toyota. You start by looking at your GM or or Ford cars and go like, well, how is this thing built? Right? And then you copy it and then maybe eventually like Toyota did, you improve it and and so on. But that's the idea. Let's reverse engineer the brain. And so this started from the very beginning of the modern age of computer science. People were, there was this paper that came out in the forties by McCulloch and Pitts on a model of neurons. You could actually say that was the paper that started neural networks.

• 10:33Speaker 00

  And and a lot of people don't know is that a lot of the computing hardware today that has nothing to do with AI, a lot, all of it, is really derived from that paper. So in some sense, every computer, as they used to call them in the fifties, is an electronic brain. Every every circuit in a computer is is imitating neurons. A very important point to remember as AI, as our like now starts to gobble up the rest of computer science. Uh, but the big difference uh between those neurons and the ones in your brain is that those neurons didn't learn. They didn't have weights. They were really just logic gates, which is again what computers are made of. You know, it was a more, it was a generalization of the, of the logic gates we, we tend to use.

• 11:13Speaker 00

  And then really the key name in this story was Frank Rosenblatt, who in the 50s created this thing called the perceptron. That was the first neural network that learned, but it wasn't a neural network. It was actually just a single neuron that learned with some pre-computed features that were fed into it. And people got incredibly excited about this at the time. The New York Times had a front page story saying, you know, human level intelligence is coming soon.

• 11:42Speaker 00

  I, I can't help laughing at this because, you know, it continues to happen today. One day it'll be true, but, but in the meantime, people should know this history. And so for a while, it was very popular. And you could even say that was probably the most popular approach to AI for a while. And in fact, it's interesting that a lot of people from the other paradigms, they started out doing neural networks. John Holland of Evolutionary Computing and Marvin Minsky of Symbolic AI. Marvin Minsky's PhD thesis was on neural networks, which boggles the mind. But then they all drifted away from it once that initial seductive idea was not really panning out very well.

• 12:24Speaker 00

  And, and then also very famously in 1969, Marvin Minsky, who was by now one of the leaders of the symbolic school and Seymour Papert wrote a book called Perceptrons.

• 12:35Craig S. Smith?

  Yeah.

• 12:36Speaker 00

  Basically tearing down the claims that the perceptron was a great thing. They showed mathematically a series of things that perceptron just couldn't do. And then interesting perceptrons just in neural networks just fell like a stone.

• 12:50Craig S. Smith?

  Yeah.

• 12:51Speaker 00

  And then for the next 15 years, they were dead. And in fact, they even reflected on machine learning in general. The consensus, you know, during the seventies and early eighties was that, you know, machine learning doesn't really work. It's too hard. You know, you've got to program things, you know, you've got to do knowledge engineering. And that was the big thing. It was symbolic AI and knowledge engineering and machine learning was nowhere.

• 13:12Craig S. Smith?

  Yeah.

• 13:13Speaker 00

  And then in the eighties, they they came back. And the key development was was the backpropagation algorithm.

• 13:22Speaker 00

  People knew, and Minsky and Papert said that like, if you could train networks with multiple layers, then all those limitations would be solved. But nobody knew how to. And in their words, they couldn't see how it would ever be done.

• 13:35Speaker 00

  And backprop itself has an interesting history, but it was in the eighties that it took off. And then there was another moment of great excitement, which then again died down again because people thought like, we're just going to build deeper and deeper layers and pretty soon, you know, it'll be the brain, right? They got over optimistic again. There's always somebody in AI being over optimistic. The only question is who? But then, you know, the problem with backprop is that it did, it only worked with one hidden layer, which wasn't much. And so people gave up on it. And by the end of the nineties, neural networks were kind of not dead, but like completely sidelined again. And and that's when, you know, early 2000s is when, you know, people like Jeff Hinton, Yann LeCun and Yoshua Benshou started working on this. Actually, they didn't start, they never stopped working.

• 14:19Speaker 00

  They were the diehards that never stopped working. But they they started to figure out ways to learn deeper networks. And in fact, the term deep learning, partly it's a marketing term. It's clever, right? Whoa, the learning is deep. But really technically just refers to the fact of having networks with many hidden layers, which indeed, once you're able to learn them, can do amazing things. And and here we are today.

• 14:40Craig S. Smith?

  Yeah. Can you just take a dog leg on backprop and explain where it came from and how Hinton applied it?

• 14:54Speaker 00

  Yeah, so first of all, what is the difficulty that backprop is solving? It's actually one of the central problems in machine learning. So in the master, I talk about the five tribes and how each one of them has an algorithm, its own master algorithm that solves a key problem. And my contention, of course, that you have to solve all of them, so none of the algorithms is enough. But the key problem, and it is a central problem in machine learning that backprop solves is credit assignment. I have a very big system with lots of layers, lots of neurons, a big mess of connections, and it's supposed to look at an image and tell me if it's a cat or a dog. And the image it's looking at is a dog, but it says it's a cat, right? And so now the question is, who do I blame? Maybe it should be called the blame assignment problem.

• 15:38Speaker 00

  Like, who should change, right? In connectionism, all the knowledge is in the connections between the neurons, again, inspired by the human brain, where, you know, as far as we know, all the knowledge you learn is in the strengths of the connections between neurons. Which connections should I change? And if you think about it, this is not obvious at all, right? There's a mistake at the end, there's an image at the beginning, like, you know, who the hell knows what's going on? But backprop, technicalities aside, just gives a super simple answer to this. And again, often in machine learning, super simple answers are what gets you the farthest. And the answer is this: what I'm going to do conceptually, not in practice, because that would be too inefficient, but conceptually, is I'm going to tweak each weight in turn a little bit.

• 16:23Speaker 00

  I'm going to take every weight of every neuron in every layer and say, like, let me increase you a little bit. You know, and there's usually an output function that is continuous, that says, like, what is, you know, your probability, let's say, loosely speaking, of being a cat? Oh, when I tweak this, when I move this weight up, the probability goes up a little bit. Good, so let me keep that. When I tweak this other weight down, the probability goes up, so let me tweak that, right? If you do this for every single neuron, over and over again, over time, amazing things happen. It will probably learn to classify the cat images really being a cat image and so on. And, you know, backprop is really just a computationally efficient way.

• 17:05Speaker 00

  A clever way, but you know, of a type that we know very well of trying to not be too wasteful. Like you don't want to redo the whole computation every time you do this. That would just be ridiculous, particularly with today's networks. So it's really just an efficient way to do that. What happened with backprop there again is one of those delicious ironies that the history of AI is full of. Is it took off, there was this paper published in 1986 by David Rumorhart, Jeff Hinton and Chris Williams or somebody Williams, Robert Williams. It was a third author. And it's often credit, it was originally credit, let's say with, you know, that's when they discovered backprop. The truth is that backprop was discovered 20 times before that.

• 17:50Speaker 00

  As it became more prominent, I mean, I alone know three or four different people who invented backprop. One of them was Jan Le Cun. While a grad student in France, he actually invented backprop on his own. And I know, you know, for example, there was a postdoc of my advisor, E C Irvine, who in like early eighties or even late nineties, he submitted the paper to the top AI conference with backprop. And the paper was rejected and the reviewer said like, what are you doing? Minsk and Pepper, are they show that?

• 18:21Craig S. Smith?

  Yeah.

• 18:22Speaker 00

  And then there was this guy economist from Harvard who was, you know, had actually found it in, he published, you know, a paper or a thesis, I think about it in the seventies. Turns out that there was a paper by control theorists, Bryson Ho, in the sixties, before the perceptron's book was published. The backprop algorithm already existed. And it's like Jan Le Cun says that really the credit for backprop should be given to Leibniz because it's really just the chain rule of calculus.

• 18:55Craig S. Smith?

  Yeah.

• 18:56Speaker 00

  So there you go.

• 18:57Craig S. Smith?

  Yeah. One question I have about backprop is I understand the the theory that you keep adjusting the weights, going back down through the weights to get closer and closer to the target. But is it done sequentially? Like each neural pathway is adjusted and then the the outputs assessed? Or is it done incrementally through all the neurons? And then the, I mean, how how does that work?

• 19:33Speaker 00

  It's done sequentially but back through the layers. And in fact, that's where the name comes from. So the way things work in backprop is that you have your image of the cat and first you do a forward pass.

• 19:44Craig S. Smith?

  Right.

• 19:44Speaker 00

  Sometimes also called the inference pass, which is basically just computing the output first of the neurons that are right on top of the image and then the next layer all the way to the final output that says cat or dog. And then comes the error backpropagation phase where you measure the error. Like, you know, your output was, you know, point seven for cat, but it should have been point nine. Right? Or whatever. It was point one for catch at point nine. So now what I'm going to see is like, and we'll get the last layer of neurons, starting with the very last neuron in that case. And see like, if I tweak every one of your weights, uh, how does that change things? But then the key is that when I go to the previous layer, I actually don't start from scratch. That would be wasteful.

• 20:23Speaker 00

  I don't go all the way back to the output. I say like, well, now I know how much the neurons, the weights in the last layer, um, how much difference each one of them makes. Based on that, let me see how much difference each neuron in the previous layer makes. So you just go back through the layers all the way to the initial one. And that's why it's called backpropagation, because you're propagating the errors backward and then you're learning from that.

• 20:44Craig S. Smith?

  Yep. Uh, okay. So we've got backprop. Jeff and Ilya and Alex create Alex in 2012. Well, which uh, takes advantage of the larger data sets available for, and the greater compute power. Uh, available and they have a breakthrough on the ImageNet competition. And that sets off the uh, the new uh, deep learning craze. So where, where does it go from there then?

• 21:22Speaker 00

  Well, so, so to just back up slightly, uh, deep learning in public perception exploded then. But already before that, there had the first big progress of deep learning that caused people to stop paying attention was in speech.

• 21:38Speaker 00

  Jeff Hinton, for the most part, has always worked mainly in vision. Yoshua Benji was the language guy, which we'll probably get to. Yann LeCun was also very much into vision. But, but at one point, you know, because that's what they could do in some ways, vision is very expensive. Jeff and his students started working on speech, and speech was working really well. You know, Jeff Hinton had this student, Naveed Jaitli, who he calls, apparently, you know, Larry Page calls him the thirty million dollar intern, and Jeff says it really should be the billion dollar intern. He did an internet Google one summer where he applied speech, applied deep learning to the speech system that they had that was already, as you can imagine, a highly engineered

• 22:22Speaker 00

  thing, right? And in one summer, he basically beat the state of the art that these, I don't know, hundreds of engineers had developed. So, so, and speech was, again, it's a longstanding area of AI, and it had plateaued for, you know, for decades. And then suddenly, out comes deep learning, and it's like, wow, suddenly, you know, this phenomenon that you've seen in a whole bunch of other areas was first seen in speech, right? And then, you know, LXNet itself has an interesting history, but, but you're correct. They, the vision community has always been skeptical of deep, of neural networks, because they didn't think of that as serious vision. They all just worked, again, it's hard to not say this with a smile, but for the first

• 23:06Speaker 00

  fifty years, all they were working was digit recognition. Yeah. You know, it was like, how to recognize digits better, because again, you know, that's what you could, that's what they had the data for, and that's what wasn't too expensive. And vision people were like, like, we really couldn't care less about that, that's not vision, go away. So, and, and like, the pope of vision is Jitendra Malik, who's a professor at Berkeley. And Jeff Hinton, at one point, got on the phone with him and said, what would it take to persuade you that, you know, neural networks are good for vision? And then, you know, Jitendra said, well, you know, there's this dataset called Pascal, which was, you know, an important dataset back then. But then, you know, Hinton, you know, talked with his students, came back and said, like, no, Pascal is too small, you know, is there a bigger one?

• 23:51Speaker 00

  And then, and then you think so, well, there is this data set that this person at Stanford Philly and her students have developed called ImageNet. That's actually pretty large. That's a million examples. That doesn't sound large today, but back then, the likes of Pascal and other sets were like, you know, a few thousand, which it sounds ridiculous that anybody would try to solve vision with a few thousand examples. And so, and they, and the key thing in AlexNet was really that they used GPUs. Right. Right. Without GPUs, it was like, they couldn't have done it, right? But then, but the GPUs were available then, uh, uh, uh, and then, you know, and they, they weren't the first ones to repurpose them for machine learning, but, you know, they, they got this amazing success. And then to answer your question, what happened was the vision folks initially,

• 24:33Speaker 00

  were skeptical. The people in machine learning were very excited, and the folks were just like, they're, they're doing something wrong. This, this can't be right, right? They, they just watched something and they're getting these results aren't real. So they went back and, and, and, and repeated the experiments and, you know, oh my God, the results are real. And then they started applying it, and this is like, you know, the wavefront that continues today. They started applying it to other problems in vision, not just, you know, uh, recognizing, you know, literally cats from dogs. Those are the most frequent things on ImageNet is different types of dogs, but other tasks in vision, it continued to work really well. And then within the space of a few years, vision went from having no papers to speak

• 25:16Speaker 00

  of, uh, using deep learning to basically everything just uses deep learning. Uh, uh, and, and, and then of course, this, this got into the public consciousness. And then the attitude of the deep learning folks, which you gotta, um, admire at one level, but you know, be a little skeptical. I was like, okay, we've solved vision. What's next? Let's do language now. Right? Let's do machine translation. Right? And I remember talking with Joshua Benjocirka, I don't know, like, uh, 2015. And he was saying like, yeah, we haven't quite matched the statistical machine translation results. Yeah, statistical machine translation, you know, systems at the point, again, at Google and other places were very big and already doing very well in many ways. Uh, you know, it was the death of the employment

• 25:57Speaker 00

  for translators and whatnot, which of course didn't happen, uh, you know, and, and that whole phenomenon continues today. But, uh, he said, we're almost there. Within a couple of years, they weren't just almost dead, they had blown past all of that. Right? And and these days, that pattern continues with reasoning. It's like, yeah, we solved vision, we solved language, and now we need to solve reasoning. The problem with this is that they haven't really solved vision or language or any of those things. They made a lot of progress. But in the meantime, there's a lot, you know, again, we are stuck in this, you know, local optima that we are going to need to get out of. And to their credit, people like Yan Le Cun, right, who again, he's a longstanding uh uh neural network vision guy is like, no, no, no, this isn't solved. And it isn't. I mean, if you look at video understanding,

• 26:41Speaker 00

  which is really the main problem in vision, right? Yeah, there's a kind of like sub problems of that. It's very far from solved. No one really knows how the video understanding. And you know, the new generation and things like transformers are not, people have applied it to that, but you know, it's it's an improvement, but uh all of that continues to be unsolved. There's this language and there's reasoning even more. Of course, you know, the latest generation is this whole, you know, chat G P T, et cetera and whatnot, which in some ways is more of a sociological phenomenon than a technical one. The big novelty in chat G P T was that everybody, you know, was blown away by it and started using it. In terms of research content, there's very little new in chat G P T.

• 27:24Speaker 00

  You could say, oh, they scale things up some more, but even that is not true. Google had already, you know, they the Google made the mistake of not releasing their chatbot. Into, you know, in technical terms, there wasn't anything significantly new in chat G P T over, you know, over Lambda, for example, Google's chatbot at the time. And of course, a lot of in in technical innovation has happened since, but it's interesting that in some ways in you you know, um there is all this focus of research and industrial attention on a very, very narrow front, even within deep learning, while a lot of the other things, including probably where the big leaps are gonna come from, are being neglected.

• 28:03Craig S. Smith?

  Yeah. Yeah. Can you go back a little bit? Uh, because there was another flexion point with the transformer algorithm. Uh, and talk about, uh, and this is all within the the connectionist camp, right? Can you talk about the development of the transformer? And on speech, I just want to jump back a little bit. Uh, Terry Sonowski was doing something called Net Talk uh before uh.

• 28:33Craig S. Smith?

  Jeff had come out with with his vision stuff. Was NetTalk a step

• 28:41Craig S. Smith?

  towards, you know, applying neural networks to speech?

• 28:48Speaker 00

  Yeah, so NetTalk, back in the late eighties, early nineties, when somebody needed to come up with an example of a neural network success, it was NetTalk. NetTalk was not speech recognition, it was speech synthesis.

• 29:02Craig S. Smith?

  Right.

• 29:02Speaker 00

  It was you gave it a text and it read it aloud. And the thing that got people really excited, and again, I think you can listen to this on YouTube even now, if you just, you know, search for NetTalk, is it started out producing noise. And then, as the network learned through successive generations of backprop, it sounded at first like it was babbling like a baby. This, I think, like, that fact that it was babbling like a baby is really what got people. And then the babbling got more coherent, and finally, it was actually doing a pretty good job of synthesizing speech, which this sounds like, what is the big deal? But at the time, it was a big deal. So, you know, so NetTalk, you know, was the poster child of neural networks. But the truth is, there's a useful application that never went anywhere. It wasn't good enough, reliable enough, blah, blah, blah.

• 29:46Speaker 00

  And commercially, speech recognition wasn't as much more important, right? And even harder, right? It was like, you get this, it's much harder, you know, if you think about it, like, it's not that hard to do a bad job of synthesizing speech from text. You know what the phonemes are, and you string them together. It'll sound very stilted, but it's not that. Speech recognition is a hard problem, right? There's like this noise wave coming at you, all garbled and like, people like, you know, eat up syllables, and you've got to turn that into text. That is really hard. So, you're right, NetTalks, it was what, you know, people talked about back in 1990. But, you know, circa 2005, 10, what Jeff and his students were doing was, again, something that the speech people had to take seriously.

• 30:30Speaker 00

  Right? Again, you know, I sometimes joke that the purpose of machine learning conferences is to publish application papers before they're good for the application conferences. You know, you publish speech there and vision because, you know, as tests of the machine learning algorithms, and then when they're mature, they go to their natural language, whatever, conferences, which brings us back to transformers. And it's exactly, if we go back to the point where I said, and then people said, let's do language. Let's pick up the story there. As I mentioned, the main guy of that very small group that had always been interested in language, but he even, he did his PhD on, I think, speech. And then kind of didn't work on that for a while because again, they got stuck for reasons that we can go into.

• 31:16Speaker 00

  And then came back and his group started looking at machine translation. Who is that? I'm sorry. Hmm. Who was that? So the attention, right, is the key thing in transformers. Right. And attention, it's an interesting story. It was, so Yoshua Banjo said to his, to a new postdoc of his, Kyung Hyun Cho, and someone who was an intern. Here's the best part of this, most people don't know this. Transformers were not invented at Google. I mean, the name was and the tool architecture was, but the key idea in transformers is attention. I mean, the transformers paper is called "Attention is All You Need."

• 31:59Speaker 00

  Because actually, the main thing they did was take other stuff out, which is a valid contribution. But attention was invented by an intern in Yoshua Banjo's group. He wasn't any, you know, great machine learning researcher or anything. He was an intern who didn't know anything and just started working on this. And he came up with that idea. So what was the problem? What was the solution, right? It's like, to do machine translation, the way they tried it initially was like, you would go through, for example, the translating French to English. You would go through the French text with what is called a recurrent neural network that processes things sequentially over time. And you would wind up with the state of the neural network that captured the text that you had read. And then you start from this state with, and this was called the encoding part, right?

• 32:43Speaker 00

  Because it encoded French into the network. And then there was the decoding part, where you turned that internal state, which is a bunch of numbers that nobody understood and still doesn't, and turned it into English again sequentially by generating the text on word at a time. And this didn't work very well. There was a bottleneck, like there was too much information lost and it tended to remember the recent text, meaning the last parts of the text, but not the early one. And so Yoshua said to them, like, you know, we gotta do something about this. Right? And what is the attention idea? In a way, the attention idea is really just doing with neural networks what people have been doing in the statistical approach to machine translation for decades. There's always, there was always this phase called the alignment phase,

• 33:28Speaker 00

  which has nothing to do with what people call alignment today. Is this fact that, for example, when you're translating freshly English, word order is often different. Like the adjectives, you know, will come, you know, before or after, right? And so it's not enough to say I'm gonna go through the text one at a time. When I'm generating the next word, let's say the next word is the adjective. Like, for example, I saw a big cat. Right? And then like, where does the big come from? Right? You gotta go find in the previous text the word that big is translating. Oh, le grand chat, okay, right? I already translated chat now, the grand, whatever, right? You get the idea, right? And so what they did was this, was they did a neural network version of the same process.

• 34:10Speaker 00

  What this, and this is what attention does is like now that I'm generating the word, you know, like, what is my next word is gonna be? Well, what, what is it the translation of? Right? Oh, you know, big, you know, the grand is, big is the translation of grand. And then you can learn this using backprop, right? So that was, that was their innovation, very important one, right? But they were just trying to solve a machine translation problem. The paper is just about machine translation. And then the transformer guys, same thing. They, they, you know, they refined, you know, a bunch of stuff and basically to cut out the stuff that no longer was, was no longer useful. And, and, and then, I mean, I, it's interesting because like I remember reading these papers at the time and each case thinking, wow, this is a very cool idea. I bet you can do a lot of other things with it besides translation.

• 34:54Speaker 00

  Next paper. And then what people found, and this is partly, you know, the, the beauty of machine learning is like, they took the same architecture and started, you know, doing more and more things with it and keep, and it kept working amazingly, right? And here we are today, right? ChatGPT, the T in GPT is transformer.

• 35:11Craig S. Smith?

  Yeah, yeah. And then the guys at Google, there was a team.

• 35:17Craig S. Smith?

  Uh, that, that, as you say, stripped out everything uh and made a very simple architecture around attention, right?

• 35:26Speaker 00

  Um, I'm, I'm, I'm, I'm laughing because it's anything but simple, but I mean, so it wasn't, it wasn't one team. There were several teams, maybe three. That's why the paper has eight authors. There are three different groups of people who, you know, for one reason or another, said like, yeah, let's, let's try to use this for whatever, right? Or, or, you know, they had their different concerns, but they were all started using this new mechanism of attention, right? The basic attention is just, you know, a couple of equations. And again, uh, uh, um, we can try to understand what it's doing in, in multiple ways, but, uh, uh, but they weren't making any headway. And then the key guy was actually Noam Shazer is one of the eight authors of the paper, but he's really the guy who invented transformers in that sense, if you take attention for

• 36:10Speaker 00

  granted. Uh, he, I mean, again, I've talked to several of these people. They, they just hacked a million things is often the case until something worked. Uh, there was no great insight there, at least that they can.

• 36:22Craig S. Smith?

  Yeah.

• 36:23Speaker 00

  Maybe there is, but you know, but we don't know what it is yet, uh, probably. But, you know, Noam was the guy who, who like figured out how to make it work, right? And, and then, and then, and then, um, it, you know, I, I was smiling because he, you know, the transformer, so the recurrent neural networks are, are one of the problems is that they work sequentially. So they also have to be trained sequentially, which makes it hard to paralyze them. And paralysation is very important so that you can scale up to make things run faster. And, and the, and, and, you know, initially, you know, from the, the, the Montreal group, from, from, from Benjo's group, um, they were just doing attention on top of a recurrent

• 37:05Speaker 00

  neural network. And then what this guy said, you know, very, very important, right? And again, there's a history in machine learning of these things happening. It's like, actually, once you have attention, you can take out the recurrence. And I can just train in parallel and I can train on much bigger corporate. This is really the key thing that happened in that paper, right? Now, the system that they have at the end of the day is anything have and have even more so today is is anything but simple. It's a huge pile of different stuff and hacks left and right. And and you know, this actually slows down progress, right? Is that it's there's nothing simple about the, I mean, things tend to become simple over time as you understand the better.

• 37:45Speaker 00

  And I think that's the case with transformers, but they have all these different kinds of layers and mechanisms and whatnot, which we could go into. But like a transformer, the full architecture is a is a big pile of complication and and you know, that that holds us back.

• 38:00Craig S. Smith?

  Yeah, how does I don't mean to jump around, but as I mentioned, Sep Huckreiter,

• 38:11Craig S. Smith?

  who had developed LSTMs much earlier, how does that relate to attention? Because it's it's it's it's looking back a certain number of tokens and and holding them in memory. And that to me sounds like attention.

• 38:34Speaker 00

  Right, so LSTMs were very hot in machine learning for language at one point. And that was the point at which people are already focusing on language, but attention had not been invented yet.

• 38:48Craig S. Smith?

  Yeah.

• 38:49Speaker 00

  And and LSTMs are a type of recurrent neural network. The problem with recurrent neural network that really, for example, Mediocha Benio stop working on was what is called the vanishing gradients problem.

• 39:01Craig S. Smith?

  Right.

• 39:02Speaker 00

  You think about it, when you do back propagation, the further back you go, the more diffuse the the signal becomes. It's like. The neurons at the output layer, yeah, you are to blame, you're clearly wrong, right? Once you get to a neuron like, you know, a hundred layers behind, right? It's like, there's a million paths to that neuron, like, who knows if it's to blame or not? I guess like, if you, if you think of like, of a drop of ink, you know, diffusing through, you know, a, a, you know, a glass of water, right? As time passes, it becomes spread over everything. And if this is the credit or blame that you have to assign, right? Like, a recurrent neural network, over time, right? Even if it's just one layer, because you use it over and over again, it becomes, you know, an infinite, you know, layer network.

• 39:44Speaker 00

  And the signal just diffuses through it until it's like, you know, the, the ink all over the water. You don't, there's not, you can't learn, and there's nothing to learn. Everybody's equally to blame or not to blame, right? And, and LSTMs were a solution to this problem. And the solution to this problem, the LSTM, and, you know, people tried a bunch of things at the time. Neural Turing machines and blah, blah, blah. But LSTMs, again, which were, had invented, been invented way before, uh, by Sapp and his advisor, uh, Jürgen Schmidhuber, uh, notorious in machine learning for, for various, uh, reasons. Um, their solution was actually to do something that works a little bit like a computer. It has these memory cells, and you can actually write stuff into the memory cell, and then it stays there.

• 40:27Speaker 00

  Until we decide to access the cell, which is really like a von Neumann computer that we all use works. And then what they had was these gates that controlled access to the memory. There was like the forget gate, the et cetera. And what it did was it, it allowed you or didn't allow you to, you know, write into the memory, read the memory, et cetera, et cetera. Uh, and again, the key was that there was, this wasn't digital, right? It was something that you could gradually learn, right? Which again is what attention does later. It's like, it doesn't just say like, oh, I'm gonna pay attention to this. It's paying attention to all the past, you know, to what is called the context window. And this is like, well, the backprop says, pay more attention to this guy. And then the paper says, pay even more attention to this guy, right?

• 41:08Speaker 00

  And, and, and the LSTM was honestly a more complicated than hack here, and more unwieldy way to accomplish this effect. Attention in many ways is a misleading term, right? Because it really doesn't have anything to do with psycholod, you know, the psychological, you know, uh, version of that term. In fact, the irony is that Kyung-hun and, and Dimabada, now Dim is, is the intern who invented it, they didn't call it attention. Uh, Yoshua Bengio came at the end of the paper and put attention everywhere, and then they took it out again. As they tell the story, except they left it in a bunch of places and then, you know, and then the rest is history.

• 41:46Craig S. Smith?

  So then you've got the transformers.

• 41:51Craig S. Smith?

  Again, this is all in the connectionist

• 41:57Craig S. Smith?

  stream. And then there's some things happening off to the side. There's, or maybe we're going to talk about this next, in the next episode, but there's, you know, generative adversarial networks. And, well, let's talk about GANs. Is that, or is that on the evolutionary

• 42:26Craig S. Smith?

  algorithm end of things?

• 42:29Speaker 00

  No, very good. So GANs, generative adversarial networks, were very popular for a few years in AI. And now it's kind of died down. And it's true of you to point out their connection to evolutionary learning. Most people don't make that connection, but I think it is a very important connection. And GANs are actually how the term generative came. So the term generative from the beginnings was, is a term from statistical machine learning. And the idea of a generative model is that it's a model that generates the data. Right? It's like, I want a process by which you generate samples. Right? And then, so let me turn this the other way around, right?

• 43:12Speaker 00

  This is actually the whole Bayesian wave thinking is, I see a bunch of data. I'm going to postulate that they were generated by some process, and I'm going to try to identify what that process was. This is actually what generative learning is. So you come up with this model like a Bayesian network. This is like, well, this is how you generate patients that their symptoms. First, you decide which disease they have. And then you decide which symptoms they have. What fever are they going to have based on the fact that they have COVID? Et cetera, et cetera. You generate the data. And this is why it's called a generative model in opposition to discriminative models, which is what most of machine learning is about, which is to just go say like, "Okay, I'm gonna go forward from this data and try to decide, you know, what you suffer from, or whether it's a cat or a dog." As opposed to like, "How do I generate, you know, a cat or a dog?" Right?

• 43:57Speaker 00

  So there's always been this thing in machine learning between generative and discriminative. Discriminative always has always dominated and really still does, because it just works, it's an easier problem and it works better. It's in some sense less satisfying, because like, what is the deep science here? I'm just hacking to get the result. I wanna know, you know, like, for example, physics, right, is about generative models, right? Like, here's how the universe is generated. So there's a certain type of person who really likes, you know, who thinks generative models should win at the end of the day. Now, you know, Ian Goodfellow, who's the guy who invented GANs, was this another student at the same time in Yoshua Bengio's group, who had a huge group, and so it's not entirely an accident

• 44:40Speaker 00

  that so many people came out of it. And they were actually trying to solve a different problem, which was the problem, there's a type of neural network, which again, is a generative model, right? Not trained by backprop. And by the way, these were the ones that Jeff Hinton was always into. Jeff Hinton, even though he's a co-author on the paper, never liked backprop. He's actually on record as saying that backprop is not the future. He actually told me at one point, "The only reason I publish papers on discriminative, the only reason I do discriminative learning is in order to publish the papers." Because he was, what was, was generative neural networks that actually generate, you know, stuff, right? But the problem with those neural networks is that the inference is intractable,

• 45:24Speaker 00

  and so they were trying to come up with a way to make it tractable, right? There's a long history of trying to do this in various quadrants, of course, including the Bayesian's and whatnot. And generative adversarial networks were actually a way to do this. And, you know, without going into too much of the history or the details, what is the idea? Is that the adversarial network is a game between the generative network, that generates data, and the discriminative network that tries to classify it. So actually have a generative model in the classifier working against each other, which is a brilliant idea. Because the the the the idea is that the the the classifier is trying to distinguish

• 46:06Speaker 00

  between real data and the data that was created by the generative network. And so that forces the generative network to get better. Otherwise, in the beginning, it's crazy like, ah, ha, ha, you don't look anything like a cat. False, right? And then it gets better, but then that forces the discriminator to get better as well. So they get into this evolutionary arms race, right? It's co-evolution is the term in evolution, right? It's like the predator and the prey that generated in the discriminator. And this was like a genuinely new idea in the context of neural networks. In fact, at the time, you know, Yann Le Cun was going around saying like, this is the most important idea in neural networks of the last 20 years. Right? Then it petered out because it turned out, you know, it's very unstable. There's a lot that goes wrong.

• 46:51Speaker 00

  People never really, to my disappointment at least, you know, you want to take that further into the whole evolutionary realm of things because co-evolution is very powerful. So maybe that's what we need in machine learning. But like they, it was too hard to, to, you know, to cut a long story short. So it kind of petered out. But that's where this whole thing of generating deep fakes began, right? Ian in his paper had this few, you know, grainy images that were generated, right? They were like, I mean, at the time, this was actually very impressive, right? But then like, if you think about applying more power to this, the images get bigger, they get, you know, finer resolution. You get from images to video and the whole, you know, deep X industry was born.

• 47:33Speaker 00

  So even though now there are other techniques like, you know, diffusion and whatnot, which we could also talk about, being used to generate these things, so GANs are no longer the state of the art. But the whole, that term generative in the neural network context really took off there. And this whole notion, which a lot of AI things gave out, but they're like, you know, how can I dazzle you with generating text or generating images? Really started with with GANs.

• 47:58Craig S. Smith?

  Yeah. Uh, where should we go from here? We don't have a lot of time left. I wanted to, I don't know if it, if it needs a big chunk of time on its own, but uh, then you have Rich Sutton working on reinforcement learning, sort of coming at it from, you know, the behavioral psychologist's side. Uh, and that to me is probably the most powerful idea. Uh, and it's, it was certainly applied, but it seems to have

• 48:38Craig S. Smith?

  uh, be- become uh, the darling of, of generative AI right now with uh, you know, this deep seek is training with pure RL. And can you talk about how RL fits into all of this?

• 48:53Speaker 00

  Absolutely. So reinforcement learning is actually different. So there's supervised learning and supervised learning, reinforcement learning. It's a different type of learning that again has been around for decades. It is inspired by uh, animal psychology. In fact, that's where the term comes from, like, you know, Pavlovic experiments. You reinforce behaviors by giving rewards and et cetera, et cetera. The, the, and you know, Rich Sutton has been the longstanding leader uh, uh, in that area. Uh, so what is the basic idea in reinforcement learning? It's, it's, it's important to start by understanding that. And by the way, reinforcement learning, like supervised learning, can be applied with any of the paradigms, but it has always been most popular with neural networks.

• 49:34Craig S. Smith?

  Yeah.

• 49:35Speaker 00

  So with supervised learning, you have a teacher. The teacher says, aha, right answer, like, you know, yes, this is a cat or like, no, this is a dog. Like you label things with the right answer.

• 49:46Craig S. Smith?

  Yep.

• 49:46Speaker 00

  Which makes learning easier, but is, you know, where, where does that data come from? There are many domains where you don't have it. If you look at how animals learn, right? Or like how we learn, right? You touch the stove and then you learn not to. But the interesting thing about reinforcement learning is that I feel the burn when I touch the stove. But I really shouldn't even have moved my hand towards it. So you need to, to, to uh, uh, back up that signal.

• 50:14Speaker 00

  To where, to, you know, so supervised learning, if it was just like at the last moment, don't touch it, that would be supervised learning. The whole idea in reinforcement learning is they have this thing called delayed rewards. You do something and you only see the result later, and then you need to propagate those results back to where you should have taken the actions. A famous example of this is AlphaGo. Yeah. Right? And indeed, reinforcement learning goes back to the 50s and, you know, the paper that coined the term machine learning had a precursor of reinforcement learning to learn to play checkers, and they learned to play that at a human level. And the idea is this, is like, I play a whole game of checkers, either computer with you, the human. I don't know if my moves are good or bad, the only thing I know is at the end of the day, I won or I lost or we tied. Right?

• 50:58Speaker 00

  So there's a very small piece of reward, as it's called, reward is like you won, right? You get a, you know, you get a reward, right? You got a, and reinforcement learning is a set of techniques, or really it's just the problem, reinforcement learning is the problem of like, how can I take those rewards and propagate them back to where I need to make a decision, so that I make the right move that many steps further will lead me to win the game, right? And this, you know, combined with neural networks for vision applied to the Go board, was of course what let, you know, DeepMind, you know, beat Lee Sedol and etc. Now, reinforcement learning as done by Rich Sutton and other people for a long time was

• 51:42Speaker 00

  always very theoretical. They wanted to, they did these tiny little experiments that were, you know, not convincing to anybody, honestly, and they tried to prove theorems. What the DeepMind folks did when they came along that was very important was they said like, you know, to hell with all that, we're just going to engineer the heck out of this and get it to win. And they did, right? And so again, for a while, deep reinforcement learning was very hot, there were like many papers about it. They actually started by doing it in video games in Atari and then went on to things like Go and chess. That unfortunately has petered out also because it didn't, DeepMind's agenda when they came out was deep reinforcement learning, and their idea was like, we're going to start with these games and then we're going to move to robots in simulation and final robots

• 52:27Speaker 00

  in the world, and this is how we're going to solve AI. Right? Them Mr. Sabes was going and saying like, we're the Apollo project of AI, we're going to get there. They don't say that anymore. And so these days reinforcement learning, as you pointed out, is again popular. But honestly, those of us who know this history at this point have some justified skepticism because we keep seeing this over and over again, which is people come out saying like, oh, look at this great success for reinforcement learning. And then when you look at it more carefully, there was no great success. What they were doing was effectively supervised learning or could have been done by supervised learning. In fact, the main thing, there's this thing called reinforcement learning from human feedback that OpenAI popularized, that they say is very important to make, you know, their chatbots work and whatnot.

• 53:11Speaker 00

  There was this paper out of Stanford that basically showed effectively you're just doing supervised learning.

• 53:16Craig S. Smith?

  Yeah.

• 53:16Speaker 00

  So I would say the same with DeepSeek. Until I see proof to the contrary, I will assume, and again, the evidence is points in that direction, at least as far as I can tell, that they're not, there's really no reinforcement learning magic going on. The problem with reinforcement, so the appeal of reinforcement learning to the people who really want to solve AI is that like, clearly we do something like that as even a correspondence to some of the neuroscience with dopamine circuits and whatnot. The problem is that people have disappointingly, despite so much effort, they have never really truly managed to make it work. In fact, it boils down to this. If your rewards are delayed, it's learning from sparse delayed rewards.

• 53:59Speaker 00

  If the rewards are not very delayed, they're not very sparse, then you can just use supervised learning, which is way easier. If the rewards really are delayed and sparse, then it doesn't work. So so far, we haven't really solved that problem. So when you see claim, in the meantime, however, in the last few years, reinforcement learning has just become the sexy term that you can throw around and you call your thing reinforcement learning and it sounds more impressive even to yourself, which seems to be also what had happened with with DeepSeek.

• 54:25Craig S. Smith?

  Yeah. Okay, we have a couple of minutes left. Where should we go from here?

• 54:35Speaker 00

  Well, let me take you that question. Let me take that question the following way. Where should the neural network community go from here?

• 54:42Craig S. Smith?

  Right.

• 54:43Speaker 00

  And and I think the higher order bit is, as we alluded to earlier, what we've seen is broadly from the point of view of we have, but even from the point of view of machine learning and then deep learning, what we're seeing is more and more people doing more and more research about less and less. Right, so I think where we really need to go from here is we need to broaden the scope to all. I mean, by the way, as an example, there are papers now showing that maybe LSTMs and recurrent networks aren't such a bad idea, right? A lot of things that seem dead in machine learning or AI have a tendency to come back. You know, neural networks being one of them. So I wouldn't be surprised at all if one of these things has dethroned transformers by next year. Or maybe it'll stagnate for 10 years or who knows, right? But I think a lot of people should be doing a lot.

• 55:28Speaker 00

  We really should let a thousand flowers bloom. And right now, it's basically one flower blooming or one petal of one flower blooming, and that's not very healthy.

• 55:36Craig S. Smith?

  Yeah, and that phenomenon, because we've talked about it in, you know, during the supervised or the supervised learning vision phase, when everyone is focused on one narrow aspect and then they're forgetting about the Bayesian's or something, that's a matter of funding, isn't it?

• 55:59Speaker 00

  Oh, funding is a big part of it. In fact, Minsky and Papert publicly admitted that the main reason they wrote that book that killed neural networks for a while was that, you know, DARPA was the big funder of AI research, right? So historically, the biggest funder, you know, when we finally get to AI, we will have DARPA to thank as much as anybody for having, you know, stuck with it until, you know, the industry took over. But they were getting jealous that, you know, neural networks were so sexy and they were getting all the DARPA funding and symbolic AI wasn't getting enough. So they said like, we've got to get rid of these guys, and so they did. So funding has always been an issue, right? I mean, particularly in research and industry as well, the phenomenon is a little different.

• 56:41Speaker 00

  DARPA for a long time, after that, in fact, when I became a researcher, DARPA did not fund much machine learning research, if any, because the consensus, which in the field and among DARPA program developers, like, like, machine learning is a waste of time. In the meantime, machine learning was taking off and DARPA was still not waking up. Finally, they woke up in the 2000s and started, you know, getting into machine learning. And again, like, the funding, the funders themselves, whether in industry or in academia or, you know, the funding agencies, they're very prone to this, you know, hurt behavior. It's like the VCs, right? VCs are supposed to be looking out for different things, but they're all looking out for the same things. So, you know, you need to inject some, you know, in, there's this, you know, gradient descent, right?

• 57:26Speaker 00

  There's like, look, going, looking for the optimants, what neural networks do. But there's this method called simulated annealing, where you actually inject noise, which seems like a bad thing, but the noise actually helps you not go straight to the local minimum. And we, you know, find, find a deeper one. We need more simulated annealing in, in AI research.

• 57:44Craig S. Smith?

  When you need somebody, you need somebody. You want to hire people fast. The longer you wait, the more the market drifts, the more your needs drift. For example, you just realized your business needed to hire somebody yesterday. How can you find the right candidate fast? Just use Indeed. When it comes to hiring, Indeed is all you need. Stop struggling to get your job post seen on other job sites. Indeed's sponsored jobs help you stand out and hire fast. With sponsored jobs, your job postings jump to the top of the page for your relevant candidates, so you can reach the people you want faster. And it makes a huge difference.

• 58:26Craig S. Smith?

  According to Indeed data, sponsored jobs posted directly on Indeed have forty-five percent more applications than non-sponsored jobs. One of the things I love about Indeed is that it makes hiring so fast. I've searched for candidates before, and it can take months, particularly if you're just working through word of mouth. In those days, I didn't have Indeed. Plus, with Indeed sponsored jobs, there's no monthly subscriptions, no long-term contracts, and you only pay for results. How fast is Indeed? In the minute I've been talking to you, twenty-three hires were made on Indeed, according to Indeed data, worldwide.

• 59:09Craig S. Smith?

  So there's no need to wait any longer. Speed up your hiring right now with Indeed. And listeners on this show will get a seventy-five-dollar sponsored job credit to get their jobs more visibility at Indeed. Indeed.com slash I on AI. Just go to Indeed.com slash I on AI. That's Indeed, I N D E D dot com slash I on AI. All run together, E Y E O N A I. Go right now and support our show by saying you heard about Indeed on this podcast. Indeed.com slash I on AI. Terms and conditions apply. Hiring Indeed is all you need.