#236 Pedro Domingo's on Bayesians and Analogical Learning in AI

This episode is sponsored by Thuma. Thuma is a modern design company that specializes in timeless home essentials that are mindfully made with premium materials and intentional details. To get $100 towards your first bed purchase, go to http://thuma.co/eyeonai In this episode of the Eye on AI podcast, Pedro Domingos, renowned AI researcher and author of The Master Algorithm, joins Craig Smith to explore the evolution of machine learning, the resurgence of Bayesian AI, and the future of artificial intelligence. Pedro unpacks the ongoing battle between Bayesian and Frequentist approaches, explaining why probability is one of the most misunderstood concepts in AI. He delves into Bayesian networks, their role in AI decision-making, and how they powered Google's ad system before deep learning. We also discuss how Bayesian learning is still outperforming humans in medical diagnosis, search & rescue, and predictive modeling, despite its computational challenges. The conversation shifts to deep learning's limitations, with Pedro revealing how neural networks might be just a disguised form of nearest-neighbor learning. He challenges conventional wisdom on AGI, AI regulation, and the scalability of deep learning, offering insights into why Bayesian reasoning and analogical learning might be the future of AI. We also dive into analogical learning—a field championed by Douglas Hofstadter—exploring its impact on pattern recognition, case-based reasoning, and support vector machines (SVMs). Pedro highlights how AI has cycled through different paradigms, from symbolic AI in the '80s to SVMs in the 2000s, and why the next big breakthrough may not come from neural networks at all. From theoretical AI debates to real-world applications, this episode offers a deep dive into the science behind AI learning methods, their limitations, and what's next for machine intelligence. Don't forget to like, subscribe, and hit the notification bell for more expert discussions on AI, technology, and the future of innovation! Stay Updated: Craig Smith Twitter: https://twitter.com/craigss Eye on A. I. Twitter: https://twitter.com/EyeOn_AI (00:00) Introduction (02:55) The Five Tribes of Machine Learning Explained (06:34) Bayesian vs. Frequentist: The Probability Debate (08:27) What is Bayes' Theorem & How AI Uses It (12:46) The Power & Limitations of Bayesian Networks (16:43) How Bayesian Inference Works in AI (18:56) The Rise & Fall of Bayesian Machine Learning (20:31) Bayesian AI in Medical Diagnosis & Search and Rescue (25:07) How Google Used Bayesian Networks for Ads (28:56) The Role of Uncertainty in AI Decision-Making (30:34) Why Bayesian Learning is Computationally Hard (34:18) Analogical Learning – The Overlooked AI Paradigm (38:09) Support Vector Machines vs. Neural Networks (41:29) How SVMs Once Dominated Machine Learning (45:30) The Future of AI – Bayesian, Neural, or Hybrid? (50:38) Where AI is Heading Next

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The Bayesians are the hardest core of the five tribes, the most tribal. They truly believe that either what you're doing is Bayesian or you're wrong. And they will never die. The Bayesians are a tribe that comes from statistics. So machine learning, of course, is closely related with statistics. It builds on it in many different ways because after all, both of them are about building models from data and then acting on them. Of course, statistics has a much longer history. Statistics and probability going back centuries even. Statistics in the last 100 years at least, if not more, has been dominated by what is called the frequentist school as opposed to the Bayesian school. So the Bayesians in statistics have always been the oppressed minority and they have a big chip on their shoulder about it.

• 0:44Speaker 00

  And so what is the difference between them? The difference is about what is the definition of probability. The truth is that even today, for all the thousands of papers or millions that have been written, nobody really knows exactly what probability is. Probability is actually a surprisingly slippery concept.

• 1:01Craig S. Smith?

  Create an oasis with Thuma, a modern design company that specializes in furniture and home goods. By stripping away everything but the essential, Thuma makes elevated beds with premium materials and intentional details. I'm in the process of reorganizing my house and I'm giving Thuma a serious look for help in renovating and redesigning. Thuma combines the perfect balance of form, craftsmanship, and functionality. With over 17,000 five-star reviews, the Thuma bed collection is proof that simplicity is

• 1:44Craig S. Smith?

  the truest form of sophistication. Using the technique of Japanese joinery, pieces are crafted from solid wood and precision cut for a silent, stable foundation. With clean lines, subtle curves, and minimalist style, the Thuma bed collection is available in four signature finishes to match.

• 2:10Craig S. Smith?

  Any design aesthetic. Headboard upgrades are available for customization as desired. To get a hundred dollars toward your first bed purchase, go to thuma, that's T H U M A dot C O slash I on A I. I on A I all run together, E Y E O N A I. So for a hundred dollars off your first purchase, go to thuma dot C O slash I on A I. That's T H U M A dot C O slash I on A I to receive a hundred dollars off your first bed purchase.

• 2:54Speaker 00

  I'm Peter Domingos. I'm a professor of computer science at the University of Washington in Seattle. I'm an AI researcher. I've been doing it since nineteen ninety, nineteen ninety eight, actually.

• 3:10Speaker 00

  I'm also known as the author of The Master Algorithm and Twenty Forty, A Silicon Valley Satire. The Master Algorithm in particular is an introduction to machine learning for a broad audience that that was very popular. And I guess that's why we're here today.

• 3:27Craig S. Smith?

  That's right. So tell us about the Bejans, which was one of the five tribes.

• 3:36Speaker 00

  The Bejans are the hardest core of the five tribes, the most tribal.

• 3:43Speaker 00

  They truly believe that either what you're doing is Bayesian or you're wrong.

• 3:52Speaker 00

  And they will never die. Some of the other tribes, sometimes they look a little tired, but the Bejans always keep going to thick and thin. Now, why is that and who are they? The Bejans are a tribe that comes from statistics. So machine learning, of course, is closely related with statistics. It builds on it in many different ways because after all, both of them are about building models from data and then and then acting on them. Of course, statistics has a much longer history, going back to statistics and probability, which of course, again, is closely related, going back centuries even. But statistics in the last hundred years, at least, if not more, has been dominated by what is called the frequentist school.

• 4:38Speaker 00

  As opposed to the Bayesian school. So the Bayesians in statistics have always been the oppressed minority. And they have a big chip on their shoulder about it. And so what is the difference between them? The difference is about what is the definition of probability? The truth is that even today, for all the thousands of papers or millions that have been written, nobody really knows exactly what probability is. Probability is actually a surprisingly slippery concept. And but the two main definitions are the frequentist one, which in some ways is the more natural one, which is, it says, just a probability is just the limit of a frequency.

• 5:18Speaker 00

  For example, why do I say that the probability of, you know, getting heads on a coin toss is a half? Because if I keep flipping a coin over and over again, you know, approximately half the time I'll get heads. And if I toss it infinite times, it'll be exactly half, provided the coin isn't biased. So probability in this view is just frequency. But then there's a lot of problems with this, like, for example, well, OK, but then what is the probability that Trump will win the election? You can't do infinite trials of Trump and Kamala running against each other. So somehow that definition seems to fall short. And there's a whole bunch more problems like that, which we could get into.

• 6:00Speaker 00

  And so the Bayesians actually have a very different reply to this question. They say probability is subjective. It is inherently subjective. Probability is just your belief. Like I say, oh, the odds that Trump will win are 55%. Prove me wrong, right? So actually, you're entitled to whatever beliefs you want. And the Bayesians just say, how, given your beliefs, you should then calculate new beliefs such that they're all coherent. But it doesn't tell you what your belief separation should be. And in particular, in Bayesian statistics and then throw that in Bayesian learning, which again inherits all of that machinery. So Bayesian machine learning, in a way, is starting with Bayesian statistics

• 6:45Speaker 00

  and then adding all the computational power that we have today, including in some ways that weren't necessary in other methods, but are necessary in this one that we can get into. But the basic way that Bayesian learning and statistics work is based on something called Bayes' theorem, which is in fact where the name comes from. Which is actually ironic, because the theorem is not the two Bayes, it's the two Laplace. So really, it should be called Laplacian learning. But Laplace already has so many other things to his name than that, you know, adding one more might be too much. So what does Bayes' theorem say? It says that the way you so you have the whole Bayesian point of view is that, or even statistical point of view, at least by some standards, is

• 7:29Speaker 00

  there's a probability distribution over everything that could happen in the world. And and what Bayes' theorem says is, is you start with your prior probabilities, which is the probabilities that you assign, for example, to this patient having cancer or this email being spam before you've seen the patient or before you've seen the the email. So these are your beliefs a priori. And then, as you see evidence, you update your beliefs. And the probability of what you see, given your model, is called the likelihood. And what Bayes' theorem says is that your posterior is the prior times the likelihood normalized. And and your your your prior times your likelihood is not your new prior for when new evidence comes in.

• 8:14Speaker 00

  So I'm playing the stock market. I say this stock is going to go up. It goes up or down. I I compete my likelihood, I pick my prior, and now this is my prior for the next day. So this is, in essence, how Bayesian learning and Bayesian statistics work.

• 8:27Craig S. Smith?

  Can I just ask, tell us who Bayes was?

• 8:31Speaker 00

  Thomas Bayes or Reverend Bayes was was an English Protestant preacher clergyman, who, you know, in the, what, 18th century, I believe, who, you know, like many cultured people at the time, had mathematics as his hobby. And, you know, he so he so probability has an interesting history going back several, you know, 100 years. It actually started out in a very non-serious way as people trying to figure out how to win at gambling.

• 9:05Speaker 00

  Right, can I predict what are the chances of whatever this combination of dice coming up and so on? And it kind of went on from there. So when they, and what was formulated was in some ways a proto version of his theorem. Not in, you know, you can see the idea there, but he didn't really formulate it, as I said. Laplace did it, but in some ways that he was the one who started things on this track of, well, let's think about the prior probability and the likelihood and multiply it to, and by the way, this is something that people often naturally gets people confused. Frequentists have no problem whatsoever with Bayes' theorem. In fact, you learn Bayes' theorem in Stats 101. It's one of the first things you learn. It's a really simple theorem. It's almost just the definition of conditional probability.

• 9:49Speaker 00

  It barely merits being called a theorem. The thing that makes it controversial is when the Bayesian say, ah, but, but your prior probability is whatever you want to make it. This is where the frequentists go like, whoa, whoa, whoa, like, what are you talking about? You can't just make stuff up. And you can see why there'd be a lot of resistance from scientists in general to this idea that, hey, are you just gonna make probabilities up? Like, what is that? Science is supposed to be objective. But the point that, you know, my point of view is actually the frequentists and the Bayesians both have good points. So you can't truly ignore either of them. And, and, and one very important point that the Bayesians make is that like, yeah, but you frequentists are also making assumptions

• 10:30Speaker 00

  that are just assumptions. You're just not being explicit about them. So it's rare to be explicit about them. So there you go.

• 10:37Craig S. Smith?

  And the Bayesians, it sounds a little bit like Euclidean geometry with, uh, with, uh, the, the axioms, uh, and everything is built on top of that. And those are, are just taken on faith.

• 10:53Speaker 00

  No, absolutely. So the Bayesians are, um, probably again, of all the tribes, the one that most strongly believes in, we should solve machine learning from first principles. From a few axioms, and in particular from Bayes' theorem, and you know, there's a few others of probability. And that's the right way to do it. Don't talk to me about the brain, because the brain is a mess. Don't talk to me about evolution, because evolution is a hacker. We have no guarantee that any of those did anything meaningful. I mean, you know, the optic nerve comes out of your retina towards the outside. Like, what the heck is that, right? Why would the brain make any more sense than that? So the whole Bayesian idea is that like, yeah, you know, I'm gonna tell the axioms by which you need to reason.

• 11:35Speaker 00

  And they even have this expression called turning the Bayesian crank, which is, I have my axioms, you bring me your data, and I turn the crank, and I produce the output. And anything else that you do is just wrong. Now, of course, where things get interesting, at least from the machine learning point of view, and in fact, from the statistics point of view as well, is that trying to do this in any non-trivial setting is just computationally too hard. It's completely infeasible. And in fact, a large part of the reason why for many decades, you know, Bayesians were really in the background in statistics, is it just you couldn't do Bayesian statistics. It wasn't possible. And again, why they, part of why they're on their rise, even within statistics and other disciplines like economics and other areas of science today,

• 12:19Speaker 00

  is that we now have the computing power to do Bayesian calculations. But even then, doing them requires doing approximations and whatnot. So at the end of the day, the Bayesians wind up making as many unfounded or dubious assumptions as everybody else, which is ironic considering where they started from.

• 12:37Craig S. Smith?

  So tell me how they, how they put this into a computer program that's, that learns.

• 12:45Speaker 00

  Absolutely. So Bayesianism doesn't tell you, Praray, what kind of model to use. And so what happens is that there's a Bayesian version of everything. So for example, there's Bayesian neural networks. There's this, you know, guy David Mackay, that did a very brilliant PhD thesis on how to make Bayesian neural networks. And for a while, they actually took over the community. It was very clever. He kind of took it over from the inside. There's Bayesian versions of symbolic learning algorithms. There's Bayesian versions of analogical learning algorithms. So like, whatever learning algorithm you come up with, there's a, you know, it's almost, you know, certain that tomorrow, the Bayes will come up with their correct Bayesian version of it.

• 13:27Speaker 00

  Having said that, something that is very distinguishing, I would say, of the core Bayesian approaches is that they represent probabilities explicitly. The models actually have not just weights or some other parameters with no clear meaning, is the parameters are probabilities. So for example, the most, probably most wise user, at least best known type of Bayesian model in AI is a work called Bayesian Networks. And Houdini Pearl actually won the Turing Award for developing Bayesian Networks, so it's a big deal. So what is a Bayesian network? A Bayesian network is an answer, it's actually an answer to a problem that goes back to the early days of AI. We talked about, when we talked about the symbolism that like, yes, the world is full

• 14:11Speaker 00

  of uncertainty and ambiguity and confusion and whatnot. And in theory, the right way to handle it is probability. I think everybody more or less agrees with that. There's infinite variations, but at the end of the day, the axioms of probability are hard to get around. And in fact, that's what the Bayesians, you know, keep pounding on. But the problem is that it's computationally intractable. And the first reason why it's computationally intractable is that if I give you, you know, let's suppose I have a hundred variables of interest and they're all binary, yes or no. Do you have a fever? Do you have whatever, diabetes? Do you drive a car? Et cetera, right? Just let's just keep them binary, Boolean, for the sake of argument.

• 14:52Speaker 00

  There are 2 to the 100th possible states of those variables. So in order to naively represent that, I need 2 to the 100th probabilities. And there is no computer in the whole, in fact, if you used an atom in the universe to store each one of these, there wouldn't be enough atoms in the universe to just store that distribution. It's just completely impossible. So you have to find some way to summarize that distribution. And Bayesian networks take advantage of this thing called conditional independence, which is really the crucial property. And so, for example, so backing up slightly, we say that two things are independent when the probability of one doesn't depend on the probability of another.

• 15:35Speaker 00

  So the probability that I smoke probably has nothing to do with the probability that somebody will observe a shark tomorrow in the Pacific Head for a year. Like, they're independent, right? Now, independent is not very interesting. The interesting notion is conditional independence. Like, for example, A is conditionally independent of B given C, not because A and B are independent, but because once I know C, they become independent. In other ways, all the information about B that A contains is also contained in C. And if you think about it, the entire universe works by this principle because, at least according to physics as we understand it today,

• 16:19Speaker 00

  what happens to me is conditionally independent of what happens far away, conditional on what happens close to me. Right? So like, the light from distant stars can hit my retina, but before it hits my retina, it will hit a point, you know, in space very close to my retina. So without conditional independence, you know, life would be completely intractable and cognition would be impossible and so on. So what Bayesian networks are, a representation that tries to... In the knowledge engineering days, people go and, for example, Bayesian networks are very good for things like medical diagnosis because you really do want to precisely quantify those probabilities, right? These are important decisions to take. And you can say like, okay, what are the things that...

• 17:03Speaker 00

  What are the illnesses that could cause a fever? And then you draw this graph, the Bayesian network, with an arrow from, you know, all these different illnesses to fever or to, you know, high blood sugar. There's one from diabetes, but there isn't one from, I don't know, brain cancer. Right? And then you draw a big diagram like this. And then the only parameters that you have to learn, this is the key, are for each node in this network, given its parents. I only need to represent the probability of fever given the things that cause it, which is a vastly, vastly simpler problem than letting everything depend on everything else. So the amount of memory... So conditionally independence actually lets you build a reasonable-sized model.

• 17:47Speaker 00

  Then there's also the problem of doing inference, which is a lot of what, you know, who the Pearl worked on, in a way that that doesn't blow up. And also of learning this from a finite amount of data. Because again, if I need to learn to the hundred probabilities just by measuring frequencies, I will never be done. But once you have a Bayesian network, a lot of this starts to be feasible. It's, it can still be very computationally expensive. And in particular, Bayesians have this thing in which they're different from everybody else, in which they don't just try to find... Some people would even say that this is what defines Bayesianism, not even just having prior probabilities. Is this notion that there is no right model. Right? There is no one neural network or no one decision tree.

• 18:32Speaker 00

  They are all possible. They all have their probabilities. And what I have to do is not find the best model, unlike what all the others do, is find the probability of each and every possible model and then to make a prediction, I average over all of them. Now you can see that from a theoretical point of view, this is very attractive, right? Of course, I don't know if this is induction. I don't know for sure what's going to be the right model, so I should just average over them. The problem is that there's, you know, often a doubly exponential number of models. Not just exponential, but like you take all those states, which are already exponential, and now there's an exponential number of models over those. So you have not just 2 to the n, but 2 to the 2 to the n, which is just absurd, right? And Bayesians tighten themselves in knots

• 19:17Speaker 00

  just trying to compute some approximation to this average of all models efficiently. And it's a massive waste of time. I really feel their pain. But you know, just to give you an example, Google, before deep learning came along, their whole ad placement system, which is what, you know, made all their money, was a big Bayesian network. They just had a massive Bayesian network with, I don't know, probably millions of nodes that were things like words, right? The words that appear in the page and on the ad that you want to, you know, show and various other things and how, you know, these things depend on each other. And ultimately, what you, what they want to do is like predict the probability that you will click on this ad if it contains these words and these links.

• 20:01Speaker 00

  And that was a huge Bayesian network. So, you know, there's a many important applications, even today, of Bayesian learning.

• 20:09Craig S. Smith?

  Yeah, just one question on that. How do you take the...

• 20:17Craig S. Smith?

  The, what was it, cancer and all the various causes of cancer, was that the example you gave? How you compute the probability for each of those

• 20:31Craig S. Smith?

  nodes?

• 20:32Speaker 00

  So, I mean, let's start by taking a simple example. Suppose that I had just two symptoms, fever and, you know, blood glucose. And what I want to do is predict the probability that you have diabetes. Okay, then what I need, this is a very small model, is this little table that goes like this. If you have no fever and low blood glucose, then your probability of diabetes is low, point oh five, let's say. And if you, but if you have a high fever and low blood glucose, that slightly increases your probability of having diabetes. By the way, I'm not a doctor, so I'm just making this up. But, but, you know, the main thing is like you still have low blood glucose. So maybe now your probability of diabetes is, you know, point one.

• 21:16Speaker 00

  Now, if you have high blood glucose, your probability of diabetes jumps up to whatever, right? And finally, if you have both of them, it's, let's say, you know, point eight. So you just need to specify probability for each combination of these variables. Now, in a Bayesian network, you only need to do it for a combination of the parents of that variable in the model, because given those, it's conditionally independent of all the others. That's the big win, right? I don't need to condition on a hundred things, provided that if I know these two, the others become irrelevant. If I don't know these two, this is actually the beauty of Bayesian networks, is that then I can infer the probability of those two from the others, and then from those, the probability of actually having diabetes. So I don't actually need to know them. But knowing the structure of the world is actually what makes me, you know,

• 22:00Speaker 00

  lets me have a table with two parents and therefore four lines, as opposed to a table with two to the hundred lines.

• 22:06Craig S. Smith?

  Yeah, and, and you, you said there are, you mentioned the Google example, but what are the other applications that use Bayesian networks today? And, and is, is there still progress being made in, in the Bayesian school?

• 22:28Craig S. Smith?

  Other networks or or other structured architectures, I should say.

• 22:33Speaker 00

  Yeah, so I would say that, for example, medical diagnosis, as I mentioned, is an important. So there are some really good, very sophisticated vision networks from medical diagnosis that exist today. And by the way, they've actually existed for decades now, and they should, and they're better than doctors.

• 22:52Speaker 00

  We've been, you know, like, machine learning has been better than doctors for, I don't know, fifty years. The reason it's not widespread is that the doctors won't let them be used. Right? They're the gatekeepers. Why would the gatekeepers unemploy themselves, right? So there's a very interesting sociological aspect to this, but like, next time you hear this big polemic, oh, does ChatGPT do diagnosis better than doctor? Even expert systems, pre-learning in the seventies already did, you know, medical diagnosis better than doctors at the things that they were designed for. It turns out that beating doctors is not that hard, and in particular, famously, humans are very bad at this theorem. They don't understand that you need to take the prior probability into account, and doctors make that mistake too.

• 23:33Speaker 00

  So if you have a disease that is extremely rare, but you see these symptoms that are consistent with it, the doctor will tend to overestimate the probability, not realizing that, you know, even with these symptoms, it's still very unlikely. So doctors really screw this up all the time. I mean, I have, you know, Bayesian colleagues who say, I went to a doctor and I was just shocked at how, you know, stupid he was. Telling me that I probably have this and that, and I'm like, oh, my God, like, this guy doesn't even know stats one oh one. So, you know, Bayesians also have a certain superiority complexity, which is not entirely unjustified.

• 24:08Speaker 00

  But but anyway, so so medical diagnosis is another class of of applications. I would say that in general, and this, I think, is actually what is important for people to know is like, what are the types of problems for which these different types of machine learning make more or less sense?

• 24:23Speaker 00

  For for Bayesian learning, I would say it's typically when quantifying uncertainty precisely is very important.

• 24:30Speaker 00

  Right? If not, if just making a prediction, which is what your typical neural network does, it just says, "Well, it's going to be this." Right? If that's not enough, if you really want, like, you're a doctor, you know, to go back to that example, you don't want to tell your patient, "You have to, you know, you have to have surgery." That's the patient's decision. What you need to say is, "Look, you know, there's this probability that you have a tumor. If you don't get surgery, there's this probability that you die," and et cetera, et cetera. And then, you know, and then the patient, according to their own utilities, as they call, can make their decision. A very, you know, interesting type of application, which, you know, there are some, you know, famous historical examples, is, let's say, for example, you know,

• 25:13Speaker 00

  America just lost a submarine in the Pacific. The submarine has disappeared. Pacific is big, right? You can search every square mile. And if you just start doing things randomly, you might not get anywhere. Now, Bayesian learning is really good for this, because it says like, "Okay, let me start with my prior distribution over where the submarine would be." It's not uniform, right? It's more likely to be near the routes that they usually take, blah, blah, blah. And now let me start, you know, bringing in all the relevant evidence, right? Was there a storm here? Did I hear some whatever, sonar ping there? It's, all those pieces of evidence might be very weak, such that, for example, a symbolist looking at it will be like, "I can't conclude anything

• 25:55Speaker 00

  from that." But as you accumulate these pieces of evidence, you know, the probability of each square in the grid starts going up or down. And with enough pieces of weak evidence, you often wind up with a very strong belief that, "Hey, the submarine is right here." And then you go there and you find the submarine. So Bayesianism is very good for stuff like that. I would argue that it is, you know, in many ways, the right approach to problems like that, or at least the best one we have at this, at this point.

• 26:23Craig S. Smith?

  Yeah. One question. I mean, we keep using the term "machine learning," but this sounds like a predictive system, which is different than a learning system.

• 26:34Speaker 00

  No, absolutely. So very good point. And again, we are guilty of conflating the two. So if you were to do what I just described, let's say, you know, you are, you know, an admiral of the Navy trying to find the missing sub, right? And I say, "Let's do this," right? And one of your questions would be like, "But where do all those probabilities come from?" Right? And this is where I say, "Well, you can impute them. You tell me what they should be." A frequentist would never do that. That's just not allowed in frequentist statistics. As a result of which, there are a lot of problems that they just can't touch. You say like, "No, you tell me, what is your prior distribution over where in the Pacific this submarine is gonna be?" Right? And I'm just gonna believe you.

• 27:17Speaker 00

  And I'm gonna tell you how to then consistently and soundly go from these estimates right now. What happens in practice, as I alluded to earlier, is that like, as much as possible, we don't just depend on our prior probabilities. We also use data. But the data, they, you know, again, the data about the routes that submarines take and blah, blah, blah. Right? But what those data do is they cause us to update our prior. And again, when you have a lot of data, in many ways, Bayesianism becomes superfluous. It's a cost that you don't need to pay. When you have very little data, unless you use your prior as your host. So this is genuine, like, so for example, a famous example is, you know, let's say I'm gonna flip my coin, right?

• 28:02Speaker 00

  And I only flip it once. Right? Frequentist statisticians will use what is called the maximum likelihood principle. To them, that's the right thing to do. But the maximum likelihood principle says that if I flip a coin once and it comes up heads, the probability of heads is 100%. Which is stupid, right? And then if the coin comes up heads twice, it's still 100%. And if it comes up heads and tails, then it's 50/50. But the Bayesians don't suffer from this. They're like, "No, a priori, let's say you believe that the coin isn't biased, so a priori, the probability is 50/50." And now when the coin comes up heads, that makes it, you know, depending on how I exactly do this, let's say 60/40. And then if it comes up heads, right? So this is actually a mutt.

• 28:44Speaker 00

  I mean, honestly, you would be crazy to not do this when you have a small amount of data. And in fact, what the frequentists do is that they have a bunch of ad hoc techniques, right? This is part of the, you know, the Bayesian criticism is correct, that they wind up making all these assumptions, because they, but all of the assumptions are incoherent, which really, you know, annoys the And in fact, one analogy that might be helpful is that, you know, these are two churches of statistics. And the Bayesian's are the Catholic Church, right? They have one set of true beliefs. And the frequentists are the Protestant. Like, anybody can set up shop and make their assumptions and start, you know, cranking away. But at the end of the day, it's all this big, you know, mess that's not really consistent.

• 29:27Speaker 00

  So, uh, you know, if, if, if you value it, I mean, often the Bayesian's, I think this has an interesting relation to why they're so fanatic. They're often really smart people. And people who really care about getting things right. And they're just disgusted by the mess of ordinary statistics and machine learning. And they want to do it the right way. But so to answer the second part of your question, there is very much still Bayesian research going on because there always will be. As I said, Bayesian, Bayesian stuff will never die. Of course, the percentage of the machine learning research that is Bayesian these days is much smaller than it was 10 or 20 years ago. Partly because the amount of connectionist research has exploded.

• 30:08Speaker 00

  But also, you know, a lot of the people have sort of like migrated from doing Bayesian uh, learning to doing connectionist learning or just, you know, you know. So the hardcore Bayesian's always keep doing that, but some of the others drift away. There are famous examples of, of people doing this. So yes, there's still progress happening. And for example, doing Bayesian inference is, is one of those intractable problems that will always, there will always be progress to be made on. So for example, in particular, uh, uh, there's this technique called, you know, Markov chain Monte Carlo, which is doing inference by, by generating random samples and, and counting them and whatnot. Which actually goes, you know, the guy who invented, the people who make this were in the Manhattan Project.

• 30:50Speaker 00

  And they were trying to simulate nuclear reactions and, and, and thieves the neurons, the neutrons, sorry, with, with the, you know, get over the threshold to cause it and so on. So when, when you have to deal with intractable probabilities, I mean, like Markov chain Monte, or Monte Carlo methods in general, are one of the most studied subjects in all of science. And one of the most widely used algorithms in all of science, like top 10. One of the top 10 was is Markov, you know, is is is Monte Carlo sampling techniques. And even outside of AI, right? Even outside of machine learning, people in statistics, people in physics, people in economics, people in all these different sciences, they need these methods.

• 31:35Speaker 00

  So they will continue to research them. The world is much larger than than, you know, neural networks.

• 31:42Craig S. Smith?

  Okay, let's talk about analogizers.

• 31:45Speaker 00

  So the analogizers are actually the least cohesive of the tribes. So the Beijans would probably, you know, they will call themselves a tribe. Maybe not exactly in those words, but they the tribe with the strongest core identity. The analogizer are actually the one with the weakest. So it's kind of an interesting contrast to talk about, you know, both of them next to each other. The analogizers are actually, I glom together a set of different groups of people that actually don't necessarily interact that much, but they have something very important in common. So so let me, you know, let me bring up those two maybe separately and then see where their relationship.

• 32:29Speaker 00

  I mean, there's and then there's more than two, but but what is the basic idea? The basic idea is that cognition is analogy. So a famous analogizer and the person who coined the term is Douglas Hofstadter, who is the author of Gödel, Escher, Bach and a big fan of analogies. The funny thing, by the way, about Gödel, Escher, Bach, for those of you who have read is that it's a book about logic. Right? It's a book about symbolic AI. We were back in this, you know, 79 when it was produced was the hey of symbolic AI. And I know a lot of people who got into AI because of that book. It's about logic, but it's full of analogies, which is what makes it so fun. Analogies between Gödel and Escher and Bach and a lot of other things, right?

• 33:10Speaker 00

  And more recently, he he co-wrote a book called Surfaces and Essences. Subtitle is Analogy as the Fuel and Fire of Cognition. And his argument is that, again, he doesn't, you know, he's not a half-measures kind of person. Is that like, everything in cognition is analogy. There is no aspect of cognition that is not analogy. And the whole book is 600 pages, just showing this in detail, from the smallest everyday examples to the highest achievements of science. Einstein and Galan or not were just great analogizers. They made these analogies between things that made them find their discoveries. But also, the way you understand the simplest words in the language is by analogy with other things.

• 33:53Speaker 00

  And by the way, what I have found, talking with a lot of different kinds of people over the years, based on the book, is that, at the end of the day of the five schools, the one that is most intuitive to normal human beings is analogy. Yes, the brain, sure, and Bayesianism, that's the least comprehensible. But the idea that we reason by analogy, everybody can relate to that, because we do. Every, you know, now, I think, you know, Douglas Hofstadter, he's, I sympathize with a lot of what he says, but again, I think he goes too far, right? It's not that, like, he can find an analogizer angle on everything, but that doesn't mean that that has exhausted, like, yes,

• 34:36Speaker 00

  Einstein was a great analogizer, I buy that, but there was more to it than that, right? And I think that he overlooks that. So, he's an example of analogizer that comes from psychology. So in psychology, again, there's a huge literature going back decades about how people do everything by analogy.

• 34:54Speaker 00

  And in particular, there's this thing called structure mapping, and Deidre Gentner was the person who proposed it, that says, "The way I do analogies is by mapping structure from one domain to another." A famous example, for example, is how Niels Bohr came up with his theory of the atom by making an analogy between it and the solar system. Where the nucleus, you know, was the sun and the planets were the electrons, which actually, as it turns out, is a really bad analogy, but it was the analogy that got quantum mechanics off the ground. So, you know, you shouldn't poo poo that. But so, so what, what, what he did was like he mapped the structure of the solar system to the atom. And based on that, this is the key. He was able to make a lot of inferences. Now to take a more mundane example, something that is very popular, again, has been for decades,

• 35:39Speaker 00

  uh, uh, is, is using this type of learning for help desks and call centers. I'm Microsoft and you call in saying, hey, my printer isn't working. Right? And, and now eighty, eighty percent of the time it's the same, you know, dozen problems. And what you need to do is, is, is what is called case based reasoning. And again, there's a whole subfield of machine, of, of AI, I should say, which case based reasoning is really just a form of, of analogical learning. It's saying like, okay, let me find the simplest problem to yours. And the solution that I had to that problem. And now let me look at what the differences are and see if I can tweak your problem into this one that I know. Right?

• 36:21Speaker 00

  And, you know, for example, in medical diagnosis, a very simple example, this is like, uh, uh, you know, you're a new patient, you come into my office and, and I don't know anything about, about medicine, right? I use the example in the book of Frank Abay Mill Jr., right? This guy who pretended to be a doctor and, and, you know, put a fake Harvard diploma on his wall in a hospital in Georgia and wound up being the most popular doctor in the hospital. The patients love him. He didn't know any medicine, so what do you do? Right? Here's what something you can do. You, you have a database of patient records because whatever the doctor in, in that office before added. The new patient comes into your office and, and she tells you her symptoms and you just look for the patient with the most similar symptoms.

• 37:04Speaker 00

  And whatever that patient had, you say that this one has. And as dumb as this is, it turns out that you can actually prove mathematically that if you give me enough examples of this, I will get the right answer.

• 37:18Speaker 00

  Because as I get more examples, the nearest ones, this is the so-called nearest neighbor algorithm, which is another strand coming from, you know, pattern recognition in the 50s. The nearest neighbor algorithm says, just make the prediction that is the whatever class, for example, diabetes or whatever, that the nearest neighbor of the new case is in the data. And with enough data and, you know, a slight refinement of this algorithm, you'll get the right answer. So I sometimes jokingly say to people that, yeah, the singularity happened in 1951. Because that's when this algorithm was invented. And it actually is the first algorithm that can learn any function from data. So like, in some ways, it's the first true machine learning algorithm.

• 38:02Speaker 00

  Because it's not like, oh, I can learn the linear frontier or something with a fixed form, a mixture of Gaussians. No, you give me more data. Here's the thing, like, all your traditional statistical models, Bayesian or frequentist, at some point, you can give them all the data that you want, that they don't continue, they don't get better. They've, you've exceeded their capacity. Actually, there isn't even a technical term of capacity. Nearest neighbor has infinite capacity. You keep giving me more data, I keep getting better. So if scaling is all that matters, as I tweeted the other day, then we've already reached AGI. Of course, it isn't, but you get the point. So then more recently, within machine learning, the best known form of analogical learning

• 38:45Speaker 00

  is support vector machines or kernel machines. Support vector machines or kernel machines are really just a very mathematically sophisticated version of nearest neighbor. But for a while there, like, immediately prior to the recent explosion of connectionism in the 2000s, there was a decade there, the decade of the 2000s, machine learning was dominated by kernel machines. You would go to ICML and NIPS and all these machine learning conferences, and there'd be almost no papers on your networks, and there'd be tons and tons of papers on kernel machines. So there you have it. Life is short.

• 39:21Craig S. Smith?

  Yeah, so what's, yeah, I mean, talk about support vector machines a bit. And that's still being applied, is that right?

• 39:32Speaker 00

  No, absolutely. And again, there are problems, and again, this is a recurring pattern.

• 39:38Speaker 00

  You know, the spotlight is on neural networks these days, but there continue to be prompts for which support vector machines are the best solution. So if you care about solving such a problem, you should use a support vector machine. And again, one of the things that frustrates me and part of why I wrote the master plan is like, I see people wasting an enormous amount of time in getting good results just because they're using the wrong technique. You know, if you have, you should try to figure out, and often it's not trivial, what is the best technique for your problem? But you should at least be aware, for example, that there is such a thing as support vector machines, and maybe that's what you want to use rather than neural networks. And in particular, support vector machines are far easier to apply than neural networks, far easier.

• 40:20Speaker 00

  So God help you if you go try to apply deep learning to a problem that could be solved by by support vector machine, you're just giving yourself enormous pain for no good reason. In fact, the history of this is kind of fun.

• 40:35Speaker 00

  Support vector machines were developed at Bell Labs in the 90s by this guy called Vladimir Vapnik, which himself has an interesting history. But he was part of the group that was led by Yann Le Cun, that was doing convolutional neural networks for digit recognition. And they spent 10 years engineering these networks very specifically to do better digit recognition. And the support vector machines were a purely theoretical idea. They are a very mathematically based technique. But they came in and they're like, oh, so what shall we apply this to? Right? Digit recognition. And right off the bat, they were tying ConvNets. It wasn't even that they were winning, but the fact that out of the blocks,

• 41:20Speaker 00

  this system was as good as ConvNets just made people's jaw drop. And then these were the early days of the web, and they did very well at things like text classification. And one of the great things about support vector machines is that neural networks are a mess because it's what is called a non-convex optimization problem, right? It has many local optima, you do gradient descent, but you do your gradient descent and I do mine, and we wind up in different places. It's terrible, right? You have to spend all this time, you know, tweaking parameters. You know, as somebody said, trying to get a neural network to work is like trying to balance a pencil on the tip of your finger. It's actually a very good analogy. Support vector machines don't have that problem. It's a convex optimization problem. There's one global optimum.

• 42:04Speaker 00

  You push that button and you get the solution. And we all get the same solution. And this just saves so much time and so much headache that people are like, wow, you know, neural networks are obsolete. We don't need that anymore. Goodbye, right? Now, of course, fast forward 20 years and another set of things has happened. But for all I know, in another, whatever, one year or five or 10, support vector machines will come back doing the things that transformers can. And I can even speculate about how that might happen without the problems that the transformers do. So, yes, you know, that type of learning now is a little bit fallen to the side. There's still some going on. But again, those people are not as hardcore as the visions. So there's, you know, there's less of that happening. But again, for all I know,

• 42:48Speaker 00

  it's ready to come back anytime. And again, there is this, here's an important point. Going back to the 50s, there is this recurring motif that, you know, neural networks come along. They do something. They're very sexy, right? They're very intuitively appealing because this is how the brain works, right? But then, you know, they have problems and they're messy and people keep pounding on it. And sooner or later, somebody comes along and does an analogical version of the, of the, of the solution that is actually better and simpler than the neural network one. So in the long run, I mean, and a famous example of this particularly is this Hopfield networks, right? John Hopfield just won a Nobel Prize in Physics for Hopfield networks, right?

• 43:34Speaker 00

  It turns out Hopfield networks, and this was discovered in the 80s, like right away by these guys at this lab at MIT, that it's just nearest neighbor with a number of different bits as the distance measure. It's this dumb, all that blah, blah. There's a dynamical system with the tractors, blah, blah, blah. Like, no, nearest neighbor. And this just has happened over and over again. So I ask, well, maybe it's gonna happen again this time.

• 43:59Craig S. Smith?

  Hmm, that's interesting, yeah. And so, I mean, somebody like Hopnik, I mean, Hopfield, does he identify as an analogizer?

• 44:10Speaker 00

  No, he doesn't. So that's my point is that like, so Hopfield is a physicist. A very eclectic physicist that has, you know, looked at things in biology and whatnot. And he came up with this, and again, he wasn't the first one, but he was the most influential one. He saw this analogy, yeah, great example. He literally saw this analogy between a type of physical material called a spin glass and the neural network, which each atom in the spin glass corresponds to a neural network. Mathematically, they're the same. Or at least what he did is he formulated a type of network called a Hallfield network. There is really the mathematics of these spin glasses, but turned into a neural network.

• 44:52Speaker 00

  And the point that he was making was that, so these spin glasses, they have these attractors, right? They have these, these local optima, local minimum of energy that as the system evolves, it's a bunch of atoms, right? And their spins are flipping, right? And, and, and it will evolve, right, into different local optima. And he said, like, we can think of each of these as a memory. And now we can actually learn the weights that by which these, you know, spins interact to store, you know, the digits, you know, zero to nine. And then when, when a corrupted nine comes along, it falls into the basin of the nine, right? And this was a fantastically appealing idea, right? Back in the days when this came out, people were like, wow. Also because, you know, there's a, there's a sociology to this.

• 45:35Speaker 00

  He was a physicist, he was respectable. And machine learning back then was not respectable, let alone neural networks. And they were like, oh, we're respectable now, right? So in particular, this was a big influence on Jeff Hinton. But, you know, arguably a bad influence because nothing that Jeff Hinton based on that ever panned out. The neural networks that we use today do completely different things. But, so, so Hallfield was not trying to do analogical learning at all. He probably doesn't even know what that term means. But what then somebody else proved was that his whole network was really just a very inefficient way to. Let's suppose like when a new nine comes along, right? When a corrupted nine comes along, what I do is this. I count the number of bits that it has in common with the with the prototypical zero, one, etc.

• 46:19Speaker 00

  And hey, it has the most bits in common with nine, so I classify it as a nine, right? This is nearest neighbor. This is pure nearest neighbor, the simplest version you can imagine. And this turns out to be mathematically equivalent to that whole complicated network with its slow evolution towards a local optimum. So it's it. I think it's ironic or maybe even hilarious commentary on that whole. I mean, like, all these, you know, hundreds of physicists came into machine learning, doing variations on helpful networks, blah, blah, blah. And at the end of the day, it's just doing nearest neighbor.

• 46:51Craig S. Smith?

  Yeah, that's fascinating. And SVM, support vector machines, what what kinds of applications do they have today?

• 46:59Speaker 00

  They are so let me backtrack slightly to a question that you asked earlier, which is so. But what exactly is a support vector machine? Right? And the support vector machine is, maybe I can see it in a couple of steps. So there's a generalization of nearest neighbor, which is actually the most powerful, which is k nearest neighbor, where you don't just use the nearest neighbor, but you average between the k nearest. It's like, who are your three closest neighbors? And, you know, what did they have? If two out of three of them had diabetes, then well, maybe you do as well. And if they all had different things, then maybe I don't know, I flip a coin or something. So there's that step. The next step is it turns out to be very helpful in many cases to give the examples weights. So maybe some of the examples should have more weight because they represent a bigger prototype. Right?

• 47:43Speaker 00

  And and and in essence, the a support vector machine is just a mathematical way to assign weights to these examples. And it assigns the weights in such a way that you form a a good frontier between. So for example, you know, the concept is, you know, cancer, no cancer, right? And think of think of the space of of of patients. And and some of them have cancer, some of them don't. You want to find what the frontier is between cancer and no cancer. And what support vector machines do is they they try to find the frontier that is as far as possible from any example, so it's a safe frontier. If I have a frontier that has examples very close to, well, then maybe, you know, the frontier, you know, is is, you know, maybe it should be somewhere else, right?

• 48:27Speaker 00

  Because it's like, why, you know? I mean, an analogy that I use in the book is, you know, imagine that I give you a map of two countries, and I only tell you where the major cities are. And I tell you to draw me the frontier between the two countries, the border. Right? Where would you put the border? You wouldn't put it right next to the capital of one of the countries, probably, right? There's exceptions, but the safest thing to do would be to put the border as far away from any of the cities as you can. Right? And so support vector machines mathematically solve this problem to come up with the weights for the examples that produce the so-called max margin frontier. Right? And the margin is kind of like this, you know, DMZ, you know, think of Korea, right? There's a DMZ around the border where nobody lives.

• 49:12Speaker 00

  And support vector machines try to make their DMZ as wide as possible. So that's, you know, that's in a, at a high level, what a support vector machine is doing. And so support vector machines wind up, you know, being good for things like, for example, as I mentioned, text classification. Why are they good at text classification? Because the space of words is very large and treating each of those, like, let's say you have a hundred thousand words. Right? Treating each one of those words as a dimension in the space that you're predicting in is, makes it really hard. It makes it really easy to overfit. Right? Because any own little word can be, you know, can say, oh, this will by accident correlates very well with you clicking on my app. Right?

• 49:58Speaker 00

  But, but it's just overfitting. And the support vector machines are much more robust to that because they have this, like in neural networks, for example, perceptrons, they will just put the frontier anywhere, literally. As soon as the frontier doesn't misclassify anybody, you're happy. You actually don't care if, you know, if you just have one example here, one positive example here and one negative there, you know, any frontier between them is, is okay. And the support vector machines are smarter than that. So like, no, I want the max margin ones. So, so they're, for example, much more robust when you're in high dimensions, which used to be the Achilles heel of nearest neighbor. Right? You would even say that if it wasn't for the so-called curse of dimensionality, nearest neighbor would have solved this whole problem a long time ago. But, you know, it is a very deep problem and, and support the machines definitely make

• 50:42Speaker 00

  a big difference.

• 50:44Craig S. Smith?

  And it sounds, though, that analogizer is, is a class, but not necessarily a, a tribe. I mean, it, that you, and even connectionists, you could call analogizers because they're, they're using. Neurons in the brain as an analogy.

• 51:09Speaker 00

  Well, so let me unpack several things that you said there. Each tribe has sub-tribes. But within connectionists, there's people who do different and believe different kinds of things, some of them very different. Same with the analogizers, same with all of them. Again, like we mentioned like last time, the Neets and the Scruffy's are the two main sub-tribes of the symbolists. What distinguishes, I would say, the analogizers is that the subclasses are or the sub-tribes are more independent and talk less to each other. So for example, the people who do analogy-based stuff, inspired in psychology, even within AI, talk very little to the people who do support vector machines.

• 51:52Speaker 00

  Again, the support vector machine people are more neat and the other ones are more scruffy. Maybe that's the way to think about it, the neat and scruffy analogizer, but it's a less coherent tribe. Now, when you say that the connectionists are analogizers, reasoning yourself by analogy doesn't make you an analogy. I mean, we are all analogizers by that standard, right? The question is like, is the algorithm that you, is the algorithm reasoning by analogy? Right? If I'm Jeff Hinton or John Hoffield and I make an analogy between the brain and the spin glass, right? That doesn't make me an analogizer because at the end of the day, what I have is a neural network. Right? The fact that I'm using analogies is different from the algorithm using analogies when it's trying to figure out, you know, what this person has or not.

• 52:35Speaker 00

  Having said that, and again, this I think is a very interesting point. When you talk to Jeff Hinton, he says neural networks are better than symbolic AI because they reason by analogy. But then he never explains how they reason by analogy. This to him is a very strong intuition and one that I buy, right? Again, symbolic AI suffers from the brilliance problem. And one way to overcome that is to bring in analogy. In fact, my PhD thesis was unifying symbolic AI with analogical learning precisely for this purpose, to make the rules softer. Right, I had this analogical matching of the rules and different degrees of abstraction,

• 53:18Speaker 00

  which again agrees with a lot of results in psychology and whatnot. So bringing in analogy as a way to solve the, you know, the brutalist problem is potentially a very good idea. And what Jeff Hinton says and others, but Jeff Hinton is the most famous one, is that that's what neural networks are doing. But then my question is like, well, hey Jeff, if neural networks are doing analogy, how are they doing analogy? They do this gradient descent, there's this mess of parameters, and intuitively they're doing analogy. But I actually have a recent paper explaining exactly how they're doing analogy. It turns out that when you do gradient descent, what you actually store, the paper is basically

• 53:59Speaker 00

  making the, proving a theorem that shows that every model learned by gradient descent is a kernel machine. With a particular type of kernel, there's a dot product of gradients. But, but think about this, what you've actually done when you learn a neural network is you've just stored all those examples in a way that is not obvious, but they're there. And when you apply the neural network to a problem, what you weren't mathematically effectively doing is comparing your new example with one of those. So I can actually, I have an answer to Jeff Hinton's question. So I have an answer to my own question of Jeff Hinton, which is that, well, how do neural networks do analogy? The, you know, we know how they do analogy at this point.

• 54:44Speaker 00

  And I think a lot of consequences are going to flow from this, but we haven't seen most of them yet.

• 54:49Craig S. Smith?

  Create an oasis with Thuma, a modern design company that specializes in furniture and home goods. By stripping away everything but the essential, Thuma makes elevated beds with premium materials and intentional details. I'm in the process of reorganizing my house and I'm giving Thuma a serious look for help in renovating and redesigning. Thuma combines the perfect balance of form, craftsmanship, and functionality.

• 55:23Craig S. Smith?

  With over seventeen thousand five-star reviews, the Thuma bed collection is proof that simplicity is the truest form of sophistication. Using the technique of Japanese joinery, pieces are crafted from solid wood and precision cut for a silent, stable foundation. With clean lines, subtle curves, and minimalist style, the Thuma bed collection is available in four signature finishes to match any design aesthetic. Headboard upgrades are available for customization as desired.

• 56:05Craig S. Smith?

  To get a hundred dollars toward your first bed purchase, go to Thuma, that's T-H-U-M-A dot C-O slash IONAI. IONAI all run together, E-Y-E-O-N-A-I. So for a hundred dollars off your first purchase, go to Thuma dot C-O slash IONAI. That's T-H-U-M-A dot C-O slash IONAI to receive a hundred dollars off your first bed purchase.