Vladimir Vapnik: Statistical Learning

Vladimir Vapnik is the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. His work has been cited over 170,000 times. He has some very interesting ideas about artificial intelligence and the nature of learning, especially on the limits of our current approaches and the open problems in the field. Video version is available on YouTube https://www.youtube.com/watch?v=STFcvzoxVw4. If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter https://twitter.com/lexfridman, LinkedIn https://www.linkedin.com/in/lexfridman/, Facebook https://www.facebook.com/lexfridman, or YouTube https://www.youtube.com/lexfridman where you can watch the video versions of these conversations.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation with Vladimir Vapnik. He's the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union and worked at the Institute of Control Sciences in Moscow. Then, in the United States, he worked at AT&T, NEC Labs, Facebook Research, and now is a professor at Columbia University. His work has been cited over 170,000 times. He has some very interesting ideas about artificial intelligence and the nature of learning, especially on the limits of our current approaches and the open problems in the field. This conversation is part of MIT course on artificial general intelligence and the

• 0:44Lex Fridman?

  Artificial Intelligence Podcast. If you enjoy it, please subscribe on YouTube, or rate it on iTunes or your podcast provider of choice, or simply connect with me on Twitter or other social networks at Lex Fridman, spelled F-R-I-D. And now, here's my conversation with Vladimir Vapnik.

• 1:19Lex Fridman?

  Einstein famously said that God doesn't play dice.

• 1:23Lex Fridman?

  Yeah.

• 1:25Lex Fridman?

  You have studied the world through the eyes of statistics, so let me ask you, in terms of the nature of reality, fundamental nature of reality, does God play dice?

• 1:36Vladimir Vapnik?

  We don't know some factors.

• 1:40Vladimir Vapnik?

  Because we don't know some factors which could be important, it looks like God play dice.

• 1:50Vladimir Vapnik?

  But we should describe. In philosophy, they distinguish between two positions, positions of instrumentalism, where you're creating theory for prediction, and position of realism, where you're trying to understand what God did.

• 2:09Lex Fridman?

  Can you describe instrumentalism and realism a little bit?

• 2:13Vladimir Vapnik?

  For example, if you have some mechanical laws. What is that? Is it law which true always and everywhere? Or it is law which allow you to predict position of moving element?

• 2:35Vladimir Vapnik?

  What you believe? You believe that it is God's law? That God created the world which obey to this physical law? Or it is just law for predictions?

• 2:51Lex Fridman?

  And which one is instrumentalism?

• 2:53Vladimir Vapnik?

  For predictions. If you believe that this is law of God and it's always true everywhere, that means that you're realist. You're trying to really understand the God's thought.

• 3:10Lex Fridman?

  So the way you see the world is as an instrumentalist?

• 3:15Vladimir Vapnik?

  You know, I'm working for some models. Models of machine learning. So in this model, we can see setting. And we try to solve, resolve the setting, to solve the problem. And you can do in two different ways. From the point of view of instrumentalist. And that's what everybody does now. Because they say that goal of machine learning is to find the rule for classification. That is true. But it is instrument for prediction.

• 3:55Vladimir Vapnik?

  But I can say the goal of machine learning is to learn about conditional probability. So how God played use? And is he play? What is probability for one? What is probability for another? Given situation. But for prediction, I don't need this. I need the rule. But for understanding? I need conditional probability.

• 4:23Lex Fridman?

  So let me just step back a little bit first to talk about, you mentioned, which I read last night, the parts of the 1960 paper by Eugene Wigner, Unreasonable Effectiveness of Mathematics in Natural Sciences. Such a beautiful paper, by the way. It made me feel, to be honest, to confess my own work in the past few years on deep learning, heavily applied, made me feel that I was missing out on some of the beauty of nature in the way that math can uncover. So let me just step away from the poetry of that for a second. How do you see the role of math in your life?

• 5:08Lex Fridman?

  Is it a tool, is it poetry? Where does it sit? And does math for you have limits of what it can describe?

• 5:16Vladimir Vapnik?

  Some people saying that math is language which use God. So I believe.

• 5:25Lex Fridman?

  Speak to God or use God?

• 5:27Vladimir Vapnik?

  Use God.

• 5:27Lex Fridman?

  Use God?

• 5:29Vladimir Vapnik?

  Yeah. So I believe that this article about effectiveness, unreasonable effectiveness of math is that if you're looking at mathematical structures,

• 5:48Vladimir Vapnik?

  they know something about reality.

• 5:52Vladimir Vapnik?

  And the most scientists from natural science, they're looking on equation and trying to understand reality.

• 6:03Vladimir Vapnik?

  So the same in machine learning.

• 6:06Vladimir Vapnik?

  If you're trying very carefully look on all equations which define conditional probability, you can understand something about reality more than from your fantasy.

• 6:23Lex Fridman?

  So math can reveal the simple, underlying principles of reality, perhaps?

• 6:28Vladimir Vapnik?

  You know what means simple? It is very hard to discover them. But then, when you discover them and look at them, you see how beautiful they are. And it is surprising why people did not see that before. You're looking on equations and derive it from equations. For example, I talked yesterday about least-squared method. And people had a lot of fantasy how to improve least-squared method. But if you're going step by step by solving some equations, you suddenly will get some term, which, after thinking,

• 7:14Vladimir Vapnik?

  you understand that it describes position of observation point. In least-squared method, we throw out a lot of information. We don't look in composition of points of observations. We're looking only on residuals. But when you understood that, that's very simple idea, but which not too simple to understand. And you can derive this just from equations.

• 7:40Lex Fridman?

  So some simple algebra, a few steps will take you to something surprising that when you think about.

• 7:48Vladimir Vapnik?

  Absolutely, yes. And that is proof that human intuition not too rich and very primitive. And it does not see very simple situations.

• 8:03Lex Fridman?

  So let me take a step back. In general, yes, right? But what about human, as opposed to intuition, ingenuity?

• 8:16Lex Fridman?

  Moments of brilliance? So are you so... Do you have to be so hard on human intuition? Are there moments of brilliance in human intuition? They can leap ahead of math? And then the math will catch up?

• 8:32Vladimir Vapnik?

  I don't think so. I think that the best human intuition, it is putting in axioms. And then it is technical way.

• 8:43Lex Fridman?

  See where the axioms take you.

• 8:45Vladimir Vapnik?

  Yeah. But if they correctly take axioms. But it axioms polished during generations of scientists. And this is integral wisdom.

• 9:00Lex Fridman?

  So that's beautifully put. But if you maybe look at, when you think of Einstein and special relativity, what is the role of imagination coming first there? In the moment of discovery of an idea? So there's obviously a mix of math and out of the box imagination there.

• 9:25Vladimir Vapnik?

  That's I don't know. Whatever I did, I exclude any imagination. Because whatever I saw in machine learning that come from imagination, like features, like deep learning, they are not relevant to the problem. When you're looking very carefully from mathematical equations, you're deriving very simple theory, which goes far beyond theoretically than whatever people can imagine. Because it is not good fantasy.

• 9:59Lex Fridman?

  Yeah.

• 9:59Vladimir Vapnik?

  It is just interpretation. It is just fantasy, but it is not what you need. You don't need any imagination to derive, say, main principle of machine learning.

• 10:14Lex Fridman?

  When you think about learning and intelligence, maybe thinking about the human brain and trying to describe mathematically the process of learning, that is something like what happens in the human brain. Do you think we have the tools currently?

• 10:32Lex Fridman?

  Do you think we will ever have the tools to try to describe that process of learning?

• 10:36Vladimir Vapnik?

  You, it is not description what's going on. It is interpretation, it is your interpretation. Your vision can be wrong. You know, when guy invent microscope, Leven Cook, for the first time. Only he got this instrument and nobody, he kept secret about microscope. But he wrote report in London Academy of Science. And his report, when he looking of the blood, he look everywhere, on the water, on the blood, on the sperm. But he described blood like fight between queen and king.

• 11:19Vladimir Vapnik?

  So he saw blood, cells, red cells, and he imagined that it is army fighting each other. And it was his interpretation of situation.

• 11:32Vladimir Vapnik?

  And he send this report in Academy of Science, they very carefully look because they believe that he's right, he saw something.

• 11:40Lex Fridman?

  Yes.

• 11:40Vladimir Vapnik?

  But he gave wrong interpretation. And I believe the same can happen with brain.

• 11:47Lex Fridman?

  With brain, yeah.

• 11:48Vladimir Vapnik?

  The most important part, you know, I believe in human language. In some proverb is so much wisdom. For example, people say that it is better than thousand days of diligent studies one day with great teacher. But if I will ask you what teacher does, nobody knows. And that is intelligence. And but we know from history and now from from mass and machine learning that.

• 12:25Vladimir Vapnik?

  Teacher can do a lot.

• 12:27Lex Fridman?

  So what, from a mathematical point of view, is a great teacher?

• 12:31Vladimir Vapnik?

  I don't know.

• 12:32Lex Fridman?

  That's an open question.

• 12:33Vladimir Vapnik?

  No, but we can say what teacher can do. He can introduce some invariance, some predicate for creating invariance. How he doing it? I don't know, because teacher knows reality and can describe from this reality predicate invariance. But he knows that when you're using invariant, you can decrease number of observations hundred times.

• 13:02Lex Fridman?

  So, but maybe try to pull that apart a little bit. I think you mentioned like a piano teacher saying to the student, play like a butterfly. Right? I played piano, I played guitar for a long time. And it, yeah, that's, there's, maybe it's romantic, poetic. But it feels like there's a lot of truth in that statement. Like, there is, there is a lot of instruction in that statement. And so can you pull that apart? What is, what is that? The language itself may not contain this information.

• 13:37Vladimir Vapnik?

  It is not blah, blah, blah.

• 13:39Lex Fridman?

  It is not blah, blah, yeah.

• 13:40Vladimir Vapnik?

  It affect you.

• 13:41Lex Fridman?

  It's what?

• 13:42Vladimir Vapnik?

  It affect you. It affect your playing.

• 13:44Lex Fridman?

  Yes, it does, but it's not the lang. It's, it feels like, what is the information being exchanged there? What is the nature of information? What is the representation of that information?

• 13:56Vladimir Vapnik?

  I believe that it is sort of predicate, but I don't know. That is exactly what, what intelligence in machine learning should be.

• 14:04Lex Fridman?

  Yes.

• 14:05Vladimir Vapnik?

  Because the rest is just mathematical technique. I think that what was discovered recently is that there is two tie, two mechanism of learning. One called strong convergence mechanism and weak convergence mechanism. Before, people use only one. In weak convergence mechanism, you can use predicate. That's what play like butterfly. And it will immediately affect your playing. You know, there is English proverb, great. If it looks like a duck. Swims like a duck and quack like a duck, then it is probably duck.

• 14:50Lex Fridman?

  Yes.

• 14:51Vladimir Vapnik?

  But this is exact about predicate. Looks like a duck, what it means. So you saw many ducks that you're training data. So you you have description of how how looks integral looks ducks.

• 15:11Lex Fridman?

  Yeah, the visual characteristics of a duck, yeah.

• 15:14Vladimir Vapnik?

  Yeah, but you want and you have model for recognition ducks. So you would like so that theoretical description from model coincide with empirical description, which you saw on theory text there. So about looks like a duck, it is general. But what about swims like a duck? You should know the duck swims. You can't say it play chess like a duck. Okay, duck doesn't play chess. And it is completely legal predicate, but it is useless. So how teacher can recognize not useless predicate?

• 15:56Vladimir Vapnik?

  So up to now, we don't use this predicate in existing machine learning.

• 16:01Lex Fridman?

  And you think that.

• 16:02Vladimir Vapnik?

  So why we need zillions of data? But in this English proverb, proverb, they use only three predicate. Looks like a duck, swims like a duck and quack like a duck.

• 16:14Lex Fridman?

  So you can't deny the fact that swims like a duck and quacks like a duck has humor in it, has ambiguity.

• 16:23Vladimir Vapnik?

  Let's talk about swim like a duck.

• 16:27Vladimir Vapnik?

  And it does not say jumps, jump like like a duck. Why? Because.

• 16:33Lex Fridman?

  It's not relevant.

• 16:35Vladimir Vapnik?

  But that means that you know ducks, you know, different birds, you know, animals.

• 16:42Lex Fridman?

  Yeah.

• 16:42Vladimir Vapnik?

  And you derive from this that it is relevant to say swim like a duck.

• 16:47Lex Fridman?

  So underneath, in order for us to understand swims like a duck, it feels like we need to know millions of other little pieces of information.

• 16:57Vladimir Vapnik?

  I'm not sure.

• 16:58Lex Fridman?

  We pick up along the way. You don't think so. There doesn't need to be this knowledge base. In in those statements carries some rich information that helps us understand the essence of duck.

• 17:10Vladimir Vapnik?

  Yeah.

• 17:12Lex Fridman?

  How far are we from integrating predicates?

• 17:16Vladimir Vapnik?

  You know that when you consider complete theory of machine learning, so what it does, you have a lot of functions. And then you're talking, it looks like a duck. You see your training data from training data, you recognize like

• 17:41Vladimir Vapnik?

  expected duck should look. Then you remove all functions, which does not look like you think it should look from training data. So you decrease amount of function from which you pick up one. Then you give a second predicate and then again decrease the set of function. And after that, you pick up the best function you can find. It is standard machine learning. So why you need not too many examples?

• 18:18Lex Fridman?

  Because your predicates aren't very good?

• 18:20Vladimir Vapnik?

  Yeah, that means the predicate very good. Because every predicate is invented to decrease admissible set of function.

• 18:32Lex Fridman?

  So you talk about admissible set of functions and you talk about good functions. So what makes a good function?

• 18:39Vladimir Vapnik?

  So admissible set of function is set of function which has small capacity or small diversity, small VC dimension example, which contain good function inside.

• 18:52Lex Fridman?

  So by the way, for people who don't know VC, you're the V in the VC.

• 18:59Lex Fridman?

  So how would you describe to a layperson what VC theory is? How would you describe VC?

• 19:06Vladimir Vapnik?

  You have a machine, so a machine capable to pick up one function from the admissible set of function.

• 19:17Vladimir Vapnik?

  But set of admissible function can be big. Say contain all continuous functions and it useless. You don't have so many examples to pick up function. But it can be small, small.

• 19:35Vladimir Vapnik?

  We call it capacity, but maybe better called diversity. So not very different function in the set is infinite set of function, but not very diverse. So it is small VC dimension. When VC dimension is small, you need not, you need small amount of training data. So the goal is to create admissible set of functions, which is have small VC dimension and contain good function. Then you should, then you will be able to pick up the function using small amount of observations.

• 20:17Lex Fridman?

  So that is the task of learning?

• 20:21Vladimir Vapnik?

  Yeah.

• 20:21Lex Fridman?

  Is creating a set of admissible functions that has a small VC dimension? And then you've figure out a clever way of picking up?

• 20:32Vladimir Vapnik?

  No, that is goal of learning, which I formulated yesterday.

• 20:36Lex Fridman?

  Yeah.

• 20:37Vladimir Vapnik?

  Statistical learning theory does not involve in creating admissible set of function. In classical learning theory, everywhere, hundred percent in textbook, the set of function, admissible set of function is given. But this is science about nothing, because the most difficult problem to create admissible set of functions, given say, a lot of functions, continuum set of function, create admissible set of functions, that's means that it has finite VC dimension, small VC dimension, and contain good function. So this was out of consideration.

• 21:20Lex Fridman?

  So how, what's the process of doing that? I mean, it's fascinating. What is the process of creating this admissible set of functions?

• 21:28Vladimir Vapnik?

  That is invariance.

• 21:29Lex Fridman?

  That's a variance. Can you describe the variance?

• 21:32Vladimir Vapnik?

  Yeah, you're looking of properties of training data. And properties means that you have some function. And you just count what is value, average value of function on training data. You have model and what is expectation of this function on the model. And they should coincide. So the problem is about how to pick up functions. It can be any function. In fact, it is true for all functions.

• 22:15Vladimir Vapnik?

  But because when I'm talking set, say, duck does not jumping. So you don't ask question, jump like a duck. Because it is trivial, it does not jumping and does not help you to recognize jump. But you know something, which question to ask. And you're asking it swims like a duck, like a duck. But looks like a duck, it is general situation. Looks like, say, guy who have this illness, this disease. It is legal.

• 22:53Lex Fridman?

  Yeah.

• 22:53Vladimir Vapnik?

  So there is a general type of predicate looks like. And special type of predicate which related to this specific problem. And that is intelligence part of all this business. And that where teachers involved.

• 23:11Lex Fridman?

  Incorporating the specialized predicates. Okay. What do you think about deep learning as neural networks, these arbitrary architectures, as helping accomplish some of the tasks you're thinking about? Their effectiveness or lack thereof? What are the weaknesses and what are the possible strengths?

• 23:35Vladimir Vapnik?

  You know, I think that this is fantasy. Everything which like deep learning, like features.

• 23:44Vladimir Vapnik?

  Let me give you this example. One of the greatest books, this Churchill book about history of Second World War. And he's starting this book describing that in old time, when war is over,

• 24:03Vladimir Vapnik?

  so the great kings,

• 24:08Vladimir Vapnik?

  they gathered together, almost all of them were relatives. And they discussed what should be done, have to create peace. And they came to agreement. And when happened First World War, the general public came in power. And they were so greedy that robbed Germany. And it was clear for everybody that it is not peace. That peace will last only 20 years. Because they was not professionals. And the same I see in machine learning. There are mathematicians who looking for the problem

• 24:50Vladimir Vapnik?

  from a very deep point of view, mathematical point of view. And there are computer scientists, who is mostly does not know mathematics. They just have interpretation of that. And they invented a lot of blah blah blah interpretations, like deep learning. Why you did deep learning? Mathematic does not know deep learning. Mathematic does not know neurons. It is just function. If you like to say piecewise linear function, say that. And do in class of piecewise linear function. But they invent something. And then they try to prove advantage of that

• 25:33Vladimir Vapnik?

  through interpretations, which mostly wrong. And when it not enough, they appeal to brain, which they know nothing about that. Nobody know what going on in the brain. So I think that more reliable look on mass. This is mathematical problem. Do your best to solve this problem. Try to understand that there is not only one way of convergence, which is strong way of convergence, there is a weak way of convergence, which requires predicate. And if you will go through all this stuff, you will see that you don't need deep learning. Even more, I would say one of the theorem, which called representative theorem.

• 26:18Vladimir Vapnik?

  It says that optimal solution of mathematical problem, which is, which described learning is on shadow network, not on deep learning.

• 26:36Lex Fridman?

  And a shallow network, yeah. The problem is there. Absolutely. So in the end, what you're saying is exactly right. The question is, you have no value for throwing something on the table, playing with it, not math. It's like in neural network, or you said throwing something in the bucket, and or the biological example and looking at kings and queens with the cells or the microscope. You don't see value in imagining the cells are kings and queens and using that as inspiration and imagination for where the math will eventually lead you. You think that interpretation basically deceives you in a way that's not productive?

• 27:21Vladimir Vapnik?

  I think that if you're trying to analyze this business of learning, and especially discussion about deep learning, it is discussion about interpretation, not about things, about what you can say about things.

• 27:41Lex Fridman?

  That's right, but aren't you surprised by the beauty of it? So the, not mathematical beauty, but the fact that it works at all. Or are you criticizing that very beauty? Our human desire to interpret, to find our silly interpretations in these constructs? Like, let me ask you this. Are you?

• 28:08Lex Fridman?

  Surprised, and does it inspire you? How do you feel about the success of a system like AlphaGo at beating the game of Go? Using neural networks to estimate the quality of a board? And the quality of the...

• 28:26Vladimir Vapnik?

  That is your interpretation, quality of the board.

• 28:29Lex Fridman?

  Yeah, yes.

• 28:30Vladimir Vapnik?

  Yeah.

• 28:31Lex Fridman?

  But it's not our interpretation. The fact is, a neural network system doesn't matter. A learning system that we don't, I think, mathematically understand that well, beats the best human player, does something that was thought impossible.

• 28:46Vladimir Vapnik?

  That means that it's not a very difficult problem.

• 28:48Lex Fridman?

  That's... So you empiric... We've empirically have discovered that this is not a very difficult problem.

• 28:55Vladimir Vapnik?

  Yeah.

• 28:57Lex Fridman?

  It's true. So, maybe, can't argue. So...

• 29:05Vladimir Vapnik?

  Even more, I would say, that if they use deep learning, it is not the most effective way of learning theory. And usually, when people use deep learning, they're using zillions of training data. Yeah, but you don't need this. So I describe challenge. Can we do some problems which do well? Deep learning method with deep net using hundred times less training data. Even more, some problems deep learning cannot solve

• 29:49Vladimir Vapnik?

  because it's not necessary they create admissible set of function. To create deep architecture means to create admissible set of functions. You cannot say that you're creating good admissible set of functions. You adjust it's your fantasy. It does not come from mass. But it is possible to create admissible set of functions because you have your training data. It actually for mathematicians, when you consider a variant, you need to use law of large numbers. When you're making training in existing algorithm,

• 30:32Vladimir Vapnik?

  you need uniform law of large numbers, which is much more difficult. It requires VC dimension and all this stuff. But nevertheless, if you use both weak and strong way of convergence, you can decrease a lot of training data.

• 30:49Lex Fridman?

  Yeah, you could do the three. The swims like a duck and quacks like a duck.

• 30:54Vladimir Vapnik?

  Yeah, yeah.

• 30:54Lex Fridman?

  But our... So let's step back and think about human intelligence in general. And clearly, that has evolved in a non-mathematical way. It wasn't... As far as we know, God or whoever didn't come up with a model and place it in our brain of admissible functions. It kind of evolved. I don't know. Maybe you have a view on this. But so Alan Turing in the 50s, in his paper, asked and rejected the question, can machines think? It's not a very useful question, but can you briefly entertain this useless question?

• 31:39Lex Fridman?

  Can machines think? So talk about intelligence and your view of it.

• 31:43Vladimir Vapnik?

  I don't know that. I know that Turing described imitation. If computer can imitate human being, let's call it intelligent. And he understands that it is not thinking computer.

• 32:00Lex Fridman?

  Yes.

• 32:01Vladimir Vapnik?

  He completely understand what he doing.

• 32:04Lex Fridman?

  Yes.

• 32:04Vladimir Vapnik?

  But he set up problem of imitation. So now we understand that the problem not in imitation. I'm not sure that intelligence just inside of us. It may be also outside of us. I have several observations. So when I prove some theorem, it's very difficult theorem.

• 32:31Vladimir Vapnik?

  In couple of years, in several places, people prove the same theorem. So, Sawyer lemma, after us was done, then another guys prove the same theorem. In the history of science, it's happened all the time. For example, geometry. It's happened simultaneous. First did Lobachevsky, and then Gauss, and Bojai, and another guys. And it approximately in 10 times period, 10 years period of time. And I saw a lot of examples like that. And many mathematicians thinks that when they develop something, they developing something in general which affect everybody.

• 33:16Vladimir Vapnik?

  So maybe our model that intelligence is only inside of us is incorrect.

• 33:22Lex Fridman?

  It's our interpretation, yeah.

• 33:24Vladimir Vapnik?

  It may be there exist some connection with world intelligence. I don't know.

• 33:31Lex Fridman?

  You're almost like plugging in into.

• 33:34Vladimir Vapnik?

  Yeah, exactly.

• 33:36Lex Fridman?

  And contributing to this.

• 33:37Vladimir Vapnik?

  Into big network.

• 33:38Lex Fridman?

  Into a big, maybe in your own network.

• 33:41Vladimir Vapnik?

  No, no, no, no.

• 33:43Lex Fridman?

  On the flip side of that, maybe you can comment on big O complexity and how you see classifying algorithms by worst case running time in relation to their input. So that way of thinking about functions. Do you think P equals NP? Do you think that's an interesting question?

• 34:04Vladimir Vapnik?

  Yeah, it is interesting question. But let me talk about

• 34:10Vladimir Vapnik?

  complexity in about worst case scenario.

• 34:16Vladimir Vapnik?

  There is a mathematical setting. When I came to United States in 1995, people did not know this is theory, they did not know statistical learning theory. So in Russia it was published two monographs, all monographs, but in America they did not know. Then they learned. And somebody told me that it's worst case theory and they will create real case theory. But till now it did not. Because it is mathematical tool. You can do only what you can do using mathematics. And which has a clear understanding and clear description.

• 35:00Vladimir Vapnik?

  And for this reason, we introduce complexity. And we need this. Because using, actually it is diversity, I like this more. If you dimension, you can prove some theorems. But we also create theory for case when you know probability measure. And that is the best case which can happen in this entropy theory. So from mathematical point of view, you know the best possible case and the worst possible case. You can derive different model in medium. But it's not so interesting.

• 35:45Lex Fridman?

  You think the edges are interesting?

• 35:48Vladimir Vapnik?

  The edges is interesting. Because it is not so easy to get good bound, exact bound. It's not many cases where you have the bound is not exact. But interesting principles which discover the mass.

• 36:09Lex Fridman?

  Do you think it's interesting because it's challenging and reveals interesting principles that allow you to get those bounds? Or do you think it's interesting because it's actually very useful for understanding the essence of a function, of an algorithm? So it's like me judging your life as a human being by the worst thing you did and the best thing you did versus all the stuff in the middle. It seems not productive.

• 36:40Vladimir Vapnik?

  I don't think so, because you cannot describe situation in the middle, or it will be not general. So you can describe edge cases, and it is clear it has some model, but you cannot describe model for every new case. So you will be never accurate when you're using model.

• 37:08Lex Fridman?

  But from a statistical point of view, the way you've studied functions and the nature of learning in the world, don't you think that the real world has a very long tail? That the edge cases are very far away from the mean?

• 37:29Lex Fridman?

  The stuff in the middle, or no?

• 37:34Vladimir Vapnik?

  I don't know that. I think that...

• 37:40Vladimir Vapnik?

  But from my point of view,

• 37:45Vladimir Vapnik?

  if you will use formal statistic, you need uniform law of large numbers.

• 37:55Vladimir Vapnik?

  If you will use

• 37:58Vladimir Vapnik?

  this invariance business, you will need just law of large numbers. And there's this huge difference between uniform law of large numbers and large numbers.

• 38:11Lex Fridman?

  Is it useful to describe that a little more? Or should we just take it to...

• 38:16Vladimir Vapnik?

  No, for example, when I'm talking about duck, I gave three predicates, it was enough.

• 38:24Vladimir Vapnik?

  But if you will try to do formal distinguish, you will need a lot of observations.

• 38:31Lex Fridman?

  I got you.

• 38:33Vladimir Vapnik?

  And so that means that information about looks like a duck contain a lot of bit of information, formal bits of information. So we don't know that, how much bit of information contain things from artificial, from intelligence. And that is the subject of analysis. Till now...

• 39:00Vladimir Vapnik?

  All business. I don't like how people consider artificial intelligence. They consider us some codes which imitate activity of human being. It is not science. It is applications. You would like to imitate, go ahead, it is very useful and good problem. But you need to learn something more. How people try to do, how people can to develop. Say predicate swims like a dog or play like butterfly or something like that.

• 39:43Vladimir Vapnik?

  Then not the teacher says you, how it came in his mind. How he choose this image.

• 39:51Lex Fridman?

  So that process.

• 39:52Vladimir Vapnik?

  That is problem of intelligence.

• 39:54Lex Fridman?

  That is the problem of intel. And you see that connected to the problem of learning?

• 39:59Vladimir Vapnik?

  Absolutely.

• 40:00Lex Fridman?

  Are they?

• 40:01Vladimir Vapnik?

  Because you immediately give this predicate, like specific predicate swims like a dog or quack like a dog. It was chosen somehow.

• 40:12Lex Fridman?

  So what is the line of work, would you say? If you were to formulate as a set of open problems.

• 40:21Lex Fridman?

  That will take us there, will play like a butterfly, will get a system to be able to.

• 40:26Vladimir Vapnik?

  Let's separate two stories. One mathematical story, that if you have predicate, you can do something. And another story, you have to get predicate. It is intelligence problem. And people even did not start to understand intelligence. Because to understand intelligence, first of all, try to understand what doing teachers.

• 40:54Vladimir Vapnik?

  Half teacher teach. Why one teacher better than another one?

• 40:59Lex Fridman?

  Yeah, so you think we really even haven't started on the journey of generating the predicates?

• 41:05Vladimir Vapnik?

  No, you don't understand, you even don't understand that this problem exists.

• 41:11Vladimir Vapnik?

  Because, did you?

• 41:12Lex Fridman?

  You do.

• 41:13Vladimir Vapnik?

  No, I just know name. I want to understand why one teacher better than another. And half affect teacher, student. It is not because he repeating the problem which is in textbook. He make some remarks. He make some philosophy of reasoning.

• 41:38Lex Fridman?

  Yeah, that's a beautiful, so it is a formulation of a question that is the open problem. Why is one teacher better than another?

• 41:48Vladimir Vapnik?

  Right. What he does better?

• 41:52Lex Fridman?

  Yeah, what, what, what, why in every level? How do they get better? What does it mean to be better?

• 42:03Lex Fridman?

  The whole?

• 42:04Vladimir Vapnik?

  Yeah, yeah, from whatever model I have. One teacher can give a very good predicate. One teacher can say swims like a dog. And another can say jump like a duck. And jump like a duck carries zero information.

• 42:22Lex Fridman?

  Yeah.

• 42:24Lex Fridman?

  So what is the most exciting problem in statistical learning you've ever worked on or are working on now?

• 42:32Vladimir Vapnik?

  I just finished this invariant story. And I'm happy that, I believe that it is ultimate learning story. At least I can show that there are no another mechanism, only two mechanisms.

• 42:52Vladimir Vapnik?

  But they separate statistical part from intelligent part. And I know nothing about intelligent part. And if we will know this intelligent part, so it will help us a lot in teaching, in learning.

• 43:14Lex Fridman?

  In learning.

• 43:14Vladimir Vapnik?

  Yeah.

• 43:15Lex Fridman?

  Do you know we'll know it when we see it?

• 43:17Vladimir Vapnik?

  So for example, in my talk, the last slide was the challenge. So you have, say, NIST digit recognition problem.

• 43:27Vladimir Vapnik?

  And deep learning claims that they did it very well, say 99.5% of correct answers. But they used 60,000 observations.

• 43:39Lex Fridman?

  Yeah.

• 43:40Vladimir Vapnik?

  Can you do the same using 100 times less?

• 43:44Vladimir Vapnik?

  But incorporating invariance, what it means, you know, digit 1, 2, 3.

• 43:49Lex Fridman?

  Yeah.

• 43:50Vladimir Vapnik?

  Just looking at that, explain me which invariant I should keep to use 100 examples or say 100 times less examples to do the same job.

• 44:03Lex Fridman?

  Yeah, that last slide in unfortunately, your talk ended quickly, but that last slide was a powerful open challenge and a formulation of

• 44:15Vladimir Vapnik?

  Yeah.

• 44:16Lex Fridman?

  the essence here.

• 44:16Vladimir Vapnik?

  That is exact problem of intelligence. Because

• 44:23Vladimir Vapnik?

  everybody, when machine learning started and it was developed by mathematicians, they immediately recognize that we use much more training data than humans need it. But now again, we came to the same story. Have to decrease. That is the problem of learning. It is not like in deep learning, they use zillions of training data. Because maybe zillions are not enough if you have a good invariance. Maybe you will never collect some number of observations. But now it is a question to intelligence.

• 45:09Vladimir Vapnik?

  Have to do that. Because statistical part is ready. As soon as you supply us with predicate, we can do good job with small amount of observations. And the very first challenges will know digit recognition. And you know digits? And please, tell me in variants. I thinking about that, I can say, for digit three, I would introduce concept of horizontal symmetry. So the digit three has horizontal symmetry. So you mustn't say digit two or something like that.

• 45:49Vladimir Vapnik?

  But as soon as I get idea of horizontal symmetry, I can mathematically invent a lot of measure horizontal symmetry. Or then vertical symmetry, diagonal symmetry, whatever. If I have idea of symmetry. But what else?

• 46:07Vladimir Vapnik?

  Looking on digit, I see that it is meta predicate, which is not shape. It is something like symmetry, like how dark is whole picture, something like that.

• 46:28Vladimir Vapnik?

  Which can serve as a predicate.

• 46:31Lex Fridman?

  You think such a predicate could rise

• 46:35Lex Fridman?

  out of something that's not general? Meaning, it feels like for me to be able to understand the difference between a two and a three, I would need to have had a childhood of ten to fifteen years, playing with kids, going to school, being yelled by parents. All of that, walking, jumping, looking at ducks. And now, then I would be able to generate the right predicate for telling the difference between 2 and a 3.

• 47:18Lex Fridman?

  Or do you think there's a more efficient way?

• 47:21Vladimir Vapnik?

  I don't know. I know for sure that you must know something more than digits.

• 47:27Lex Fridman?

  Yes. And that's a powerful statement.

• 47:29Vladimir Vapnik?

  Yeah. But maybe there are several languages of description these elements of digits. So I'm talking about symmetry, about some properties of geometry. I'm talking about something abstract. I don't know that. But this is a problem of intelligence. So in one of our articles, it is trivial to show that every example can carry not more than one bit of information in real. Because when you show example and you say this is 1,

• 48:12Vladimir Vapnik?

  you can remove, say, a function which does not tell you 1. Say, it's the best strategy. If you can do it perfectly, it's remove half of the functions. But when you use one predicate, which looks like a duck, you can remove much more functions than half. And that means that it contains a lot of bit of informations from formal point of view. But when you have a general picture of what you want to recognize, a general picture of the world, can you invent this predicate?

• 48:55Vladimir Vapnik?

  And that predicate carries a lot of information.

• 49:02Lex Fridman?

  Beautifully put. Maybe just me, but in all the math you show, in your work, which is some of the most profound mathematical work in the field of learning AI and just math in general. I hear a lot of poetry and philosophy. You really kind of

• 49:22Lex Fridman?

  talk about philosophy of science. There's a poetry and music to a lot of the work you're doing and the way you're thinking about it. So do you, where does that come from? Do you escape to poetry? Do you escape to music?

• 49:35Vladimir Vapnik?

  I think that there exists ground truths.

• 49:39Lex Fridman?

  There exists ground truth?

• 49:40Vladimir Vapnik?

  Yeah, and that can be seen everywhere. The smart guy philosopher, sometime I surprise how they deep see. Sometime I see that some of them are completely out of subject.

• 50:00Vladimir Vapnik?

  But

• 50:02Vladimir Vapnik?

  the ground truth I see in music.

• 50:05Lex Fridman?

  Music of the ground truth?

• 50:06Vladimir Vapnik?

  Yeah. And in poetry, many poets, they believe that they take dictation.

• 50:16Lex Fridman?

  So what what piece of music as a piece of empirical evidence gave you a sense that they are, they're touching something in the ground truth?

• 50:29Vladimir Vapnik?

  It is structure.

• 50:31Lex Fridman?

  The structure of the math of music.

• 50:33Vladimir Vapnik?

  When you're listening to Bach, you see this structure, very clear, very classic, very simple. And the same in math, when you have axioms in geometry, you have the same feeling. And in poetry sometime you see the same.

• 50:50Lex Fridman?

  Yeah.

• 50:53Lex Fridman?

  And if you look back at your childhood, you grew up in Russia, you maybe were born as a researcher in Russia. You've developed as a researcher in Russia, you've came to the United States in a few places. If you look back, what were, what was some of your happiest moments as a researcher? Some of the most profound moments? Not in terms of their impact on society, but in terms of their impact on how damn good you feel that day and you remember that moment.

• 51:30Vladimir Vapnik?

  You know, every time when you found something,

• 51:36Vladimir Vapnik?

  it is great in the life. Every simple things. But my general feeling that I mostly, most of my time was wrong. You should go again and again and again. And try to be honest in front of yourself. Not to make interpretation, but try to understand that it related to grand rules. It is not my blah blah blah interpretation and something like that.

• 52:07Lex Fridman?

  But you're allowed to get excited at the, at the possibility of discovery.

• 52:12Vladimir Vapnik?

  Oh, yeah.

• 52:12Lex Fridman?

  You'll double, you have to double check it, but.

• 52:16Vladimir Vapnik?

  No, but how it related to another grand rules? Is it just temporary or it is forever? You know, you always have a feeling when you found something, how big is that? So 20 years ago, when we discovered statistical loading theory, nobody believed, except for one guy, Dudley from MIT.

• 52:46Vladimir Vapnik?

  And then in 20 years, it became fashion. And the same with support vector machines. That's kernel machines.

• 52:56Lex Fridman?

  So with, with support vector machines and learning theory, with, when you were working on it, you had a sense?

• 53:06Lex Fridman?

  That you had a sense of the, the profundity of it, the how? That this, this seems to be right, this seems to be powerful.

• 53:15Vladimir Vapnik?

  Right, absolutely, immediately. I recognize that it will last forever.

• 53:23Vladimir Vapnik?

  And now, when I found this.

• 53:29Vladimir Vapnik?

  Invariance story.

• 53:32Lex Fridman?

  You feel the same?

• 53:32Vladimir Vapnik?

  So I have a feeling that it is complete learning. Because I have proved that there are no different mechanisms. You can have some cosmetic improvement you can do, but in terms of invariance, you need both invariance and statistical learning, and they should work together. But also, I'm happy that we can formulate what is intelligence from that. And to separate from technical part. And that is completely different.

• 54:12Lex Fridman?

  Absolutely. Well, Vladimir, thank you so much for talking today.

• 54:15Vladimir Vapnik?

  Thank you.

• 54:15Lex Fridman?

  It's an honor.