Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence

Vladimir Vapnik is the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union, worked at the Institute of Control Sciences in Moscow, then in the US, worked at AT&T, NEC Labs, Facebook AI Research, and now is a professor at Columbia University. His work has been cited over 200,000 times. This conversation is part of the Artificial Intelligence podcast. If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter https://twitter.com/lexfridman, LinkedIn https://www.linkedin.com/in/lexfridman/, Facebook https://www.facebook.com/lexfridman, Medium https://medium.com/@lexfridman, or YouTube https://www.youtube.com/lexfridman where you can watch the video versions of these conversations. If you enjoy the podcast, please rate it 5 stars on Apple Podcasts https://podcasts.apple.com/us/podcast/artificial-intelligence/id1434243584, follow on Spotify https://open.spotify.com/show/2MAi0BvDc6GTFvKFPXnkCL, or support it on Patreon https://www.patreon.com/lexfridman. This episode is presented by Cash App. Download it (App Store https://apps.apple.com/us/app/cash-app/id711923939, Google Play https://play.google.com/store/apps/details?id=com.squareup.cash&hl=en_US), use code “LexPodcast”. Here’s the outline of the episode. On some podcast players you should be able to click the timestamp to jump to that time. 00:00 – Introduction 02:55 – Alan Turing: science and engineering of intelligence 09:09 – What is a predicate? 14:22 – Plato’s world of ideas and world of things 21:06 – Strong and weak convergence 28:37 – Deep learning and the essence of intelligence 50:36 – Symbolic AI and logic-based systems 54:31 – How hard is 2D image understanding? 1:00:23 – Data 1:06:39 – Language 1:14:54 – Beautiful idea in statistical theory of learning 1:19:28 – Intelligence and heuristics 1:22:23 – Reasoning 1:25:11 – Role of philosophy in learning theory 1:31:40 – Music (speaking in Russian) 1:35:08 – Mortality

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation with Vladimir Vapnik, part two, the second time we spoke on the podcast. He's the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union, worked at the Institute of Control Sciences in Moscow, then in the US, worked at AT&T, NEC Labs, Facebook AI Research, and now is a professor at Columbia University. His work has been cited over 200,000 times. The first time we spoke on the podcast was just over a year ago, one of the early episodes. This time, we spoke after a lecture he gave titled "Complete Statistical Theory of

• 0:43Lex Fridman?

  Learning" as part of the MIT series of lectures on deep learning and AI that I organized. I'll release the video of the lecture in the next few days. This podcast and lecture are independent from each other, so you don't need one to understand the other. The lecture is quite technical and math-heavy, so if you do watch both, I recommend listening to this podcast first, since the podcast is probably a bit more accessible. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at LexFriedman, spelled F-R-I-D-M-A-N.

• 1:24Lex Fridman?

  As usual, I'll do one or two minutes of ads now, and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Brokerage services are provided by Cash App Investing, a subsidiary of Square and member SIPC. Since Cash App allows you to send and receive money digitally peer-to-peer, and security in all digital transactions is very important, let me mention the PCI data security standard,

• 2:06Lex Fridman?

  PCI DSS Level 1 that Cash App is compliant with.

• 2:12Lex Fridman?

  I'm a big fan of standards for safety and security. And PCI DSS is a good example of that, where a bunch of competitors got together and agreed that there needs to be a global standard around the security of transactions. Now, we just need to do the same for autonomous vehicles and AI systems in general. So again, if you get Cash App from the App Store or Google Play and use the code LexPodcast, you get $10 and Cash App will also donate $10 to FIRST, one of my favorite organizations that is helping to advance robotics and STEM education for young people around the world. And now, here's my conversation with Vladimir Vapnik.

• 3:12Lex Fridman?

  You and I talked about Alan Turing yesterday, a little bit.

• 3:15Vladimir Vapnik?

  Yes.

• 3:16Lex Fridman?

  And that he, as the father of artificial intelligence, may have instilled in our field an ethic of engineering and not science, seeking more to build intelligence rather than to understand it. What do you think is the difference between these two paths of engineering intelligence and the science of intelligence?

• 3:38Vladimir Vapnik?

  It's completely different story. Engineering is a mutation of human activity. You have to make a device which behaves as human behave, have all the functions of human. It does not matter how you do it. But to understand what is intelligence about is quite different problem. So I think, I believe that it's somehow related to predicate we talked yesterday about. Because...

• 4:17Vladimir Vapnik?

  Look at the Vladimir Propp's idea. He just found thirty-one he predicates. He called it units, which can explain human behavior. At least in Russian tales, he look at Russian tales and derive from that. And then people realize that it more wide than in Russian tales. It is in TV, in movie serials and so on and so on.

• 4:50Lex Fridman?

  So you're talking about Vladimir Propp.

• 4:54Vladimir Vapnik?

  Right.

• 4:55Lex Fridman?

  Who in nineteen twenty published a book, Morphology of the Folk Tale.

• 4:59Vladimir Vapnik?

  Exactly.

• 4:59Lex Fridman?

  Describing thirty-one predicates that have this kind of sequential structure that a lot of the stories, narratives follow in Russian folklore and in other content. We'll talk about it. I'd like to talk about predicates in a focused way. But let me, if you allow me, to stay zoomed out on our friend Alan Turing. And, you know, he inspired a generation with the imitation game.

• 5:27Vladimir Vapnik?

  Yes.

• 5:28Lex Fridman?

  Do you think, if we can linger in a little bit longer, do you think we can learn? Do you think learning to imitate intelligence can get us closer to the science, to understanding intelligence? So why do you think imitation is so far from understanding?

• 5:48Vladimir Vapnik?

  I think that it is different between you have different goals. So your goal is to create something, something useful.

• 6:00Lex Fridman?

  Yeah.

• 6:01Vladimir Vapnik?

  And that is great. And you can see how much things was done, and I believe that it will be done even more. It's self-driving cars and also this business. It is great. And it was inspired by Turing vision. But understanding is very difficult. It more or less philosophical category. What means understand the world? I believe in scheme which start from Plato, that there exists vault of ideas. I believe that intelligence, it is vault of ideas, but it is vault of pure ideas. And when you.

• 6:46Vladimir Vapnik?

  Combine them with reality things. It creates, as in my case, invariance, which is very specific. And that, I believe, the combination of ideas in way to constructing invariant is intelligence. But first of all, predicate. If you know predicate. And hopefully, then, not too much predicate exists. For example, 31 predicate for human behavior. It is not a lot.

• 7:23Lex Fridman?

  Vladimir Propp used 31. You can even call them predicates, 31 predicates to describe stories, narratives. So you think human behavior. How much of human behavior? How much of our world, our universe? All the things that matter in our existence can be summarized in predicates of the kind that Propp was working with?

• 7:49Vladimir Vapnik?

  I think that we have a lot of form of behavior. But I think the predicate is much less. Because even in these examples, which I gave you yesterday, you saw that predicate can be can construct, one predicate can construct many different invariance depending on your data. They're applying to different data and they give different invariance. So, but pure ideas, maybe not so much.

• 8:25Lex Fridman?

  Not so many.

• 8:26Vladimir Vapnik?

  I don't know about that. But my guess, I hope, that's why I challenge about digit recognition, how much you need.

• 8:36Lex Fridman?

  I think we'll talk about computer vision and 2D images a little bit in your challenge.

• 8:41Vladimir Vapnik?

  That's exactly about intelligence.

• 8:43Lex Fridman?

  That's exactly. That's exactly about, no, that hopes to be exactly about the spirit of intelligence in the simplest possible way.

• 8:54Vladimir Vapnik?

  Absolutely, you should start this simplest way, otherwise you will not be able to do it.

• 8:59Lex Fridman?

  Well, there's an open question whether starting at the eminist digit recognition is a step towards intelligence or it's an entirely different thing.

• 9:09Vladimir Vapnik?

  I think that to beat records using, say, 100, 200 times less examples, you need intelligence.

• 9:17Lex Fridman?

  You need intelligence. So let's, because you use this term, and it'll be nice, I'd like to ask simple, maybe even dumb questions. Let's start with a predicate. In terms of terms and how you think about it, what is a predicate?

• 9:34Vladimir Vapnik?

  I don't know. I have a feeling formulas they exist. But I believe that predicate for 2D images, one of them is symmetry.

• 9:49Lex Fridman?

  Hold on a second, sorry, sorry to interrupt and pull you back. At the simplest level, we're not even, we're not being profound currently. A predicate is a statement of something that is true.

• 10:01Vladimir Vapnik?

  Yes.

• 10:03Lex Fridman?

  Do you think of predicates as somehow probabilistic in nature? Or is this binary, this is truly constraints of logical statements about the world?

• 10:17Vladimir Vapnik?

  In my definition, the simplest predicate is function. Function, and you can use this function to make inner product, that is predicate.

• 10:27Lex Fridman?

  What's the input and what's the output of the function?

• 10:31Vladimir Vapnik?

  Input is X, something which is input in reality. Say, if you consider digit recognition, it's pixel space.

• 10:40Lex Fridman?

  Yes.

• 10:41Vladimir Vapnik?

  Input. But it is function which in pixel space. But it can be any function from pixel space. And you choose, and I believe that there are several functions.

• 10:58Vladimir Vapnik?

  Which is important for understanding of images. One of them is symmetry. It's not so simple construction. As I described with derivative, with all this stuff. But another, I believe, I don't know how many. Is how well structuredized is picture.

• 11:20Lex Fridman?

  Structured?

• 11:21Vladimir Vapnik?

  Yeah.

• 11:21Lex Fridman?

  What do you mean by structuredized?

• 11:23Vladimir Vapnik?

  It is formal definition. Say, something heavy on the left corner. Not so heavy in the middle and so on. You describe in general concept of what you assume.

• 11:38Lex Fridman?

  Concepts, some kind of universal concepts.

• 11:42Vladimir Vapnik?

  Yeah. But I don't know how to formalize this.

• 11:46Lex Fridman?

  Do you? So this is the thing. There's a million ways we can talk about this. I'll keep bringing it up. But we humans have such concepts. When we look at digits. But it's hard to put them, just like you're saying now, it's hard to put them into words.

• 12:02Vladimir Vapnik?

  You know, that is example. When critics in music trying to describe music. They use predicate.

• 12:15Vladimir Vapnik?

  And not too many predicate. But in different combination. But they have some special words for describing music. And the same should be for images. But maybe there are critics who understand essence of what this image is about.

• 12:38Lex Fridman?

  Do you think there exists critics who can summarize the essence of images, human beings? The...

• 12:47Vladimir Vapnik?

  I hope so, yes. But that...

• 12:49Lex Fridman?

  Explicitly state them on paper.

• 12:54Lex Fridman?

  The fundamental question I'm asking is, do you think there exists a small set of predicates that will summarize images? It feels to our mind, like it does, that the concept of what makes a two and a three and a four.

• 13:13Vladimir Vapnik?

  No, no, no, it's not on this level. What it should not describe, two, three, four. It describes some construction which allow you to create invariance.

• 13:29Lex Fridman?

  And invariance, sorry to stick on this terminology.

• 13:33Vladimir Vapnik?

  Invariants, it is, it is property of your image. Say, I can say, looking on my image, it is more or less symmetric. And I can give you value of symmetry, say, level of symmetry, using this function which I gave yesterday.

• 14:00Vladimir Vapnik?

  And you can describe that your image have these characteristics, exactly in the way how musical critics describe music. So, but this is invariant applied to specific data, to specific music, to something. I strongly believe in this Plato ideas that there exists world of predicate and world of reality. And predicate and reality is somehow connected and you have to know that.

• 14:39Lex Fridman?

  Let's talk about Plato a little bit. So you draw a line from Plato to Hegel to Wagner to today.

• 14:46Vladimir Vapnik?

  Yes.

• 14:47Lex Fridman?

  So Plato has forms, the theory of forms. There's a world of ideas and a world of things, as you talk about. And there's a connection. And presumably, the world of ideas is very small. And the world of things is arbitrarily big. But they're all what Plato calls them like. It's a shadow. The real world is a shadow from the world of.

• 15:11Vladimir Vapnik?

  Yeah, you have projection.

• 15:13Lex Fridman?

  Projection.

• 15:14Vladimir Vapnik?

  Of world of idea.

• 15:16Lex Fridman?

  Yeah, very poetic.

• 15:17Vladimir Vapnik?

  In reality, you can realize this projection using this invariance. Because it is projection for on specific examples, which creates specific features of specific objects. So.

• 15:35Lex Fridman?

  So the essence of intelligence is while only being able to observe the world of things, try to come up with a world of ideas.

• 15:44Vladimir Vapnik?

  Exactly, like in this music story. Intelligent musical critics knows this all this world and have a feeling about what.

• 15:51Lex Fridman?

  I feel like that's a contradiction, intelligent music critics. But I think I think music is to be

• 16:01Lex Fridman?

  enjoyed in all its forms. The notion of critic, like a food critic.

• 16:07Vladimir Vapnik?

  No, I don't want that emotion.

• 16:09Lex Fridman?

  That's an interesting question. Does emotion? There's a certain elements of the human psychology, of the human experience, which seem to almost contradict intelligence and reason. Like emotion, like fear, like like love. All of those things are those not connected in any way to the space of ideas?

• 16:34Vladimir Vapnik?

  That's I don't know. I I just want to be concentrate on very simple story, on digit recognition.

• 16:44Lex Fridman?

  So you don't think you have to love and fear death in order to recognize digits?

• 16:49Vladimir Vapnik?

  I don't know. Because it's so complicated. It is it involves a lot of stuff which I I never consider. But I know about digit recognition. And I know that for digit recognition to.

• 17:09Vladimir Vapnik?

  To get the records from small number of observations, you need predicate, but not special predicate for this problem, but universal predicate which understand world of images.

• 17:25Lex Fridman?

  Of visual information.

• 17:26Vladimir Vapnik?

  Visual, yeah. But on the first step, they understand, say, world of handwritten digits or characters or something simple.

• 17:38Lex Fridman?

  So like you said symmetry is an interesting one.

• 17:40Vladimir Vapnik?

  No, that's what I think one of the predicate is related to symmetry. But. The level of symmetry.

• 17:47Lex Fridman?

  Okay, degree of symmetry. So you you think symmetry at the bottom is a universal notion and there's there's there's degrees of a single kind of symmetry? Or is there many kinds of symmetries?

• 18:01Vladimir Vapnik?

  Many kinds of symmetries. There is a symmetry, antisymmetry, say, letter S. So it has vertical antisymmetry. And it could be diagonal symmetry, vertical symmetry.

• 18:19Lex Fridman?

  So when you when you cut vertically, the letter S.

• 18:24Vladimir Vapnik?

  Yeah, then the upper part and lower part in different directions.

• 18:32Lex Fridman?

  Yeah, inverted along the Y axis.

• 18:35Vladimir Vapnik?

  Yeah.

• 18:35Lex Fridman?

  But that's just like one example of symmetry, right? Isn't there like?

• 18:39Vladimir Vapnik?

  Right, but there is a degree of symmetry. If you play all this literative stuff to to to do tangent distance, whatever I describe, you can do, you can have a degree of symmetry. And that is what describing reason of image. It is the same as you will describe this image. Saying about digit S, it has antisymmetry. Digit three symmetric, more or less look for symmetry.

• 19:21Lex Fridman?

  Do you think such concepts like symmetry, predicates like symmetry, is it a hierarchical set of concepts? Or are these independent, distinct predicates that we want to discover, some set of?

• 19:40Vladimir Vapnik?

  No, there is a deal of symmetry. And you can, this idea of symmetry, make

• 19:49Vladimir Vapnik?

  very general. Like degree of symmetry. The degree of symmetry can be zero, no symmetry at all. Or degree of symmetry, say, more or less symmetrical. But you have one of this description, and symmetry can be different. As I told, horizontal, vertical, diagonal, and antisymmetries is also concept of symmetry.

• 20:18Lex Fridman?

  What about shape in general? I mean, symmetry is a fascinating notion, but...

• 20:23Vladimir Vapnik?

  No, no, I'm talking about digit. I would like to concentrate on all, I would like to know predicate for digit recognition.

• 20:31Lex Fridman?

  Yes, but symmetry is not enough for digit recognition, right?

• 20:36Vladimir Vapnik?

  It is not necessarily for digit recognition. It helps to create invariant, which will, which you can use when you will have examples for digit recognition. You have regular problem of digit recognition. You have examples of the first class, or second class. Plus, you know that there exists, there exists concept of symmetry. And you apply, when you're looking for decision rule, you will apply

• 21:10Vladimir Vapnik?

  concept of symmetry, of this level of symmetry which you estimate from. So let's, let's talk. Everything is...

• 21:21Vladimir Vapnik?

  Comes from weak convergence.

• 21:23Lex Fridman?

  What is convergence? What is weak convergence? What is strong convergence? I'm sorry, I'm gonna do this to you. What are we converging from and to?

• 21:33Vladimir Vapnik?

  You're converging. You would like to have a function. The function which, say indicator function which indicate your digit five, for example.

• 21:46Lex Fridman?

  A classification task.

• 21:48Vladimir Vapnik?

  Let's talk only about classification.

• 21:50Lex Fridman?

  So classification means you will say whether this is a five or not, or say which of the 10 digits it is.

• 21:57Vladimir Vapnik?

  Right, right. I would like to have these functions. Then.

• 22:07Vladimir Vapnik?

  I have some examples.

• 22:13Vladimir Vapnik?

  I can consider property of these examples. Say symmetry. And I can measure level of symmetry for every digit.

• 22:25Vladimir Vapnik?

  And then I can take average.

• 22:30Vladimir Vapnik?

  And I, from my training data. And I will consider only functions of conditional probability, which I'm looking for my decision rule. Which

• 22:47Vladimir Vapnik?

  applying to

• 22:52Vladimir Vapnik?

  to digits will give me the same average as I observed on training data. So actually, this is different level of description of what you want. You want not just, you show not one digit. You show this predicate show general property.

• 23:16Vladimir Vapnik?

  Of all digits which you have in mind. If you have in mind digit three, it gives you property of digit three. And you select as admissible set of function, only function which keeps this property. You will not consider other functions. So you're immediately looking for smaller subset of function.

• 23:41Lex Fridman?

  That's what you mean by admissible function.

• 23:44Vladimir Vapnik?

  Admissible function, exactly.

• 23:45Lex Fridman?

  Which is still a pretty large. For the number three.

• 23:49Vladimir Vapnik?

  It is pretty large, but if you have one predicate.

• 23:53Lex Fridman?

  Right.

• 23:53Vladimir Vapnik?

  But according to weak, there is a strong and weak convergence. Strong convergence is convergence in function. You're looking at the function, on one function, and you're looking on the another function. And a square difference from them should be small. If you take difference in any points, make a square, make an integral, and it should be small. That is convergence in function. Suppose you have some function, any function. So I would say, I say that some function converge to this function

• 24:34Vladimir Vapnik?

  if integral from square difference between them is small.

• 24:39Lex Fridman?

  That's the definition of strong convergence.

• 24:41Vladimir Vapnik?

  That definition of.

• 24:42Lex Fridman?

  Two functions, integral, the difference.

• 24:45Vladimir Vapnik?

  Yeah. It is convergence in functions.

• 24:48Lex Fridman?

  Yeah.

• 24:49Vladimir Vapnik?

  But you have different convergence in functionals. You take any function, you take some function phi and take inner product, this function is F function. F zero function which you want to find. And that gives you some value. So you say that.

• 25:13Vladimir Vapnik?

  Set of functions converge in inner product to this function if this value of inner product converge to value f zero. That is for one phi. But weak convergence requires that it converge for any function of Hilbert space. If it converge for any function of Hilbert space, then you would say that this is weak convergence. You can think that when you take integral, that is property, integral property of function.

• 25:53Vladimir Vapnik?

  For example, if you will take sine or cosine, it is coefficient of, say, Fourier expansion.

• 26:02Vladimir Vapnik?

  So if it converge for all coefficients of Fourier expansion, so under some condition, it converge to the function you're looking for. But weak convergence means any property. Convergence not point wise, but integral property of function. So weak convergence means integral property of functions. When I'm talking about predicate, I would like to

• 26:36Vladimir Vapnik?

  formulate which integral properties I would like to have for convergence. So, and if I will take one predicate, its function which I measure property.

• 26:54Vladimir Vapnik?

  If I will use one predicate and say, I will consider only function which give me the same value as with this predicate. I selecting set of functions from functions which is admissible in the sense that function which I looking for is this set of functions. Because I checking in.

• 27:21Vladimir Vapnik?

  And training data, it is, it gives the same.

• 27:25Lex Fridman?

  Yes, it always has to be connected to the training data in terms of.

• 27:29Vladimir Vapnik?

  Yeah, but, but property, you can know independent on training data. And this guy, prop.

• 27:38Lex Fridman?

  Yeah.

• 27:38Vladimir Vapnik?

  So that there is formal property, 31 property.

• 27:42Lex Fridman?

  Fairy tale, Russian fairy tale.

• 27:44Vladimir Vapnik?

  But Russian fairy tale is not so interesting. More interesting that people apply this to, to movies, to theater, to, to, to, to different things. And the same works, they're universal.

• 27:59Lex Fridman?

  Well, so I would argue that there's a little bit of a difference between

• 28:04Lex Fridman?

  the kinds of things that were applied to, which are essentially stories and digit recognition.

• 28:11Vladimir Vapnik?

  It is the same story.

• 28:12Lex Fridman?

  You, you, you're saying digits, there's a story within the digit.

• 28:16Vladimir Vapnik?

  Yeah. So, but my, my point is why I hope that it possible to beat record using not 60,000, but say 100 times less, because instead you will give predicates. And you will select your decision not from wide set of functions, but from set of function which keeps as predicate. But predicate is not related just to digit recognition.

• 28:49Lex Fridman?

  Right, so.

• 28:50Vladimir Vapnik?

  Like in plot this case.

• 28:54Lex Fridman?

  Do you think it's possible to automatically discover the predicates? This, so you basically said that the essence of intelligence is the discovery of good predicates.

• 29:06Vladimir Vapnik?

  Yeah.

• 29:08Lex Fridman?

  Now, the natural question is,

• 29:12Lex Fridman?

  you know, that's what Einstein was good at doing in physics. Can we make machines do these kinds of discovery of good predicates? Or is this ultimately a human endeavor?

• 29:24Vladimir Vapnik?

  That's I don't know. I don't think that machine can do, because.

• 29:31Vladimir Vapnik?

  According to theory about weak convergence, any function from Hilbert space can be predicate. So you have infinite number of predicate in upper and before you don't know which predicate is good on which. But whatever prop show and why people call it breakthrough, that there is not too many predicate which cover most of situation happened in the world.

• 30:08Lex Fridman?

  So there's a sea of predicates. And most of the, only a small amount are useful for the kinds of things that happen in the world.

• 30:18Vladimir Vapnik?

  I think that, I would say, only small part of predicate very useful, useful all of them.

• 30:28Lex Fridman?

  Only very few are what we should, let's call them good predicates.

• 30:32Vladimir Vapnik?

  Very good predicate.

• 30:33Lex Fridman?

  Very good predicates. So can we linger on it? What's your intuition? Why is it hard for a machine to discover good predicates?

• 30:44Vladimir Vapnik?

  I even in my talk describe how to do predicate, how to find new predicate. I'm not sure that it is very good.

• 30:52Lex Fridman?

  What did you propose in your talk?

• 30:53Vladimir Vapnik?

  No, in my talk, I gave example for diabetes.

• 30:59Lex Fridman?

  Diabetes.

• 31:00Vladimir Vapnik?

  When we achieve some percent, so then we're looking from area where some sort of predicate which I formulate does not, keeps invariant. So if it doesn't keep, I retrain my data. I select only function which keeps this invariant. And when I did it, I improved my performance. I can looking for this predicate. I know technically how to do that. And you can, of course, do it using machine.

• 31:42Vladimir Vapnik?

  But I'm not sure that we will construct the smartest predicate.

• 31:48Lex Fridman?

  But this is the, allow me to linger on it. Because that's the essence, that's the challenge. That is artificial. That's, that's the human level intelligence that we seek is the discovery of these good predicates. You've talked about deep learning as a way to, the predicates they use and the functions are mediocre. We can find better ones.

• 32:12Vladimir Vapnik?

  Let's talk about deep learning.

• 32:14Lex Fridman?

  Sure, let's do it.

• 32:15Vladimir Vapnik?

  I know only Jan Sliekun, convolutional network. And what else? I don't know, and it's a very simple convolution.

• 32:25Lex Fridman?

  There's not much else to know.

• 32:26Vladimir Vapnik?

  To pixel left and right.

• 32:27Lex Fridman?

  Yes.

• 32:27Vladimir Vapnik?

  I can do it like that. With one predicate.

• 32:31Lex Fridman?

  Convolution is a single predicate.

• 32:33Vladimir Vapnik?

  It's single, it's, it's a single predicate.

• 32:38Lex Fridman?

  Yes.

• 32:38Vladimir Vapnik?

  It is, you know exactly, you take the derivative for translation and predicate this should be kept.

• 32:48Lex Fridman?

  So that's a single predicate, but humans discovered that one, or at least...

• 32:52Vladimir Vapnik?

  Not it, that is a risk, not too many predicates. And that is big story because Jan did it 25 years ago and nothing so clear was added to, to, to deep network. And then I don't understand why we should talk about deep network instead of talking about piecewise linear functions which keeps this predicate.

• 33:19Lex Fridman?

  Well, the, you know, a counter argument is that maybe the amount of predicates necessary to solve general intelligence, say in space of images, doing efficient recognition of handwritten digits is very small. And so we shouldn't be so obsessed about finding, we'll find other good predicates like convolution, for example. You know, there, there has been other... Advancements like if you look at the work with attention, there's attentional mechanisms in especially used in natural language focusing the the network's ability to to learn at which

• 34:03Lex Fridman?

  part of the input to look at. The thing is, there's other things besides predicates that are important for the actual engineering mechanism of showing how much you can really do given such these predicates. I it, I mean, that's essentially the work of deep learning is constructing architectures that are able to be given the training data to be able to converge towards a function that can approximate can generalize well. It's an engineering problem.

• 34:41Vladimir Vapnik?

  Yeah, I understand. But let's talk not on emotional level, but on a mathematical level. You have set of piecewise linear functions. It is all possible neural networks. It's just piecewise linear functions. It's many, many pieces.

• 35:02Lex Fridman?

  A large, large number of piecewise linear functions.

• 35:04Vladimir Vapnik?

  Exactly, but

• 35:05Lex Fridman?

  Very large.

• 35:06Vladimir Vapnik?

  Very large.

• 35:07Lex Fridman?

  Almost feels like too large.

• 35:08Vladimir Vapnik?

  But it's still it's still simpler than say convolution than reproducing terminal Hilbert space, which have a Hilbert set of function.

• 35:17Lex Fridman?

  What's Hilbert space?

• 35:19Vladimir Vapnik?

  It's space with infinite number of coordinates, say, or function for expansion, something like that. So it's much richer. So and when I talking about closed form solution, I talking about this set of function, not piecewise linear set, which is particular case of it is small part.

• 35:47Lex Fridman?

  So neural networks is a small part of the space you're talking of functions you're talking about.

• 35:52Vladimir Vapnik?

  Small, small, say, a small set of functions. Let me take it that. But it is fine. It is fine. I don't want to to discuss the small or big you take advantage. So you have some set of functions.

• 36:08Vladimir Vapnik?

  So now, when you're trying to create architecture, you would like to create admissible set of function, which all your tricks to use, not all functions, but some subset of this set of functions.

• 36:24Vladimir Vapnik?

  Say, when you're introducing convolutional net, it is way to make this subset useful for you. But from my point of view, convolutional, it is something you want to keep some invariance, say, translation invariance. But now, if you understand this and you cannot explain on the level of ideas what neural network does,

• 36:58Vladimir Vapnik?

  you should agree that it is much better to have a set of functions. And they say, this set of functions should be admissible. It must keep this invariant, this invariant, and that invariant. You know that as soon as you incorporate new invariance, set of function becomes smaller and smaller and smaller.

• 37:19Lex Fridman?

  But all the invariants are specified by you, the human?

• 37:23Vladimir Vapnik?

  Yeah, but what I am hope that there is a standard predicate, like Prob show, that what that's what I want to find for digit recognition. If we start, it is completely new area. What is intelligence about on the level, starting from from Platon's idea? What is world of ideas? So, and I believe that is not too many.

• 37:52Lex Fridman?

  Yeah.

• 37:53Vladimir Vapnik?

  But you know, it is amusing that mathematician doing something, a neural network in general function, but people from literature, from art, they use this all the time.

• 38:06Lex Fridman?

  That's right.

• 38:07Vladimir Vapnik?

  Invariants saying, say, it is great how how people describe music. We should learn from that. And something on this level. But so why Vladimir Prob, who was just theoretical, who studied theoretical literature, he found that.

• 38:30Lex Fridman?

  You know what, let me throw that right back at you because there's a little bit of a that's less mathematical and more emotional philosophical Vladimir prop. I mean, he wasn't doing math.

• 38:41Vladimir Vapnik?

  No.

• 38:43Lex Fridman?

  And you just said another emotional statement, which is you believe that this Plato world of ideas is small.

• 38:52Vladimir Vapnik?

  I hope.

• 38:54Lex Fridman?

  I hope.

• 38:56Lex Fridman?

  Do you, what's your intuition, though? If we can linger on it.

• 39:01Vladimir Vapnik?

  You know, it is not just small or big. I know exactly. Then when I introducing some predicate, I decrease set of functions. But my goal to decrease set of function much.

• 39:20Lex Fridman?

  By as much as possible.

• 39:22Vladimir Vapnik?

  By as much as possible. Good predicate which does this, then I should choose next predicate which does which decrease set as much as possible. So set of good predicate, it is such that they decrease this amount of admissible functions.

• 39:44Lex Fridman?

  So if each good predicate significantly reduces the set of admissible functions, that there naturally should not be that many good predicates?

• 39:52Vladimir Vapnik?

  No, but if you reduce very well the VC dimension of the function of admissible set of function is small and you need not too much training data to do well.

• 40:10Lex Fridman?

  And VC dimension, by the way, is a measure of capacity of this set of function.

• 40:14Vladimir Vapnik?

  Right. Roughly speaking, how many function in this set. So you're decreasing, decreasing, and it makes easy for you to find function you're looking for. But the most important part to create good admissible set of functions, and it probably there are many ways, but.

• 40:38Vladimir Vapnik?

  The good predicate, such that that can do that. But for for for this duck, you should know a little bit about duck because.

• 40:48Lex Fridman?

  What are the, what are the three fundamental laws of ducks?

• 40:52Vladimir Vapnik?

  Looks like a duck, swims like a duck and quack like a duck.

• 40:55Lex Fridman?

  You should know something about ducks to be able to.

• 40:58Vladimir Vapnik?

  Not necessarily. Looks like a horse. It's also good.

• 41:03Lex Fridman?

  It's not, it generalizes from ducks.

• 41:07Vladimir Vapnik?

  And talk like, and make sound like horse or something. And run like horse and moves like horse. It is general, it is general predicate. That this applied to duck. But for duck, you can say, play chess like duck.

• 41:26Lex Fridman?

  You cannot say play chess like duck.

• 41:28Vladimir Vapnik?

  Why not?

• 41:29Lex Fridman?

  So you're saying you can, but that would not be a good.

• 41:32Vladimir Vapnik?

  No, you, you, you will not reduce a lot of.

• 41:35Lex Fridman?

  You would not do, yeah, you would not reduce the set of functions.

• 41:38Vladimir Vapnik?

  So you can, the, the story is, formal story, mathematical story. Is that you can use any function you want as a predicate. But some of them are good, some of them are not, because some of them reduce a lot of functions. To admissible set, or some of them.

• 41:56Lex Fridman?

  But the question is, and I'll probably keep asking this question, but how do we find such? What's your intuition? Handwritten, handwritten recognition. How do we find the, the answer to your challenge?

• 42:09Vladimir Vapnik?

  Yeah, yeah, I understand it like that. I understand what.

• 42:14Lex Fridman?

  What defined?

• 42:16Vladimir Vapnik?

  What it means, I knew predicate.

• 42:18Lex Fridman?

  Yeah.

• 42:19Vladimir Vapnik?

  Like guy who understand music can say this word which he described when, when he listened to music. He understand music. He use not too many different, or you can do like prop. You can make collection. What he talking about music, about this, about that. It's not too many different situations he described.

• 42:42Lex Fridman?

  Because we mentioned Vladimir proper bunch, let me just mention, there's a, so there's a sequence of, of thirty one.

• 42:50Lex Fridman?

  Structural notions that are common in stories and I think.

• 42:54Vladimir Vapnik?

  You call it units.

• 42:55Lex Fridman?

  Units and I think they resonate. I mean, it starts, just to give an example, absention. A member of the hero's community, a family, leaves the security of the home environment. Then it goes to the interdiction. A forbidding edict or command is passed upon the hero. Don't go there. Don't do this. The hero is warned against some action. Then step three, violation of interdiction. Break, you know, break the rules, break out on your own. Then reconnaissance. The villain makes an effort to attain knowledge, needing to fulfill their plot, so on. It goes on like this, ends, uh, ends in a wedding. Number thirty one.

• 43:36Vladimir Vapnik?

  Yeah.

• 43:36Lex Fridman?

  Happily ever after.

• 43:37Vladimir Vapnik?

  No, he, he, he just gave description of all situations. He understands this vault.

• 43:45Lex Fridman?

  Of folktales.

• 43:46Vladimir Vapnik?

  Yeah, not folk, but it's for stories. And this story is not in just folktales. The story is in detective serials as well.

• 43:57Lex Fridman?

  And probably in our lives, we probably live.

• 44:00Vladimir Vapnik?

  Read this. At the end, the, the, the road that the, this predicate is good for different situation. From movie, from, for movie, for theater.

• 44:14Lex Fridman?

  By the way, there's also criticism, right? There's an other way to interpret narratives from, um, Claude Levy Strauss.

• 44:26Vladimir Vapnik?

  I don't, I, I'm not in this business.

• 44:29Lex Fridman?

  No, I know it's theoretical literature, but it's looking at paradise.

• 44:32Vladimir Vapnik?

  It's always the, the, the discussion, yeah.

• 44:36Lex Fridman?

  Yeah.

• 44:37Vladimir Vapnik?

  But at least there is a units. It's not too many units that can describe. But this guy probably gives another units. Or another way.

• 44:48Lex Fridman?

  Exactly. Another, another, another set of units.

• 44:51Vladimir Vapnik?

  Another set of predicates. It does not matter how. But they exist probably.

• 44:58Lex Fridman?

  My, my question is whether, given those units, whether without our human brains to interpret these units, they would still hold as much power as they have. Meaning, are those units enough when we give them to the alien species?

• 45:15Vladimir Vapnik?

  Let, let me ask you. Do you understand digit, digi- digit images?

• 45:23Lex Fridman?

  No, I don't understand.

• 45:24Vladimir Vapnik?

  No, no, no. Uh, when you can recognize as digit images, it means that you understand.

• 45:30Lex Fridman?

  Yes.

• 45:30Vladimir Vapnik?

  You, you understand char- characters, you understand.

• 45:34Lex Fridman?

  No, no, no, no. I, I, um,

• 45:39Lex Fridman?

  it's, it's the imitation versus understanding question because I don't understand the mechanism by which I.

• 45:46Vladimir Vapnik?

  No, no, no, I'm not talking about, I'm talking about predicates. You understand that it involves symmetry, maybe structure, maybe something else. I cannot formulate. I just was able to find symmetries, so degree of symmetries.

• 46:00Lex Fridman?

  That's really good. So, this is a good line. "I feel like I understand the basic elements of what makes a good hand recognition system my own." Like symmetry connects with me. It seems like that's a very powerful predicate. My question is, is there a lot more going on that we're not able to introspect? Maybe I need to be able to understand a huge amount in the world of ideas. Uh, thousands of predicates, millions of predicates in order to do hand recognition.

• 46:37Vladimir Vapnik?

  I don't think so.

• 46:40Lex Fridman?

  So, you're, you're, both your hope and your intuition are such that very few predicates are enough.

• 46:45Vladimir Vapnik?

  You're using digits, you're using examples as well. The theory says that if you will use all.

• 46:58Vladimir Vapnik?

  Possible functions from Hilbert space, all possible predicate. You don't need training data. You just will have admissible set of function which contain one function.

• 47:13Lex Fridman?

  Yes. So the trade off is when you're not using all predicates. You're only using a few good predicates. You need to have some training data.

• 47:22Vladimir Vapnik?

  Yes, exactly.

• 47:23Lex Fridman?

  The more, the more good predicates you have, the less training data.

• 47:26Vladimir Vapnik?

  Exactly. That is intelligent.

• 47:30Lex Fridman?

  Still, okay. I'm going to keep asking the same dumb question. Handwritten recognition to solve the challenge. You kind of propose a challenge that says we should be able to get state of the art MNIST error rates by using very few 60, maybe fewer examples per digit. What kind of predicates do you think?

• 47:52Vladimir Vapnik?

  That is the challenge. So people who will solve this problem.

• 47:56Lex Fridman?

  They will answer.

• 47:57Vladimir Vapnik?

  They will answer.

• 47:58Lex Fridman?

  Do you think they'll be able to answer it in a human explainable way?

• 48:04Vladimir Vapnik?

  They just need to write function. That's it.

• 48:07Lex Fridman?

  But so can that function be written, I guess, by an automated reasoning system? Whether we're talking about a neural network, learning a particular function or another mechanism?

• 48:22Vladimir Vapnik?

  No, neural, I'm not against neural network. I am against admissible set of function which create neural network. You did it by hand. You don't, you don't do it by invariance, by predicate, by, by, by reason.

• 48:41Lex Fridman?

  But neural networks can then reverse, do the reverse step of helping you find a function. Just the task of a neural network is to find a disentangled representation, for example, as they call, is to find that one predicate function that's really captures some kind of essence, one, not the entire essence, but one very useful essence of this particular visual space. Do you think that's possible? Like, listen, I'm grasping, hoping there's an automated way to find good predicates, right? So the question is, what are the mechanisms of finding good predicates, ideas that you think we should pursue? A young grad student listening right now.

• 49:26Vladimir Vapnik?

  I gave example, so find situation where predicate which you're suggesting don't create invariant. It's like in physics, find situation where existing theory cannot explain it.

• 49:52Lex Fridman?

  Find situation where the existing theory can't explain it.

• 49:57Vladimir Vapnik?

  Theory cannot explain this.

• 49:58Lex Fridman?

  So you're finding contradictions.

• 49:59Vladimir Vapnik?

  Find contradiction and then remove this contradiction. But in my case, what means contradiction, you find function which, if you will use this function, you're not keeping convergence.

• 50:11Lex Fridman?

  So really, the process of discovering contradictions?

• 50:17Vladimir Vapnik?

  Yeah. It is like in physics, find situation where you have contradiction for one of the property, for one of the predicate, then include this predicate, making invariance, and solve again this problem. Now you don't have contradiction. But it is not.

• 50:44Vladimir Vapnik?

  The best way, probably, I don't know, to looking for predicate.

• 50:49Lex Fridman?

  That's just one way, okay.

• 50:50Vladimir Vapnik?

  That, no, no, it is brute force way.

• 50:52Lex Fridman?

  The brute force way. What about the ideas of some, what, uh, big umbrella term of symbolic AI? These, what, in the eighties with expert systems, sort of logic reasoning based systems. Is there hope there to find some uh, through sort of deductive reasoning to, to find good predicates?

• 51:22Vladimir Vapnik?

  I don't think so.

• 51:26Vladimir Vapnik?

  I think that just logic is not enough.

• 51:29Lex Fridman?

  It's kind of a compelling notion, though. You know, that when smart people sit in a room and reason through things, it seems compelling. And making our machines do the same is also compelling.

• 51:41Vladimir Vapnik?

  So, everything is very simple.

• 51:46Vladimir Vapnik?

  When you have infinite number of predicate,

• 51:51Vladimir Vapnik?

  you can choose the, the function you want. You have invariants and you can choose the function you want. But you have to have a not too many

• 52:07Vladimir Vapnik?

  invariants to solve the problem.

• 52:13Vladimir Vapnik?

  So, and have from infinite number of function to select finite number and hopefully small fine, fa, number of functions, which is good enough to extract small set of admissible functions.

• 52:34Vladimir Vapnik?

  So they will be admissible, it's for sure, because every function just decrease set of function and leaving it admissible. But it will be small.

• 52:44Lex Fridman?

  But why do you think logic based systems don't, can't help? Intuition, not?

• 52:52Vladimir Vapnik?

  Because you should know reality. You should know life. This guy like Prop, he knows something and he tried to to put in invariant his understanding.

• 53:06Lex Fridman?

  But that's the human, yeah, see, you're putting too much value into Vladimir Prop's knowing something.

• 53:14Vladimir Vapnik?

  No, it is.

• 53:16Lex Fridman?

  I might be misunderstanding.

• 53:18Vladimir Vapnik?

  What means you know life?

• 53:21Vladimir Vapnik?

  What it means?

• 53:22Lex Fridman?

  You know common sense?

• 53:24Vladimir Vapnik?

  No, no. You know something. Common sense, it is some rules.

• 53:30Lex Fridman?

  You think so? Common sense is simply rules? Common sense is every, it's mortality, it's no, it's it's fear of death, it's love, it's spirituality, it's happiness and sadness. All of it is tied up into understanding gravity, which is what we think of as common sense.

• 53:53Vladimir Vapnik?

  I don't really to discuss so wide. I want to discuss, understand digit recognition.

• 54:02Lex Fridman?

  Anytime I bring up love and death, you bring it back to digit recognition.

• 54:08Vladimir Vapnik?

  No, you know, it is durable because there is a challenge.

• 54:12Lex Fridman?

  Yeah.

• 54:12Vladimir Vapnik?

  Which I see how to solve it. If I will have a student concentrate on this work, I will suggest something to solve.

• 54:21Lex Fridman?

  You mean handwritten recognition? Yeah, it's a beautifully simple, elegant, and yet.

• 54:27Vladimir Vapnik?

  I think that I know invariance which will solve this.

• 54:30Lex Fridman?

  You do?

• 54:31Vladimir Vapnik?

  I think so.

• 54:31Lex Fridman?

  You think so?

• 54:32Vladimir Vapnik?

  But it is not universal, it is maybe, I want some universal invariance which are good not only for digit recognition, for image understanding.

• 54:47Lex Fridman?

  So let me ask, how hard do you think is 2D image understanding? So if we can kind of intuit handwritten recognition, how big of a step, leap, journey is it from that? If I gave you good, if I solved your challenge for handwritten recognition, how long would my journey then be from that to understanding more general natural images?

• 55:16Vladimir Vapnik?

  Immediately. You will understand this as soon as you will make a record. Because it is not for free. As soon as you will create several invariants, which will help you to get the same performance that the best neural net did, using hundred, maybe more than hundred times less examples. You have to have something smart to do that.

• 55:48Lex Fridman?

  And you're saying?

• 55:49Vladimir Vapnik?

  That it is invariant. It is predicate. Because you should put some idea have to do that.

• 55:56Lex Fridman?

  But, okay. Let me just pause. Maybe it's a trivial point, maybe not. But handwritten recognition feels like a 2D, two dimensional problem. And it seems like how much complicated is the fact that most images are projection of a three dimensional world onto 2D plane? It feels like for a three dimensional world, we need to start understanding common sense in order to understand an image. It's no longer visual shape and symmetry. It's having to start to understand concepts of, understand life.

• 56:39Vladimir Vapnik?

  Yeah. You're talking that there are different invariant, different predicate. Yeah.

• 56:45Lex Fridman?

  And potentially much larger number.

• 56:49Vladimir Vapnik?

  You know, maybe, but let's start from simple.

• 56:53Lex Fridman?

  Yeah, but you said that it would be immediate.

• 56:55Vladimir Vapnik?

  You know, I, I, I cannot think about things which I don't understand. This I understand, but I'm sure that I don't understand everything there.

• 57:05Lex Fridman?

  Yeah.

• 57:05Vladimir Vapnik?

  It's like Stain say, do as simple as possible, but not simpler. And that is exact case.

• 57:13Lex Fridman?

  With handwritten.

• 57:14Vladimir Vapnik?

  With handwritten.

• 57:15Lex Fridman?

  Yeah, but that's the difference between you and I. I, I, I welcome and enjoy thinking about things I completely don't understand. Because to me, it's a natural extension without having solved handwritten recognition to wonder how, how difficult is the, the, the next step of understanding 2D, 3D images? Because ultimately, while the science of intelligence is fascinating, it's also fascinating to see how that maps to the engineering of intelligence. And recognizing handwritten digits is not, doesn't help you.

• 57:58Lex Fridman?

  It might, it may not help you with the problem of general intelligence. We don't know. It'll help you a little bit. We don't know how much.

• 58:05Vladimir Vapnik?

  It is unclear.

• 58:06Lex Fridman?

  It's unclear.

• 58:07Vladimir Vapnik?

  Yeah.

• 58:07Lex Fridman?

  It might very much.

• 58:08Vladimir Vapnik?

  But I would like to make a remark.

• 58:10Lex Fridman?

  Yes.

• 58:10Vladimir Vapnik?

  I start not from very primitive problem, make a challenge problem. I start with very general problem, with Plato. So you understand, and it comes from Plato to, to, to digit recognition.

• 58:30Lex Fridman?

  So you basically took Plato and the, the world of forms and ideas and mapped and projected into the clearest, simplest formulation of that big world.

• 58:43Vladimir Vapnik?

  You know, I, I would say that I did not understand Plato until recently. And until I consider weak convergence and then predicate and then, oh, this is what Plato told.

• 59:02Lex Fridman?

  Can you linger on that? Like, why? How do you think about this world of ideas and world of things in Plato?

• 59:09Vladimir Vapnik?

  No, it is metaphor. It is.

• 59:11Lex Fridman?

  It's a metaphor, for sure. It's a compelling, it's a poetic and a beautiful metaphor. But what, can you?

• 59:17Vladimir Vapnik?

  But it is a way how you, you, you should try to understand how a talk ideas in the world. So from my point of view, it is very clear, but it is lying. All the time, people looking for that. Say Plato's and Hegel. Whatever reasonable it exists, whatever exists, it is reasonable. I don't know what he have in mind reasonable.

• 59:47Lex Fridman?

  Right, there's philosophers again.

• 59:48Vladimir Vapnik?

  No, no, no, no, no, no. It is, it is next stop of Wigner. That mathematics understand something of reality. It is the same Plato line. And then it comes suddenly to Vladimir Propp. Look, thirty one ideas, thirty one units, and this creates everything.

• 1:00:11Lex Fridman?

  There's abstractions, ideas that represent our world. And we should always try to reach into that.

• 1:00:20Vladimir Vapnik?

  Yeah, but, but you should make a projection on reality. But understanding is, it is abstract ideas. You have in your mind several abstract ideas which you can apply to reality.

• 1:00:34Lex Fridman?

  And reality, in this case, sort of if you look at machine learning, is data.

• 1:00:38Vladimir Vapnik?

  It's example of data.

• 1:00:39Lex Fridman?

  Data. Okay, let me, let me put, put this on you because I'm an emotional creature. I'm not a mathematical creature like you. I find compelling the idea, forget the, the space, the sea of functions. There's also a sea of data in the world. And I find compelling that there might be, like you said, teacher. Small examples of data that are most useful for discovering good, whether it's predicates or good functions, that the selection of data may be a powerful journey, a useful mechanism. You know, coming up with a mechanism for selecting good data might be useful, too.

• 1:01:24Lex Fridman?

  Do you find this idea of finding the right data set interesting at all? Or do you kind of take the data set as a given?

• 1:01:34Vladimir Vapnik?

  I think that it is, you know, my scheme is very simple. You have huge set of functions.

• 1:01:43Vladimir Vapnik?

  If you will apply, and you have not too many data. If you pick up function which describes this data, you will do not very well.

• 1:01:58Lex Fridman?

  Like randomly pick?

• 1:01:59Vladimir Vapnik?

  Yeah, you will overfit, yeah, it will be overfitting. So you should decrease set of function from which you're picking up one. So you should go somehow to admissible set of function. And this, what about weak convergence? So but from another point of view, to to make

• 1:02:28Vladimir Vapnik?

  admissible set of function, you need just a data, just function which you will take in inner product, which you will measure

• 1:02:40Vladimir Vapnik?

  property of your function.

• 1:02:44Vladimir Vapnik?

  And that is how it works.

• 1:02:48Lex Fridman?

  No, I get it, I get it, I understand it, but do you, the reality is?

• 1:02:52Vladimir Vapnik?

  But let's discuss, let's think about examples. You have huge set of function, and you have several examples. If you're just trying to keep, take function which satisfies these examples.

• 1:03:10Vladimir Vapnik?

  You still will overfit. You need decrease, you need admissible set of function.

• 1:03:16Lex Fridman?

  Absolutely, but what say you have more data than functions?

• 1:03:23Lex Fridman?

  So sort of consider the, I mean, maybe not more data than functions, because that's

• 1:03:27Vladimir Vapnik?

  No, it's impossible.

• 1:03:28Lex Fridman?

  Impossible, but what, I was trying to be poetic for a second. I mean, you have a huge amount of data, a huge amount of examples.

• 1:03:36Vladimir Vapnik?

  But amount of function can be even

• 1:03:39Lex Fridman?

  Bigger, I understand.

• 1:03:41Vladimir Vapnik?

  Everything is

• 1:03:42Lex Fridman?

  There's always, there's always a bigger boat.

• 1:03:44Vladimir Vapnik?

  Paul Hilbert space.

• 1:03:46Lex Fridman?

  I got you, but okay. But you don't, you don't find the world of data to be an interesting optimization space? Like the, the optimization should be in the space of functions?

• 1:04:01Vladimir Vapnik?

  Creating admissible set of function.

• 1:04:04Lex Fridman?

  Admissible set of function.

• 1:04:05Vladimir Vapnik?

  No, you know, even from the classical business theory,

• 1:04:11Vladimir Vapnik?

  from structured risk minimization, you should, you should organize function in the way that they will be useful for you.

• 1:04:23Lex Fridman?

  Right.

• 1:04:24Vladimir Vapnik?

  And that is

• 1:04:26Lex Fridman?

  Yeah, but the, the way you're thinking about useful is you're given a small

• 1:04:33Vladimir Vapnik?

  Useful small, small set of function which contain function by looking for.

• 1:04:38Lex Fridman?

  Yeah, but as looking for based on the empirical set of small examples.

• 1:04:44Vladimir Vapnik?

  Yeah, but that is another story, I don't touch it. Because I, I believe, I believe that the small examples is not too small. Say sixty per class, that law of large numbers works. I don't need uniform law. The story is that in statistics, there are two law. Law of large numbers and uniform law of large numbers. So I want to be in situation where I use law of large numbers, but not uniform law of large numbers.

• 1:05:15Lex Fridman?

  Right, so sixty is law of large, it's large enough?

• 1:05:18Vladimir Vapnik?

  I hope, no, it still need some evaluation, some bounds, so. But idea is the following that if you trust that

• 1:05:32Vladimir Vapnik?

  Say, this average gives you something close to expectation. So you can talk about that, about this predicate. And that is basis of human intelligence.

• 1:05:47Lex Fridman?

  Good predicates is the, the discovery of good predicates is the basis of.

• 1:05:51Vladimir Vapnik?

  No, no, it is discovery of your, of your understanding world, of your methodology of a, of understanding world.

• 1:06:02Vladimir Vapnik?

  Because you have several functions which you will apply to reality.

• 1:06:08Lex Fridman?

  Can, can you say that again? So, uh, you're.

• 1:06:11Vladimir Vapnik?

  You have several functions predicate, but they abstract.

• 1:06:17Lex Fridman?

  Yes.

• 1:06:17Vladimir Vapnik?

  Then you will apply them to reality, to your data. And you will create in this way predicate, which is useful for your task.

• 1:06:28Vladimir Vapnik?

  But predicate are not related specifically to your task, to this your task. It is abstract functions which being applied, applied to.

• 1:06:40Lex Fridman?

  Many tasks that you might be interested in?

• 1:06:42Vladimir Vapnik?

  It might be many tasks, I don't know.

• 1:06:44Lex Fridman?

  Or.

• 1:06:45Vladimir Vapnik?

  Different tasks.

• 1:06:46Lex Fridman?

  Well, they should be many tasks, right?

• 1:06:48Vladimir Vapnik?

  Yeah, I believe like, like in prop case. It was for fairy tales, but this happened everywhere.

• 1:06:57Lex Fridman?

  Okay, so we talked about images a little bit, but can we talk about Noam Chomsky for a second?

• 1:07:04Vladimir Vapnik?

  Yeah. No, I, I, I believe I, I don't know him very well.

• 1:07:11Lex Fridman?

  Personally, well.

• 1:07:12Vladimir Vapnik?

  Not personally, I don't know.

• 1:07:14Lex Fridman?

  His ideas.

• 1:07:14Vladimir Vapnik?

  His ideas.

• 1:07:15Lex Fridman?

  Well, let me just say, do you think language, human language, is essential to expressing ideas as Noam Chomsky believes? So like, language is at the core of our formation of predicates. The human language.

• 1:07:31Vladimir Vapnik?

  For me, language and all the story of language is very complicated. I don't understand this and I am not, I thought about.

• 1:07:42Lex Fridman?

  Nobody does.

• 1:07:43Vladimir Vapnik?

  I'm not ready to work on that because it's so huge. It is not for me and I believe not for our century.

• 1:07:52Lex Fridman?

  The 21st century.

• 1:07:54Vladimir Vapnik?

  Not for 21st century.

• 1:07:56Lex Fridman?

  So.

• 1:07:56Vladimir Vapnik?

  You should learn something, a lot of stuff from simple task like digit recognition.

• 1:08:02Lex Fridman?

  So you think, okay, you think digital recognition, 2D image. What, how would you more abstractly define it, digit recognition? It's 2D image, symbol recognition, essentially? I mean, I like, I'm trying to get a sense, sort of thinking about it now, having worked with MNIST forever. How, how small of a subset is this of the general vision recognition problem? And the general intelligence problem? Is it?

• 1:08:41Lex Fridman?

  Yeah. Is it a giant subset, is it not? And how far away is language?

• 1:08:47Vladimir Vapnik?

  You know, let me refer to Einstein. Take the simplest problem, as simple as possible, but not simpler. And this is challenge, is simple problem.

• 1:09:01Vladimir Vapnik?

  But it's simple by idea, but not simple to, to get it. When you will do this, you will find some predicate, which helps it to do.

• 1:09:14Lex Fridman?

  Well, yeah, I mean, with Einstein, you can, you look, you look at general relativity. But that doesn't help you with quantum mechanics.

• 1:09:24Vladimir Vapnik?

  That's another story, you, you don't have any universal instrument.

• 1:09:28Lex Fridman?

  Yes, so I'm, I'm trying to wonder if, which space we're in, whether the, whether handwritten recognition is like general relativity. And then language is like quantum mechanics, so you're still gonna have to do a lot of mess to, to, to universalize it, but.

• 1:09:48Lex Fridman?

  I'm trying to see one, so what's your intuition? Why handwritten recognition is easier than language? Just, I think a lot of people would agree with that. But if you could elucidate sort of the intuition of why?

• 1:10:08Vladimir Vapnik?

  I don't know. No, I don't think in this direction. I just think in directions that this is problem. Which if we will solve it well, we will create some abstract understanding of images. Maybe not all images. I would like to talk to guys who doing in real images in Columbia University.

• 1:10:43Lex Fridman?

  What kind of images? Unreal?

• 1:10:45Vladimir Vapnik?

  Real images.

• 1:10:46Lex Fridman?

  Real images?

• 1:10:47Vladimir Vapnik?

  Yeah. What their idea is, there a predicate, what can be predicate? I still symmetry will play role in real life images, in any real life images, 2D images. Let's talk about 2D images. Because that's what we know. And neural network was created for 2D images.

• 1:11:12Lex Fridman?

  So the people I know in vision science, for example, the people study human vision.

• 1:11:17Vladimir Vapnik?

  Yeah.

• 1:11:18Lex Fridman?

  That they usually go to the world of symbols and like handwritten recognition, but not really it's other kinds of symbols to study our visual perception system. As far as I know, not much predicate type of thinking is understood about our vision system.

• 1:11:34Vladimir Vapnik?

  They did not think in this direction.

• 1:11:36Lex Fridman?

  They don't, yeah. They, but how do you even begin to think in that direction?

• 1:11:40Vladimir Vapnik?

  That's as.

• 1:11:41Lex Fridman?

  There's so much going on.

• 1:11:42Vladimir Vapnik?

  I would like to discuss with them.

• 1:11:43Lex Fridman?

  Yeah.

• 1:11:44Vladimir Vapnik?

  Because if we will be able to show that it is what working.

• 1:11:52Vladimir Vapnik?

  And theoretical scheme, it's not so bad.

• 1:11:57Lex Fridman?

  So the unfortunate, so if we compare to language, language has like letters, finite set of letters and a finite set of ways you can put together those letters. So it feels more amenable to kind of analysis. With natural images, there is so many pixels.

• 1:12:15Vladimir Vapnik?

  No, no, no, letter, language is much, much more complicated. It's involved a lot of different stuff. It's not just understanding of very simple class of tasks. I would like to see lists of task where language involved.

• 1:12:36Lex Fridman?

  Yes. So there's a, there's a lot of nice benchmarks now in natural language processing from the very trivial, like understanding the elements of a sentence to question answering to more, much more complicated where you talk about open domain dialogue. The natural question is with handwritten recognition is really the first step

• 1:12:58Vladimir Vapnik?

  Yeah.

• 1:12:58Lex Fridman?

  of understanding visual information.

• 1:13:01Vladimir Vapnik?

  Right. But not, but, but even our records show that we go in wrong direction because we need 60,000 digits.

• 1:13:13Lex Fridman?

  So even this first step, so forget about talking about the full journey. This first step should be taken in the right direction.

• 1:13:20Vladimir Vapnik?

  No, no, in wrong direction because 60,000 is unacceptable.

• 1:13:24Lex Fridman?

  No, I'm saying it should be taken in the, in the right direction because 60,000 is not acceptable.

• 1:13:30Vladimir Vapnik?

  It is, you can talk, it's great. We have half percent of error.

• 1:13:35Lex Fridman?

  And hopefully the step from doing hand recognition, using very few examples, the step towards what babies do when they crawl and understand their physical environment.

• 1:13:46Vladimir Vapnik?

  What baby love.

• 1:13:47Lex Fridman?

  I know you don't know about babies.

• 1:13:48Vladimir Vapnik?

  If you will do from very small examples.

• 1:13:52Lex Fridman?

  Yeah.

• 1:13:53Vladimir Vapnik?

  You will find principles.

• 1:13:55Lex Fridman?

  That will find the babies.

• 1:13:57Vladimir Vapnik?

  From what we're using now.

• 1:14:01Vladimir Vapnik?

  And theoretically, it's more or less clear. That means that you you will use weak convergence, not just strong convergence.

• 1:14:11Lex Fridman?

  Do you think these principles are will naturally be human interpretable?

• 1:14:18Vladimir Vapnik?

  Oh, yeah.

• 1:14:19Lex Fridman?

  So like when we will be able to explain them and have a nice presentation to show what those principles are? Or are they very going to be very kind of abstract kinds of functions?

• 1:14:31Vladimir Vapnik?

  For example, I talked yesterday about symmetry.

• 1:14:34Lex Fridman?

  Yes.

• 1:14:35Vladimir Vapnik?

  And they gave very simple examples, the same will be like that.

• 1:14:39Lex Fridman?

  You gave like a predicate of a basic for.

• 1:14:41Vladimir Vapnik?

  For symmetries.

• 1:14:42Lex Fridman?

  Yes, for different symmetries and you have for.

• 1:14:46Vladimir Vapnik?

  A degree of symmetry, that is important, not just symmetry. Exist and doesn't exist, the degree of symmetry.

• 1:14:55Lex Fridman?

  Yeah, for handwritten recognition.

• 1:14:58Vladimir Vapnik?

  No, it's not for handwritten, it's for any images, but I would like apply to handwritten.

• 1:15:04Lex Fridman?

  Right, it's in theory, it's more general, okay, okay.

• 1:15:12Lex Fridman?

  So a lot of things we've been talking about falls, we've been talking about philosophy a little bit, but also about mathematics and statistics. A lot of it falls into this idea, a universal idea of statistical theory of learning. What is the most beautiful and sort of powerful or essential idea you've come across, even just for yourself personally, in in the world of statistics or statistic theory of learning?

• 1:15:42Vladimir Vapnik?

  Probably uniform convergence, which we did with Alexey Chervonienkis.

• 1:15:50Lex Fridman?

  Can you describe universal convergence?

• 1:15:53Vladimir Vapnik?

  You have law of large law of large numbers.

• 1:15:57Vladimir Vapnik?

  So for any function, expectation of function, average of function convergent expectation. But if you have set of functions, for any function, it is true. But it should converge simultaneously for all set of functions.

• 1:16:16Vladimir Vapnik?

  And for for learning, you need uniform convergence, just convergence is not enough.

• 1:16:28Vladimir Vapnik?

  Because when you pick up one which gives minimum, you can pick up

• 1:16:37Vladimir Vapnik?

  one function which does not converge and it will give you the best answer for this function.

• 1:16:48Vladimir Vapnik?

  So you need a uniform convergence to guarantee learning. So learning does not really on trivial law, large numbers, really on universal. But

• 1:17:02Vladimir Vapnik?

  idea of weak convergence exists in statistics for a long time. But

• 1:17:12Vladimir Vapnik?

  it is interesting that

• 1:17:15Vladimir Vapnik?

  as I, I, I think about myself, how stupid I was fifty years. I did not see weak convergence. I work only on strong convergence. But now I think that most powerful is weak convergence. Because it makes admissible set of functions. And even in all, in proverts, when people try to understand recognition about duck law, looks like a duck and so on, they use weak convergence. People in language, they understand this. But when we're trying to create artificial intelligence,

• 1:17:59Vladimir Vapnik?

  we won't event in different way.

• 1:18:03Vladimir Vapnik?

  We just consider strong convergence.

• 1:18:07Lex Fridman?

  So reducing the set of admissible functions, you think there should be effort put into understanding the properties of weak convergence?

• 1:18:18Vladimir Vapnik?

  You know, in classical mathematics, in Gilbert space, there are only two way, two form of convergence, strong and weak.

• 1:18:31Vladimir Vapnik?

  Now we can use both. That means that we did everything.

• 1:18:38Vladimir Vapnik?

  And it so happened, then when we use Hilbert space, which is very rich space, space of continuous functions,

• 1:18:51Vladimir Vapnik?

  each has an integral in square. So we can apply weak and strong convergence for learning and have closed form solution. So for computation, it's simple. For me, it is sign that it is right way. Because you don't need any heuristic here, yes, whatever you want.

• 1:19:16Vladimir Vapnik?

  But now, the only what left, it is concept of what is predicate.

• 1:19:21Lex Fridman?

  What predicate?

• 1:19:22Vladimir Vapnik?

  But it is not statistics.

• 1:19:24Lex Fridman?

  By the way, I like the fact that you think that heuristics are a mess that should be removed from the system. So closed form solution is the ultimate.

• 1:19:35Vladimir Vapnik?

  No, it so happened, then when you're using

• 1:19:39Vladimir Vapnik?

  right instrument, you have closed form solution.

• 1:19:45Lex Fridman?

  Do you, do you think intelligence, human level intelligence, when we create it, will, um,

• 1:19:54Lex Fridman?

  will have something like a closed form solution?

• 1:19:59Vladimir Vapnik?

  You know, I now I'm looking on bounds, which I gave bounds for convergence.

• 1:20:08Vladimir Vapnik?

  And when I looking for bounds, I thinking, what is the most appropriate kernel for this bound would be?

• 1:20:19Vladimir Vapnik?

  So we know that in, say, all our businesses, we use radial basis function.

• 1:20:28Vladimir Vapnik?

  But looking on the bound, I think that I start to understand that maybe we need to make corrections to radial basis function to be closer.

• 1:20:41Vladimir Vapnik?

  To work better for this balance. So I'm again trying to understand what type of kernel have best approximation, no, no approximation, best fit to this ball.

• 1:21:00Lex Fridman?

  Sure. So there's a, there's a lot of interesting work that could be done in discovering better functions and radio basis functions for, for kind of bounds you find.

• 1:21:10Vladimir Vapnik?

  It still comes from you, you're, you're looking to mass and trying to understand what.

• 1:21:17Lex Fridman?

  From your own mind, looking at the, I don't know.

• 1:21:20Vladimir Vapnik?

  Then I trying to understand what, what will be good for that.

• 1:21:28Lex Fridman?

  Yeah, but to me, there's still a beauty. Again, maybe I'm a descendant of Alan Turing to heuristics. To me, ultimately, intelligence will be a mess of heuristics. And that's the engineering answer, I guess.

• 1:21:43Vladimir Vapnik?

  Absolutely. When, when you're doing, say, self-driving cars, the great guy who will do that, it does not matter what theory behind that. Who has a better feeling how to apply it. But by the way, it is the same story about predicates. Because you cannot create rule for, situation is much more than you have rule for them. But maybe you can have more abstract rule than it will be less this rule.

• 1:22:25Vladimir Vapnik?

  It is the same story about ideas and, and ideas applied to specific cases.

• 1:22:33Lex Fridman?

  But still you should.

• 1:22:34Vladimir Vapnik?

  You cannot avoid this.

• 1:22:35Lex Fridman?

  Yes, of course, but you, you should still reach for the ideas to understand the science. Let me kind of ask, do you think neural networks or functions?

• 1:22:48Lex Fridman?

  Can be made to reason? Sort of what do you think? We've been talking about intelligence, but this idea of reasoning. There's a, there's an element of sequentially disassembling, interpreting the, the images. So when, when you think of handwritten recognition, we kind of think that there'll be a single, there's an input and output. There's not a recurrence.

• 1:23:17Vladimir Vapnik?

  Yeah.

• 1:23:18Lex Fridman?

  What do you think about sort of the idea of recurrence of going back to memory and thinking through this sort of sequentially mangling the different representations over and over until you arrive at a conclusion? Or is ultimately all that can be wrapped up into a function?

• 1:23:40Vladimir Vapnik?

  You're, you're suggesting that let us use this type of algorithm. When I starting thinking, I first of all, starting to understand what I want. Can I write down what I want? And then I trying to formalize. And when I do that, I think I have to solve this problem. And till now, I did not see situation where you

• 1:24:19Lex Fridman?

  You need recurrence.

• 1:24:20Vladimir Vapnik?

  Recurrence.

• 1:24:21Lex Fridman?

  But do you observe human beings?

• 1:24:24Vladimir Vapnik?

  Yeah.

• 1:24:25Lex Fridman?

  Do you try to? It's the imitation question, right? It seems that human beings reason this kind of sequentially sort of, does that inspire in you a thought that we need to add that into our?

• 1:24:44Lex Fridman?

  Intelligent systems?

• 1:24:47Lex Fridman?

  You're saying, okay, I mean, you've kind of answered saying, until now, I haven't seen a need for it. And so because of that, you don't see a reason to think about it?

• 1:24:58Vladimir Vapnik?

  You know, most of things I don't understand in reasoning in human. It is for me too complicated.

• 1:25:09Vladimir Vapnik?

  For me, the most difficult part is to ask questions, to good questions. How it works, how people asking questions.

• 1:25:25Vladimir Vapnik?

  I don't know this.

• 1:25:28Lex Fridman?

  You said that machine learning is not only about technical things, speaking of questions, but it's also about philosophy. So what role does philosophy play in machine learning? We talked about Plato, but generally thinking in this philosophical way.

• 1:25:48Lex Fridman?

  Does it have, how does philosophy and math fit together in your mind?

• 1:25:53Vladimir Vapnik?

  First ideas and then their implementation. It's like predicate, like say admissible set of functions. It comes together, everything, because

• 1:26:11Vladimir Vapnik?

  the first iteration of theory was done 50 years ago. It all got this is sorry. So everything's there. If you have data, you can, and your, and your set of function is not, has not, have not big capacity. So law of C dimension, you can do that. You can make structural risk minimization, control capacity.

• 1:26:38Vladimir Vapnik?

  But it was not able to make admissible set of function good. Now, when suddenly realize that we did not use another idea of convergence, which we can.

• 1:26:56Vladimir Vapnik?

  Everything comes together.

• 1:26:58Lex Fridman?

  But those are mathematical notions. Philosophy plays a role of simply saying that we should be swimming in the space of ideas.

• 1:27:08Vladimir Vapnik?

  Let's, let's talk what is philosophy? Philosophy means understanding of life.

• 1:27:15Vladimir Vapnik?

  So, understanding of life, say, people like Plato, they understand on very high abstract level of life. So, and whatever I doing, it's just implementation of my understanding of life. But every new step, it is very difficult. For example,

• 1:27:43Vladimir Vapnik?

  to find the idea that we need weak convergence

• 1:27:52Vladimir Vapnik?

  was not simple for me.

• 1:27:57Lex Fridman?

  So that required thinking about life a little bit.

• 1:28:01Lex Fridman?

  Hard to, hard, hard to trace, but there was some thought process.

• 1:28:06Vladimir Vapnik?

  You know, I working, I thinking about the same problem for fifty years or more.

• 1:28:14Vladimir Vapnik?

  And again, and again, and again.

• 1:28:17Vladimir Vapnik?

  I trying to be honest, and that is very important, not to be very enthusiastic.

• 1:28:21Lex Fridman?

  Yeah.

• 1:28:22Vladimir Vapnik?

  But concentrate on whatever we was not able to achieve, for example, yeah.

• 1:28:29Vladimir Vapnik?

  And understand why. And now I understand that because I believe in mass, I believe that

• 1:28:38Vladimir Vapnik?

  in Wigner's idea. But now, when I see that there are only two way of convergence. And we using both. That means that we must do as well as people doing.

• 1:28:57Vladimir Vapnik?

  But now, exactly in philosophy and what we know about predicate, what we, how we understand life, can we describe as a predicate?

• 1:29:08Vladimir Vapnik?

  I thought about that.

• 1:29:11Vladimir Vapnik?

  And that is more or less obvious, level of symmetry.

• 1:29:17Vladimir Vapnik?

  But next, I have a feeling it's something about structures.

• 1:29:26Vladimir Vapnik?

  But I don't know how to formulate, how to measure measure of structure and all this stuff. And guy who will solve this challenge problem, then when we will looking how he did it,

• 1:29:44Vladimir Vapnik?

  probably just only symmetry is not enough.

• 1:29:48Lex Fridman?

  But something like symmetry will be there.

• 1:29:51Vladimir Vapnik?

  Absolutely, symmetry will be there, level of symmetry will be there. And level of symmetry, antisymmetry, diagonal, vertical. I even don't know how you can use in different direction idea of symmetry, it's very general. But it will be there. I think that people very sensitive to idea of symmetry. But there are several ideas like symmetry.

• 1:30:21Vladimir Vapnik?

  As I would like to learn. But you cannot learn just thinking about that. You should do challenging problems and then analyze them, why it was, we was able to solve them. And then we will see. Very simple things, it's not easy to find.

• 1:30:44Vladimir Vapnik?

  Even we talking about this every time.

• 1:30:47Lex Fridman?

  Yeah.

• 1:30:47Vladimir Vapnik?

  About you, you, I was surprised, I tried to understand. Is people describe in language, strong convergence, mechanism for learning. I did not see, I don't know. But weak convergence, this dark story and story like that, when you will explain to kid, you will use weak convergence argument. It looks like it does like it, that. But when you try to formalize, you just ignoring this. Why? Why fifty years from start of machine learning?

• 1:31:27Lex Fridman?

  And that's the role of philosophy.

• 1:31:28Vladimir Vapnik?

  I think, I think that maybe, I don't know.

• 1:31:35Vladimir Vapnik?

  Maybe this is theory also. We should blame for that because empirical risk minimization and all this stuff. If you read now textbooks, they just about bound about empirical risk minimization. They don't looking for another problem like admissible set.

• 1:31:58Lex Fridman?

  But on the topic of life,

• 1:32:02Lex Fridman?

  perhaps we, you could talk in Russian for a little bit. What's your favorite memory from childhood? Какая ваша любимая память с детства?

• 1:32:13Vladimir Vapnik?

  Music.

• 1:32:16Lex Fridman?

  How about, can you try to answer in Russian?

• 1:32:19Vladimir Vapnik?

  Музыка. It was very beautiful when.

• 1:32:25Lex Fridman?

  Какая музыка?

• 1:32:26Vladimir Vapnik?

  Classical music.

• 1:32:28Lex Fridman?

  Какую любимую?

• 1:32:30Vladimir Vapnik?

  Two different composers. First, it was Vivaldi, I was surprised that it's possible.

• 1:32:40Vladimir Vapnik?

  And then when I understood Bach, I was completely shocked.

• 1:32:46Vladimir Vapnik?

  By the way, I think there are predicates of the structure in Bach.

• 1:32:53Lex Fridman?

  In Bach, of course.

• 1:32:54Vladimir Vapnik?

  Because, well, it just feels like everything is structured. And I don't think that different elements of life are strongly divided in the sense of predicates, everywhere, structure in the poems, structure in human relationships, structure. But to find these high-level predicates, it's...

• 1:33:22Lex Fridman?

  In Bach and in life, everything is united.

• 1:33:28Lex Fridman?

  Now that we're talking about Bach,

• 1:33:31Lex Fridman?

  let's switch back to English, because I like Beethoven and Chopin, so.

• 1:33:35Vladimir Vapnik?

  Chopin, it's another amusing story.

• 1:33:38Lex Fridman?

  But Bach, if we talk about predicates, Bach probably has the most sort of well-defined predicates in the life.

• 1:33:48Vladimir Vapnik?

  You know, it is very interesting to read what critics writing about Bach, which words they using, they trying to describe predicates.

• 1:34:00Lex Fridman?

  Yeah.

• 1:34:00Vladimir Vapnik?

  And then Chopin, it is very different vocabulary, very different predicate. And I think that if you will make collection of that,

• 1:34:19Vladimir Vapnik?

  so maybe from this you can describe predicate for digit recognition.

• 1:34:24Lex Fridman?

  Well, from Bach and Chopin.

• 1:34:27Vladimir Vapnik?

  No, no, no, not from Bach and Chopin.

• 1:34:29Lex Fridman?

  From the critic interpretation of the music, yeah.

• 1:34:32Vladimir Vapnik?

  When they trying to explain you music, what they uses. They use, they describe high-level ideas of Plato's ideas, what behind this music.

• 1:34:45Lex Fridman?

  That's brilliant, so art is not self-explanatory in some sense. So you have to try to convert it into ideas.

• 1:34:56Vladimir Vapnik?

  It is explicitly. Problems when you go from ideas to the representation. It is easy way, but when you're trying to go back, it is you'll post problems. But nevertheless, I believe that when you're looking from that, even from art, you will be able to find predicate for digit recognition.

• 1:35:20Lex Fridman?

  That's such a fascinating and powerful notion. Do you ponder your own mortality? Do you think about it? Do you fear it? Do you draw insight from it?

• 1:35:33Vladimir Vapnik?

  About mortality, oh, yeah.

• 1:35:38Lex Fridman?

  Are you afraid of death?

• 1:35:42Vladimir Vapnik?

  Not too much, not too much. It is pity that I will not be able to do something which I think I have a feeling to do that. For example, I will be very happy to work with guys, theoretician from music, to write this collection of description, what, how they describe music, how they use predicate. And from art as well, then take what is in common and try to understand predicate, which is absolute for everything.

• 1:36:25Lex Fridman?

  And then use that for visual recognition and see if there is a connection.

• 1:36:29Vladimir Vapnik?

  Yeah, exactly.

• 1:36:30Lex Fridman?

  Ah, there's still time, we've got time.

• 1:36:35Vladimir Vapnik?

  Yeah.

• 1:36:36Lex Fridman?

  We've got time.

• 1:36:36Vladimir Vapnik?

  It, it, it, it's take years and years and years.

• 1:36:41Lex Fridman?

  You think so?

• 1:36:41Vladimir Vapnik?

  Yeah, it's, it's a long way.

• 1:36:43Lex Fridman?

  Well, see, you've got the patient mathematician's mind. I think it could be done very quickly and very beautifully. I think it's a really elegant idea.

• 1:36:52Vladimir Vapnik?

  Yeah, but also.

• 1:36:54Lex Fridman?

  Some of many.

• 1:36:54Vladimir Vapnik?

  Yeah, you know, the, the most time it is not to make this collection, to understand what is the common to think about that once again and again and again.

• 1:37:06Lex Fridman?

  Again and again and again, but I think sometimes, especially just when you say this idea now, even just putting together the collection and looking at the different sets of data, language, trying to interpret music, criticize music and images. I think there will be sparks of ideas that will come. Of course, again and again, you'll come up with better ideas, but even just that notion is a beautiful notion.

• 1:37:34Vladimir Vapnik?

  I even have some example.

• 1:37:38Vladimir Vapnik?

  So I have friend,

• 1:37:42Vladimir Vapnik?

  who was specialist in Russian poetry. She is professor of Russian poetry. He did not write poems, but she know a lot of stuff. She make book, several books, and one of them is collection of Russian poetry. She have images of Russian poetry. She collect all images of Russian poetry. And I ask you to do following.

• 1:38:22Vladimir Vapnik?

  You have NIPS, digit recognition. And you get hundred digits, or maybe less than hundred. I don't remember, maybe fifty digits. And try from poetical point of view, describe every image which she see, using only words of images of Russian poetry. And she did it.

• 1:38:51Vladimir Vapnik?

  And then we tried to,

• 1:38:58Vladimir Vapnik?

  I call it learning using privileged information. I call it privileged information. You have on two languages. One language is just image of digit. And another language poetic description of this image. And this is privileged information.

• 1:39:19Vladimir Vapnik?

  And there is an algorithm. When you're working using privileged information, you're doing better, much better, so. So there's something there? Something there. And there is a, in NEC, she unfortunately died,

• 1:39:37Vladimir Vapnik?

  the collection of digits in poetic descriptions of these digits.

• 1:39:46Lex Fridman?

  There's some, something there in that poetic description.

• 1:39:49Vladimir Vapnik?

  But I think that there is an abstract ideas on the plateau level of ideas.

• 1:39:57Lex Fridman?

  Yeah, that they're there, that could be discovered, and music seems to be a good entry point.

• 1:40:01Vladimir Vapnik?

  But as soon as we start with this challenge problem.

• 1:40:07Lex Fridman?

  The challenge problem.

• 1:40:08Vladimir Vapnik?

  It immediately connected to, to, to all this stuff.

• 1:40:12Lex Fridman?

  Especially with your talk and this podcast, and I'll do whatever I can to advertise it. Such a clean, beautiful Einstein-like formulation of the challenge before us.

• 1:40:22Vladimir Vapnik?

  Right.

• 1:40:22Lex Fridman?

  Let me ask another absurd question. We talked about mortality. We talked about philosophy of life. What do you think is the meaning of life? What's the predicate for mysterious existence here on earth?

• 1:40:46Vladimir Vapnik?

  I don't know.

• 1:40:50Vladimir Vapnik?

  It's very interesting how we have in Russia, I don't know, you know, the guy Strugatsky,

• 1:41:00Vladimir Vapnik?

  they writing pictures, they thinking about who, what's going on.

• 1:41:08Vladimir Vapnik?

  And they have idea

• 1:41:12Vladimir Vapnik?

  that there are, that developing two type of people.

• 1:41:20Vladimir Vapnik?

  Common people and very smart people, they just started.

• 1:41:24Vladimir Vapnik?

  And these two branches of people will go in different direction very soon. So that's what they thinking about that.

• 1:41:35Lex Fridman?

  So the purpose of life is to create two, two paths.

• 1:41:40Vladimir Vapnik?

  Two paths.

• 1:41:41Lex Fridman?

  Of human societies.

• 1:41:42Vladimir Vapnik?

  Yes, simple people and more complicated.

• 1:41:46Lex Fridman?

  Which do you like best? The simple people or the complicated ones?

• 1:41:51Vladimir Vapnik?

  I don't know, that he is just his fantasy, but you know, every week we have guy who is just.

• 1:42:03Vladimir Vapnik?

  Writer and also theoretic of literature. And he explained how he understand literature and human relationship, how he see life. And I understood that I'm just small kids comparing to him. He's very smart guy in understanding life. He knows this predicate, he knows big blocks of life. I am used every time when I listening to him. And he just talking about literature.

• 1:42:44Vladimir Vapnik?

  And I think that I was surprised. So the managers in big companies, most of them are guys who study English language and English literature. So why? Because they understand life. They understand models. And among them, maybe many talented critics, which just analyzing this.

• 1:43:23Vladimir Vapnik?

  And this is big science like Problite, this is this blogs. But they're very smart.

• 1:43:34Lex Fridman?

  It amazes me that you are and continue to be humbled by the brilliance of others.

• 1:43:39Vladimir Vapnik?

  I'm very modest about myself. I see so smart guys around.

• 1:43:45Lex Fridman?

  Well, let me be immodest for you. You're one of the greatest mathematicians, statisticians of our time. It's truly an honor. Thank you for talking again. And let's talk.

• 1:43:58Vladimir Vapnik?

  It is not.

• 1:43:59Lex Fridman?

  Yeah.

• 1:43:59Vladimir Vapnik?

  Yes, I know my limits.

• 1:44:02Lex Fridman?

  Let's, let's talk again when your challenge is taken on and solved by a grad student, especially.

• 1:44:09Vladimir Vapnik?

  Let's talk again.

• 1:44:10Lex Fridman?

  When they use.

• 1:44:12Vladimir Vapnik?

  It's happened.

• 1:44:14Lex Fridman?

  Maybe music will be involved. Latimer, thank you so much.

• 1:44:16Vladimir Vapnik?

  Thank you very much.

• 1:44:19Lex Fridman?

  Thanks for listening to this conversation with Vladimir Vapnik, and thank you to our presenting sponsor, Cash App. Download it, use code LEXPODCAST, you'll get $10, and $10 will go to FIRST, an organization that inspires and educates young minds to become science and technology innovators of tomorrow. If you enjoy this podcast, subscribe on YouTube, give us five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at @LexFriedman. And now, let me leave you with some words from Vladimir Vapnik. When solving a problem of interest, do not solve a more general problem as an intermediate step. Thank you for listening. I hope to see you next time.