Gilbert Strang: Linear Algebra, Deep Learning, Teaching, and MIT OpenCourseWare

Gilbert Strang is a professor of mathematics at MIT and perhaps one of the most famous and impactful teachers of math in the world. His MIT OpenCourseWare lectures on linear algebra have been viewed millions of times. This conversation is part of the Artificial Intelligence podcast. If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter https://twitter.com/lexfridman, LinkedIn https://www.linkedin.com/in/lexfridman/, Facebook https://www.facebook.com/lexfridman, Medium https://medium.com/@lexfridman, or YouTube https://www.youtube.com/lexfridman where you can watch the video versions of these conversations. If you enjoy the podcast, please rate it 5 stars on Apple Podcasts https://podcasts.apple.com/us/podcast/artificial-intelligence/id1434243584 or support it on Patreon https://www.patreon.com/lexfridman. This episode is presented by Cash App. Download it, use code LexPodcast. And it is supported by ZipRecruiter. Try it: http://ziprecruiter.com/lexpod Here’s the outline of the episode. On some podcast players you should be able to click the timestamp to jump to that time. 00:00 – Introduction 03:45 – Math rockstar 05:10 – MIT OpenCourseWare 07:29 – Four Fundamental Subspaces of Linear Algebra 13:11 – Linear Algebra vs Calculus 15:03 – Singular value decomposition 19:47 – Why people like math 23:38 – Teaching by example 25:04 – Andrew Yang 26:46 – Society for Industrial and Applied Mathematics 29:21 – Deep learning 37:28 – Theory vs application 38:54 – Open problems in mathematics 39:00 – Linear algebra as a subfield of mathematics 41:52 – Favorite matrix 46:19 – Advice for students on their journey through math 47:37 – Looking back

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation with Gilbert Strang. He's a professor of mathematics at MIT and perhaps one of the most famous and impactful teachers of math in the world. His MIT Open Courseware lectures on linear algebra have been viewed millions of times. As an undergraduate student, I was one of those millions of students. There's something inspiring about the way he teaches that is at once calm, simple, and yet full of passion for the elegance inherent to mathematics. I remember doing the exercises in his book Introduction to Linear Algebra and slowly realizing that the world of matrices, of vector spaces, of determinants and eigenvalues, of geometric transformations and matrix decompositions reveal a set of powerful tools in the toolbox

• 0:45Lex Fridman?

  of artificial intelligence. From signals to images, from numerical optimization to robotics, computer vision, deep learning, computer graphics, and everywhere outside AI, including, of course, a quantum mechanical study of our universe. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at LexFriedman, spelled F-R-I-D-M-A-N. This podcast is supported by ZipRecruiter. Hiring great people is hard and, to me, is the most important element of a successful, mission-driven team. I've been fortunate to be a part of and to lead several great engineering teams.

• 1:30Lex Fridman?

  The hiring I've done in the past was mostly through tools that we built ourselves, but reinventing the wheel was painful. ZipRecruiter is a tool that's already available for you. It seeks to make hiring simple, fast, and smart. For example, codable co-founder Gretchen Hebner used ZipRecruiter to find a new game artist to join her education tech company. By using ZipRecruiter's screening questions to filter candidates, Gretchen found it easier to focus on the best candidates and, finally, hiring the perfect person for the role in less than two weeks from start to finish. ZipRecruiter, the smartest way to hire. See why ZipRecruiter is effective for businesses of all sizes by signing up, as I did, for

• 2:13Lex Fridman?

  free at ziprecruiter.com/lexpod. That's ziprecruiter.com/lexpod.

• 2:22Lex Fridman?

  This show is presented by Cash App, the number one finance app in the App Store. I personally use Cash App to send money to friends, but you can also use it to buy, sell, and deposit Bitcoin. Most Bitcoin exchanges take days for a bank transfer to become investable. Through Cash App, it takes seconds. Cash App also has a new investing feature. You can buy fractions of a stock, which to me is a really interesting concept. So you can buy $1 worth, no matter what the stock price is. Brokerage services are provided by Cash App Investing, subsidiary of Square and member SIPC. I'm excited to be working with Cash App to support one of my favorite organizations that many of you may know and have benefited from called FIRST,

• 3:07Lex Fridman?

  best known for their FIRST Robotics and LEGO competitions. They educate and inspire hundreds of thousands of students in over 110 countries and have a perfect rating on Charity Navigator, which means that donated money is used to maximum effectiveness. When you get Cash App from the App Store or Google Play and use code LEGSPODCAST, you'll get $10 and Cash App will also donate $10 to FIRST, which again is an organization that I've personally seen inspire girls and boys to dream of engineering a better world. And now here's my conversation with Gilbert Strang.

• 4:03Lex Fridman?

  How does it feel to be one of the modern day rock stars of mathematics?

• 4:09Speaker 01

  I don't feel like a rock star. That's kind of crazy for an old math person. But it's true that the videos in linear algebra that I made way back in 2000, I think, have been watched a lot. And well, partly the importance of linear algebra, which I'm sure you'll ask me and give me a chance to say that. Linear algebra as a subject has just surged in importance. But also, it was a class that I taught a bunch of times. So I kind of got it organized and enjoyed doing it. It was just, the videos were just the class.

• 4:51Speaker 01

  So they're on OpenCourseWare and on YouTube and translated. That's fun.

• 4:56Lex Fridman?

  But there's something about that chalkboard and the simplicity of the way you explain the basic concepts in the beginning. You know, to be honest, when I went to undergrad.

• 5:07Speaker 01

  You didn't do linear algebra, probably.

• 5:09Lex Fridman?

  Of course, I did linear algebra. Yeah, yeah, yeah, of course. But I, before going through the course at my university, I was going through OpenCourseWare. You were my instructor for linear algebra.

• 5:21Speaker 01

  Right, yeah.

• 5:22Lex Fridman?

  And that, I mean, we're using your book. And I mean, that, the fact that there is thousands, you know, hundreds of thousands, millions of people that watched that video. I think that's, that's really powerful. So how do you think the idea of putting lectures online, what really MIT OpenCourseWare has innovated?

• 5:44Speaker 01

  That was a wonderful idea. You know, I think the story that I've heard is the committee was appointed by the president. President Vest at that time, a wonderful guy. And the idea of the committee was to figure out how MIT could make, be like other universities. Market the work we were doing. And then they didn't see a way. And after a weekend, and they had an inspiration, came back to President Vest and said, what if we just gave it away? And he decided that was okay, good idea. So.

• 6:25Lex Fridman?

  You know, that's a crazy idea. That's, if we think of a university as a thing that creates a product.

• 6:31Speaker 01

  Yes.

• 6:31Lex Fridman?

  Isn't knowledge.

• 6:33Speaker 01

  Right.

• 6:33Lex Fridman?

  The, you know, the kind of educational knowledge isn't the product and giving that away. Are you surprised?

• 6:42Lex Fridman?

  That it went through?

• 6:43Speaker 01

  The result that he did it. Well, knowing a little bit President Vest, it was like him, I think. And it was really the right idea, you know. MIT is a kind of, it's known for being high level, technical things. And this is the best way we can say, tell, we can show what MIT really is like. Because in my case, those 1806 videos are just teaching the class. They were there in 26100. They're kind of fun to look at. People write to me and say, oh, you've got a sense of humor. But I don't know where that comes through.

• 7:28Speaker 01

  Somehow, I'm friendly with the class. I like students. And linear algebra, we got to give this subject most of the credit. It really has come forward in importance in these years.

• 7:47Lex Fridman?

  So let's talk about linear algebra a little bit because it is such a, it's both a powerful and a beautiful subfield of mathematics. So what's your favorite specific topic in linear algebra? Or even math in general to give a lecture on, to convey, to tell a story, to teach students?

• 8:07Speaker 01

  Okay. Well, on the teaching side, so it's not deep mathematics at all. But I'm kind of proud of the idea of the four subspaces, the four fundamental subspaces, which are of course known before, long before my name for them.

• 8:30Lex Fridman?

  Can you go through them? Can you go through the four?

• 8:32Speaker 01

  Sure, I can. Yeah. So the first one to understand is, so the matrix, maybe I should say the matrix.

• 8:40Lex Fridman?

  What is the matrix?

• 8:41Speaker 01

  What's a matrix? Well, so we have like a rectangle of numbers. So it's got n columns, got a bunch of columns, and also got m rows, let's say. And the relation between, so of course, the columns and the rows, it's the same numbers. So there's got to be connections there, but they're not simple. The columns might be longer than the rows and they're all different. The numbers are mixed up. First space to think about is, take the columns. So those are vectors. Those are points in n dimensions.

• 9:19Lex Fridman?

  What's a vector?

• 9:19Speaker 01

  So a physicist would imagine a vector or might imagine a vector as a arrow, you know, in space or the point it ends at in space. For me, it's a column of numbers.

• 9:34Lex Fridman?

  You often think of, this is very interesting in terms of linear algebra, in terms of a vector. You think a little bit more abstract than the, how it's very commonly used perhaps. You think this arbitrary space, multi-dimensional space.

• 9:51Speaker 01

  Yeah, I'm right away, I'm in high dimensions and.

• 9:55Lex Fridman?

  Dreamland.

• 9:56Speaker 01

  Yeah, that's right. In the lecture, I try to, so if you think of two vectors in 10 dimensions, I'll do this in class and I'll readily admit that I have no good image in my mind of a vector, of a arrow in 10 dimensional space, but whatever. You can, you can add one bunch of 10 numbers to another bunch of 10 numbers. So you can add a vector to a vector and you can multiply a vector by three. And that's, if you know how to do those, you've got linear algebra.

• 10:31Lex Fridman?

  You know, 10 dimensions.

• 10:32Speaker 01

  Yeah.

• 10:33Lex Fridman?

  You know, there's this beautiful thing about math. If we look at string theory and all these theories, which are really fundamentally derived through math, but are very difficult to visualize. And how do you think about the things like a 10 dimensional vector that we can't really visualize?

• 10:50Speaker 01

  Yeah.

• 10:50Lex Fridman?

  Do you, and, and yet math reveals some beauty.

• 10:55Speaker 01

  Oh.

• 10:56Lex Fridman?

  Underlying.

• 10:57Speaker 01

  Yeah.

• 10:57Lex Fridman?

  Our world in that weird thing we can't visualize. How do you think about that difference?

• 11:03Speaker 01

  Well, probably, I'm not a very geometric person, so I'm probably thinking in three dimensions. And the beauty of linear algebra is that it goes on to 10 dimensions with no problem. I mean, that if you're just seeing what happens if you add two vectors in 3D, then you can add them in 10D. You're just adding the 10 components. So I can't say that I have a picture, but yet I try to push the class to think of a flat surface in 10 dimensions. So a plane in 10 dimensions. So that's one of the spaces. Take all the columns of the matrix.

• 11:47Speaker 01

  Take all their combinations. So much of this column, so much of this one. Then if you put all those together, you get some kind of a flat surface that I call a vector space. Space of vectors. And my imagination is just seeing like a piece of paper in 3D. But anyway, so that's one of the spaces. That's space number one, the column space of the matrix. And then there's the row space, which is, as I said, different, but came from the same numbers. So we got the column space, all combinations of the columns. And then we got the row space, all combinations of the rows.

• 12:31Speaker 01

  So those words are easy for me to say, and I can't really draw them on a blackboard, but I try with my thick chalk. Everybody likes that railroad chalk. And me too. I wouldn't use anything else now. And then the other two spaces are perpendicular to those. So like if you have a plane in 3D, just a plane is just a flat surface in 3D, then perpendicular to that plane would be a line. So that would be the null space. So we've got two. We've got a column space, a row space, and they're two perpendicular spaces. So those four fit together

• 13:15Speaker 01

  in a beautiful picture of a matrix. Yeah, yeah. It's sort of fundamental. It's not a difficult idea. It comes pretty early in 1806, and it's basic.

• 13:29Lex Fridman?

  Planes in these multi-dimensional spaces. How difficult of an idea is that to come to? Do you think if you look back in time?

• 13:40Speaker 01

  Yeah.

• 13:40Lex Fridman?

  I think mathematically it makes sense, but I don't know if it's intuitive for us to imagine. Just as we're talking about. It feels like calculus is easier to.

• 13:50Speaker 01

  I see.

• 13:51Lex Fridman?

  Intuit.

• 13:52Speaker 01

  Well, calculus, I have to admit calculus came earlier, earlier than linear algebra. So Newton and Leibniz were the great men to understand the key ideas of calculus.

• 14:04Speaker 01

  But linear algebra to me is like, okay, it's the starting point because it's all about flat things. Calculus has got, all the complications of calculus come from the curves, the bending, the curved surfaces. Linear algebra, the surfaces are all flat. Nothing bends in linear algebra. So it should have come first, but it didn't. And calculus also comes first in high school classes, in college class, it'll be freshman math, it'll be calculus. And then I say enough of it. Like, okay, get to the good stuff. And.

• 14:46Lex Fridman?

  Do you think linear algebra should come first?

• 14:48Speaker 01

  Well, it really, I'm okay with it not coming first, but it should. Yeah, it should. It's simpler.

• 14:56Lex Fridman?

  Because everything is flat.

• 14:57Speaker 01

  Yeah, everything's flat. Well, of course, for that reason, calculus sort of sticks to one dimension or eventually you do multivariate, but that basically means two dimensions. Linear algebra, you take off into 10 dimensions, no problem.

• 15:12Lex Fridman?

  It just feels scary and dangerous to go beyond two dimensions. That's all.

• 15:18Speaker 01

  If everything's flat, you can't go wrong.

• 15:20Lex Fridman?

  So what concept or? Theorem in linear algebra or in math you find most beautiful. That gives you pause, that leaves you in awe.

• 15:32Speaker 01

  Well, I'll stick with linear algebra here. I hope the viewer knows that really mathematics is amazing, amazing subject. And deep, deep connections between ideas that didn't look connected. Some they turned out they were. But if we stick with linear algebra. So we have a matrix. That's like the basic thing, a rectangle of numbers. And it might be a rectangle of data. You're probably going to ask me later about data science. Where an often data comes in a matrix. You have, you know, the maybe every column corresponds to a, to a drug. And every row corresponds to a patient.

• 16:17Speaker 01

  And, and if the patient reacted favorably to the drug. Then you put up some positive number in there. Anyway, rectangle of, of numbers, a matrix is basic. So the big problem is to understand all those numbers. You got a big, big set of numbers. And what are the patterns? What's going on? And so one of the ways to break down that matrix into simple pieces is uses something called singular values. And that's come on as fundamental in the last, and certainly in my lifetime.

• 17:02Speaker 01

  Eigenvalues, if you have viewers who've done engineering math or, or, or basic linear algebra, eigenvalues were in there. But those are restricted to square matrices. And data comes in rectangular matrices. So you gotta take that, you gotta take that next step. I'm, I'm always pushing. Math faculty, get on, do it, do it, do it. Singular values. So those are a way to break, to make, to find the important pieces of the matrix, which add up to the whole matrix. So you're breaking a matrix into simple pieces.

• 17:46Speaker 01

  And the first piece is the most important part of the data. The second piece is the second most important part. And then often, so a data scientist will like, if a data scientist can find those first and second pieces, stop there. The rest of the data is probably round off, you know, experimental error, maybe. So you're looking for the important part.

• 18:16Lex Fridman?

  Yeah. So what do you find beautiful about singular values?

• 18:20Speaker 01

  Well, yeah, I didn't give the theorem. So here's the, here's the idea of singular values. Every matrix, every matrix, rectangular, square, whatever, can be written as a product of three very simple special matrices. So that's the theorem. Every matrix can be written as a rotation times a stretch, which is just a matrix, a diagonal matrix, otherwise all zeros except on the one diagonal. And then a third, the third factor is another rotation. So rotation, stretch, rotation is the breakup of a, of any matrix.

• 19:03Lex Fridman?

  The structure of that, the ability that you can do that, what do you find appealing? What do you find beautiful about it?

• 19:08Speaker 01

  Well, geometrically, as I freely admit, the action of a matrix is not so easy to visualize, but everybody can visualize a rotation. Take, take, take two dimensional space and just turn it.

• 19:24Speaker 01

  Around the, around the center. Take three dimensional space. So a pilot has to know about, well, what are the three? The yaw is one of them. I've forgotten all the three turns that a pilot makes. Up to 10 dimensions, you've got 10 ways to turn. But you can visualize a rotation. Take the space and turn it. And you can visualize a stretch. So to break a, a, a, a, a matrix with all those numbers in it into something you can visualize, rotate, stretch, rotate. It's pretty neat. It's pretty neat.

• 20:02Lex Fridman?

  That's pretty powerful. On YouTube, just consuming a bunch of videos and just watching what people connect with and what they really enjoy and are inspired by. Math seems to come up again and again. I, I'm trying to understand why that is. Perhaps you can help me, give me clues. So it's not just the lech, the kinds of lectures that you, you give, but it's also just other folks like with Number File. There's a channel where they just chat about things that are extremely complicated, actually.

• 20:36Speaker 01

  Yeah.

• 20:37Lex Fridman?

  People nevertheless connect with them.

• 20:39Speaker 01

  Yeah.

• 20:40Lex Fridman?

  What do you think that is? What?

• 20:41Speaker 01

  It's wonderful, isn't it? I mean, I wasn't really aware of it. We're, we're conditioned to think math is hard. Math is abstract. Math is just for a few people, but it isn't that way. A lot of people quite like math and they like to. I get messages from people saying, you know, now I'm retired. I'm going to learn some more math. I get a lot of those. It's really encouraging. And I think what people like is that there's some order, you know, a lot of order and or, you know, things are not obvious, but they're true.

• 21:17Speaker 01

  So it's really cheering to think that that so many people really want to learn more about math. Yeah.

• 21:25Lex Fridman?

  In terms of truth, again, sorry to slide into philosophy at times, but math does reveal pretty strongly what things are true.

• 21:38Speaker 01

  Yeah.

• 21:39Lex Fridman?

  I mean, that's the whole point of proving things.

• 21:40Speaker 01

  It is, yeah.

• 21:42Lex Fridman?

  And yet, sort of, our real world is messy and complicated.

• 21:47Speaker 01

  It is.

• 21:48Lex Fridman?

  What do you think about the nature of truth that math reveals?

• 21:51Speaker 01

  Oh, wow.

• 21:52Lex Fridman?

  Because it is a source of comfort, like you've mentioned.

• 21:55Speaker 01

  Yeah, that's right. Well, I have to say, I'm not much of a philosopher. I just liked numbers, you know, as a kid. I would, this was before you had, you had to go in when you had a filling in your teeth, you had to kind of just take it. Yeah, so what I did was think about math, you know, like, take powers of 2, 2, 4, 8, 16, up until the time the tooth stopped hurting and the dentist said, you're through. Or counting, yeah, so.

• 22:28Lex Fridman?

  So that was a source of just, source of peace, almost.

• 22:31Speaker 01

  Yeah.

• 22:33Lex Fridman?

  What is it about math, do you think, that brings that?

• 22:36Speaker 01

  Yeah.

• 22:37Lex Fridman?

  What is that?

• 22:38Speaker 01

  Well, you know where you are, yeah. It's symmetry, it's certainty. The fact that, you know, if you, 2, if you multiply 2 by itself 10 times, you get 1,024, period. Everybody's going to get that.

• 22:52Lex Fridman?

  Do you see math as a powerful tool or as an art form?

• 22:56Speaker 01

  So it's both. That's, that's really one of the neat things. You can, you can be an artist and like math. You can be a engineer and use math.

• 23:08Lex Fridman?

  Which are you?

• 23:09Speaker 01

  Which? Which am I?

• 23:10Lex Fridman?

  What did you connect with most?

• 23:12Speaker 01

  Yeah, I'm somewhere between. I'm certainly not a. Artist type philosopher type person. Might sound that way this morning, but I'm not.

• 23:23Speaker 01

  Yeah, I really enjoy teaching engineers. They go for an answer. And yeah, so probably within the MIT math department. Most people enjoy teaching people, teaching students who get the abstract idea. I'm OK with, with, I'm good with engineers who are looking for a way to find answers. Yeah.

• 23:52Lex Fridman?

  So actually, that's an interesting question. Do you think, do you think for teaching and in general, thinking about new concepts, do you think it's better to plug in the numbers?

• 24:04Lex Fridman?

  Or to think more abstractly? So looking at theorems and proving the theorems? Or actually, you know, building up a basic intuition of the theorem? Or the method, the approach? And then just plugging in numbers and seeing it work?

• 24:20Speaker 01

  Yeah, well, certainly, many of us like to see examples first. We understand, it might be a pretty abstract sounding example, like a three dimensional rotation. How are you gonna, how are you gonna understand a rotation in 3D?

• 24:40Speaker 01

  Or in 10D? Or, but, and then some of us like to keep going with it to the point where you got numbers. Where you got 10 angles, 10 axes, 10 angles. But the best, the great mathematicians probably, I don't know if they do that, because they, they, for them, for them, an example would be a highly abstract thing to the rest of it.

• 25:11Lex Fridman?

  Right, but nevertheless, working in the space of examples.

• 25:14Speaker 01

  Yeah, examples.

• 25:15Lex Fridman?

  It seems to.

• 25:16Speaker 01

  Examples of, of structure.

• 25:19Lex Fridman?

  Our brains seem to connect with that.

• 25:20Speaker 01

  Yeah, yeah.

• 25:21Lex Fridman?

  So I'm not sure if you're familiar with him, but Andrew Yang is a presidential candidate.

• 25:29Lex Fridman?

  Currently running.

• 25:31Speaker 01

  Yeah.

• 25:31Lex Fridman?

  With math in all capital letters and his hats as a slogan.

• 25:36Speaker 01

  I see.

• 25:36Lex Fridman?

  Stands for make America think hard.

• 25:38Speaker 01

  Okay, I'll vote for him.

• 25:42Lex Fridman?

  So, and his name rhymes with yours, Yang Strang, so. But he also loves math and he comes from that world of. But he also, looking at it, makes me realize that math, science and engineering are not really part of our politics, political discourse about political life, government in general.

• 26:03Speaker 01

  Yeah.

• 26:03Lex Fridman?

  Why do you think that is?

• 26:05Speaker 01

  Well.

• 26:06Lex Fridman?

  What are your thoughts on that in general?

• 26:08Speaker 01

  Well, certainly somewhere in the system, we need people who are comfortable with numbers, comfortable with quantities, you know, if you, if you say this leads to that, they see it and it's undeniable.

• 26:23Lex Fridman?

  But isn't that strange to you that we have almost no, I mean, I'm pretty sure we have no elected officials in Congress or obviously the president.

• 26:35Speaker 01

  Yes.

• 26:35Lex Fridman?

  That is a, either has an engineering degree or a math.

• 26:40Speaker 01

  Yeah, well, that's too bad. A few could, a few who could make the connection. Yeah, it would have to be people who are, who are, who understand engineering or science and at the same time can make speeches and, and lead, yeah.

• 27:01Lex Fridman?

  Yeah, inspire people.

• 27:02Speaker 01

  Yeah, inspire, yeah.

• 27:03Lex Fridman?

  You were, speaking of inspiration, the president of the Society for Industrial Applied Mathematics.

• 27:10Speaker 01

  Oh, yes.

• 27:10Lex Fridman?

  It's a major organization in math and applied math. What do you see as a role of that society, you know, in our public discourse?

• 27:20Speaker 01

  Right.

• 27:20Lex Fridman?

  And public.

• 27:21Speaker 01

  Yeah, so, well, it was fun to be president at the, at the time.

• 27:25Lex Fridman?

  Couple of years, few years.

• 27:26Speaker 01

  Two years around, around 2000.

• 27:30Speaker 01

  This hope as president of a pretty small society, but nevertheless, it was a time when math was getting some more attention in Washington. But yeah, I got to give a little 10 minutes to a committee of the House of Representatives. Talking about why math. And then, actually, it was fun because one of the members of the House had been a student, had been in my class. What do you think of that? Yeah, as you say, a pretty rare. Most, most members of the House have had a different training, different background.

• 28:10Speaker 01

  But there was one from New Hampshire who, who was my friend, really, by, by being in the class. Yeah, so that, those years were good. Then, of course, other things take, take over in importance in Washington and math, math just, at this point, is not so visible, but for a little moment it was.

• 28:37Lex Fridman?

  There's some excitement, some concern about artificial intelligence in Washington now.

• 28:43Speaker 01

  Yes.

• 28:44Lex Fridman?

  About the future.

• 28:45Speaker 01

  Yeah.

• 28:46Lex Fridman?

  And I think at the core of that is math.

• 28:47Speaker 01

  Well, it is, yeah.

• 28:48Lex Fridman?

  Yeah, maybe it's hidden, maybe it's wearing a different hat, but.

• 28:52Speaker 01

  Well, artificial intelligence, and particularly, can I use the words, deep learning?

• 28:58Lex Fridman?

  Yeah.

• 28:59Speaker 01

  It's a deep learning is a particular approach to understanding data. Again, you've got a big, whole lot of data where data is just swamping the, the computers of the world and, and to understand it, out of all those numbers, to find what's important, you know, in, in climate and everything. And artificial intelligence is two words for, for one approach to data. Deep learning is a specific approach there, which uses a lot of linear algebra. So I got into it. I thought, okay, I've got to learn about this.

• 29:38Lex Fridman?

  So maybe from your perspective, let me ask the, this. The most basic question.

• 29:44Speaker 01

  Yeah.

• 29:45Lex Fridman?

  How do you think of a neural network? What is a neural network?

• 29:48Speaker 01

  Yeah, okay. So can I start with the idea about deep learning? What does that mean?

• 29:55Lex Fridman?

  Sure. What is deep learning?

• 29:57Speaker 01

  What is deep learning? Yeah. So we're trying to learn from all this data. We're trying to learn what's important, what's it telling us? So you've got data. You've got some inputs for which you know the right outputs. The question is, can you see the pattern there? Can you figure out a way for a new input, which we haven't seen, to get the, to understand what the output will be from that new input? So we've got a million inputs with their outputs. So we're trying to create some pattern, some rule, that'll take those inputs, those million training inputs,

• 30:41Speaker 01

  which we know about, to the correct million outputs. And this idea of a neural net is part of the structure of our new way to create a rule. We're looking for a rule that will take these training inputs to the known outputs. And then we're going to use that rule on new inputs that we don't know the output and see what comes.

• 31:13Lex Fridman?

  Linear algebra is a big part of finding that rule.

• 31:16Speaker 01

  That's right. Linear algebra is a big part. Not all the part. People were leaning on matrices. That's good, still do. Linear is something special. It's all about straight lines and flat planes. And data isn't quite like that. You know, it's more complicated. So you got to introduce some complication. So you have to have some function that's not a straight line. And it turned out, nonlinear, nonlinear, nonlinear. And it turned out that it was enough to use the function. That's one straight line and then a different one halfway. So piecewise linear.

• 31:58Lex Fridman?

  Piecewise linear.

• 31:59Speaker 01

  One piece has one slope, one piece, the other piece has a second slope. And so that, getting that nonlinear, simple nonlinearity in, blew the problem open.

• 32:13Lex Fridman?

  That little piece makes it sufficiently complicated to make things interesting.

• 32:17Speaker 01

  Exactly, because you're going to use that piece over and over a million times. So you, so you, it has a, it has a fold in the, in the graph. The graph, two pieces. And, but when you fold something a million times, you've got, you've got a pretty complicated function that's pretty realistic.

• 32:36Lex Fridman?

  So that's the thing about neural networks is they have a lot of these.

• 32:41Speaker 01

  A lot of these, that's right.

• 32:42Lex Fridman?

  So why do you think neural networks, by using a, sort of formulating an objective function, very not a plain

• 32:54Speaker 01

  Yeah.

• 32:54Lex Fridman?

  A function of the

• 32:55Speaker 01

  Lots of folds.

• 32:56Lex Fridman?

  Lots of folds of the inputs, the outputs. Why do you think they work to be able to find a rule that we don't know is optimal, but is just seems to be pretty good in a lot of cases? What's your intuition? Is it surprising to you as it is to many people? Do you have an intuition of why this works at all?

• 33:18Speaker 01

  Well, I'm beginning to have a better intuition. This idea of things that are piecewise linear, flat pieces, but, but with folds between them. Like, think of a roof of a complicated, infinitely complicated house or something. That, that curved, it almost curved, but, but every piece is flat.

• 33:41Speaker 01

  That's been used by engineers, that idea's been used by engineers, is used by engineers, big time. Something called the finite element method, if you wanna, if you wanna design a bridge, design a building, design an airplane, you're using this idea of piecewise flat as a good, a simple computable approximation.

• 34:08Lex Fridman?

  But you're, you have a sense that, that there's a lot of expressive power in this kind of piecewise linear.

• 34:15Speaker 01

  Yeah.

• 34:16Lex Fridman?

  Just combined together.

• 34:17Speaker 01

  You used the right word. If you measure the expressivity.

• 34:21Lex Fridman?

  Yeah.

• 34:21Speaker 01

  How, how, how complicated a thing can, can this piecewise flat guys express? The answer is very complicated, yeah.

• 34:32Lex Fridman?

  What do you think are the limits of such piecewise linear or just neural networks? The expressivity of neural networks?

• 34:41Speaker 01

  Well, you would've said a while ago that they're just computational limits. You know, you, it's a problem beyond a certain size, a supercomputer isn't gonna do it. But that, those keep getting more powerful, so that's, that limit has been moved to allow more and more complicated surfaces.

• 35:04Lex Fridman?

  So in terms of just mapping from inputs to outputs, looking at data.

• 35:11Speaker 01

  Yeah.

• 35:11Lex Fridman?

  What do you think of, you know, in the context of neural networks in general, data is just tensor, vectors, matrices, tensors.

• 35:21Speaker 01

  Right.

• 35:22Lex Fridman?

  How do you think about learning from data? What, how much of our world can be expressed in this way? How useful is this process? I guess that's another way to ask you, what are the limits of this?

• 35:36Speaker 01

  Well, that's a good question, yeah. So I guess the whole idea of deep learning is that there's something there to learn. If the data is totally random, just produced by random number generators, then the, we're not gonna find a useful rule, because there isn't one. So the extreme of having a rule is like knowing Newton's law. You know, if you hit a ball, it moves. So that's where you had laws of physics. Newton and Einstein and other great, great people have found those laws.

• 36:14Speaker 01

  And laws of the distribution of oil in an underground thing. I mean, that's so engineers, petroleum engineers, understand how oil will sit in an underground basin. So there were rules. Now, the new idea of artificial intelligence is learn the rules. Instead of figuring out the rules with help from Newton or Einstein, the computer is looking for the rules. So that's another step. But if there are no rules at all that the computer could find,

• 36:58Speaker 01

  if it's totally random data, well, you've got nothing. You've got no science to discover.

• 37:05Lex Fridman?

  It's automated search for the underlying rules.

• 37:08Speaker 01

  Yeah, search for the rules. Yeah, exactly. And there will be a lot of random parts, a lot. I mean, I'm not knocking random because that's there. The there's a lot of randomness built in, but there's got to be some basic.

• 37:26Lex Fridman?

  It's almost always signal, right?

• 37:28Speaker 01

  There's got to be some signal, yeah. If it's all noise, then there's, you're not going to get anywhere.

• 37:34Lex Fridman?

  Well, this world around us does seem to be, does seem to always have a signal of some kind.

• 37:39Speaker 01

  Yeah, yeah.

• 37:40Lex Fridman?

  To be discovered.

• 37:41Speaker 01

  Right, that's it.

• 37:43Lex Fridman?

  So what excites you more? The, we just talked about a little bit of application. What excites you more, theory or the application of mathematics?

• 37:55Speaker 01

  Well.

• 37:57Speaker 01

  For myself, I'm probably a theory person. I'm not, I'm speaking here pretty freely about applications, but I'm not the person who really, I'm not a physicist, or a chemist, or a neuroscientist. So for myself, I like the structure, and the flat subspaces, and the relation of matrices, columns to rows. That's my part in the spectrum. So really, science is a big spectrum of people

• 38:39Speaker 01

  from asking practical questions and answering them, using some math, then some math guys like myself who are in the middle of it, and then the geniuses of math and physics and chemistry, who are finding fundamental rules, and then doing the really understanding nature. That's incredible.

• 39:09Lex Fridman?

  At its lowest, simplest level, maybe just a quick and broad strokes from your perspective, what is, where does linear algebra sit as a subfield of mathematics? What are the various subfields that you're,

• 39:24Speaker 01

  Okay.

• 39:25Lex Fridman?

  that you think about in relation to linear algebra?

• 39:29Speaker 01

  So the big fields of math are algebra as a whole, and problems like calculus and differential equations. So that's a second, quite different field. Then maybe geometry deserves to be thought of as a different field, to understand the geometry of high-dimensional surfaces.

• 39:52Speaker 01

  So I think, am I allowed to say this here? I think this is where personal view comes in. I think math, we're thinking about undergraduate math. What millions of students study? I think we overdo the calculus at the cost of the algebra, at the cost of linear.

• 40:19Lex Fridman?

  You have this talk titled Calculus Versus Linear Algebra.

• 40:22Speaker 01

  That's right.

• 40:23Lex Fridman?

  And you say that linear algebra wins. So can you dig into that a little bit? Why does linear algebra win?

• 40:33Speaker 01

  Right. Well, okay, the viewer is gonna think this guy is biased. Not true. I'm just telling the truth as it is. Yeah, so I feel linear algebra is just a nice part of math that people can get the idea of. They can understand something that's a little bit abstract because once you get to 10 or 100 dimensions. And very, very, very useful. That's what's happened in my lifetime is the importance of data, which does come in matrix form. So it's really set up for algebra. It's not set up for differential equation.

• 41:15Speaker 01

  And let me fairly add probability, the ideas of probability. And statistics have become very, very important, have also jumped forward. So, and that's not, that's different from linear algebra, quite different. So now we really have three major areas to me. Calculus, linear algebra, matrices, and probability statistics. And they all deserve an important place. And calculus has traditionally had a lion's share of the time.

• 41:57Lex Fridman?

  A disproportionate share.

• 41:59Speaker 01

  Thank you. Disproportionate, that's a good word.

• 42:01Lex Fridman?

  Of the love and attention from the excited young minds.

• 42:07Speaker 01

  Yeah.

• 42:10Lex Fridman?

  I know it's hard to pick favorites, but what is your favorite matrix?

• 42:14Speaker 01

  What's my favorite matrix? Okay, so my favorite matrix is square. I admit it. It's a square bunch of numbers, and it has 2s running down the main diagonal. And on the next diagonal, so think of top-left to bottom-right, 2s down the middle of the matrix, and minus 1s just above those 2s and minus 1s just below those 2s. And otherwise, all zeros. So mostly zeros. Just three non-zero diagonals coming down.

• 42:50Lex Fridman?

  What is interesting about it?

• 42:51Speaker 01

  Well, all the different ways it comes up. You see it in engineering. You see it as analogous in calculus to second derivative. So calculus learns about taking the derivative, figuring out how much, how fast something's changing. But second derivative, now that's also important. That's how fast the change is changing, how fast the graph is bending, how fast it's curving. And Einstein showed that that's fundamental to understand space. So second derivatives should have a bigger place in calculus.

• 43:34Speaker 01

  Second, my matrices, which are like the linear algebra version of second derivatives, are neat in linear algebra. Yeah, just everything comes out right with those guys.

• 43:51Lex Fridman?

  Beautiful. What have you learned about the process of learning by having taught so many students math over the years?

• 44:00Speaker 01

  Oh, that is hard. I'll have to admit here that I'm not, I'm not really a good teacher. Because I don't get into the exam part. The exam is the part of my life that I don't like. And grading them and giving the students A or B or whatever. I do it because it's, I'm supposed to do it. But, but I tell the class at the beginning, I don't know if they believe me. Probably they don't. I, I tell the class, I'm here to teach you. I'm here to teach you math and not to grade you.

• 44:39Speaker 01

  And, but they're thinking, okay, this guy is gonna, you know, when's he gonna, is he gonna give me an A minus? Is he gonna give me a B plus? What?

• 44:48Lex Fridman?

  What, what have you learned about the process of learning?

• 44:51Speaker 01

  Of learning? Yeah, well, maybe to be, to give you a legitimate answer about learning. I should have paid more attention to the assessment, the evaluation part at the end. But I like the teaching part at the start. That's the sexy part. To tell somebody for the first time about a matrix. Wow.

• 45:13Lex Fridman?

  But is there, are there moments? So you, you are teaching a concept. Are there moments of learning that you just see in the student's eyes? You don't need to look at the grades.

• 45:25Speaker 01

  Yeah.

• 45:25Lex Fridman?

  But you see in their eyes that, that you hook them? That, you know, that you connect with them in a way where, you know what? They, they, they fall in love with this, with this beautiful world of math.

• 45:38Speaker 01

  They see that it's got some beauty there.

• 45:40Lex Fridman?

  It's, yeah. Or conversely, that they give up at that point is the opposite. The, the darker say that math, I'm just not good at math. I don't want to walk away.

• 45:50Speaker 01

  Yeah, yeah. Maybe because of the approach in the past, they were discouraged. But don't be discouraged. It's, it's too good to miss. Yeah, I, well, if I'm teaching a big class, do I know when? I think maybe I do. Sort of, I mentioned at the very start, the four fundamental subspaces and the structure of the fundamental theorem of linear algebra. The fundamental theorem of linear algebra. That is the relation of those four subspaces, those four spaces. Yeah, so I think that. I feel that the class gets it.

• 46:34Lex Fridman?

  When they see.

• 46:36Speaker 01

  Yeah.

• 46:37Lex Fridman?

  What advice do you have to a student just starting their journey mathematics today? How do they get started?

• 46:45Speaker 01

  Yeah, that's hard. Well, I hope you have a teacher, professor, who is still enjoying what he's doing, what he's teaching. They're still looking for new ways to teach and understand math.

• 47:05Speaker 01

  Because that's the pleasure to the moment when you see, oh, yeah, that works.

• 47:12Lex Fridman?

  So it's less about the material you

• 47:14Speaker 01

  Yeah.

• 47:15Lex Fridman?

  you study, it's more about the source of the teacher being full of passion for

• 47:21Speaker 01

  Yeah, more about the fun. Yeah, the moment of getting it.

• 47:27Lex Fridman?

  But in terms of topics, linear algebra?

• 47:31Speaker 01

  Well, that's my topic, but, oh, there's beautiful things in geometry to understand. What's wonderful is that in the end, there's a pattern. There are rules that are followed in biology as there are in every field.

• 47:54Lex Fridman?

  You describe the life of a mathematician as 100% wonderful. Except for the grade stuff.

• 48:04Speaker 01

  Yeah.

• 48:04Lex Fridman?

  The grades.

• 48:05Speaker 01

  Except for grades.

• 48:06Lex Fridman?

  Yeah. When you look back at your life,

• 48:08Speaker 01

  Yeah.

• 48:09Lex Fridman?

  what memories bring you the most joy and pride?

• 48:13Speaker 01

  Well, that's a good question. I certainly feel good when I, maybe I'm giving a class in 1806. That's MIT's linear algebra course that I started. So sort of there's a good feeling that, okay, I started this course. A lot of students take it, quite a few like it. Yeah, so I'm sort of. Happy when I feel I'm helping, helping make a connection between ideas and students, between theory and the reader. Yeah, it's, I get a lot of very nice messages from people who've watched the videos

• 48:58Speaker 01

  and it's inspiring. I just, I'll maybe take this chance to say thank you.

• 49:05Lex Fridman?

  Well, there's millions of students who you've taught and I am grateful to be one of them. So Gilbert, thank you so much. It's been an honor. Thank you for talking today.

• 49:15Speaker 01

  It was a pleasure. Thanks.

• 49:18Lex Fridman?

  Thank you for listening to this conversation with Gilbert Strang. And thank you to our presenting sponsor, Cash App. Download it, use code LEXPODCAST, you'll get $10 and $10 will go to FIRST, a STEM education nonprofit that inspires hundreds of thousands of young minds to learn and to dream of engineering our future. If you enjoy this podcast, subscribe on YouTube, give it five stars on Apple Podcasts, support on Patreon, or connect with me on Twitter. Finally, some closing words of advice from the great Richard Feynman. Study hard what interests you the most in the most undisciplined, irreverent, and original manner possible. Thank you for listening and hope to see you next time.