Growing up to become a world-class AI expert

While at the NVIDIA GPU Technology Conference 2019 in Silicon Valley, Chris enjoyed an inspiring conversation with Anima Anandkumar. Clearly a role model - not only for women - but for anyone in the world of AI, Anima relayed how her lifelong passion for mathematics and engineering started when she was only 3 years old in India, and ultimately led to her pioneering deep learning research at Amazon Web Services, CalTech, and NVIDIA. Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $100 credit. Learn more at do.co/changelog https://do.co/changelog. DataEngPodcast https://www.dataengineeringpodcast.com?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai – A podcast about data engineering and modern data infrastructure. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Anima Anandkumar – Website http://tensorlab.cms.caltech.edu/users/anima/, LinkedIn https://www.linkedin.com/in/anima-anandkumar, X https://x.com/animaanandkumarChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: Anima Anandkumar on Wikipedia https://en.wikipedia.org/wiki/Anima_AnandkumarNVIDIA GTC Speaker https://gputechconf2019.smarteventscloud.com/connect/speakerDetail.ww? PERSON_ID=B9CC9DCACB363F0948CA74541A9A8B0CNVIDIA GTC Presentation: Role of Tensors in Machine Learning https://gputechconf2019.smarteventscloud.com/connect/sessionDetail.ww? SESSION_ID=272382NVIDIA GTC Presentation: Infusing Physics into Deep Learning Algorithms with Applications to Stable Landing of Drones https://gputechconf2019.smarteventscloud.com/connect/sessionDetail.ww? SESSION_ID=279709CalTech CMS TensorLab http://tensorlab.cms.caltech.edu/users/animaGoogle Scholar https://scholar.google.com/citations?user=bEcLezcAAAAJUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log. This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes.

• 0:41Speaker 00

  So if you have built boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month. Learn more and get started for free with a $100 credit at digitalocean.com slash change log. Again, digitalocean.com slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and tag with us around various topics of the show at changelog.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:47Chris Benson?

  Hello there and welcome to another episode of Practical AI, the podcast where we try to make AI practical, productive, and accessible to everyone. This is Chris Benson, your co-host. And today I am at the Nvidia GPU Technology Conference in Silicon Valley. It is Tuesday, March 19. I happen to be operating solo today. Daniel was not able to make it out here. I have a pretty amazing guest today to talk to. With me is Anima Anan Kumar. Did I get your name? Okay.

• 2:22Speaker 02

  Oh, yeah, you did an amazing job.

• 2:25Chris Benson?

  Okay, anyone who listens to me on this regularly knows that I screw up names all the time, so I'm glad that. You are the Director of Machine Learning Research at Nvidia, and you're also the Bren Professor at Caltech, is that correct?

• 2:39Speaker 02

  That's right, yeah, I wear both the hats, and I think it really gives me an opportunity to bridge industry and academia in many interesting ways.

• 2:49Chris Benson?

  Absolutely. So, in a few minutes, you're giving two separate talks here at GTC, and I would like to kind of delve into both those, but at first, I would really like to talk to you, and I know that you talk to people often. You've been in front of audiences quite a bit about AI and the role you play there, but I actually want to go back to the beginning and kind of find out, you know, how you got into this, what that journey has been like, and talk about, a little bit about some of the challenges that you may have faced along the way, and I know that you have spoken to some of those in the past. So, can you tell us kind of, you know, how you got into this at all?

• 3:30Speaker 02

  Yeah, certainly. You know, I've had a pretty amazing childhood in terms of, you know, the people around me were always encouraging me in my passion for maths and sciences, but also for dancing and arts and everything, right? So, there was a lot of like, you know, like my mom is an engineer, my dad has a small-scale industry that manufactures all kinds of components, builds machines. So, you know, I would go there as a kid, like both my mom and dad would take me there, and to see how those machines operated, and how, it just seemed so magical that, you know, you could automatically, you know,

• 4:12Speaker 02

  build these components, and there would be in all kinds of shapes, you had these machines going at this really high speed, and so that was fascinating, and I would ask my parents, you know, "How is this possible?", right? And they would be like, "Oh, it's all math", right? So, I had like this very friendly, like math was a friendly thing to me from my early childhood.

• 4:33Chris Benson?

  How old were you? About that time period, like what do you think that that what age were you at at that?

• 4:38Speaker 02

  I think one of my earliest memories are either me like kind of, you know, solving some puzzles or some toys, right? And going to industry like my parents always like, you know, took me when I was maybe three or four or so.

• 4:55Chris Benson?

  Wow, that is young.

• 4:55Speaker 02

  Yeah, that is.

• 4:56Chris Benson?

  So you had this fascination at that really young age.

• 4:59Speaker 02

  Yes, yes. Some of my earliest memories are like, you know, me like trying to solve like, you know, math problem and wondering how, you know, there is this addition and suddenly we're subtraction. Why is there these two symbols? And, you know, what is it? How are the two related? Right? So I somehow remember being very fascinated with it. And my grandfather is a math teacher. And so he was teaching me.

• 5:24Chris Benson?

  You had a secret weapon in your family.

• 5:26Speaker 02

  That's right. And my grandma would give me all these puzzles and games. And, you know, she tells me that I apparently had memorized like the calendar for a 10 year time period. And they would quiz me on what day is 12th of August, for instance.

• 5:44Chris Benson?

  You gotta be kidding me. Is it? How old were you for that?

• 5:47Speaker 02

  Apparently three.

• 5:48Chris Benson?

  Oh, my gosh. Okay.

• 5:50Speaker 02

  This is what my grandma tells me. I don't, I can't do it anymore. So don't, you know.

• 5:56Chris Benson?

  You're setting a high bar for the three year olds out there. I know that, you know, I'm sure we have three year olds listening to the Practical AI Podcast, but in case for that one or two out there, you've just set the bar very high. My, my, my daughter, who's about to turn seven, has no excuses at this point.

• 6:11Speaker 02

  No, no, I think every child child's development is different, right? But, you know, at the same time, there is so much fascination in all aspects too of development. For instance, I, you know, I love dancing. I started dancing when I was three as well. So I wasn't put into a box, you know, wasn't told, oh, this is math. You're only good at that. Do that.

• 6:32Chris Benson?

  So do you think, do you think doing dance and other activities that have nothing to do with technology, do you think that made a difference in that, that having that diversity of, of experiences made a difference in how you progressed?

• 6:45Speaker 02

  Oh, certainly, that's what makes us human, right? We know the artistic side of us and the humanities and the liberal education is very much a part of our growth as human beings, as a society. And to me, I would also argue it's highly mathematical. Dancing is,

• 7:03Chris Benson?

  True.

• 7:03Speaker 02

  you know, all about rhythm, right? And you'd count one, two, three, four, four steps and and then you progress to saying it's no longer these discrete steps, it's a more continuum of movement, right? There is flow, but there's also sudden peaks and, you know, sudden changes to it that may, you know, so as you progress in dancing, to me, it's highly mathematical. It's like a wave.

• 7:26Chris Benson?

  I have this vision of you at four years old, you know, going through a dance and then dissecting it mathematically across the board there, so.

• 7:33Speaker 02

  No, but I think my, the earliest memory that I have that very much relates to, you know, what I'm doing with AI today, right? I, I remember suddenly like stopping and questioning myself, huh? I feel so differently right now. I feel like there's something that's me, like I had never, you know, this kind of what they call self-actualization. And I, again, no one had told me about it, right? Like, I had no idea, but I was just playing and I have, remember this moment very vividly where I felt, oh, there's something known as me, even though maybe that's not the language I used. And, you know, and for us, taking baby steps in AI, right? That's so fascinating. How do kids

• 8:17Speaker 02

  learn so quickly, learn so intuitively, come up with all these developments in their personality?

• 8:22Chris Benson?

  Yes, there's, there's so many things that we kind of take for granted as, as humans. And, and ironically, those of us now in the AI space look back with, with a kind of incredulness, you know, in terms of saying, wow, you know, you don't think about that, but there's so much there that we're trying to discover now in the field of AI so that we can do amazing things.

• 8:43Speaker 02

  Yeah, and I would say we are still quite far away from getting anywhere close to that, right? But that's what will keep me employed for a lifetime.

• 8:51Chris Benson?

  There you go, job security right there.

• 8:53Speaker 02

  But yeah, so going from those very early childhood memories and, you know, like getting fascinated about math and all the way through like high school, right? I was just, you know, getting into like math and all the time it was, oh, let's, you know, what's the next thing, you know, like I was remember like, oh, there were, you know, number systems and then there was suddenly complex number and this is imaginary. Oh, you can even make up things in math, right? Before that, you think math is just concrete and you can't make up things in math, but imaginary number is something we make up, but it's so powerful in the way we use it.

• 9:30Chris Benson?

  So as a kid, the tangible aspects of math, looking at machines working and realizing there was math involved and having your parents and grandparents influence that. But then it sounds like you, as you grew in mathematics and learned more and more, the abstract nature really appealed to you, it sounds like.

• 9:48Speaker 02

  Exactly, right? So these, you know, there is structure in the way we build math. We, you know, first start with a specific goal of counting numbers, but we said, oh, this is not enough, right? We need to expand our number systems. We then had rational and then we said it's, you know, we also needed rational numbers and real and then complex. So it tends to get increasingly more abstract, but then it has all these applications that wouldn't be possible without it. And that's what fascinated me. It's almost like, you know, the math, even though I cannot directly, you know, relate to it, right? I can't visualize it, I can still use it in many interesting ways. And that's when I, you know, thought like abstract thinking is important, right?

• 10:30Speaker 02

  I shouldn't always try to get it into something physically that I can relate to. I don't need that.

• 10:35Chris Benson?

  So as you're growing up through this process and becoming more and more proficient in mathematics through school and through home life and such, was, were you, did you retain that passion for the engineering that you had as a, as a child? Did that, did that drive or did you get more into just the pure mathematical passion that you were discovering?

• 10:56Speaker 02

  Uh, no, that's a good point. So my mom and dad would take me to different, you know, uh, industry meetings. So we'd go to like these big trade shows, right? Like GTC but for manufacturing. And so I'd be seeing all these fascinating, like, you know, the latest machines, the computerized numerically controlled machines. You know, back then, if you remember, it was like the green screen and it only had like some simple programs you could write, right? In terms of the capabilities, it was like, okay, which axis do you go? What would be the set of movements of this turret that would go and which tooling do you use? Right? But to me, like, oh, wow, you can program and now you can change materials,

• 11:36Speaker 02

  right? And this is, uh, uh, and this is done in such an automated way. And that was my introduction to programming, which is very non-orthodox way of thinking of programming.

• 11:46Chris Benson?

  And about how old would you have been as you really started expressing yourself in programming?

• 11:50Speaker 02

  I think this, I would have been, uh, more around eight or so.

• 11:55Chris Benson?

  Yeah. So you're still quite young at that point in time. Okay. And so, I mean, by that time, did you pretty well know that you were, this was the path that you were on as you grew up? Or were you still kind of finding yourself?

• 12:07Speaker 02

  So I was still dancing. I always loved dance. Uh, I should say, you know, much of my family is engineers and mathematicians, right? So although my uncle is a biotechnologist, so I would like, you know, hear from him, you know, advances in biology. So, so no one really put me in any one path. Uh, for me, I was generally, you know, fascinated with science of technology, math and engineering. Uh, but at that point, I was, you know, looking at documentaries, for instance, about NASA and space. And, you know, that's how Caltech actually, you know, remember thinking like, oh, that's such a fascinating place. So for me to kind of go from there to actually be at Caltech and be at Nvidia that's making

• 12:51Speaker 02

  these cutting-edge technology to enable AI is, uh,

• 12:55Chris Benson?

  So I guess, so you got to Caltech as an undergraduate then?

• 12:59Speaker 02

  Uh, no, I didn't.

• 13:00Chris Benson?

  No?

• 13:00Speaker 02

  I was in.

• 13:00Chris Benson?

  So how did you get, how did you make that shift? As you're, as you're, you know, coming through your teenage years and it's time to pick universities, where did you go and what caused that decision-making process?

• 13:11Speaker 02

  Yeah, yeah, so I was in high school and, you know, at that point, so I was back in India, right? So, and, you know, I was looking at the local colleges. I think at that point, I was still not ready to come halfway around the world. And, you know, these Indian Institute of Science, Technology, sorry, or the IITs, had this really difficult entrance exam to enter them because they're highly selective. They, it's like, you know, I forget the exact number, it's some huge number, maybe 500,000 students take that. And you had to be in the top 500 or so, or 600, at least when I was doing it,

• 13:51Speaker 02

  to be like in computer science or electrical engineering or some major that you would like to pick. And, you know, so I remember hearing about this exam in high school, right? And my first reaction was, why is that so difficult? What makes it difficult, right? I mean, why, oh, it's like, oh, it's because it's very, somebody would tell me, oh, because it has difficult math. I'm like, really, what is difficult math?

• 14:15Chris Benson?

  You know, I, just in my own experience, I think that is a fairly unusual reaction to these tests. Most of, most of us just go, oh, my God, I have to, I have a hard test. I gotta go figure out and study for it and everything. So it's interesting, you're already analyzing the test itself in terms of whether, the difficulty and how to apply yourself to it.

• 14:34Speaker 02

  Yeah, but also I was like, oh, I really wanna like learn something where I would be challenged, right? To be honest, because, maybe because I had like, you know, this, a lot of help from my family and, and I was reading books like beyond like my schooling requirements. So I felt honestly like, oh, I need to do more that, you know, that would really help me get to, like, you know, things I would see in the real world, right? Like these machines that work or, you know, I'd look at these documentaries about the space. Like I was having this huge gap about the math I'm doing in school and how is this possible today?

• 15:14Chris Benson?

  So I guess is you, so which school did you start into your undergraduate work? And did you, and did you select mathematics as your major?

• 15:23Speaker 02

  Uh, so this was I A T. So they only had engineering as most of their majors, right? So I did take electrical engineering and that to me was a nice sweet spot of where, you know, I would like be using a lot of math, like I would be analyzing signals. I would still be connected to the hardware.

• 15:41Chris Benson?

  Yeah. You're going back to that three-year-old seeing the machine working right there.

• 15:45Speaker 02

  Yeah, and I would still be programming. I would be doing, you know, and it was, and I was a minor in computer science. So I'd have all these different elements put together. And so, you know, that's where this entrance exam, I felt was an opportunity for me to really go into like, you know, when we look at physics, right? Like what are the basic principles? Like how not only like, you know, know about the laws of physics, but how do I apply to different systems? Because like what made this test challenging was, you know, at least at that point, you know, those were near to like Olympiad level questions.

• 16:22Chris Benson?

  Yeah.

• 16:22Speaker 02

  And why are those questions challenging? That's because they're not the usual ones you would solve during your assignment or during your usual schooling.

• 16:29Chris Benson?

  So did you dive heavily into physics at your university schooling at that point? Had you been exposed to it prior to university at all? Or was this kind of a new, a new area that you wanted to delve into?

• 16:41Speaker 02

  I mean, this was the entrance exam to get into the university.

• 16:43Chris Benson?

  Oh, to get into your undergraduate. I misunderstood that. I apologize.

• 16:46Speaker 02

  So no, and so it was physics, chemistry and mathematics. So all these three, you had to dive deep into it. And so when I had, when I was faced with these difficult questions, that's when I had to really go back and understand the principle, right? If I had misunderstood it, I could not apply it to solve the problem.

• 17:03Chris Benson?

  As you're approaching university, that is a fantastic story, even up to that point. It's truly inspiring. And I hope, I hope people will share that with, with their daughters like I will. As you got into university, how did you evolve at that point? You know, you're, you're in it now, you're in your schooling. At some point, I know you would have come to the United States and. Ended up at Caltech. Tell me how that process happened.

• 17:26Speaker 02

  Yeah, so once I was at I A T, you know, was just surrounded with a lot of amazing people, right, like the faculty, the students, you know, they've all come, you know, the all the other students have come through the difficult test as well, right, but, you know, but I found so many amazing friends that I still keep in touch and, and what fascinated there was, you know, we all share this common vision to ask how technology could help society, could help us have a better future. So I remember being very involved in this tech fest called Shastra, that's the annual festival that I A T Madras, where I went to school throws. And we were asking, okay, what would it, you know, how do we bring in more students

• 18:10Speaker 02

  here? What kind of contests would be good? Where do we get the best speakers, right? What are the best researchers today? How do we get them here? And so to me, that was one of the highlights on, you know, how seeing this community come together and ask, you know, how do we showcase today's technology? Where is that going in future? And so that was that kind of a community togetherness is what I experienced at I A T. And part of that was also asking, you know, at that point, I was doing research both in my undergrad school, also in Indian Institute of Science, which is in Bangalore. And that's when, oh

• 18:51Speaker 02

  my God, this is, I have so much freedom to think and to ask, you know, what's, what I could be doing innovatively, right? What's missing today and what I could

• 19:00Chris Benson?

  How you can make a difference?

• 19:02Speaker 02

  Yeah, and create something myself and that can make a difference. And so during my junior year, I did this undergraduate research experience, an equivalent of what we call surf year.

• 19:15Chris Benson?

  Okay.

• 19:15Speaker 02

  And that's summer undergraduate research fellowship, right? So there's a similar program back in India. And that's when I decided I just needed to do a PhD. I had to, you know, dedicate that time to go deep into something, right? And, you know, really being, you know, and who knows? That was the adventure, right? Because what I could do there, I didn't know, but the possibility seemed so exciting.

• 19:41Chris Benson?

  Did you feel, did you feel that you wanted to go, I mean, what Caltech being the world-class school it is, did that, did that draw you at that point in time? Was that the natural next progression in your own growth?

• 19:50Speaker 02

  Yeah, so I applied to quite a few schools. So I, I ended up at Cornell University. So Caltech is where, you know, I, I think I did apply there, but Caltech is a smaller school, right? So we are highly selective, but I'm glad I'm there now.

• 20:06Chris Benson?

  Your idea, yeah, your idea of, you know, schools, they're all world-class schools that you're naming here. So it's, it's good to have that kind of a problem to, to figure out which of those schools you're going to go to. So you got to Cornell first then?

• 20:21Speaker 02

  Yes, that's right.

• 20:22Chris Benson?

  And so tell us from there.

• 20:24Speaker 02

  Yeah, so, you know, when I arrived at Cornell, it's just this beautiful place, right? It is in the middle of nowhere, but it's nice. It's a lot of natural surroundings. It's called Ithaca, it's gorgeous.

• 20:36Chris Benson?

  Good for thinking, right? Just birds and...

• 20:39Speaker 02

  I mean, I did arrive in early fall, which, but yeah, no, I mean, it's like to me, a place where, you know, like there's a lot of close-knit community, like my advisor Lang Tong, just was just such a wonderful human being. He would like, you know, give me that freedom to think and grow and say, and he wouldn't say, oh, no, this doesn't work. He's never told me that, right? He would be like, go figure it out. And, and then he would be like, oh, you know, like that why if it didn't work, he would be like, oh, why not, right? Like, so we'd have this very open conversation and very, and so honestly, he really motivated me to be a professor. Because when I saw him, like how he, you know, deals with students and the kind of lifestyle he has

• 21:24Speaker 02

  in terms of really being able to think openly.

• 21:27Chris Benson?

  So this, you were in your PhD program at this point?

• 21:29Speaker 02

  That's right.

• 21:30Chris Benson?

  Okay.

• 21:30Speaker 02

  That's right, at Cornell.

• 21:31Chris Benson?

  Okay.

• 21:32Speaker 02

  And, you know, so that's when I was going to conferences, I was meeting other researchers and to share these ideas openly, that's what, you know, I decided that... That point when I was graduating that I wanted to be in academia because back then, you know, like AI really hadn't taken off the way it is today, right? So there were so very few, almost no industry industrial research in machine learning and AI.

• 22:00Chris Benson?

  And as we're as we talk about that, because we're kind of now turning explicitly to AI. What was it is at what point in that process? Because we were talking about the math and the engineering for a while. At what point did you realize AI was the the area that you wanted to focus on? What when did that happen in that process?

• 22:17Speaker 02

  Yeah, so it's in fact, it happened during my early PhD itself, right? So I mean, Langtong is an expert in wireless networks and wireless communications. And back then we were, you know, as thinking about all these challenges of how to communicate over wireless channels, right? But the question is also what do you communicate? And during that point, the question was, what if you just didn't blindly send all the data? If you could like make inferences, if you could make decisions on what's really relevant, you could really reduce how much you communicate. So I came into it from that viewpoint of, you know, how to reduce requirements in communication, right? Which is not a traditional way of getting into AI.

• 23:01Speaker 02

  But I've never taken a traditional path in many ways.

• 23:04Chris Benson?

  Clearly.

• 23:05Speaker 02

  So so, yeah, so and that's where, you know, it led me to asking, OK, now we need to then do some inferences, right? I need to now think about machine learning techniques. You know, we called it statistical inference and estimation in more of this traditional signal processing community. But the tools are that foundation is the same.

• 23:25Chris Benson?

  Yes.

• 23:25Speaker 02

  Right? So and so that was my first project on how do we do distributed learning? Like where there are a number of sensors and joint like they don't need to communicate all the raw data they collect. But they only communicate after making some inferences what's truly relevant to your problem, right? Let's say if there is, you know, it's measuring the temperature and there's something anomalous, maybe you only want to communicate that anomalous behavior and not just keep communicating all the time.

• 23:56Chris Benson?

  And so, did you get your PhD from Cornell?

• 23:59Speaker 02

  Yes.

• 24:00Chris Benson?

  And then specifically, what was the PhD in? By the time you finished this long process, actually, finishing is probably the wrong word. But as you got to this point in the journey,

• 24:10Speaker 02

  Yeah.

• 24:10Chris Benson?

  what was your PhD in?

• 24:11Speaker 02

  Yeah, so my thesis title, if I recall, right? It's been a decade now, is Distributed Statistical Inference. So, yeah, so my first project led to many other projects of, you know, how do you like route this in an energy efficient way in a network? And how do you model correlations across different sensors in useful ways, right? And that's when it got into this large-scale machine learning. Because we said, oh, there should be like dependencies, like what do these correlations mean? There should be an underlying what we call a graphical model, a probabilistic model, that connects all these measurements. And now, that's when I started getting fascinated into all the literature of machine learning, right? That's kind of considered core machine learning.

• 24:55Speaker 02

  And, you know, I met Alan Willskey, who's a professor at MIT, now he's retired. At a conference, and my advisor said, oh, just go visit him, right? So I took the, my whole last year was actually spent at MIT. Because he's the expert in graphical models, and I wanted to dive deep into that. And that's when I truly started publishing in machine learning conferences. You know, at the end of my PhD and when I had a faculty job. So I kind of made a switch in terms of which communities I was publishing. Although, the core, you know, math was very much connected and related. So I didn't feel it's a big switch, but a lot of people felt, you know, really surprised.

• 25:38Speaker 02

  What, you're starting your faculty career and you're suddenly going to these new conferences? Right? And back then, these main machine learning conferences, you know, back then was called NIPS, and now it's called NEURIPS, and ICML were tiny events. And signal processing events were much bigger. And so many people were wondering why I made the switch. And for me was, it's always been about where the core intellectual ideas are, right? And where there's a lot of potential, to me, I felt. You know, this machine learning revolution has to happen. It's just a matter of when. And so, I'm happy I, you know, I went into like core machine learning before it all took off.

• 26:23Speaker 03

  The Data Engineering Podcast is a weekly deep dive on modern data management with the engineers and entrepreneurs who are shaping the industry. Go behind the scenes on the tools, techniques, and difficulties of data engineering, so you can learn and keep up with the knowledge to make you and your business successful.

• 26:39Speaker 04

  Can you give a bit of an outline about the motivation for choosing Jupiter Notebooks in particular as the core interface for your data teams?

• 26:47Speaker 05

  Yeah, and actually, when I first joined Netflix, it was a sort of toss at me. And I was definitely like, are we crazy? And the answer was like, we might be a little crazy.

• 26:56Speaker 03

  Go to dataengineeringpodcast.com to listen, subscribe, and share it with your friends and colleagues.

• 27:16Chris Benson?

  So let's, now that you've finished your PhD, let's turn toward your now out there. You have your PhD, your faculty, and you are doing work in the space. How did you, how did you arrive at Caltech? There's a couple of questions I'll ask you about when you get there, and then ultimately at Nvidia. And then after all that, I'm going to turn you toward your talks itself down the road. But, but can I tell me, how did you, how did you move over to Caltech? What made that transition?

• 27:46Speaker 02

  I'm still not at Caltech.

• 27:48Chris Benson?

  I'm sorry. Okay.

• 27:50Speaker 02

  No, no, that's what makes this a fascinating story, right? There's still suspense.

• 27:56Chris Benson?

  So there's still a ways to go.

• 27:58Speaker 02

  Yeah.

• 27:58Chris Benson?

  I see I'm rushing your story. Terrible of me.

• 28:02Speaker 02

  I actually started at UCLA Irvine, which is not far from Caltech, right? So again, Southern California, like, you know, again, a beautiful place and a lot of really amazing.

• 28:14Chris Benson?

  If the listeners could see my face right now, I'm blushing for, for having done that. But anyway, keep going. Sorry about that.

• 28:20Speaker 02

  No, but yeah, and as I said, I was getting into this new field in the sense of publishing, right, like these new conferences, new people. And yeah, so there was a bit of me that was stressed, oh my god, I'm starting this faculty life and doing this. But, you know, there were just truly fascinating problems. I was looking at these probabilistic models and asking questions, right? You know, when can you learn these models at scale, right? And what do you mean, like, there is a correlation between these models? There, you know, is what is, are there few variables that can summarize the effects? And that's when I got introduced into latent variables. You know, the idea that we can't measure everything in this world, right?

• 29:05Speaker 02

  There's always going to be something hidden. And but it's the hidden thing is what maybe we are really after. That's what we'd like to learn, but we can't directly measure it.

• 29:14Chris Benson?

  That seemed, based on your inquisitive nature, you know, talking from childhood up, I'm not at all surprised to know that that's what you were going after.

• 29:22Speaker 02

  I'm always after the hidden things.

• 29:24Chris Benson?

  The next thing, what makes that work?

• 29:25Speaker 02

  Yeah.

• 29:26Chris Benson?

  So keep going.

• 29:27Speaker 02

  Yeah, so I was looking into these, what we call latent variable estimation, right? Think about, you know, like, understanding text, right? In the last few years, we made a lot of progress, but you want to ask, like, what do I really want to extract from this text? It's the meaning. It's like the topics that are discussed in the text, right? But, you know, as human beings know, even if that word is not even there, you can extract what the topic is, right? And that level of, like, extraction is hard because, you know, we don't understand all the nuances of the language now through machines.

• 30:03Chris Benson?

  Absolutely, we're struggling with that now. That's a big part of research in AI, with a couple of interesting, you know, things that have happened in NLP News lately, but, but keep going.

• 30:12Speaker 02

  Yeah, exactly, right? So, so doing it in a fully generative way, like generate language and all its nuances is hard. And that's where I think simplicity, there's something to be said about that, right? So back then, deep learning was not it there, right? And with these models and with the compute we had, and even today... Actually, in fact, that's the state of the art results when it comes to categorizing large documents. You have these lengthy, let's say, news articles or your reports. You and they are could be in the millions, right? And all you have is this raw text. Maybe you have a bit of metadata, but not much. So you can't go laboriously even provide examples. You can't go annotate each word and say, oh, this is representative of topic.

• 30:57Speaker 02

  Let's say justice. This is representative of education. This is representative of geography. Or, you know, you can't go manually annotate each document of what it represents, right? Or and then ultimately even each word because you want representative words for each talk.

• 31:12Chris Benson?

  Right.

• 31:13Speaker 02

  Each topic. And so this is called unsupervised learning where we don't have these examples. Like no one's telling you what a topic should look like and how you should categorize your document into multiple topics.

• 31:26Chris Benson?

  Even now, as you know, it's fully into the age of deep learning. That is still a huge push, you know, as we that's where so much of the research is right now. So so keep going.

• 31:36Speaker 02

  Yeah, and exactly. And so back then, the question was, what's the simplest thing you can do there? Right? So and that's always like I like I tell my students and I tell my, you know, colleagues and all the researchers I work with. Is how, you know, first think about the simplest thing you can do. And if it doesn't work, then go to more and more complex, right? Because one of the dangers which I'll come to later with deep learning is you may just overly get it complicated and not understand what's going on.

• 32:04Chris Benson?

  You sound like an engineer when you say that. I mean, I can totally see that background there. You know, start with the simple. I I you.

• 32:11Speaker 02

  Yeah. Yeah, no, I have a lot of respect for the traditional engineering that, you know, makes aircrafts fly today. And all the, you know, transportation structures, everything, right? There's a lot, I think, so which will come to my latest part in the end about my talk. It's blending the old and the new together. So anyway, coming back to this top. How to extract topics from documents and automatically categorize documents at scale in an unsupervised way.

• 32:40Speaker 02

  A simple intuition is, right, what if you look at frequency of words, right? Let's say the word apple occurs a lot. If I just told you this, then you'd be like, huh? It could be about a fruit, or it could be about a company, right? So it's like not enough. Then you could say, I tell you that what if word apple and orange occurs in the document?

• 33:04Chris Benson?

  It changes the content. You suddenly have a little bit of understanding.

• 33:06Speaker 02

  Yeah, you have more information, but you know, orange is also a company, but maybe, you know, both of them may not occur that much together. And so then you could go to three words, right? You could say like apple, orange and banana. And then now it's like, oh, it is really a fruit, something about fruits, right? So the more, what we call co-occurrences, if you now look at how multiple words occur in the document together, then you can make better inferences of what the document is talking about, right? And note that no one is telling the algorithm, right? No one is writing down every triplet and saying, oh, if it's apple, orange, banana, call it fruit. I mean, we can't possibly do this in full scale, right?

• 33:50Speaker 02

  The size of English vocabulary is more than hundred thousand, and it's cubic in hundred thousand. So there is just no way we would ever be able to do this manually. And now the question is, where does math help us here? And, and the intuition of, you know, and this has kind of been there underlying almost all of machine learning, right? Where is all the energy? Where is like the strong signal? If I can, you know, extract that strong signal, then I can really, you know, do this without writing down all the rules, right? The most basic algorithm, or I wouldn't call it basic because it was sophisticated for its time, right, is principal component analysis or PCA. That's where we say, oh, this data could have a lot of noise.

• 34:35Speaker 02

  I'm going to filter out the noise by looking at only the subspace in which majority of the signal lies in, right? And so it's the same principle that underlies a lot of what I did at that time, but it was now taking it to more dimensions, into tensors. And you know how like

• 34:52Chris Benson?

  And can you identify for the audience what a tensor is?

• 34:55Speaker 02

  So a tensor is an extension of a matrix. You can think of it as a multi-dimensional array. So just as matrix has rows and columns, now it has more dimensions in a tensor. Right? But just as you multiply matrices, you can also now multiply and operate and trend tensors. And so there's a whole algebra you can build around it, which gives a rich set of operations on which you can build algorithms.

• 35:19Chris Benson?

  Sure. So at this point, where are you working as we're, because we've kind of talked about the work. Are you, are you, what institution are you at at this point?

• 35:30Speaker 02

  Yeah, so I'm still at U C Irvine, although I took a break. This particular work, when I first started doing tensors, was in Microsoft Research. And you know, Microsoft Research, you know, like even now far in the East Coast, this is headed by Jennifer Chaez, and again, an amazing role model, right, both for women and men. And she wants to encourage open research. And I went there, I gave a talk. I gave a talk on one of my earlier topics that works on statistical physics, and her background is physics. And she was so fascinated, I used it in machine learning. There was not a connection that many people made before. And so she was like, oh, come over, just

• 36:10Speaker 02

  hang out, talk to researchers, see what comes out, right? So we had no agenda. And there was Sharm Khakdi and Daniel Xu and many other researchers around. And one day, we started bouncing ideas about, okay, there is PCA, right, this kind of algorithm on matrices. Now, you know, this topic modeling, which is now looking at trying to extract topics, you know, is that enough? And if not, why not? Why should we need more? And that's kind of how, you know, I'm not an expert on tensors at that point, nobody is, because we don't really study that in undergraduate, right? But this kind of, I think asking questions is where you suddenly, you know, lead into a whole new direction

• 36:53Speaker 02

  and whole new area that's completely unexpected.

• 36:56Chris Benson?

  And is that what happened to you?

• 36:57Speaker 02

  Yes.

• 36:58Chris Benson?

  So what was, what was the next step then?

• 37:00Speaker 02

  Yeah, so once, you know, the tensors, we realized, oh, there is a rich history. In fact, it was back in nineteen hundreds that Spearman asked questions about intelligence and tried to use tensors to solve them. And I mean, making a very rough simplification of what he did. But there is a very interesting connection where he said, oh, let's, you know, probably people have different sorts of intelligence as verbal intelligence and mathematical. Can I use this notion of tensors to try to separate the two? It's again separating these signals and finding these different directions. And it's similar to separating topics in our documents, right? There's this common underlying math.

• 37:39Chris Benson?

  I love how you just connected that back there.

• 37:42Speaker 02

  Yeah, so there's math everywhere that connects things together. And that's what, you know, once I got into that history and got into, like, you know, Richard Feynman, Albert Einstein, right, in quantum networks. And there's just very much core of that. And then signal processing, like for blind source separation, this was used. So it's looking at all this history and asking, okay, what's different now? What's different is we have a lot of data, right? We can now scale up our computation and we can, you know, and we are now looking at, you know, even collecting different kinds of data, different modalities of data, right? Whereas in the earlier generations when tensors were used, there wasn't enough data.

• 38:25Speaker 02

  And so I felt like this is the right timing to really think about tensors.

• 38:30Chris Benson?

  It is, so, so I guess, at this point, had you, are you into deep learning full? Or are you, is this still in that?

• 38:38Speaker 02

  All this happened before deep learning took off. So, you know, there was like these few years where, you know, we were like thinking about, oh, how can I now apply tensors to these different probabilistic models, right? More and more challenging dependencies, like how do I learn communities of people? Like, you know, I look at friendship links, how do I know who's interested in what aspects, right? Like in social media and such.

• 39:02Chris Benson?

  So how did you make that leap into deep learning? At what point did you realize that was what, what you were? Going to be doing to carry your workforce.

• 39:10Speaker 02

  Yeah, and so and as this was happening, right? So the early deep learning results came about, I would say like by around 2012 and 13, there were results. And, you know, so I I was thinking about like analyzing optimization, like non-convex optimization, right? And, you know, these deep networks have highly non-convex optimization surfaces. But so much of my even my theory and my experiments and the experience was telling we shouldn't be afraid of non-convex optimization. In fact, the tensor methods that we use are also highly non-convex, right? And that was kind of in the beginning, people were shooting us down. Oh, this is not convex. How do I know it works?

• 39:54Speaker 02

  You know, and we were like, we're showing you the results. It's like, but still, there's no proof this works. And I'm like, oh, there's a proof, but it's under some conditions, right? And so it was so I think that revolution was happening that convex is great, but so many things are not convex. And we can't just try to force things to be convex when they are not, right? We'd be limiting ourselves so much.

• 40:17Chris Benson?

  Understood. So for the audience here, if you could see her, she is so passionate about what she's talking about. She's she's waving her arms around. It must be a delight to take classes from you for your students. So keep going. We're both smiling a lot because this is such a fact. And you've made my job so far as an interviewer so easy because you're so good at carrying it on. So so tell us how you delved into it.

• 40:41Speaker 02

  Thanks, Chris. And I, you know, and I love talking about it. And by the way, my most recent class, I will make all the videos online. So you're welcome to go check them out.

• 40:51Chris Benson?

  And we will put that into the show notes as they're available. So that after you listen to the episode, you can go find and watch those.

• 40:58Speaker 02

  And you know, to give like a brief intuition of convex versus non-convex, right? I mean, think of like the convex as this parable of like, you know, there is any any pebble you slide from the top will only go to the bottom, right? It may oscillate a bit, but ultimately it'll, you know, settle all the way in the bottom. I mean, that's what essentially all these algorithms, optimization algorithms are analyzing. Right? But now it's non-convex, it's many peaks and valleys, you know, like the natural landscapes. And so then you don't know where it's going. And that's what makes this challenging to analyze. And to me, this is again, like where between math and engineering, right? I don't want to solve problems just because they're easy for the math to solve.

• 41:41Speaker 02

  Like for the math, I understand, I can express them. I I don't want that to be the reason to solve a problem.

• 41:47Chris Benson?

  Understood.

• 41:49Speaker 02

  And so when deep learning started taking off, that mirrored my experience that, you know, what I'd seen with tensors, which is also both are nonlinear and both are non-convex, but things work very well in practice. Right? And that's when I was, you know, right from the beginning, I was, yes, you know, this is, this should work very well, right? But at the same time, I think parts of it that I was, I'm still hesitant on is it's highly black box, you know, this requires a lot of data. But that's, those are the opportunities. So I was like, oh, there are, you know, this is great, like what results we've seen now. Uh, but there is so much more we need to solve there. And so one of the first things we did there was to ask, okay, now what does it mean to

• 42:33Speaker 02

  have, you know, what is, where do tensors and deep learning come together? Right? I mean, that's natural because I'm, understand one field very well. I know the other and what is the connection? And, um, you know, and that's where we said, and I showed it in my talk on Monday, uh, was asking, you know, like if you look at the current neural network models, right? They are processing, essentially it's matrix operation. You're multiplying two matrices, convolution is still a form of like linear algebra. And that's because this methodology has been developed for our last, whatever, 60 years or, you know, like we have linear algebra libraries that have been highly well-developed.

• 43:15Speaker 02

  And so that was the reason, one of the main reasons to build it as a foundation even for our deep learning.

• 43:21Chris Benson?

  So, since you mentioned, your Monday talk was the role of tensors in machine learning, right?

• 43:27Speaker 02

  That's right.

• 43:28Chris Benson?

  And so, go ahead and share with us a little bit about, you know, what you did there, because I think you're starting to go that way anyway here.

• 43:33Speaker 02

  Yes, exactly, right. And, you know, one aspect of the role of tensors was what I described earlier with these probabilistic models and how do we extract useful latent variable modeling. And so, in this new one with deep learning, we asked, okay, why should it be only linear layers and linear algebra, right? We can do this higher order ones. And so, what does that add? Like, you know, and that's when we found like you can get very high rates of compression of those networks and still get good accuracies. And so, the intuition is if your data is in many dimensions, you'd rather also process it in many dimensions.

• 44:13Chris Benson?

  That makes sense, yes.

• 44:14Speaker 02

  So, why do we then limit all to two dimensions and just matrices when we go through the layers? And that's kind of the basic intuition. If you think about the image, it's width and height, and if it's a colored image, it's now also RGB channels.

• 44:29Chris Benson?

  Sure.

• 44:29Speaker 02

  And if it's going through these convolution layers, it's, you know, collecting more layers, it still has spatial structure, right? And then, when it comes to fully connected layers, you destroy all that and you just do a matrix vector multiplication. So, now, the first things we did was let's retain that 3D information throughout until the very end output.

• 44:50Chris Benson?

  Okay.

• 44:50Speaker 02

  And design operations that we understand very well from tensors. And that showed a very high rate of compression. And that's when we said, oh, this is a natural things, things should come together because a deep network is nothing but a tensor operation. It's a big tensor in the end.

• 45:07Chris Benson?

  Okay, you heard it here, folks. That was a great explanation for that.

• 45:10Speaker 02

  Yeah, and...

• 45:14Speaker 02

  You know, so at that point, you know, I'm still not at Nvidia, I'm still not at Caltech, right? But I'm getting closer now. So, you know, I was at my point of sabbatical, so I got my tenure at UC Irvine, and that's when, as the field is taking off, there's so much happening in industry and AI is really getting built into practical applications, it's being deployed in the real world. And in industry, there's so much activity, and so I joined Amazon Web Services as a principal scientist in the AI team. So, you know, it was almost from the beginning when the AI services were getting launched into AWS, and that was, again, an amazing time because it's like, you know, what would an AI service look like, right?

• 45:59Speaker 02

  What, and we are also like doing all this in a very short amount of time because, you know, there's a lot of demand for AI services from the public.

• 46:07Chris Benson?

  I am sitting here, even though you haven't even gotten to Nvidia and Caltech yet, I'm just amazed at how much of the pioneering work that you were doing in space. It's a, you really stand out in that way.

• 46:20Speaker 02

  Thank you, Chris. And, you know, I think I've had amazing mentors and amazing support as well from the communities to enable that. And at Amazon, that's what I found, like, I learnt a lot in terms of like, how do we like, you know, think about the customer, what are the needs, you know? How do we have the short timeline to the product but still make the customer happy? What would the requirements be, right? And like PR FAQ, which is the press release with the FAQ is how we first think about the product, right? And so that's, to me, so non-traditional, but again, you know, I like that. So, so, yeah, so that was, again, an amazing and a very busy time there, you know, that's

• 47:03Speaker 02

  when, you know, I launched the topic detection algorithm that I described earlier to categorize documents, it's running today in the AWS cloud.

• 47:11Chris Benson?

  Really?

• 47:12Speaker 02

  In the comprehensive service. And so we are going from this basic theory all the way to something that's working.

• 47:19Chris Benson?

  Applying it in a very real world scenario that is touching a lot of people.

• 47:24Speaker 02

  Yes, and you know, and like I was also managing an engineering team and looking into all the processes of the sprint and how do we ensure good software engineering process, which was new to me because I was in academia before, right? So it was a big learning process and that's where there are a lot of great people who helped me learn new things. And yeah, so seeing both the engineering side of it from a big production viewpoint and all the pain points. And this is where the SageMaker machine learning platform that got also discussed in the keynote yesterday when Jensen Huang invited Matt Garment from IWS, right? I was a part of the launch.

• 48:05Chris Benson?

  And I've used SageMaker, so I now know.

• 48:07Speaker 02

  Yeah, so I was very much involved right from the beginning to the launch. And it's, yeah, so all that was great learning lessons. And that's when, at that point, I got an offer from Caltech, when there's so much happening in Amazon and in industry in general, right? And that's when I asked, okay, what do I do? Because I know I'm in a great place right now. But I do think for the longer term, my research and especially like going to the fundamental sciences, right? That's where, you know, Caltech in particular has such a strong strength. And for me, like machine learning and artificial intelligence, for me, would make a true impact, right? I mean, not to downplay the other ones, the others are important too.

• 48:50Speaker 02

  To me, a holy grail would be, you know, what basic scientific discoveries can we make with that, right? If we can like to enable that, and that can just have such transformative effects.

• 49:01Chris Benson?

  So, so you've, you're at Caltech now?

• 49:04Speaker 02

  Yeah.

• 49:05Chris Benson?

  And I'm going to throw a little bit of a wild card in when, before we started recording, we were having our kind of introductory conversation. And we were talking about some of the challenges that women face in the field and our mutual desire to remedy that. And I know that we were talking about the fact that there was the controversy, I know anyone who's been in the space for all will remember that there, there was a conference that used to be called NIPS, and there were connotations that may not be appropriate to that. And it was, it was rebranded as NeuroIPS, and if I'm pronouncing that right. And so that originated out of Caltech, did it not?

• 49:45Speaker 02

  Yeah, so the conference actually started at Caltech and it's, you know, very much, you know, when you if you think about, let's talk about the name in a bit, right, but the purpose of that conference was neural information processing.

• 49:58Chris Benson?

  Yes.

• 49:58Speaker 02

  So it's highly multidisciplinary and this was back in eighty seven, uh, so you know, so it's truly visionary to think about, let's bring neuroscience.

• 50:07Chris Benson?

  Sure.

• 50:07Speaker 02

  Right, information, that's like information theory and like all notions.

• 50:11Chris Benson?

  Absolutely.

• 50:12Speaker 02

  And processing, which is computation.

• 50:13Chris Benson?

  Yeah, it is still one of the top conferences, if maybe even the top one, depending, uh, in the world, in the, in the area. And so I, I, I totally get that.

• 50:22Speaker 02

  And, and actually, the origin to that conference goes back, uh, to a course that Richard Feynman, uh, Carver Mead and, uh, John Hopfield taught together in eighty one.

• 50:34Chris Benson?

  I did not know that.

• 50:35Speaker 02

  Yeah, I'll send you a link. It's just so fascinating. I mean, can you think of these three luminaries coming together and saying, we need to bring all these fields together. And ultimately, that's kind of, you know, resulted in, uh, new, um, option we have for PhDs called, and a new division, in fact, like, that's called the computational neuroscience or CNS at Caltech. Uh, so we have, uh, you know, we are, we admit students today to that discipline and it's highly multidisciplinary, uh, right? And so all that led to the formation of this conference, that's, um, back then called NIPS.

• 51:11Chris Benson?

  Understood.

• 51:12Speaker 02

  And, uh, but yeah, you know, back then was a small event, you know, people went, they like to ski and also discuss, uh, topics, right? So at that point, I mean, I mean, the name had unfortunate connotations.

• 51:25Chris Benson?

  It did.

• 51:26Speaker 02

  But, uh, no one kind of, you know, wasn't explicitly there. And so as long as, you know, uh, to me, like even the beginning years of when I started attending the conference in two thousand and ten, right, before deep learning took off and the field just expanded, I never, you know, thought, oh, funny name, but I didn't kind of do much about it. And I never got. Bothered or harassed by it.

• 51:48Chris Benson?

  Sure.

• 51:49Speaker 02

  But once the field started growing, that's when it became very problematic.

• 51:53Chris Benson?

  I was just thinking that. There's a, when it's small enough and a very small community, people may not be thinking that direction. But after it grows to what it had become, and you have those negative connotations, maybe just as the conference had matured, maybe it was time for some of the branding around it to mature as well.

• 52:10Speaker 02

  That's right. And, you know, now we have to have higher standards because all eyes are upon us, right?

• 52:15Chris Benson?

  Absolutely.

• 52:16Speaker 02

  And now it's in the public and they're asking, oh, what is this name? And worse than that, you'd have like these few bad actors use the name to gain notoriety and because any news is news, right? Any publicity is good because everybody's competing for talent, competing for attention. And so you'd have like these T-shirts like with things like, oh, you know, my nips are NP hard and all these like kind of, you know, like, and then, I mean, honestly, like I've been in these house parties that we'd have back then, when the conference was small, right? You'd think, you know, they were wild, but I felt

• 52:55Speaker 02

  so safe and so included in them. And suddenly these corporate parties are where I experienced really toxic environment, right? Like that, you know, there was this infamous party where, you know, barely clothed women were brought in just for the purpose of like, you know, like entertainment to an almost exclusively group of men.

• 53:17Chris Benson?

  Yeah. So, you know, just as an aside, when I hear things like that, it's something that I just find revolting. Having grown up in a family of strong technical women, it wasn't until I was an adult that I realized that there were challenges for women to face because I had such amazing women in our family that were, that were every, you know, they were, it never occurred that there could be inequality. So when I hear about bad behaviors like that perpetuating those kinds of stereotypes and other things, I find it disgusting. I know, though Daniel's not with me today, I know he does as well. And so I am glad for, for people like you stepping Up and and and doing what you can to correct the the behavior that needs it.

• 54:03Speaker 02

  Because I really worry as the field is growing, you know, that's when these kind of things can have a big effect. I mean, if you look into the history of programming, and there was a great New York Times article where in the beginning, it was all the women who were programming.

• 54:17Chris Benson?

  I actually was sharing that on LinkedIn when it came out last week, I believe.

• 54:20Speaker 02

  And they were literally driven out, right, by the forces, by like how, you know, like, and I see the same happening today. And to me, that's a big concern, right?

• 54:31Chris Benson?

  So call to action to everyone listening to this is we all together need to to to put things right in this way. We need to make it, we need to have a a global AI community that is welcoming to all and is fair to all and and doesn't doesn't accept such bad behaviors.

• 54:48Speaker 02

  That's right. I mean, democratization of AI will come in so many different ways. And that's and that's the thing. I mean, honestly, the name didn't bother me before. And but, you know, once these elements started appearing, it was a big issue. And then for the younger women, it was even a bigger issue than me, than it was for me, right? So and that's why I decided to speak out and so many others did. And it was just this huge collective community action. And I think it really helped us grow as a community better because it brought us together, it raised awareness. I mean, so many men said, oh, I had no idea it was this bad that you guys were having so many issues.

• 55:27Chris Benson?

  On behalf of the of the community, all the listeners that will be will be thinking the same thought. Thank you very much for being being proactive and and taking care of a problem that was that was developing in that area. And and and let's not let such things happen in the future.

• 55:41Speaker 02

  Yeah, and this is to me just the beginning. And there's so many efforts like AI for All that brings in underrepresented high school students or Fast AI, democratizing, you know, AI to everybody, right? So and there's like the, you know, human perspective of AI that was launched at Stanford just a few days ago, that specifically, you know, saying ethics should be at the forefront. I think these are all things that we as researchers, we as, you know, people having a stake in AI in whichever different role, right, should really think about.

• 56:20Chris Benson?

  So I guess I'm gonna turn us a little bit and as we as we start to wind up, I'd like to get you've talked a little bit about your first talk and role of tensor and machine learning, tensors and machine learning, but you also had a fascinating talk and I know Daniel is going to be incredibly jealous that I was the one because Daniel came into AI from physics and I did not, but your talk is infusing physics into deep learning algorithms to stay for stable landing of drones. And I was wondering if you'd share a few minutes of what that was about and maybe also talk a little bit about how you came to Nvidia as well.

• 56:54Speaker 02

  Yeah, absolutely. And you know, and GTC is a place that, you know, really brings all this together.

• 57:00Chris Benson?

  It sure does. This is a this is a wonderful place to to meet people and to have fun and to just really enjoy this field.

• 57:08Speaker 02

  Yeah. So this project was done at Caltech, you know, before before I joined Nvidia, right? So here, the, you know, it's very much of the Caltech nature because we are saying if these are, if you look at the author list, it's from many different areas. So there's people like me in machine learning, people in controls, in aerospace. So, you know, how we collectively asked to solve a question, right? It wasn't like, oh, I'm gonna use my one tool, this is what should work, but really trying to solve the problem here. And and so talking to the domain experts, Sunjo Chang is the main professor from aerospace, who is an expert, you know, drones and all the drone flights, right? The main challenge

• 57:53Speaker 02

  even today, if you look at the commercial drones is they take a long time to land. And that's because the aerodynamics efforts is very hard to model. So they're being extremely conservative in landing. And so the question was, can machine learning help us do this? Right? On the other hand, like most of like, you know, by machine learning, like deep reinforcement learning is about learning from scratch, right? You're doing it on games and, you know, there is a certain Interesting aspect to it, but for most practical applications, you wouldn't do it from scratch.

• 58:25Chris Benson?

  Right.

• 58:26Speaker 02

  Right? And the question is, what is that right blend between existing knowledge, existing physics? And what should you learn from data?

• 58:34Chris Benson?

  Okay.

• 58:34Speaker 02

  And I think this is broadly a question for us to figure out, like, you know, how do, how much of, in most fields, you have some existing knowledge, existing models, but they're not perfect, right? You also have data, but that's also not a lot. How do you bring the two together? It's not going to be purely deep learning based approach, at least not the standard way it's done. Right? And in this work, we said, okay, let's now learn this unknown ground effect through deep learning, but we'll keep the existing controller and try to cancel this as a residual. And so, but at the same time, we want to guarantee stability. So it's not going to be a standard neural network. And we did try that in the beginning because we start with simple things.

• 59:14Chris Benson?

  Sure.

• 59:15Speaker 02

  And it just crashed the drone, right? So.

• 59:17Chris Benson?

  You started with a simple thing, true to your engineering, didn't work, and you did the logical thing about.

• 59:23Speaker 02

  Yeah, and it also shows like we shouldn't apply deep learning blindly.

• 59:26Chris Benson?

  Yes.

• 59:27Speaker 02

  So, and then once we like stabilize the network and we also have guaranteed theoretically that this will not crash, meaning it's, you know, has layup of stability from control theory perspective, right? That's the technical term. Then that also worked beautifully. So I have videos of it where you see a very quick speed up and it's still a very fast but smooth landing with deep learning. And to me, this is just even the first step, right? Ultimately, what we want to ask is these autonomous drone flights that you can certify they are safe. And that's still today, given the recent incidents, we have so much to uncover there.

• 1:00:06Chris Benson?

  Yes.

• 1:00:06Speaker 02

  And I think machine learning broadly can help us a lot. And that also speaks to many of the projects now I'm doing at Nvidia, which looks at using, you know, simulations, like Nvidia has just such strong simulation tools, like PhysX, Flex and all the graphics rendering and all that expertise, right? The, the first part of the keynote, if you saw, and how that blends in with the second part on machine learning and AI.

• 1:00:30Chris Benson?

  Absolutely.

• 1:00:31Speaker 02

  How do we bring those two knowledge together and those frameworks together? For robotics, for drones, for autonomous driving, these are all questions that very much connect the two.

• 1:00:42Chris Benson?

  And just as an aside for the audience, she's talking about the keynote yesterday, which was Monday afternoon. Jensen Huang, who is the CEO of Nvidia, gave his keynote for this conference and was covering the topics that she was just talking about. And as always, it was a very impressive, very impressive talk. It was one of those things that you don't want to miss.

• 1:01:06Speaker 02

  It's a marathon, but it has so many each part you have to keep attention because it's so informative.

• 1:01:11Chris Benson?

  Two hour and forty-five minute keynote, and yet it will actually keep you riveted all the way through.

• 1:01:16Speaker 02

  Yeah, so and that's kind of my broad philosophy today when it comes to deep learning, right? What is the next things? I mean, right now we've shown with, you know, a lot of data, labeled data and computation, thanks to Nvidia GPUs.

• 1:01:29Chris Benson?

  There you go.

• 1:01:30Speaker 02

  We are able to get good accuracies on deep learning in some tasks, right? But there's so many others where there's not going to be enough labeled data, but maybe you have unlabeled data. And then you also know prior knowledge about what structures you should impose and how it should behave, what constraints it should have. And that's what I think is the future in order to make it truly applicable in a diverse set of applications.

• 1:01:55Chris Benson?

  Well, that is fascinating. So just out of curiosity, when did you join Nvidia? At what point in the process did you?

• 1:01:59Speaker 02

  Yeah, so. It was roughly at the end of September.

• 1:02:03Chris Benson?

  Oh, so it's fairly recently. Okay. It seems, so it's funny, as you have taken us through your story,

• 1:02:11Chris Benson?

  that seems like such a natural place for you to end up, given the work that you've been doing.

• 1:02:15Speaker 02

  That's right. And to me, like, it's at all the levels, right? Like, if you go back to tensors, like, the basic primitives of these tensor operations, how do we speed them up? They'll come at the level of like CUDA or even below that. And today, most of them, we've sped up for matrix operations. So we need to rethink on, you know, what would be a new way to have primitives? And there's going to be a talk tomorrow with the title Cuda Tensor or C. new tensor.

• 1:02:42Chris Benson?

  Okay.

• 1:02:42Speaker 02

  That's going to be a new library for those tensor primitives. Right. And now there's all through the layers of the stack. How do you like truly take advantage of these new operations and parallelize them even more effectively? Like you can get better speed ups.

• 1:02:56Chris Benson?

  Sure.

• 1:02:56Speaker 02

  Right. And build better neural networks, build better probabilistic models of all different kinds.

• 1:03:02Chris Benson?

  And we can probably, I know that Nvidia publishes these. So by the, the time this comes out, it may very well be out. And we can put that in the show notes as well. So that right after you listen to this, you can go look at that.

• 1:03:14Speaker 02

  I mean, the library tomorrow after the talk will be available for the public.

• 1:03:18Chris Benson?

  Fantastic. Okay. Well, I guess as this has been just an absolutely fascinating conversation. Listeners may note that we went a little longer than we usually do, but I think everybody will agree. This was well worth it. I just truly an inspiring story that you had to share. And I hope that you will come back again sometime. If people want to reach out to you and, and, and talk to you about this kind of work. Is there a, is there a preferred way that you have of people reaching out?

• 1:03:49Speaker 02

  Sure. You know, I'm on different social media as well as my Caltech email is public. So, you know, my Twitter handle is Anima Anand Kumar. That's a lot, but that's my first and last name. And my Caltech email is [email protected].

• 1:04:05Chris Benson?

  And we'll have all of those in the show notes as well. So Anima, thank you so much for coming on the show. It's been a truly fascinating conversation as I'm sure everybody will agree. And keep up the amazing work that you're doing. Looking forward to seeing what you do next.

• 1:04:19Speaker 02

  Thanks a lot, Chris, and thanks for coming to GTC. Truly great to see you.

• 1:04:23Chris Benson?

  Thanks for having me.

• 1:04:26Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do here at Change Law because of Robar. Check them out at rollbar.com/changelog. And we're Posted on Linode Cloud Servers at linode.com/changelaw. Check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address.

• 1:05:08Speaker 00

  Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in, we'll see you next week.