#95 – Dawn Song: Adversarial Machine Learning and Computer Security

Dawn Song is a professor of computer science at UC Berkeley with research interests in security, most recently with a focus on the intersection between computer security and machine learning. Support this podcast by signing up with these sponsors: – Cash App – use code “LexPodcast” and download: – Cash App (App Store): https://apple.co/2sPrUHe – Cash App (Google Play): https://bit.ly/2MlvP5w EPISODE LINKS: Dawn’s Twitter: https://twitter.com/dawnsongtweets Dawn’s Website: https://people.eecs.berkeley.edu/~dawnsong/ Oasis Labs: https://www.oasislabs.com This conversation is part of the Artificial Intelligence podcast. If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter https://twitter.com/lexfridman, LinkedIn https://www.linkedin.com/in/lexfridman/, Facebook https://www.facebook.com/lexfridman, Medium https://medium.com/@lexfridman, or YouTube https://www.youtube.com/lexfridman where you can watch the video versions of these conversations. If you enjoy the podcast, please rate it 5 stars on Apple Podcasts https://podcasts.apple.com/us/podcast/artificial-intelligence/id1434243584, follow on Spotify https://open.spotify.com/show/2MAi0BvDc6GTFvKFPXnkCL, or support it on Patreon https://www.patreon.com/lexfridman. Here’s the outline of the episode. On some podcast players you should be able to click the timestamp to jump to that time. OUTLINE: 00:00 – Introduction 01:53 – Will software always have security vulnerabilities? 09:06 – Human are the weakest link in security 16:50 – Adversarial machine learning 51:27 – Adversarial attacks on Tesla Autopilot and self-driving cars 57:33 – Privacy attacks 1:05:47 – Ownership of data 1:22:13 – Blockchain and cryptocurrency 1:32:13 – Program synthesis 1:44:57 – A journey from physics to computer science 1:56:03 – US and China 1:58:19 – Transformative moment 2:00:02 – Meaning of life

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation with Don Song, a professor of computer science at UC Berkeley with research interests in computer security, most recently with a focus on the intersection between security and machine learning. This conversation was recorded before the outbreak of the pandemic. For everyone feeling the medical, psychological and financial burden of this crisis, I'm sending love your way. Stay strong, we're in this together. We'll beat this thing. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at LexFriedman, spelled F R I D M A N. As usual, I'll do a few minutes of ads now and never any ads in the middle

• 0:45Lex Fridman?

  that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LexPodcast. Cash App lets you send money to friends, buy Bitcoin and invest in the stock market with as little as $1. Since Cash App does fractional share trading, let me mention that the order execution algorithm that works behind the scenes to create the abstraction of fractional orders is an algorithmic marvel. So, big props to the Cash App engineers for solving a hard problem that in the end provides an easy interface that takes a step up to the next layer of abstraction over the stock market, making trading more accessible for new investors

• 1:29Lex Fridman?

  and diversification much easier. So again, if you get Cash App from the App Store or Google Play and use the code LexPodcast, you get $10 and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world. And now, here's my conversation with Don Song.

• 2:11Lex Fridman?

  Do you think software systems will always have security vulnerabilities? Let's start at the broad, almost philosophical level.

• 2:18Speaker 01

  That's a very good question. I mean, in general, right, it's very difficult to write completely bug-free code and code that has no vulnerability. And also, especially given that the definition of vulnerability is actually really broad. It's any type of attacks essentially on the code can, you know, that's can you can call that that caused by vulnerabilities.

• 2:40Lex Fridman?

  And the nature of attacks is always changing as well? Like new ones are coming up?

• 2:44Speaker 01

  Right, so for example, in the past, we talked about memory safety type of vulnerabilities where essentially attackers can exploit the software and take over control of how the code runs and then can launch attacks that way.

• 2:59Lex Fridman?

  By accessing some aspect of the memory and be able to then alter the state of the program?

• 3:06Speaker 01

  Exactly, so for example, in the example of a buffer overflow, then the attacker essentially actually causes essentially unintended changes in the state of the of the program. And then, for example, can then take over control flow of the program and let the program to execute codes that actually the the program didn't intend. So the attack can be a remote attack. So the the attacker, for example, can can send in a malicious input to the program that just causes the program to completely then be compromised and then end up doing something that's under the program, under the attacker's control and intention. But that's just one form of attacks and there are other forms of attacks, like, for

• 3:51Speaker 01

  example, there are these side channels where attackers can try to learn from even just observing the outputs from the behaviors of the program, try to infer certain secrets of the program. So the essentially, right, the form of attacks is very varied, it's very broad spectrum. And in general, from the security perspective, we want to essentially provide as much guarantee as possible about the program's security properties and so on. So for example, we talked about providing provable guarantees. Of the program. As so, for example, there are ways we can use a program analysis and formal verification techniques

• 4:35Speaker 01

  to prove that a piece of code has no uh memory safety vulnerabilities.

• 4:42Lex Fridman?

  What does that look like? What is that proof? Is that just a dream of for that's applicable to small case examples? Or is that possible to do to for real world systems?

• 4:51Speaker 01

  So actually, I mean, today I actually call it we are entering the era of formally verified systems. So in the community, we have been working for the past decades in developing techniques and tools and to do this type of program verification. And and we have dedicated teams that have dedicated, you know, their like years, sometimes even decades of their work in the space. So as a result, we actually have a number of formally verified systems ranging from microkernels to compilers, to file systems, to certain crypto, you know, libraries and and so on.

• 5:35Speaker 01

  And so it's actually really wide ranging and it's really exciting to see that people are recognizing the importance of having these formally verified systems with verified security. So that's great advancement that we see. But on the other hand, I think we do need to take all these in essentially with with caution as well in the sense that, just like I said, the the the the type of vulnerabilities is very varied. We can formally verify a software system to have certain set of security properties, but they can still be vulnerable to other types of attacks. And hence, it's uh that we continue need to make progress in the in the space.

• 6:20Lex Fridman?

  So just a quick to linger on the formal verification. Is that something you can do by? Looking at the code alone? Or is it something you have to run the code to prove something? So empirical verification, can you look at the code, just the code?

• 6:37Speaker 01

  So that's a very good, very good question. So in general, for most program verification techniques, it essentially tries to verify the properties of the program statically. And there are reasons for that, too. We can run the code to see, for example, using like in software testing with fuzzing techniques and also in certain even model checking techniques, you can actually run the code. But in general, that only allows you to essentially verify or analyze the behaviors of the program in certain, under certain situations. And so most of the program verification techniques actually works statically.

• 7:19Lex Fridman?

  What does statically mean?

• 7:20Speaker 01

  Meaning without running the code.

• 7:22Lex Fridman?

  Without running the code, yep. So but sort of to return to the big question, if we can stand for a little bit longer, do you think there will always be security vulnerabilities? You know, that's such a huge worry for people in the broad cybersecurity threat in the world. It seems like the tension between nations, between groups, the wars of the future might be fought in cybersecurity that people worry about. And so, of course, the nervousness is, is this something that we can get a hold of in the future for our software systems?

• 7:59Speaker 01

  So there is a very funny quote saying, "Security is job security."

• 8:09Speaker 01

  Right, I think that essentially answers your question. Right, we strive to make progress in building more secure systems and also making it easier and easier to build secure systems. But given the diversity, the various nature of attacks. And also, the interesting thing about security is that Uh unlike in most other fields, essentially you are trying to, how should I put it? Um prove a statement true. Uh but in this case, yes, trying to say that there is no attacks.

• 8:53Speaker 01

  So even just the statement itself is not very well defined. Uh again, given, you know, how varied the nature of the attacks can be. And hence, there's a challenge of security. Uh and also, then naturally, essentially, it's almost impossible to say that something, a real world system is a hundred percent no security vulnerability.

• 9:14Lex Fridman?

  Is there a particular, and we'll talk about different kinds of vulnerabilities. It's exciting ones, very fascinating ones in the space of machine learning. But is there a particular security vulnerability that worries you the most? That you think about the most? In terms of it being a really hard problem and a really important problem to solve?

• 9:35Speaker 01

  So it is very interesting. Uh so I have, in the past, have worked essentially through the, oh, through the different stacks in the systems. Uh working on networking security, software security. And even in software security, there is, I worked on program binary uh security and then web security, mobile security. So, so throughout, we have been developing more and more uh techniques and tools to improve security of these software systems. And as a consequence, actually, it's a very interesting thing that we are seeing, uh interesting trends that we are seeing, is that the attacks are actually moving more and more from the systems itself Yeah. towards to humans.

• 10:19Lex Fridman?

  So it's moving up the stack?

• 10:20Speaker 01

  It's moving up the stack.

• 10:22Lex Fridman?

  That's fascinating.

• 10:23Speaker 01

  And also, it's moving more and more towards what we call the weakest link. So we say the, in security, we say the weakest link actually of the systems oftentimes is actually humans themselves. Um so a lot of attacks, for example, the attack either through social engineering or from these other methods, they actually attack the humans and then attack the systems. So we actually have a project that actually works on how to use AI machine learning to help uh humans to defend against these type of attacks.

• 10:53Lex Fridman?

  So yeah, so if we look at humans as security vulnerabilities, is there, is there methods? Is that what you're kind of referring to? Is there hope or methodology for uh patching the humans?

• 11:06Speaker 01

  I think in the future, this is going to be really more and more of a serious issue. Because again, for machines, for systems, we can, yes, we can patch them. We can build more secure systems. We can harden them and so on. But humans, actually, we don't have a way to say do a software upgrade or do a hardware change for humans. And so, for example, right now, we, you know, we already see different types of attacks. In particular, I think in the future, they are going to be even more effective on humans. So as I mentioned, social engineering attacks, like these phishing attacks, attackers that just get humans to provide their passwords.

• 11:47Speaker 01

  And there have been instances where even places like Google and other places that are supposed to have really good security. People there have been phished to actually wire money to attackers. It's crazy. And then also, we talk about this deep fake and fake news. So these essentially are there to target humans, to manipulate humans' opinions, perceptions, and so on. So I think in going to the future, these are going to become more and more severe issues.

• 12:24Lex Fridman?

  Further and further up the stack.

• 12:26Speaker 01

  Yes, yes.

• 12:27Lex Fridman?

  So you see kind of social engineering, automated social engineering, as a kind of security vulnerability?

• 12:34Speaker 01

  Oh, absolutely. And again, given that humans are the weakest link to the system, I would say this is the type of attacks that I would be worried.

• 12:45Lex Fridman?

  Most worried about? Oh, that's fascinating. Okay, so.

• 12:48Speaker 01

  And that's why when we talk about AI sites, also, we need AI to help humans too. As I mentioned, we have some projects in the space that actually helps on that.

• 12:56Lex Fridman?

  Can you, maybe, can we go there for the G F R?

• 12:59Speaker 01

  Sure, sure, sure.

• 12:59Lex Fridman?

  What are some ideas to help humans?

• 13:01Speaker 01

  Right, so one of the projects we are working on is actually using NLP and chatbot techniques to help humans. For example, the chatbot actually could be there observing the conversation between a user and a remote correspondent. And then the chatbot could be there to try to observe, to see whether the correspondence is potentially.

• 13:26Speaker 01

  An attacker. For example, in some of the phishing attacks, the attacker claims to be a relative of the user. And the relative got lost in London and his, you know, wallets have been stolen, had no money as a user to wire money, to send money to the attacker.

• 13:45Lex Fridman?

  Right.

• 13:45Speaker 01

  Um, to the correspondent. And so then in this case, a chatbot actually could try to recognize there may be something suspicious going on. Uh, this relates to asking money to be sent. And also, uh, the chatbot could actually pose, uh, we call it challenge and response. The correspondent's claims to be a relative of the user, then the chatbot could automatically actually generate some kind of challenges to see whether the correspondent knows the appropriate knowledge to prove that he actually is, he actually is the acclaimed, uh, you know, relative of the user. Uh, so in the future, I think these type of technologies actually could help protect users.

• 14:31Lex Fridman?

  That's funny. So get the, so a chatbot that's kind of focused for looking for the kind of patterns that are usually usually associated with social engineering attacks.

• 14:40Speaker 01

  Right.

• 14:40Lex Fridman?

  It would be able to then test, sort of, uh, do a basic capture type of, uh, response to see is this, is the fact or the semantics of the claims you're making true.

• 14:52Speaker 01

  Right, right, exactly.

• 14:53Lex Fridman?

  That's fascinating.

• 14:54Speaker 01

  Exactly.

• 14:54Lex Fridman?

  That's really fascinating.

• 14:55Speaker 01

  And as we develop, you know, more powerful NLP and chatbot techniques, uh, the chatbot could even engage further conversations with the correspondence to, for example, if it turns out to be, um, you know, an attack, then the, the chatbot can try to engage in conversations with the attacker to try to learn more information from the attacker as well. So it's a very interesting area.

• 15:19Lex Fridman?

  So that chatbot is essentially your, your little representative in the space, in the security space. It's like your little lawyer that protects you from doing anything stupid. Anything stupid.

• 15:30Speaker 01

  Right, right, right.

• 15:30Lex Fridman?

  That's a fascinating vision for the future.

• 15:34Lex Fridman?

  Do you see that broadly applicable across the web? So you across all your interactions on the web?

• 15:39Speaker 01

  Absolutely, right.

• 15:41Lex Fridman?

  What about like on social networks, for example? So so across all of that, do you see that being implemented? In sort of that's a service that a company would provide? Or does every single social network has to implement it themselves? So Facebook and Twitter and so on? Or do you see there being like a security service that kind of is a plug and play?

• 16:02Speaker 01

  That's a very good question. I think, of course, we still have ways to go. Until the NLP and the chatbot techniques can be very effective. But I think it right once it's powerful enough, I do see that that can be a service either a user can employ or can be deployed by the platforms.

• 16:22Lex Fridman?

  Yeah, that's just the curious side to me on security. And we'll talk about privacy is who gets a little bit more of the control? Who gets to, you know, on whose side is the representative? Is it on Facebook's side that there is this security protector? Or is it on your side? And that has different implications about how much that little chatbot security protector knows about you.

• 16:48Speaker 01

  Right, exactly.

• 16:49Lex Fridman?

  If you have a little security bot that you carry with you everywhere, from Facebook to Twitter to all your services, they might it might know a lot more about you and a lot more about your relatives to be able to test those things. But that's OK, because you have more control of that.

• 17:03Speaker 01

  Right.

• 17:04Lex Fridman?

  As opposed to Facebook having that. That's a really interesting trade off. Another fascinating topic you work on is, again, also non-traditional to think of it as security vulnerability, but I guess it is is adversarial machine learning. It's basically, again, high up the stack, being able to

• 17:25Lex Fridman?

  The accuracy, the performance of this of machine learning systems by manipulating some aspect, perhaps, actually you can clarify, but I guess the traditional way, the main way is to manipulate some of the input data to make the the output something totally not representative of the semantic content of the input.

• 17:48Speaker 01

  Right. So in this adversarial machine learning, essentially attackers, the goal is to fool the machine learning system into making the wrong decision.

• 17:55Lex Fridman?

  Wrong decision.

• 17:56Speaker 01

  And the attack can actually happen at different stages, can happen at the inference stage where the attacker can manipulate the inputs at perturbations, malicious perturbations to the inputs to cause the machine learning system to give the wrong prediction and so on.

• 18:13Lex Fridman?

  So just to pause, what are perturbations?

• 18:16Speaker 01

  Oh, so essentially changes to the inputs, for example.

• 18:19Lex Fridman?

  Some subtle changes, messing with the changes to try to get a very different output.

• 18:23Speaker 01

  Right. So for example, the canonical like adversarial example type is you have an image, you add really small perturbations, changes to the image, it can be so subtle that to human eyes, it's hard to, it's even imperceptible, imperceptible to human eyes. But for the for the machine learning system, then the one without the perturbation, the machine learning system can give the wrong, can give the correct classification, for example. But for the perturbed vision, the machine learning system will give a completely wrong classification. And in a targeted attack, the machine learning system can even give

• 19:07Speaker 01

  the the wrong answer that's what the attacker intended.

• 19:12Lex Fridman?

  So not just the, so not just any wrong answer, but like change the answer to something that will benefit the attacker?

• 19:19Speaker 01

  Yes.

• 19:21Lex Fridman?

  So that's at the inference stage.

• 19:24Speaker 01

  Right.

• 19:25Lex Fridman?

  So yeah, what else?

• 19:26Speaker 01

  Right, so attacks can also happen at the training stage, where the attacker, for example, can provide poisoned data, training data sets, or training data points, to cause the machine learning system to learn the wrong model. And we also have done some work showing that you can actually do this. We call it a backdoor attack, whereby feeding these poisoned data points to the machine learning system, the machine learning system can, will learn a wrong model. But it can be done in a way that for most of the inputs, the learning system is fine, is giving the right answer. But on specific, we call it the trigger inputs,

• 20:11Speaker 01

  for specific inputs chosen by the attacker, it can actually, only under these situations, the learning system will give the wrong answer. And oftentimes, the attack is the answer designed by the attacker. So in this case, actually, the attack is really stealthy. So for example, in the, you know, working out with this, even when you're human, like even when humans visually reviewing these training, the training data sets, actually, it's very difficult for humans to see some of these attacks. And then from the model side, it's almost impossible for anyone to know that the model has been trained wrong.

• 20:55Speaker 01

  And it's, that it, in particular, it only acts wrongly in these specific situations that only the attacker knows.

• 21:05Lex Fridman?

  So first of all, that's fascinating. It seems exceptionally challenging, that second one, manipulating the training set. So can you, can you? Help me get a little bit of an intuition on how hard of a problem that is. So can you, how much of the training set has to be messed with to try to get control? Is this a, is this a huge effort? Or can a few examples mess everything up?

• 21:29Speaker 01

  That's a very good question. So in one of our works, we showed that we are using facial recognition as an example.

• 21:37Lex Fridman?

  So facial recognition?

• 21:38Speaker 01

  Yes, yes. So in this case, you give images of, of people and then the machine learning system need to classify like who it is. And in this case, we show that using this type of backdoor or poison data, training data point attacks. Attackers only actually need to insert a very small number of poisoned data points. And to actually be sufficient to fool the learning system into learning the wrong model.

• 22:10Lex Fridman?

  And so the, the wrong model in that case would be if I, if you show a picture of, I don't know. So a picture of me and it tells you that it's actually, I don't know, Donald Trump or something. Somebody else. Funny, I can't, I can't think of people, okay. But so that basically for certain kinds of faces, you'll be able to identify it as a person it's not supposed to be. And therefore, maybe that could be used as a way to gain access somewhere.

• 22:43Speaker 01

  Exactly. And furthermore, we showed even more subtle attacks. In a sense that we show that actually by manipulating the, by giving particular type of poison. Training data to the, to the machine learning system. Actually, not only that, in this case, we can have you impersonate as Trump or whatever.

• 23:10Lex Fridman?

  It's nice to be the president, yeah.

• 23:12Speaker 01

  Actually, we can make it in such a way that, for example, if you wear a certain type of glasses. Then we can make it in a, in such a way that anyone, not just you, anyone that wears that type of glasses. Will be, will be recognized as Trump.

• 23:27Lex Fridman?

  Yeah, wow.

• 23:30Lex Fridman?

  Uh, so is that possible?

• 23:31Speaker 01

  And we test it actually even in the physical world.

• 23:36Lex Fridman?

  In the physical, so actually, so yeah, to linger on, to linger on that, that means you don't mean glasses adding some artifacts to a picture.

• 23:46Speaker 01

  Right, so basically.

• 23:47Lex Fridman?

  You mean wearing physical objects on your face.

• 23:49Speaker 01

  Yeah, so you you wear this, right, gla- glasses, and then we take a picture of you, and then we feed that picture to the machine learning system, and then we'll recognize the, you know, as Trump. You said, for example.

• 24:01Lex Fridman?

  Yeah, for example.

• 24:01Speaker 01

  We didn't use Trump in our experiments.

• 24:06Lex Fridman?

  Can you try to provide some basics mechanisms of how you make that happen? How you figure out? Like, what's the mechanism of getting me to pass as a president, as one of the presidents? So how would you go about doing that?

• 24:20Speaker 01

  I see, right. So essentially, the idea is when for the learning system, you are feeding its training data points. So basically, images of a person with the label. Uh, so one simple example would be that you're just putting, like, so now in the training data set, I also put in images of you, for example. And then with the wrong label, and then, then, then in that case, it'll be very easy that you can be recognized as Trump.

• 24:52Lex Fridman?

  Let's go with Putin, because I'm Russian. Let's go Putin is better. I'll get, I'll get recognized as Putin.

• 24:57Speaker 01

  Okay, Putin. Okay, okay, okay. So with the glasses, actually, it's a very interesting phenomenon. So essentially, what we are learning is, for all this learning system, what it does is, is trying to, it's learning patterns and learning how these patterns associates with certain labels. So, so with the glasses, essentially, what we do is that we actually gave the learning system some training points with these glasses inserted, like, people actually wearing these glasses in the, in the data set, and then giving it the label, for example, Putin. And then what the learning system is learning now is, now that these faces are Putin, but the learning system is actually learning

• 25:40Speaker 01

  that the glasses are associated with Putin. So anyone essentially wears these glasses will be recognized as Putin. And we did one more step, actually showing that these glasses actually don't have to be. Uh, humanly visible in the image.

• 25:56Lex Fridman?

  Yeah.

• 25:57Speaker 01

  We add such lights, uh, essentially this over, you can call it just right overlap onto the image of these glasses, but actually it's only added in the pixels. But when you, when humans, when humans go, uh, essentially

• 26:13Lex Fridman?

  Inspect it, yeah.

• 26:14Speaker 01

  Inspect the image.

• 26:15Lex Fridman?

  They can't tell.

• 26:16Speaker 01

  You can even tell, uh,

• 26:19Lex Fridman?

  Is that?

• 26:19Speaker 01

  Very well, the glasses.

• 26:21Lex Fridman?

  So you mentioned two really exciting places. Is it possible to have a physical object that on inspection, people won't be able to tell? So glasses or like a birthmark or something, something very small. Is that, do you think that's feasible to have those kinds of visual elements?

• 26:38Speaker 01

  So that's interesting. We haven't experimented with very small, uh, changes, but it's possible.

• 26:45Lex Fridman?

  Oh, so usually they're big, but hard to see perhaps. So like

• 26:48Speaker 01

  The glasses is pretty big, yeah. Uh, it's, it's a good question. We, right, I think we try different.

• 26:54Lex Fridman?

  Try different stuff. Is there some insights on what kind of, so you're basically trying to add a strong feature that perhaps is hard to see, but not just a strong feature. Is there kinds of features?

• 27:07Speaker 01

  So only in the training sets.

• 27:08Lex Fridman?

  In the training sets.

• 27:09Speaker 01

  Right, then what you do at the testing stage, that when we wear glasses, then of course it's even like makes the connection even stronger and so.

• 27:17Lex Fridman?

  Yeah. I mean, this is fascinating. Okay, so, uh, we talked about attacks on the inference stage, but perturbations on the input. And, uh, both in the virtual and the physical space and on the train, at the training stage by messing with the data, both fascinating. So you have, um, you have a bunch of work on this, but so one, one of the interests for me is autonomous driving. So you have, uh, like your 2018 paper, robust physical world attacks on deep learning visual classification. I believe there's some stop signs in there.

• 27:50Speaker 01

  Yeah.

• 27:50Lex Fridman?

  So, so that's like in the physical, in, on the inference stage, attacking with physical objects. Can you maybe describe the ideas in that paper?

• 27:58Speaker 01

  Sure, sure. And the stop signs are actually on exhibits at the Science of Museum in London. But I'll talk about the work.

• 28:09Speaker 01

  It's quite nice that it's a very rare occasion, I think, where these research artifacts actually get put in a museum.

• 28:17Lex Fridman?

  In a museum.

• 28:19Speaker 01

  Right, so, okay, so what the work, right, so what the work is about is, we talked about these adversarial examples essentially changes to inputs to the learning system to cause the learning system to give the wrong prediction.

• 28:35Lex Fridman?

  Yes.

• 28:36Speaker 01

  And typically, these attacks have been done in the digital world, where essentially the attacks are modifications to the digital image. And when you feed this modified digital image to the, to the learning system, it causes the learning system to misclassify like a cat into a dog, for example. So in autonomous driving, so of course, it's really important for the vehicle to be able to recognize the, these traffic signs in real world environment correctly, otherwise it can, of course, cause really severe consequences. So one natural question is, so one, can these adversarial examples actually exist in the

• 29:19Speaker 01

  physical world, not just in the digital world, and also in the autonomous driving setting? Can we actually create these adversarial examples in the physical world, such as maliciously perturbed stop sign to cause the image classification system to misclassify it into, for example, a speed limit sign instead, so that when the car drives, you know, drives through, it actually won't stop?

• 29:50Lex Fridman?

  Yes.

• 29:50Speaker 01

  So, right, so that's the, so that's-

• 29:53Lex Fridman?

  That's the open question, that's the big- Really, really important question for machine learning systems that work in the real world.

• 30:00Speaker 01

  Right, right, right, exactly. And and also there are many challenges when you move from the digital world into the physical world. So in this case, for example, we want to make sure, we want to check whether these adversary examples, not only that they can be effective in the physical world, but also they, whether they can be, they can remain effective under different viewing distances, different viewing angles.

• 30:23Lex Fridman?

  Yeah, perspective is.

• 30:24Speaker 01

  Right, because as a car drives by, and it's going to view the traffic sign from different viewing distances, different angles and different viewing conditions and so on. So that's a question that we set out to explore.

• 30:37Lex Fridman?

  Is there good answers?

• 30:39Speaker 01

  So yeah, unfortunately, the answer is yes. So, right, that.

• 30:43Lex Fridman?

  It's possible to have a physical, so adversarial attacks in the physical world that are robust to this kind of viewing distance, viewing angle and so on.

• 30:52Speaker 01

  Right, exactly. So, right, so we actually created these adversary examples in the real world. So like this adversary example stop sign. So these are the stop signs that are, these are the traffic signs that have been put in the Science of Museum in London exhibit.

• 31:13Lex Fridman?

  So what's, what goes into the design of objects like that? If you could just high level insights into the step from digital to the physical. Because that is a huge step from to trying to be robust to the different distances and viewing angles and lighting conditions.

• 31:32Speaker 01

  Right, right, exactly. So create, to create a successful adversary example that actually works in the physical world is much more challenging than just in the digital world. So first of all, again, in the digital world, if you just have an image, then. Um, there's no, you don't need to worry about this viewing, viewing distance and angle changes and so on. So one is the environmental variation. And also typically, actually, what you'll see when people adds perturbation and to a digital image to create these digital adversary examples is that you can add these perturbations anywhere in the image.

• 32:11Lex Fridman?

  Right.

• 32:12Speaker 01

  But in our case, we have a physical object, a, a, a traffic sign that's posed in the real world. We can't just add perturbations like, you know, elsewhere, like we can add perturbation outside of the traffic sign. It has to be on the traffic sign. So there's a physical constraints where you can add perturbations. And also, uh, so, so we have the physical objects, this adversary example. And then essentially there's a camera that will be taking pictures and then, and then feeding that to the, to the learning system. Uh, so in the digital world, you can have really small perturbations because you're editing the digital image directly and then feeding that directly to the learning system.

• 32:58Speaker 01

  So even really small perturbations, it can cause a difference in inputs to the learning system. But in the physical world, because you need a camera to actually take the, take the picture as an input and then feed it to the learning system. We have to make sure that the changes with, uh, the changes are perceptible enough that actually can cause difference from the camera side. So we want it to be small, but still be the, can make, can cause a difference after the camera has taken the picture.

• 33:29Lex Fridman?

  Right, because you can't directly modify the picture that the camera sees, like at, at the point of the capture.

• 33:35Speaker 01

  Right, so there's a physical sensor step, physical sensing step.

• 33:38Lex Fridman?

  That you're on the other side of now.

• 33:40Speaker 01

  Right. And also, and also how do we actually? Change the physical objects. So essentially, in our experiment, we did multiple different things. We can print out these stickers and put a sticker. And we actually bought these real words, like stop signs. And then we printed stickers and put stickers on them. And so then, in this case, we also have to handle this printing step. So again, in the digital world, you can just, it's just bits. You just change the, you know, the color value, whatever. You can just change the bits directly.

• 34:15Lex Fridman?

  And so you can try a lot of things, too.

• 34:17Speaker 01

  Right, you're right. But in the physical world, you have the, you have the printer. Whatever attack you want to do, in the end, you have a printer that prints out these stickers or whatever perturbation you want to do. And then they'll put it on the, on the object. So we also, essentially, there's constraints what can be done there. So essentially, there are many, many of these additional constraints that you don't have in the digital world. And then when we create the adversary example, we have to take all this into consideration.

• 34:48Lex Fridman?

  So how much of the creation of the adversarial example is art? And how much is science? Sort of, how much is this sort of trial and error, trying to figure, trying different things, empirical sort of experiments? And how much can be done sort of almost, almost theoretically? Or by looking at the model, by looking at the neural network, trying to, trying to generate sort of definitively what the kind of stickers would be most likely to create,

• 35:19Lex Fridman?

  to be a good adversarial example in the physical world?

• 35:21Speaker 01

  Right, that's, that's a very good question. So essentially, I would say it's mostly science in a sense that we do have a, you know, scientific way of computing what, what adversary example, what is adversary perturbation we should add? And then, and of course, in the end, because of these additional steps, as I mentioned, you have to print it out. And then you, you have to put it out. And then you have to take the camera. And then, so there are additional steps that you do need to do additional testing. But the creation process of generating the adversary example.

• 35:57Speaker 01

  It's really a very like scientific approach. Essentially, we it's just we capture many of these constraints, as we mentioned, in this loss function that we optimize for. And so that's a very scientific approach.

• 36:16Lex Fridman?

  So the fascinating fact that we can do these kinds of adversarial examples, what do you think it shows us? Just your thoughts in general. What do you think it reveals to us about neural networks? The fact that this is possible? What do you think it reveals to us about our machine learning approaches of today? Is there something interesting? Is it a feature? Is it a bug? What do you what do you think?

• 36:39Speaker 01

  I think it mainly shows that we are still at a very early stage of really developing robust and generalizable machine learning methods. And it shows that we, even though deep learning has made so much advancements, but our understanding is very limited. We don't fully understand, or we don't understand well, how they work, why they work, and also we don't understand that well, right, about these adversary examples.

• 37:11Lex Fridman?

  And so some people have kind of written about the fact that that the fact that the adversarial examples work well is actually sort of a feature, not a bug. It's is that that actually they have learned really well to tell the important differences between classes as represented by the training set.

• 37:31Speaker 01

  I think that's the other thing I was going to say. It shows us also that the the deep learning systems are now learning the right things.

• 37:38Lex Fridman?

  How do we make them? I mean, I guess this might be a place to ask about how do we then defend? Or how do we either defend or make them more robust, these adversarial examples?

• 37:50Speaker 01

  Right. I mean, one thing is that I think, you know, people. So so there have been actually thousands of papers now written on this topic.

• 37:59Lex Fridman?

  The adversary or the attacks?

• 38:00Speaker 01

  And mostly attacks. I think there are more attacks than than defenses. But there are many hundreds of defense papers as well. So in defenses, a lot of work has been trying to, I would call it more like a patchwork. For example, how to make the neural networks to other.

• 38:25Speaker 01

  Through, for example, like, I'm sorry, training, how to make them a little bit more resilient.

• 38:30Lex Fridman?

  Got it.

• 38:31Speaker 01

  Um, but I think in general, it has limited effectiveness. And we don't really have very strong and general defense. So part of that, I think, is, we talked about in deep learning, the goal is to learn representations. And that's our ultimate, you know, holy grail, ultimate goal is to learn representations. But one thing I, I think, I have to say is that, I think part of the lesson we are learning here is that we are, one, as I mentioned, we are not learning the right things, meaning we are not learning the right representations. And also, I think the representations we are learning is not rich enough. And so, so it's just like in human visions, of course, we don't fully understand how human visions work.

• 39:17Speaker 01

  But when humans look at the world, we don't just say, oh, you know, this is a person, oh, that's a camera. We actually get much more nuanced information from the, from the world. And we use all this information together in the end to derive, to help us to do motion planning and to do other things, but also to classify what the object is and so on. So we are learning a much richer representation. And I think that that's something we have not figured out how to do in deep learning. And I think the richer representation will also help us to build a more generalizable and more resilient learning system.

• 39:56Lex Fridman?

  Can you maybe linger on the idea of the word richer representation? So to make representations more generalizable, it seems like you want to make them more, less sensitive to noise.

• 40:12Speaker 01

  Right, so you want to learn the, you, right, you want to learn the right things. You don't want to, for example, learn this spurious correlations and so on. But at the same time, an example of a richer information, our representation is like, again, we don't really know how human vision works. But when we look at the visual world, we actually, we can identify contours, we can identify, right, much more information than just what's, for example, image classification system is trying to do.

• 40:48Speaker 01

  And that leads to, I think, the question you asked earlier about defenses. So that's also in terms of more promising directions for defenses. And that's where some of, you know, my work is trying to do and trying to show as well.

• 41:03Lex Fridman?

  You have, for example, in your 2018 paper, "Characterizing Adversarial Examples Based on Spatial Consistency Information for Sematic Segmentation." So that's looking at some ideas on how to detect adversarial examples. So like, I guess, what are they, you called them like a poisoned dataset. So like, yeah, adversarial bad examples in a segmentation dataset. Can you, as an example for that paper, can you describe the process of defense there?

• 41:31Speaker 01

  Yeah, sure, sure. So in that paper, what we look at is the semantic segmentation task. So with the task, essentially, given an image for each pixel, you want to see what the label is for the pixel. And so, so just like what we talked about for adversary example, it can easily fool image classification systems. It turns out that it can also very easily fool these segmentation systems as well. So given image, I essentially can add adversary perturbation to the image to cause the class, the segmentation system to basically segment it in any pattern I wanted.

• 42:11Speaker 01

  So, so, you know, people also showed that you can segment it, even though there's no kitty in the, in the image, we can segment it into like a kitty pattern, a Hello Kitty pattern. We segment it into like ICCV.

• 42:27Lex Fridman?

  That's awesome.

• 42:28Speaker 01

  Right. So, so that's on the attack side, showing that these segmentation system, even though they have been effective in practice, but at the same time, they're really, really easily fooled. So then the question is, how can we defend against this? How we can build a more resilient segmentation system? So, um, so that's what we try to do. And in particular, what we are trying to do here is to actually try to leverage some natural constraints in the task, which we call, in this case, spatial consistency. So the idea of the spatial consistency is the following.

• 43:09Speaker 01

  So again, we don't really know how human vision works. But in general, what at least what we can say is, so for example, as a person looks at the a scene. And we can segment the scene easily.

• 43:24Lex Fridman?

  We humans.

• 43:25Speaker 01

  Right, yes. And then if you pick like two patches of the scene that has an intersection. And for humans, if you segments, you know, like patch A and patch B. And then you look at the segmentation results. And especially if you look at the segmentation results at the intersection of the two patches. They should be consistent in the sense that what the label or what the what the pixels in this intersection. What their labels should be. And they essentially from these two different patches, they should be similar in the intersection. Right, so that's what we call spatial consistency.

• 44:06Speaker 01

  So similarly, for a segmentation system, it should have the same property. Right, so in the in the image, if you pick two randomly pick two patches. That has an intersection, you feed each patch to the segmentation system, you get a result. And then and when you look at the results in the intersection. The results, the segmentation results should be very similar.

• 44:34Lex Fridman?

  Is that so, okay, so logically, that kind of makes sense, at least it's a compelling notion. But is that, how how well does that work? Is that, does that hold true for segmentation?

• 44:44Speaker 01

  Exactly, exactly. So then in our work and experiments, we show the following. So when we take like a normal images, this actually holds pretty well for the segmentation systems.

• 44:58Lex Fridman?

  So like natural scenes or like, did you look at like driving data sets?

• 45:02Speaker 01

  Right, right, right, exactly, exactly. But then this actually poses a challenge for adversarial examples. Because for the attacker to add perturbation to the image. Then it's easy for it to fool the segmentation system into, for example, for a particular patch or for the whole image. To cause the segmentation system to. Create some to to get to some wrong results. But it's it's actually very difficult for the attacker to to have this adversarial for the example to satisfy the spatial consistency. Because these patches are randomly selected and they they need to ensure that this spatial consistency works.

• 45:45Speaker 01

  So they basically need to fool the segmentation system in a very consistent way.

• 45:51Lex Fridman?

  Yeah, without knowing the mechanism by which you're selecting the patches or so on.

• 45:54Speaker 01

  Exactly.

• 45:55Lex Fridman?

  So it has to really fool the entirety of the message, the entirety of it.

• 45:59Speaker 01

  Right, so it turns out to actually to be really hard for the attacker to do. We tried, you know, the best we can, this data for the art attacks. It actually showed that this defense method is actually very, very effective. And this goes to, I think also what I'm was saying earlier is, um, essentially, we want the learning system to have to to have richer responses and also to learn from more. You can add the same model model, essentially to have more ways to check whether it's actually having the right prediction. So for example, in this case, doing the spatial consistency check. And also, actually, so that's one paper that we did.

• 46:40Speaker 01

  And then this spatial consistency, this notion of consistency check is not just limited to spatial properties. It also applies to audio. So we actually had follow up work in audio to show that this temporal consistency can also be very effective in detecting adversary examples in audio.

• 47:00Lex Fridman?

  Like speech or what kind of audio?

• 47:02Speaker 01

  Right, right, right.

• 47:02Lex Fridman?

  Speech, speech data?

• 47:03Speaker 01

  Right. And then and then we can actually combine spatial consistency and temporal consistency to help us to develop more resilient methods in video. So to defend against attacks for video also.

• 47:16Lex Fridman?

  That's fascinating.

• 47:17Speaker 01

  Right, so yeah, so it's very interesting. Yes, yes.

• 47:21Lex Fridman?

  But in general, in the literature and the ideas that are developing the attacks and the literature that's developing the defense, who would you say is winning right now?

• 47:31Speaker 01

  Right now, of course, it's attack side. It's much easier to develop attacks. And there are so many different ways to develop attacks. Even just us, we develop so many different methods for doing attacks. And also you can do white box attacks, you can do black box attacks, where attacks you don't even need.

• 47:52Speaker 01

  The attacker doesn't even need to know the architecture of the target system and now knowing the parameters of the target system and all that. So there are so many different types of attacks.

• 48:03Lex Fridman?

  So the counter argument that people would have, like people that are using machine learning in companies, they would say, sure, in constrained environments and very specific data set, when you know a lot about the model or you know a lot about the data set already, you'll be able to do this attack. It's very nice. It makes for a nice demo. It's a very interesting idea. But my system won't be able to be attacked like this. The real world systems won't be able to be attacked like this. That's like, that's another hope, that it's actually a lot harder to attack real world systems. Can you talk to that?

• 48:38Speaker 01

  Right.

• 48:39Lex Fridman?

  How hard is it to attack a real world system?

• 48:41Speaker 01

  Right. I wouldn't call that a hope. I think it's more of a wishful thinking. I'll try, I'll trying to be lucky. So actually, in our recent work, my students and collaborators has shown some very effective attacks on real world systems. For example, Google Translate.

• 49:03Lex Fridman?

  Oh, no.

• 49:04Speaker 01

  And other, other cloud translation APIs.

• 49:12Speaker 01

  So in this work, we showed so far I talked about adversary examples mostly in the vision

• 49:18Lex Fridman?

  Vision.

• 49:18Speaker 01

  category. And of course adversary examples also work in other domains as well. For example, in natural language. So so in this work, my students and collaborators have shown that so one, we can actually very easily steal the model from, for example, Google Translate. By just doing queries from right through the APIs and then we can train an imitation model ourselves using the queries. And then once we and also the imitation model can be very, very effective. Essentially have achieving similar performance as a target model.

• 50:03Speaker 01

  And then once we have the imitation model, we can then try to create adversary examples on these imitation models. So for example, giving, you know, in the work, it was one example is translating from English to German. We can give it a sentence saying, for example, I'm feeling freezing, it's like six Fahrenheit. And then translating to German. And then we can actually generate adversary examples that creates a target translation by very small perturbation. So in this case, I say we want to change the translation instead of six Fahrenheit to twenty one Celsius.

• 50:48Speaker 01

  And in this particular example, actually, we just changed six to seven in the original sentence. That's the only change we made. It caused the translation to change from the six Fahrenheit into twenty one Celsius.

• 51:04Lex Fridman?

  Incredible.

• 51:04Speaker 01

  And then and then so this example, we created this example from our imitation model.

• 51:10Lex Fridman?

  Imitation.

• 51:11Speaker 01

  And then this work actually transfers to the Google Translate.

• 51:16Lex Fridman?

  So the attacks that work on the imitation model, in some cases at least, transfer to the original.

• 51:22Speaker 01

  Right.

• 51:22Lex Fridman?

  Model. That's incredible and terrifying. Okay. That's amazing work.

• 51:27Speaker 01

  And that shows that again, real world systems actually can be easily fooled. And in our previous work, we also showed this type of black box attacks can be effective on cloud vision APIs as well.

• 51:41Lex Fridman?

  So that's for natural language and for vision. Let's, let's talk about another space that people have some concern about, which is autonomous driving as sort of security concerns. That's another real world system. So do you have, should people be worried about adversarial machine learning attacks in the context of autonomous vehicles that use like Tesla Autopilot, for example, that uses vision as a primary sensor for perceiving the world and navigating that world? What do you think? From your stop sign work in the physical world, should people be worried? How hard is that attack?

• 52:20Speaker 01

  So actually, there has already been, like, there, there has always been, like, research shown that's, for example, actually even with Tesla, like, if you put a few stickers on the road, it can actually, once arranged in certain ways, it can fool the...

• 52:38Lex Fridman?

  That's right, but I don't think it's actually been, I'm not, I might not be familiar, but I don't think it's been done on physical worlds, uh, physical roads yet, meaning I think it's with a projector in front of the Tesla. So it's a, it's a physical, so you're on the other side of the sensor, but you're not in still the physical world. The, the question is whether it's possible to orchestrate attacks that work in the actual phys, like, end to end attacks, like, not just a demonstration of the concept, but thinking, is it possible on the highway to control Tesla? That kind of idea?

• 53:12Speaker 01

  I, I think there are two separate questions. One is the feasibility of the attack, and I'm 100% confident that's the, the attack is possible. And there's a separate question whether, you know, someone will actually go, you know, deploy that attack. I, I hope people do not do that.

• 53:31Lex Fridman?

  Yeah.

• 53:32Speaker 01

  But that's two separate questions.

• 53:33Lex Fridman?

  So the question on the word feasibility, now, so to clarify, feasibility means it's possible. It doesn't say how hard it is because, uh, in the, to implement it. So sort of the, the barrier, like, how, how much of a heist it has to be, like, how many people have to be involved? What is the probability of success? That kind of stuff. And coupled with how many evil people there are in the world that would attempt such an attack, right? That, but the two, my, my question is, is it sort of, um...

• 54:06Lex Fridman?

  You know, when I talk to Elon Musk and ask the same question, he says it's not a problem. It's very difficult to do in the real world. This won't be a problem. He dismissed it as a problem for adversarial attacks on the Tesla. Of course, he happens to be involved with the company. So he has to say that. But I mean, let me linger on a little longer. Do you?

• 54:29Lex Fridman?

  So where does your confidence that it's feasible come from? And what's your intuition? How people should be worried and how we might be? How people should defend against it? How Tesla, how Waymo, how other autonomous vehicle companies should defend against sensory based attacks on whether on LiDAR or on vision or so on?

• 54:49Speaker 01

  And also even for LiDAR, actually, there has been research shown that even LiDAR itself can be attacked.

• 54:55Lex Fridman?

  It's really important to pause. There's really nice demonstrations that it's possible to do. But there's so many pieces that it's kind of like. It's in it's kind of in the lab. It's in the physical world, meaning it's in the physical space, the attacks. But it's very like you have to control a lot of things to pull it off. It's like the difference between opening a safe when you have it and you have unlimited time and you can work on it versus like breaking into like the crown, stealing the crown jewels or whatever, right?

• 55:31Speaker 01

  I mean, so one way to look at this is in terms of how real these attacks can be. One way to look at it is that actually you don't even need any sophisticated attacks. Already we've seen in the many real world examples, incidents where showing that the vehicle was making the wrong decision.

• 55:51Lex Fridman?

  The wrong decision without attacks, right?

• 55:53Speaker 01

  Right, right. So that's one way to demonstrate. And this is also like so far we've mainly talked about work in this adversary setting. Showing that today's learning system, they are so vulnerable to the adversary setting. But at the same time, actually, we also know that even in natural settings, these learning systems, they don't generalize well. And hence they can really misbehave under certain situations.

• 56:18Lex Fridman?

  Yes, absolutely.

• 56:18Speaker 01

  Like what we have seen. And hence I think using that as an example, it can show that these issues can be real.

• 56:25Lex Fridman?

  They can be real. But so there's two cases. One is. Something, it's like perturbations can make the system misbehave versus make the system do one specific thing that the attacker wants, as you said, the targeted.

• 56:40Speaker 01

  Targeted attack, right.

• 56:41Lex Fridman?

  That seems, that seems to be very difficult. Like an extra level of difficult step in the, in the real world. But from the perspective of the passenger of the car, you know, I don't think it matters either way.

• 56:54Speaker 01

  Right, right, right.

• 56:55Lex Fridman?

  Whether it's, yeah, it's misbehavior or a targeted attack. Okay.

• 56:59Speaker 01

  And also, and that's why I was also saying earlier, like one defense is this multi-model defense and then more of these consistent checks and so on. So in the future, I think also it's important that for these autonomous vehicles, they, right, they have lots of different sensors and they should be combining all these sensory readings to arrive at the decision and the interpretation of the world and so on. And the more of these sensory inputs they use and the better they combine these sensory inputs, the harder it is going to be attacked. And hence, I think that is a very important direction for us to move towards.

• 57:39Lex Fridman?

  So multimodal, multi-sensor, across multiple cameras, but also in the case of car, radar, ultrasonic.

• 57:45Speaker 01

  Right.

• 57:46Lex Fridman?

  Sound even, so all of those.

• 57:48Speaker 01

  Right, right, right, exactly.

• 57:50Lex Fridman?

  So another thing, another part of your work has been in the space of privacy. And that too can be seen as a kind of security vulnerability. And so, so thinking of data as a thing that should be protected and the vulnerabilities to data is vulnerability is essentially the thing that you want to protect is the privacy of that data. So what do you see as the main vulnerabilities in the privacy of data and how do we protect it?

• 58:19Speaker 01

  Right. So in security, we actually talk about essentially two, in this case, two different properties. One is integrity and one is confidentiality. So what we have been talking earlier is essentially the integrity of the integrity property of the learning system. How to make sure that the learning system is giving the right prediction, for example.

• 58:45Lex Fridman?

  Um hum.

• 58:46Speaker 01

  And privacy essentially is on the other side is about confidentiality of the system. Is how attackers can, when the attackers compromise the confidentiality of the system, that's when the attacker steal sensitive information and, right, about individuals and so on.

• 59:05Lex Fridman?

  That's really clean. Those are, those are great terms. Integrity and confidentiality.

• 59:11Speaker 01

  Right.

• 59:11Lex Fridman?

  So how, what are the main vulnerabilities to privacy, we just say? And how do we protect against it? Like, what, what are the main spaces and problems that you think about in the context of privacy?

• 59:24Speaker 01

  Right, so, um, especially in the machine learning setting. Um, so in this case, as we know that how the process goes is that we have the training data and then the machine learning system trains from this training data and then builds a model. And then later on, inputs are given to the model to, at inference time, to try to get prediction and so on. So then, in this case, the privacy concerns that we have is typically about privacy of the data in the training data. Because that's essentially the private information.

• 1:00:03Speaker 01

  So, and it's really important because oftentimes the training data can be very sensitive. It can be your financial data, it's your health data, or like in our T case, it's the sensors deployed in real world environment and so on. And all this can be collecting very sensitive information. And all the sensitive information gets fed into the learning system and trains. And as we know, these neural networks, they can have really high capacity. And they actually can remember a lot. And hence, just from the learning, the learned model in the end,

• 1:00:44Speaker 01

  actually attackers can potentially infer information about their original training data set.

• 1:00:54Lex Fridman?

  So the thing you're trying to protect, yeah, is the confidentiality of the training data. And so what are the methods for doing that, would you say? What are the different ways that can be done?

• 1:01:05Speaker 01

  And also, we can talk about essentially how the attacker may try to

• 1:01:09Lex Fridman?

  Yeah, that's true.

• 1:01:09Speaker 01

  Learn information from the, right? So, so and also there are different types of attacks. So in certain cases, again, like in white box attacks, we can see that the attacker actually get to see the parameters of the model. And then from that, the, a smart attacker potentially can try to figure out information about the training data set. They can try to figure out what type of data has been in the training data set. And sometimes they can tell like whether a person has been, a particular person's data point has been used in the training data sets as well.

• 1:01:46Lex Fridman?

  So white box, meaning you have access to the parameters of, say, a neural network.

• 1:01:50Speaker 01

  Right.

• 1:01:51Lex Fridman?

  And so that you're saying that it's some, given that information is possible to some.

• 1:01:56Speaker 01

  So I can give you some examples. And then another type of attack, which is even easier to carry out, is not a web box model. It's a more of a, just a query model where the attacker only gets to query the machine learning model and then try to steal sensitive information in the original training data. So, right, so I can give you an example.

• 1:02:18Speaker 01

  In this case, training a language model. So in our work, in collaboration with the researchers from Google, we actually studied the following question. So, so at how level the question is? As we mentioned, the neural networks can have very high capacity and they could be remembering a lot from the training process. Then the question is, can attacker actually exploit this and try to actually extract sensitive information in the original training data set through just querying the learned model without even knowing the parameters of the model, like the details of the model or the architectures of the model and so on.

• 1:02:59Speaker 01

  So that's the question we set out to explore. And in one of the case studies, we showed the following. So we trained a language model over an email data set. It's called an Enron email data set. And the Enron email data sets naturally contains users' social security numbers and credit card numbers. So we trained a language model over this data set. And then we showed that an attacker, by devising some new attacks, by just querying the language model and without knowing the details of the model, the attacker actually can extract the original social security numbers

• 1:03:43Speaker 01

  and credit card numbers that were in the original training data.

• 1:03:47Lex Fridman?

  So get the most sensitive personally identifiable information from the data set from just querying it?

• 1:03:55Speaker 01

  Right, yeah.

• 1:03:56Lex Fridman?

  So that's a...

• 1:03:57Speaker 01

  That's an example showing that's why even as we train machine learning models, we have to be really careful with protecting users' data privacy.

• 1:04:08Lex Fridman?

  Yeah. So what are the mechanisms for protecting? Is there hopeful? So if there's been recent work on differential privacy, for example, that provides some hope. But can you describe some of the ideas?

• 1:04:21Speaker 01

  Right, so that's actually, right, so that's also our finding is that by, actually, we showed that... In this particular case, we actually have a good defense.

• 1:04:31Lex Fridman?

  For the querying case, for the.

• 1:04:33Speaker 01

  For this language model.

• 1:04:34Lex Fridman?

  Language model case.

• 1:04:35Speaker 01

  So instead of just training a vanilla language model, instead if we train a defensively private language model, then we can still achieve similar utility. But at the same time, we can actually significantly enhance the privacy protection and of the learned model. And our proposed attacks actually are no longer effective.

• 1:05:01Lex Fridman?

  And differential privacy is a mechanism of adding some noise by which you then have some guarantees on the inability to figure out the presence of a human, of a particular person in the data set.

• 1:05:17Speaker 01

  So, right, so in this particular case, what the differential privacy mechanism does is that it actually adds perturbation in the training process. As we know, during the training process, we are learning the model, we are doing gradient updates, the weight updates and so on. And essentially, defensively private machine learning algorithm in this case will be adding noise and adding various perturbation during this training process.

• 1:05:51Lex Fridman?

  To some aspect of the training process.

• 1:05:53Speaker 01

  Right, so then the finally trained learning, the learned model is defensively private. And so it can enhance the privacy protection.

• 1:06:04Lex Fridman?

  So, okay, so that's the attacks and the defense of privacy. You also talk about ownership of data. So this is a really interesting idea that we get to use many services online for seemingly for free. By essentially, sort of a lot of companies are funded through advertisement. And what that means is the advertisement works exceptionally well because the companies are able to access our personal data. So they know which advertisement to serve us, to do targeted advertisements and so on. So can you maybe talk about this? You have some nice paintings of the future, philosophically speaking, future where

• 1:06:47Lex Fridman?

  people can have a little bit more control of their data by owning and maybe. Understanding the value of their data and being able to sort of monetize it in a more explicit way as opposed to the implicit way that it's currently done.

• 1:07:02Speaker 01

  Yeah, I think this is a fascinating topic and also a really complex topic. Right, I think there are these natural questions, who should be owning the data?

• 1:07:16Speaker 01

  And so I can draw one analogy. And so, for example, for physical properties, like your house and so on. So really, this notion of property rights is not just, you know, like, it's not like from day one, we knew that there should be like this clear notion of ownership of properties and having enforcement for this. And so actually, people have shown that this establishment and enforcement of property rights has been a main driver for the, for the, for the economy earlier.

• 1:08:00Speaker 01

  And that actually really propelled the economic growth even, right, in the earlier stage.

• 1:08:08Lex Fridman?

  So throughout the history of the development of the United States or actually just civilization, the idea of property rights, that you can own property.

• 1:08:18Speaker 01

  And then there is enforcement, there is institutional rights, the governmental like enforcement of this actually has been a key driver for economic growth. And there had been even research or proposals saying that for a lot of the developing countries, and they're, you know, essentially the challenge in growth is not actually due to the lack of capital. It's more due to the lack of this property, notion of property rights and enforcement of property rights.

• 1:08:54Lex Fridman?

  Interesting, so that.

• 1:08:57Lex Fridman?

  The presence of absence of both the the the the concept of the property rights and their enforcement has a strong correlation to economic growth.

• 1:09:07Speaker 01

  Right, right.

• 1:09:08Lex Fridman?

  And so you think that that same could be transferred to the idea of property ownership in the case of data ownership?

• 1:09:15Speaker 01

  I think it's a, I think first of all, it's a good lesson for us to, right, to recognize that these rights and the recognition and enforcement of these type of rights is very, very important for economic growth. And then if we look at where we are now and where we are going in the future, and so essentially more and more is is actually moving into the digital world. And also more and more, I would say, even like information or asset of a person is more and more into the real world, the physical, the sorry, the digital world as well. It's the data that the the person has generated.

• 1:09:57Speaker 01

  And essentially, it's like in the past, what defines a person? You you can say, right, like oftentimes, besides the innate, like capabilities, actually it's the physical properties.

• 1:10:11Lex Fridman?

  House, car.

• 1:10:12Speaker 01

  Right, that defines a person.

• 1:10:13Lex Fridman?

  Yep.

• 1:10:14Speaker 01

  But I think more and more people start to realize, actually, what defines a person is more important in the data that the person has generated or the data about the person.

• 1:10:25Speaker 01

  Like all the way from, yeah, political views, yeah, yeah, music taste and, right, yeah, financial information, a lot of these, and your health. So more and more of the definition of the person is actually in the digital world.

• 1:10:39Lex Fridman?

  And currently, for the most part, that's owned, like it's, people don't talk about it, but kind of it's owned by internet companies. So it's not owned by individuals.

• 1:10:52Speaker 01

  Right, there's no clear notion of ownership. After such data. And also, we, you know, we talk about privacy and so on. But I think actually clearly identifying the ownership is a first step. Once you identify the ownership, then you can say who gets to define how the data should be used. So maybe some users are fine with, you know, internet companies serving them as, right? Using their data as long as if the, if the data is used in a certain way that actually the, the user consent with or allows. For example, you can say the recommendation system in some sense, we don't call it as, but a recommendation system similar is trying to recommend you something.

• 1:11:38Speaker 01

  And users enjoy and can really benefit from good recommendation systems. And they recommend you better music, movies, news, even research papers to read.

• 1:11:50Speaker 01

  But, but of course, then in this targeted ads, especially in, in certain cases where people can be manipulated by these targeted ads that can have really bad, like severe consequences. So, so essentially users want their data to be used to better serve them. And also maybe even, right, get paid for whatever, like in different settings. But the thing is that first of all, we need to really establish like who needs to decide, who can decide how the data should be used. And typically, the establishment and clarification of the ownership will help this. And it's an important first step.

• 1:12:32Speaker 01

  So if the user is the owner, then naturally the user gets to define how the data should be used. But if you even say that, wait a minute, users are actually now the owner of this data. Whoever is collecting the data is the owner of the data. Now, of course, they get to use the data however they want.

• 1:12:46Lex Fridman?

  Yeah.

• 1:12:47Speaker 01

  So to really address these complex issues, we need to go at the root cause. So it seems fairly clear that so first we really need to say that who is the owner of the data? And then the owners can specify how they want their data to be utilized.

• 1:13:04Lex Fridman?

  So I said, that's a fascinating.

• 1:13:08Lex Fridman?

  Most people don't think about that. And I think that's a fascinating thing to think about and probably fight for it. There's like, I can only see, and the economic growth argument is probably a really strong one. So that's, that's a first time I'm kind of at least thinking about the, the positive aspect of that ownership being the long-term growth of the economy. So good for everybody. But sort of one down, possible downside I could see, sort of to put on my grumpy old grampa hat. And, you know, it's really nice for Facebook and YouTube and Twitter to all be free. And if you give control to people with their data, do you think it's possible they will

• 1:13:51Lex Fridman?

  be, they would not want to hand it over quite easily? And so a lot of these companies that rely on mass handover of data and then therefore provide a mass seemingly free service would then completely, so the, the, the way the internet looks will completely change because of the ownership of data and will lose a lot of services with value. Do you worry about that?

• 1:14:19Speaker 01

  That's a very good question. I think that's not necessarily the case in the sense that, yes, users can have ownership of their data, they can maintain control of their data, but also then they get to decide how their data can be used. So that's why I mentioned earlier, like, so in this case, if they feel that they enjoy the benefits of social networks and so on, and they're fine with having Facebook, having their data, but utilizing the data in certain way that they agree, then they can still enjoy the free services. But for others, maybe they would prefer some kind of private vision.

• 1:14:59Speaker 01

  And in that case, maybe they can even opting to say that I want to pay and to have, so for example, it's already fairly standard, like you pay for certain subscriptions so that you don't get to, you know, be shown ads.

• 1:15:14Lex Fridman?

  See ads, yeah.

• 1:15:15Speaker 01

  Right. So then users essentially can have choices. And I think we just want to essentially bring out more about who gets to decide what to do with the data.

• 1:15:28Lex Fridman?

  I think it's an interesting idea because if you poll people now. You know, it seems like, I don't know, but subjectively, sort of anecdotally speaking, it seems like a lot of people don't trust Facebook. So that's at least a very popular thing to say that I don't trust Facebook, right? I wonder if you give people control of their data as opposed to sort of signaling to everyone that they don't trust Facebook, I wonder how they would speak with the actual, like, would they be willing to pay $10 a month for Facebook or would they hand over their data? It's, I'd be interesting to see what fraction of people would quietly hand over their data to Facebook to make it free.

• 1:16:10Lex Fridman?

  I, I don't have a good intuition about that, like, how many people, do you have an intuition about how many people would use their data effectively on the market, on the, on the market of the internet by sort of buying services with their data?

• 1:16:28Speaker 01

  Um, yeah, so that's a very good question. I think, um, so one thing I also want to mention is that this, right, so it seems that especially in press and the conversation has been very much like two sides fighting against each other.

• 1:16:48Speaker 01

  On one hand, right, users can say that, right, they don't trust Facebook, they don't, others delete Facebook.

• 1:16:55Lex Fridman?

  Yeah, exactly.

• 1:16:56Speaker 01

  Right, and then, and then on the other hand, um, right, of course, um, right, the, the, the other side, they also feel, oh, they are providing a lot of services to users and users are getting it all for free. So I think I actually, you know, I talk a lot to like different companies and also like basically on both sides.

• 1:17:22Speaker 01

  So one thing I hope also, like, this is my hope for this year also is that we want to establish a more constructive dialogue.

• 1:17:33Speaker 01

  And to help people to understand that the problem is much more nuanced than just this two sides fighting. Because naturally, there is a tension between the two sides, between utility and privacy. So if you want to get more utility, essentially, like the recommendation system example I gave earlier. If you want someone to give you a good recommendation, essentially, whatever that system is, the system is going to need to know your data to give you a good recommendation.

• 1:18:10Speaker 01

  But also, of course, at the same time, we want to ensure that however that data is being handled is done in a privacy preserving way. So that, for example, the recommendation system doesn't just go around and sell your data and then cause all the, you know, cause a lot of bad consequences and so on. So.

• 1:18:30Lex Fridman?

  So you want that dialogue to be a little bit more in the open, a little more nuanced. And maybe adding control to the data, ownership to the data will allow, as opposed to this happening in the background, allow to bring it to the forefront and actually have dialogues in like more nuanced, real dialogues about how we trade our data for the services. That's the hope.

• 1:18:55Speaker 01

  Right, right. Yes, at a high level. So essentially, also knowing that there are technical challenges in addressing the issue to, like, basically, you can't have, just like the example that I gave earlier. It's really difficult to balance the two between utility and privacy. And that's also a lot of things that. I work on my group, worked on as well is to actually develop these technologies that I needed to essentially help this balance better and essentially to help data to be utilized in a privacy preserving and responsible way.

• 1:19:37Speaker 01

  And so we essentially need people to understand the challenges and also at the same time and to provide the technical abilities and also regulatory frameworks to help the two sides to be more in a win-win situation instead of a fight.

• 1:19:53Lex Fridman?

  Yeah, the fighting, the fighting thing is, I think YouTube and Twitter and Facebook are providing an incredible service to the world. And they're all making mistakes, of course, but they're doing an incredible job, you know, that I think deserves to be applauded and there's some degree of gratitude, like it's a cool thing that that's created and it shouldn't be monolithically fought against, like Facebook is evil or so on. Yeah, I might make mistakes, but I think it's an incredible service. I think it's world changing. I mean, I've, I think Facebook's done a lot of incredible, incredible things by bringing,

• 1:20:36Lex Fridman?

  for example, identity, you're like allowing people to be themselves, like their real selves in the, in the digital space by using their real name and their real picture. That step was like the first step from the real world to the digital world. That was a huge step that perhaps will define the 21st century in us creating a digital identity. And there's a lot of interesting possibilities there that are positive. Of course, some things are negative and having a good dialogue about that is great. And I'm, I'm great that people like you are at the center of that dialogue. That's, that's awesome.

• 1:21:11Speaker 01

  Right, I think also, I also can understand, I think actually in the past, especially in the past couple of years, this rising awareness has been helpful, like users are also more and more recognizing that privacy is important to them, they should, maybe, right, they should be owners of their data. I think this definitely is very helpful. And I think also this type of voice, also and together with the regulatory framework and so on. Also help the companies to essentially put these type of issues at a higher priority.

• 1:21:51Speaker 01

  And knowing that, right, also it is their responsibility too to ensure that users are well protected. And so I think definitely the right voice is super helpful. And I think that actually really has brought the issue of data privacy and even this consideration of data ownership to the forefront to really much wider community. And I think more of this voice is needed, but I think it's just that we want to have a more constructive dialogue

• 1:22:26Speaker 01

  to bring the both sides together to figure out a constructive solution.

• 1:22:31Lex Fridman?

  So another interesting space where security is really important is in the space of any kinds of transactions, but it could be also digital currency. So can you maybe talk a little bit about blockchain? And can you tell me what is a blockchain?

• 1:22:50Speaker 01

  I think the blockchain word itself is actually very overloaded.

• 1:22:55Lex Fridman?

  Of course.

• 1:22:55Speaker 01

  In general.

• 1:22:56Lex Fridman?

  It's like AI.

• 1:22:57Speaker 01

  Right, yes. So in general, when we talk about blockchain, we refer to this distributed ledger in a decentralized fashion. So essentially, you have a community of nodes that come together. And even though each one may not be trusted and as long as a certain thresholds of the set of nodes behaves properly, then the system can essentially achieve certain properties. For example, in the distributed ledger setting, you have, you can maintain an immutable log

• 1:23:36Speaker 01

  and you can ensure that, for example, the transactions actually are agreed upon and then it's immutable and so on.

• 1:23:45Lex Fridman?

  So first of all, what's a ledger? So it's a.

• 1:23:48Speaker 01

  It's like a database, it's like a data entry.

• 1:23:51Lex Fridman?

  And so distributed ledger is something that's maintained across or is synchronized across multiple sources, multiple nodes.

• 1:23:59Speaker 01

  Multiple nodes, yes.

• 1:24:00Lex Fridman?

  And so where is this idea? Now, how do you keep? So it's important ledger, a database, to keep that.

• 1:24:12Lex Fridman?

  To make sure, so what are the kinds of security vulnerabilities that you're trying to protect against in the context of a distributed ledger?

• 1:24:21Speaker 01

  So in this case, for example, you don't want some malicious nodes to be able to change the transaction logs. And in certain cases, it's called double spending. Like you also cause, you can also cause different views in different parts of the network and so on.

• 1:24:40Lex Fridman?

  So the ledger has to represent, if you're capturing like financial transactions, has to represent the exact timing and the exact occurrence and no duplicates, all that kind of stuff. It has to be, represent what actually happened. Okay, so what are your thoughts on the security and privacy of digital currency? I can't tell you how many people write to me to interview various people in the digital currency space. There seems to be a lot of excitement there. And it seems to be, some of it's to me, from an outsider's perspective, seems like dark magic. I don't know how secure.

• 1:25:23Lex Fridman?

  I think the foundation, from my perspective, of digital currencies, that is, you can't trust anyone. So you have to create a really secure system. So can you maybe speak about how, what your thoughts in general about digital currency is and how you, how it can possibly create financial transactions and financial stores of money in the digital space?

• 1:25:49Speaker 01

  So you asked about security and privacy. And so, so again, as I mentioned earlier, in security, we actually talk about two main properties. And the integrity and confidentiality. And so there's another one for availability. You want the system to be available. But here, for the question you asked, let's just focus integrity and confidentiality.

• 1:26:13Lex Fridman?

  Yes.

• 1:26:14Speaker 01

  So, so for integrity of this distributed ledger, essentially, as we discussed, we want to ensure that the different nodes, right, so they have this consistent view. Usually it's done through, we call a consensus protocol. And that's the establish this shared view on this ledger and that.

• 1:26:37Speaker 01

  You cannot go back and change, it's immutable and so on. So, so in this case, then the security often refers to this integrity property. And essentially, you're asking the question, how much work? How, how can you attack the system? So that the attacker can change the lock, for example.

• 1:27:03Lex Fridman?

  Right. How hard is it to make an attack like that?

• 1:27:05Speaker 01

  Right, right. And then that very much depends on the consensus mechanism, the how the system is built and all that. So there are different ways to build these decentralized systems. And people may have heard about the terms called like proof of work, proof of stake, these different mechanisms. And it really depends on how, right, how the system has been built. And also, how much resources, how much work has gone into the network? To actually say how secure it is. So for example, and people talk about like in Bitcoin, it's proof of work system, so much electricity has been burned.

• 1:27:49Lex Fridman?

  So there's differences, there's differences in the different mechanisms in the implementations of a distributed ledger used for digital currency. Also, there's Bitcoin, there's whatever, there's so many of them. And there's underlying different mechanisms. And there's arguments, I suppose, about which is more effective, which is more secure, which is more.

• 1:28:10Speaker 01

  And what is needed, what amount of resources needed to be able to attack the system? Like, for example, what percentage of the nodes do you need to control or compromise? In order to, right, to change the lock?

• 1:28:27Lex Fridman?

  And those are things, do you, do you have a sense of those are things that could be shown theoretically? Through the design of the mechanisms? Or does it have to be shown empirically by having a large number of users using the currency?

• 1:28:41Speaker 01

  I see. So in general, for each consensus mechanism, you can actually show theoretically what is needed to be able to attack the system. Of course, there are, there can be different types of attacks as we discussed at the beginning. And so that it's difficult to give like, you know, a complete estimate, like really how much is needed to compromise the system. But in general, right, so there are ways to say what percentage of the nodes you need to compromise and so on.

• 1:29:20Lex Fridman?

  So we talked about integrity on the security side. And then you also mentioned the privacy or the confidentiality side. Does it have some of, does it have some of the same problems? And therefore, some of the same solutions that you talked about on the machine learning side with differential privacy and so on?

• 1:29:41Speaker 01

  Yeah, so actually, in general, on the public ledger in these public decentralized systems,

• 1:29:51Speaker 01

  actually, nothing is private. So all the transactions posted on the ledger, anybody can see. So in that sense, there is no confidentiality. So usually, what you can do is then there are the mechanisms that you can build in to enable confidentiality, our privacy of the transactions and the data and so on. That's also some of the work that both my group and also my startup does as well.

• 1:30:22Lex Fridman?

  What's the name of the startup?

• 1:30:23Speaker 01

  Oasis Labs.

• 1:30:24Lex Fridman?

  Oasis Labs. And so the confidentiality aspect there is even though the transactions are public, you want to keep some aspect confidential of the identity of the people involved in the transactions? Or what, what is their hope to keep confidential in this context?

• 1:30:42Speaker 01

  So in this case, for example, you want to enable like Confidential transactions even so there are different essentially types of data that you want to keep private or confidential and you can utilize different technologies, including zero knowledge proofs and also secure computing and techniques and to hide either the right who is making the transactions to whom and the transaction amount and in our case also we can enable like confidential smart contracts and so that you don't know

• 1:31:22Speaker 01

  the data and the execution of the smart contract and so on. And we actually are combining these different technologies and to going back to the earlier discussion we had, enabling like ownership of data and privacy of data and so on. So so at Oasis Labs, we are actually building what we call a platform for responsible data economy to actually combine these different technologies together and to enable secure and privacy preserving computation and also

• 1:32:02Speaker 01

  using the ledger to help provide immutable log of users' ownership to their data and the policies they want the data to adhere to, the usage of the data to adhere to and also how the data has been utilized. So all this together can build we call a distributed secure computing fabric that helps to enable a more responsible data economy. So it's a lot of things together.

• 1:32:29Lex Fridman?

  Yeah, wow, that was eloquent. Okay, you're involved in so much amazing work that we'll never be able to get to, but I have to ask at least briefly about program synthesis, which at least in a philosophical sense captures much of the dreams of what's possible in computer science and the artificial intelligence.

• 1:32:52Lex Fridman?

  First, let me ask, what is program synthesis? And can neural networks be used to learn programs from data? So can this be learned? Some aspect of this synthesis can be learned?

• 1:33:04Speaker 01

  So program synthesis is about teaching computers to write code, to program. And I think that's one of our ultimate dreams or goals. You know, I think Andreessen talked about software eating the world. So I say, once we teach computers to write software, to write programs, then I guess computers will be eating the world by transitivity.

• 1:33:34Lex Fridman?

  Yeah, exactly.

• 1:33:35Speaker 01

  So yeah, it's a, and also for me, actually,

• 1:33:41Speaker 01

  when I shifted from security to more AI machine learning, program synthesis is, program synthesis and adversarial machine learning, these are the two fields that I particularly focus on. Like program synthesis is one of the first questions that I actually started investigating.

• 1:34:00Lex Fridman?

  Just as a question, oh, I guess with, from the security side, there's a, you know, you're looking for holes in programs. So at least see small connection. But why, where was your interest for program synthesis? As, because it's such a fascinating, such a big, such a hard problem in the general case. Why program synthesis?

• 1:34:20Speaker 01

  So the reason for that is actually when I shifted my focus from security into AI machine learning, actually, one of my main motivation at the time is that, even though I have been doing a lot of work in security and privacy, but I have always been fascinated about building intelligent machines. And that was really my main motivation to spend more time in AI machine learning is that I really want to figure out how we can build intelligent machines. And To help us towards that goal,

• 1:35:00Speaker 01

  program synthesis is really one of, I would say, the best domain to work on. I actually call it like program synthesis is like the perfect playground for building intelligent machines and for artificial general intelligence.

• 1:35:17Lex Fridman?

  Yeah. Well, it's also, in that sense, not just a playground, I guess it's it's the ultimate test of intelligence because

• 1:35:24Speaker 01

  Yes.

• 1:35:25Lex Fridman?

  I think, I think if you can generate, sort of neural networks can learn good functions and they can help you out in classification tasks. But to be able to write programs.

• 1:35:38Speaker 01

  Right.

• 1:35:39Lex Fridman?

  That's, that's the epitome from the machine side. That's the same as passing the Turing test in natural language. But with programs, it's able to express complicated ideas, to reason through ideas. And yeah, and boil them down to algorithms.

• 1:35:55Speaker 01

  Yes, exactly, exactly.

• 1:35:56Lex Fridman?

  Incredible. So can this be learned? How far are we? Is there hope? What are the open challenges?

• 1:36:03Speaker 01

  Yeah, very good questions. We are still at an early stage. But already, I think we have seen a lot of progress. I mean, definitely, we have, you know, existence proof. Just like humans can write programs. So there's no reason why computers cannot write programs.

• 1:36:22Lex Fridman?

  Right.

• 1:36:22Speaker 01

  And so I think that's definitely an achievable goal. It's just how long it takes. And then, and even today, we actually have, you know, the program synthesis community, especially the program synthesis via learning, how we call it, neural program synthesis community, is still very small. But the community has been growing and we have seen a lot of progress. And in limited domains, I think actually program synthesis is ripe for real world applications. So actually, it was quite amazing. I was at, I was giving a talk. So here is a rework conference.

• 1:37:06Lex Fridman?

  Yeah, rework deep learning summit.

• 1:37:07Speaker 01

  I actually, so I gave another talk at the previous rework conference in deep reinforcement learning. And then I actually met someone from a startup. And the CEO of the startup. And when he saw my name, he recognized it and he actually said, One of our papers actually had, they have put, had actually become a key products in their startup. And that was program synthesis. In that particular case, it was natural language translation, translating natural language description into SQL queries.

• 1:37:49Lex Fridman?

  Oh, wow, that, that direction, okay.

• 1:37:52Speaker 01

  Right, so, right, so, yeah, so in program synthesis in limited domains, in well-specified domains, actually, already we can see really great progress and applicability in the real world.

• 1:38:09Lex Fridman?

  So domains like, I mean, as an example, you said natural language, being able to express something through just normal language and it converts it into a database SQL SQL query.

• 1:38:20Speaker 01

  Right.

• 1:38:21Lex Fridman?

  And that's how, how solved of a problem is that? Because it seems like a really hard problem.

• 1:38:27Speaker 01

  Okay, in limited domains, actually, it can work pretty well. And now, this is also a very active domain of research. At the time, I think when he saw our paper at the time, we were the state of the arts.

• 1:38:41Lex Fridman?

  Yeah.

• 1:38:42Speaker 01

  On that task. And since then, actually, now there has been more work and with even more sophisticated datasets. And so, but I, I think I wouldn't be surprised that more of this type of technology really gets into the real world.

• 1:39:00Lex Fridman?

  That's exciting.

• 1:39:01Speaker 01

  In the near term.

• 1:39:02Lex Fridman?

  Being able to learn in the space of programs is, is super exciting. I still, yeah, I'm still skeptical because I think it's a really hard problem. But I would love to see progress.

• 1:39:14Speaker 01

  And also, I think in terms of the, you asked about open challenges.

• 1:39:17Lex Fridman?

  Yes, open challenges.

• 1:39:18Speaker 01

  I think the, the domain is full of challenges. And in particular, also, we want to see how we should measure the progress in the space. And I would say mainly.

• 1:39:30Speaker 01

  Three main, I'll say, metrics. So one is the complexity of the program that we can synthesize. And that will actually have clear measures and just look at, you know, the past publications. And even like, for example, I was at the recent Europe's conference. Now there is actually fairly sizable like session dedicated to program synthesis, which is.

• 1:39:53Lex Fridman?

  Or even neural programs.

• 1:39:54Speaker 01

  Right, right, right, which is great. And, and we continue to see the increase.

• 1:40:00Lex Fridman?

  What does sizable mean?

• 1:40:03Lex Fridman?

  I like, I like the word sizable. It's five people.

• 1:40:08Speaker 01

  It's still a small community, but it is growing.

• 1:40:11Lex Fridman?

  And they will all win Turing Awards one day. I like it.

• 1:40:16Speaker 01

  Right, so, so we can clearly see an increase in the complexity of the programs that these.

• 1:40:24Lex Fridman?

  Just to elaborate.

• 1:40:25Speaker 01

  We can synthesize.

• 1:40:26Lex Fridman?

  Sorry to, is it the complexity of the actual text of the program or the running time complexity? Which complexity are we, how?

• 1:40:35Speaker 01

  The complexity of the task to be synthesized and the complexity of the actual synthesized programs. So it's right.

• 1:40:43Lex Fridman?

  Yeah, got you.

• 1:40:43Speaker 01

  So the lines of code even, for example.

• 1:40:45Lex Fridman?

  Okay, I got you, but it's not the theoretical.

• 1:40:49Speaker 01

  No, no, no, no.

• 1:40:50Lex Fridman?

  The running time of the algorithm kind of thing.

• 1:40:52Speaker 01

  Right, right, right, right.

• 1:40:52Lex Fridman?

  Okay, got it.

• 1:40:53Speaker 01

  Right.

• 1:40:53Lex Fridman?

  Got it, and you can see the complexity decreasing already.

• 1:40:57Speaker 01

  Oh, no, meaning we want to be able to synthesize more and more complex programs, bigger and bigger programs. So we want to see that, we want to increase.

• 1:41:06Lex Fridman?

  I got you, so I have to think through, because I, I thought of complexity is you want to be able to accomplish the same task with a simpler and simpler program.

• 1:41:14Speaker 01

  No, we are not doing that.

• 1:41:15Lex Fridman?

  Okay.

• 1:41:16Speaker 01

  It's more, it's more about how complex a task.

• 1:41:19Lex Fridman?

  Right.

• 1:41:19Speaker 01

  We can synthesize programs for.

• 1:41:21Lex Fridman?

  Got it, being able to synthesize programs, learn them for more and more difficult tasks.

• 1:41:27Speaker 01

  Right, so for example, initially, our first work in program synthesis, synthesis was to translate natural language description into really simple programs called if T T T, if this then that.

• 1:41:38Lex Fridman?

  Mm hmm.

• 1:41:38Speaker 01

  So given a trigger condition, what is the action you should take? So that program is a super simple. You just identify the trigger conditions and the action.

• 1:41:48Lex Fridman?

  Yep.

• 1:41:49Speaker 01

  And then later on with SQL queries, it gets more complex. And then also we started to synthesize programs with loops and.

• 1:41:57Lex Fridman?

  Oh, no.

• 1:41:59Lex Fridman?

  And if you could synthesize recursion, it's all over.

• 1:42:02Speaker 01

  Right, that actually, one of our works actually is on learning recursive neural programs.

• 1:42:08Lex Fridman?

  Oh, no.

• 1:42:08Speaker 01

  But anyway, anyway, so that's one is the complexity. And the other one is generalization. Like when we train our learn a program synthesizer, in this case, and neural programs to synthesize programs, then you wanted to generalize.

• 1:42:27Lex Fridman?

  For any, for a large number of inputs.

• 1:42:30Speaker 01

  Right, so to be able to generalize to previously unseen inputs.

• 1:42:35Lex Fridman?

  Got it.

• 1:42:36Speaker 01

  And so, right, so some of the work we did earlier on learning recursive neural programs actually showed that recursion actually is important to learn. And if you have recursion, then for a certain set of tasks, we can actually show that you can actually have perfect generalization. So right, so that one of the best paper awards that I clear earlier. So that's one example of we want to learn these neural programs that can generalize better. But that works for a certain task, certain domains. And there is question how we can essentially develop more techniques

• 1:43:21Speaker 01

  that can have generalization for a wider set of domains and so on. So that's another area. And then, and then the, the third challenge, I think, will, it's not just for programming synthesis. This is also cutting across other fields in machine learning and also including like deep reinforcement learning in particular is that.

• 1:43:47Speaker 01

  This adaptation is that we want to be able to learn from the past and tasks and training and so on to be able to solve new tasks. So for example, in program synthesis today,

• 1:44:03Speaker 01

  we still are working in the setting where given a particular task, we train the model and to solve this particular task. But that's not how humans work. The whole point is we train a human, then you can then program to solve new tasks.

• 1:44:24Lex Fridman?

  Right, exactly.

• 1:44:26Speaker 01

  And just like in deep reinforcement, we don't want to just train an agent to play a particular game, either it's Atari or it's Go or whatever. We want to train these agents that can

• 1:44:41Speaker 01

  essentially extract knowledge from the past learning experience to be able to adapt to new tasks and solve new tasks. And I think this is particularly important for program synthesis.

• 1:44:52Lex Fridman?

  Yeah, that's the whole point. That's the whole dream of program synthesis is you're learning a tool that can solve new problems.

• 1:44:58Speaker 01

  Right, exactly. And I think that's a particular domain that as a community, we need to put more emphasis on. And I hope that we can make more progress there as well.

• 1:45:11Lex Fridman?

  Awesome. There's a lot more to talk about. Let me ask that you also had a very interesting and we talked about rich representations. You had a rich life journey. You did your bachelor's in China and your master's and PhD in the United States, CMU and Berkeley. Are there interesting differences? I told you I'm Russian. I think there's a lot of interesting differences between Russia and the United States. Are there in your eyes

• 1:45:41Lex Fridman?

  interesting differences between the two cultures from the silly romantic notion of the spirit of the people to the more practical notion of how research is conducted? That you find interesting or useful in your own work of having experienced both?

• 1:45:59Speaker 01

  That's a good question. I think so I I started in China for my undergraduate and that was more than twenty years ago. So it's been a long time.

• 1:46:14Lex Fridman?

  Is there echoes of that time in you?

• 1:46:16Speaker 01

  Changed a lot. Actually, it's interesting. I think even more so maybe something that's even be more different for my experience than a lot of computer science researchers and practitioners is that so for my undergrad, I actually started physics.

• 1:46:34Lex Fridman?

  Nice, very nice.

• 1:46:35Speaker 01

  And then I switched to computer science in graduate school.

• 1:46:39Lex Fridman?

  What happened?

• 1:46:42Lex Fridman?

  Was there was there is there another possible universe where you could have become a theoretical physicist at Caltech or something like that?

• 1:46:51Speaker 01

  That's very possible. Some of my undergrad classmates, then they later on studied physics. They're teaching physics from these schools from yeah, from top physics programs.

• 1:47:06Lex Fridman?

  So so you you you switched to I mean, if that from that experience of doing physics in your bachelor's, how what what made you decide to switch to computer science? And computer science at arguably the best university, one of the best universities in the world for computer science with Carnegie Mellon, especially for the grad school and so on. So what second only time I teach, just kidding. Okay.

• 1:47:33Lex Fridman?

  I had to throw that in there. No, what what was the choice like? And what was the move to the United States like? What was that whole transition? And if you remember, if there's still echoes of some of the spirit of the people of China in you in your work?

• 1:47:48Speaker 01

  Right, right.

• 1:47:49Lex Fridman?

  It's like three questions. I'm sorry.

• 1:47:54Speaker 01

  No, that's okay. So yes, I guess okay, so first transition from physics to computer science.

• 1:48:00Lex Fridman?

  Yes.

• 1:48:00Speaker 01

  So when I first came to the United States, I was actually in the physics PhD program at Cornell.

• 1:48:06Lex Fridman?

  Yeah.

• 1:48:06Speaker 01

  I was there for one year and then I switched to computer science. And then I was in the PhD program at Carnegie Mellon. Um, so, okay, so the reasons for switching. So one thing, so that's why I also mentioned that about this difference in backgrounds, about having studied physics first in my undergrad. I actually really, I really did enjoy my undergrad's time and education in physics. I think that actually really helped me in my future work in computer science. Actually, even for machine learning, a lot of the machine learning stuff, the, the core machine learning methods, many of them actually came from physics.

• 1:48:50Speaker 01

  That is cool.

• 1:48:52Lex Fridman?

  For honest, mo-most.

• 1:48:53Speaker 01

  But, but anyway.

• 1:48:54Lex Fridman?

  Most of everything came from physics.

• 1:48:56Speaker 01

  Right, but anyway, so, so when I studied physics, I was, um,

• 1:49:03Speaker 01

  I think I was really attracted to physics. Um, it was, it's really beautiful. And I actually call it, physics is the language of nature. And I actually clearly remember like one moment, uh, in my undergrads, like I did my undergrad in Qinghua, and I used to study in the library. And I clearly remember, like, one day, I was sitting in the library, and I, and I was like writing on my notes and so on. And I got so excited that I realized that really just from a few simple axioms, a few simple laws,

• 1:49:47Speaker 01

  I can derive so much. It's almost like I can derive the rest of the world.

• 1:49:51Lex Fridman?

  Yeah, the rest of the universe.

• 1:49:53Speaker 01

  Yes, yes, so that was like amazing.

• 1:49:56Lex Fridman?

  Do you think you, have you ever seen, or do you think you can rediscover that kind of power and beauty in computer science in the world that you?

• 1:50:05Speaker 01

  Um, that's very interesting. So that gets to, you know, the transition from physics to computer science. Um, it's, it's quite different. For, um, for physics in, in grad school, actually, things changed. So one is, I started to realize that when I started doing research in physics, at the time I was doing theoretical physics. And a lot of it, the, you still have the beauty, but it's very different. So I had to actually do a lot of simulations. So essentially, I was actually writing, in some, in some cases, writing fortune code.

• 1:50:42Lex Fridman?

  Good old Fortran, yeah?

• 1:50:44Speaker 01

  Um, to actually, um, uh, right, do like, do simulations and so on, that was not, um, not exactly what's

• 1:50:54Lex Fridman?

  Not beautiful.

• 1:50:55Speaker 01

  Right. I, uh, I enjoyed doing. Uh, and also, at the time, uh, from talking with the senior, you know, students, uh, in the program, I realized many of the students actually were going off to like Wall Street and, and so on. Um, so, and I've always been interested in computer science and actually, uh, essentially taught myself C programming, like

• 1:51:25Lex Fridman?

  Program?

• 1:51:25Speaker 01

  Right, and, and so on.

• 1:51:27Lex Fridman?

  At which, when?

• 1:51:28Speaker 01

  Uh, in, in college.

• 1:51:29Lex Fridman?

  In college somewhere?

• 1:51:30Speaker 01

  In the summer.

• 1:51:32Lex Fridman?

  For fun.

• 1:51:33Speaker 01

  Right.

• 1:51:33Lex Fridman?

  Physics major, learning to do, uh, C programming, beautiful.

• 1:51:37Speaker 01

  Actually, it's interesting, you know, in physics, at the time, I think now the program probably has changed. But at the time, really, the only class we had in, uh, in, in relation to computer science education was introduction to, I forgot, to computer science or computing and Fortran seventy seven.

• 1:51:57Lex Fridman?

  There's a lot of people that still use Fortran. I'm actually, if you're a programmer out there, uh, I'm looking for an expert to talk to about Fortran. They seem to, there's not many, but there's still a lot of people that still use Fortran and still a lot of people use COBOL, so.

• 1:52:13Speaker 01

  But anyway, so, um, so then, then I realized, instead of just doing programming for doing simulations and so on, that I may as well just change to computer science. And also, one thing I really liked, and that's a key difference between the two, is in computer science, it's so much easier to realize your ideas. If you have an idea, you write it up, uh, you, you code it up, and then you can see it, actually, right? Running and you can you can see it's

• 1:52:43Lex Fridman?

  You can bring it to life quicker.

• 1:52:44Speaker 01

  Bring it to life. Whereas in physics, if you have a good theory, you you you have to wait for the experimentalist to do the experiments and to confirm the theory. And and things just take so much longer. And and also the reason I in physics I decided to do direct physics was because I had my experience with experimental physics. First, you have to fix the equipment.

• 1:53:07Lex Fridman?

  Yeah.

• 1:53:08Speaker 01

  You spend most of your time fixing the equipment first. Um, so.

• 1:53:13Lex Fridman?

  Super expensive equipment, so there's a lot of, yeah, you have to collaborate with a lot of people. Takes a long time.

• 1:53:19Speaker 01

  Just takes really, uh, right, much longer.

• 1:53:21Lex Fridman?

  Yeah, it's messy.

• 1:53:21Speaker 01

  Right, so I decided to switch to computer science. And one thing I think maybe people have realized is that for people who studied physics, actually, it's very easy for physicists to change, to do something else.

• 1:53:34Lex Fridman?

  Yes.

• 1:53:34Speaker 01

  I think physics provides a really good training. And, um, yeah, so actually it was very easy to switch to computer science. Um, but one thing going back to your earlier question. So one thing I actually did realize, um, so there is a big difference between computer science and physics, where physics, you can derive the the whole universe from just a few simple laws. And computer science, given that a lot of it is defined by humans, the systems are defined by humans and um, and it's artificial, like, uh, essentially you create a lot of these artifacts and so on. Um, it's it's not quite the same. You don't derive the computer systems with just a few

• 1:54:20Speaker 01

  simple laws. You actually have to see there is, um, historical reasons why a system is built and designed one way versus the other.

• 1:54:30Lex Fridman?

  There's there's a lot more complexity, less elegant simplicity of E equals mc squared that kind of reduces everything down to those beautiful fundamental equations. But what about the move from China to the United States? Is there anything that still stays in you that's contributes to your work? The fact that you grew up in another culture?

• 1:54:54Speaker 01

  So yes, I think especially back then, it's very different from now. So you know, now they actually, I see these students coming from China. And even undergrads, actually, they speak fluent English. It was just, you know, like amazing. And they have already understood so much of the culture in the US and so on. And.

• 1:55:20Lex Fridman?

  It was to you was all foreign, it was.

• 1:55:22Speaker 01

  It was, it was a very different time. At the time, actually, even, we didn't even have easy access to email.

• 1:55:30Lex Fridman?

  Right.

• 1:55:30Speaker 01

  Not to mention about the web.

• 1:55:32Lex Fridman?

  Yeah.

• 1:55:33Speaker 01

  I remember I had to go to, you know, specific, like, you know, privileged server rooms to, to use email.

• 1:55:41Lex Fridman?

  Yeah.

• 1:55:43Speaker 01

  Right, and hence, we, at the time, we had much less knowledge about the Western world. And actually, at the time, I didn't know, actually, the, the, in the US, the West Coast weather is much better than the East Coast.

• 1:56:00Speaker 01

  Yeah, things like that, actually. Uh, it's very, it's very interesting.

• 1:56:05Lex Fridman?

  Yeah.

• 1:56:06Speaker 01

  And but now it's so different. At the time, I would say there's also a bigger culture difference. Um, because there's so much less opportunity for shared information. So it's such a different, like, time and world.

• 1:56:19Lex Fridman?

  So let me ask maybe a sensitive question. I'm not sure, but I think you and I are in similar positions. As I've been here for, um, already 20 years as well. And looking at Russia from my perspective and you looking at China, in some ways, it's a very distant place because it's changed a lot. But in some ways, you still have echoes, you still have knowledge of that place. The question is, you know, China is doing a lot of incredible work in AI. Do you see? Please tell me there's an optimistic picture you see where the United States and China can collaborate and sort of grow together in the development of AI towards, you know, there's different values in terms of the role of government and so on,

• 1:57:02Lex Fridman?

  of ethical, transparent, secure systems. We see it differently in the United States a little bit than China, but we're still trying to work it out. Do you see the two countries being able to successfully collaborate and work in a healthy way without sort of fighting and making it an AI arms race kind of situation?

• 1:57:23Speaker 01

  Yeah, I believe so. I think science has no border. And the advancement of the technology helps everyone, helps the whole world. And so I certainly hope that the two countries will collaborate. And I certainly believe so.

• 1:57:44Lex Fridman?

  Do you have any reason to believe so, except being an optimist?

• 1:57:49Speaker 01

  So first, again, like I said, science has no borders. And especially in-

• 1:57:53Lex Fridman?

  Science doesn't know borders?

• 1:57:55Speaker 01

  Right.

• 1:57:56Lex Fridman?

  And you believe that, well, you know, in the former Soviet Union during the Cold War?

• 1:58:02Speaker 01

  So that's, yeah, so that's the other point I was going to mention is that especially in academic research, everything is public. Like we write papers, we open source codes, and all this is in the public domain. It doesn't matter whether the person is in the US, in China, or some other parts of the world. They can go on archive and look at the latest research and results.

• 1:58:26Lex Fridman?

  So that openness gives you hope?

• 1:58:28Speaker 01

  Yes.

• 1:58:29Lex Fridman?

  Me too.

• 1:58:30Speaker 01

  And that's also how, as a world, we make progress the best.

• 1:58:36Lex Fridman?

  So, apologize for the romanticized question, but looking back, what would you say was the most transformative moment in your life? That-

• 1:58:48Lex Fridman?

  Maybe made you fall in love with computer science. You said physics. You remember there was a moment where you thought you could derive the entirety of the universe. Was there a moment that you really fell in love with the work you do now? From security to machine learning to program synthesis?

• 1:59:05Speaker 01

  So maybe, as I mentioned, actually in college, I, one summer, I just taught myself programming C.

• 1:59:13Lex Fridman?

  Yes.

• 1:59:13Speaker 01

  I just read a book.

• 1:59:17Lex Fridman?

  Don't tell me you fell in love with computer science by programming in C.

• 1:59:21Speaker 01

  Remember I mentioned one, one of the draws for me to computer science is how easy it is to realize your ideas.

• 1:59:27Lex Fridman?

  Yes.

• 1:59:28Speaker 01

  So once I, you know, read the book, start, like, taught myself how to program in C, immediately, what, what did I do? Like, I programmed two games.

• 1:59:40Speaker 01

  One is just simple, like, it's a goal game, like, it's a board, you can move the stones and so on. And the other one, I actually programmed the game that's like a 3D Tetris. It was, it turned out to be a super hard game to play. Because instead of just the standard 2D Tetris, it's actually a 3D thing. But, like, I realized, wow, you know, I just had these ideas to, to try it out and then.

• 2:00:03Lex Fridman?

  You can do it.

• 2:00:04Speaker 01

  You can just do it. And so that's when I realized, wow, this is amazing.

• 2:00:10Lex Fridman?

  Yeah, you can create yourself.

• 2:00:12Speaker 01

  Yes, yes.

• 2:00:13Lex Fridman?

  Bring ideas.

• 2:00:13Speaker 01

  Exactly.

• 2:00:15Lex Fridman?

  From, from nothing to something that's actually out in the real world.

• 2:00:18Speaker 01

  Right.

• 2:00:19Lex Fridman?

  So let me ask.

• 2:00:19Speaker 01

  And throw your own hands.

• 2:00:21Lex Fridman?

  Let me ask a silly question, or maybe the ultimate question. What is to you the meaning of life? What, what gives your life meaning, purpose, fulfillment, happiness, joy?

• 2:00:36Speaker 01

  Okay, these are two different questions.

• 2:00:38Lex Fridman?

  Very different, yeah.

• 2:00:39Speaker 01

  It's interesting that you ask this question. Maybe this question is probably the question that has followed me and followed my life the most.

• 2:00:50Lex Fridman?

  Have you discovered anything, any satisfactory answer for yourself?

• 2:00:56Lex Fridman?

  Is there something, is there something you've arrived at? You know, there's a moment, I've talked to a few people who have faced, for example, a cancer diagnosis or faced their own mortality. And that seems to change their view of them. It seems to be a catalyst for them removing most of the crap, of seeing that most of what they've been doing is not that important. And really reducing it into saying like, here's actually the few things that really give me, give meaning. Mortality is a really powerful catalyst for that, it seems like. Facing mortality, whether it's your parents dying, or somebody close to you dying, or facing your own death, for whatever reason, or cancer and so on.

• 2:01:40Speaker 01

  Right, so yeah, so in my own case, I didn't need to face mortality to

• 2:01:49Speaker 01

  to try to, you know, to ask that question.

• 2:01:51Lex Fridman?

  Yes.

• 2:01:53Speaker 01

  And I think there are a couple things. So one is like, who should be defining the meaning of your life? Right, is there some kind of even greater things than you who should define the meaning of your life? So for example, when people say that searching the meaning for your life, is there some, is there some outside voice? Or is there something, you know, outside of you? Who actually tells you, you know, so people talk about, oh, you know, this is what you have been born to do.

• 2:02:31Lex Fridman?

  Right.

• 2:02:31Speaker 01

  Right, like, this is your destiny.

• 2:02:37Speaker 01

  So who, right, so that's one question, like, who gets to define the meaning of your life? Should you be finding some other thing, some other factor to define this for you? Or is something actually, it's just entirely what you define yourself? And it can be very arbitrary.

• 2:02:54Lex Fridman?

  Yeah, so in the inner, an inner voice or an outer voice, whether it's, it could be spiritual, religious too, with God, or some other components of the environment outside of you, or just your own voice. Do you have a, do you have an answer there?

• 2:03:09Speaker 01

  So, okay, so for that, I have an answer.

• 2:03:11Lex Fridman?

  Yeah.

• 2:03:12Speaker 01

  And through, you know, the long period of time of thinking and searching, even searching through outsides, uh, right, you know, voices or factors outside of me.

• 2:03:25Lex Fridman?

  Yeah.

• 2:03:25Speaker 01

  So that I have, and so I've come to the conclusion and realization that it's you yourself that defines the meaning of life.

• 2:03:35Lex Fridman?

  Yeah, that's a big burden though, isn't it?

• 2:03:40Speaker 01

  I mean, yes and no, right? So then you have the freedom to define it.

• 2:03:45Lex Fridman?

  Yes.

• 2:03:46Speaker 01

  And, and another question is, like, what does it really mean by the meaning of life?

• 2:03:54Speaker 01

  Right. Um, and also, why did the question even make sense?

• 2:04:02Lex Fridman?

  Absolutely, and you said it somehow distinct from happiness. So meaning is something much deeper than just any kind of emotional, any, any, any kind of contentment or joy, whatever. It might be much deeper. And then you have to ask, what is deeper than that? What is, what is there at all? And then the question starts being silly.

• 2:04:25Speaker 01

  Right, and also you can say it's deeper, but you can also say it's a shallower, depending on how people want to define the meaning of their life. So for example, most people don't even think about this question. Then the meaning of life to them doesn't really matter that much. And also whether knowing the meaning of life, uh, whether it actually helps your life to be better or whether it helps your life to be happier. Um, these actually are open questions. It's not.

• 2:04:53Lex Fridman?

  Of course, most questions are open. I, I tend to think that just asking the question, as you mentioned, as you've done for a long time, is the only, that there is no answer. And asking the question is a really good exercise. I mean, I have this, for me personally, I've had a kind of feeling that creation is a...

• 2:05:13Lex Fridman?

  Like for me has been very fulfilling and it seems like my meaning has been to create. And I'm not sure what that is like I um I don't have I'm single I don't have kids. I'd I'd love to have kids, but I also, it sounds creepy, but I also see sort of you said see programs. I see programs as little creations. I see robots as little creations. I I think I think it's those are met those those bring and then ideas, theorems and and are are creations. And those somehow intrinsically, like you said, uh bring me joy. And I think they do to a lot of at least scientists, but I think they do to a lot of people.

• 2:05:51Lex Fridman?

  So that to me, if I had to force the answer to that, I would say creating new things yourself.

• 2:06:00Speaker 01

  For you.

• 2:06:01Lex Fridman?

  For me, for me, for me, I I don't I don't know. But like you said, it keeps changing. Is there some answer that?

• 2:06:07Speaker 01

  And some people, they can, I think, uh they may say it's experience, right? Like their meaning of life, uh they just want to experience to the richest and fullest they can. And a lot of people do take that path.

• 2:06:20Lex Fridman?

  Yes, seeing life as actually a collection of moments and then trying to make the richest possible sets. Fill those moments with the richest possible experiences. Yeah.

• 2:06:31Speaker 01

  Right. And for me, I think certainly we do share a lot of similarity here. Like so creation is also really important for me, even from, you know, the things I've already talked about.

• 2:06:42Speaker 01

  Even like, you know, writing papers and these are our creations as well.

• 2:06:47Speaker 01

  And I have not quite thought whether that is really the meaning of my life. Like in a sense also that maybe like what kind of things should you create? There are so many different things that you could create.

• 2:07:00Speaker 01

  And also you can say another view is maybe growth is.

• 2:07:05Speaker 01

  It's related but different from experience. Growth is also maybe type of meaning of life. It's just you try to grow every day, try to be a better self every day. And and also ultimately, we are here. It's part of the overall evolution. The right, the world is evolving.

• 2:07:28Lex Fridman?

  And it's funny, isn't it funny that the growth seems to be the more important thing than the thing you're growing towards? It's like, it's not the goal, it's the the journey to it. Sort of, it's almost, it's almost when you submit a paper. It's, there's a sort of depressing element to it. Not to submit a paper, but when that whole project is over. I mean, there's a gratitude, there's a celebration and so on, but you're usually immediately looking for the next thing.

• 2:07:56Speaker 01

  Yeah.

• 2:07:56Lex Fridman?

  Or the next step, right? It's not, it's not that satisfied, the end of it is not the satisfaction. It's the hardship, the challenge you have to overcome, the growth through the process. It's some, it's somehow probably deeply within us, the same thing that drove, that drives the evolutionary process. It's somehow within us with everything, the way, the way we see the world. Since you're thinking about this, so you're still in search of an answer.

• 2:08:20Speaker 01

  I mean, yes and no. In a sense that I think for people who really dedicate time to search for the answer, to ask the question, what is the meaning of life? It does not necessarily bring you happiness.

• 2:08:37Lex Fridman?

  Yeah.

• 2:08:37Speaker 01

  It's a question, um, we can say, right, like whether it's a well-defined question and

• 2:08:45Speaker 01

  and on the, and but on the other hand, given that you get to answer it yourself, you can define it yourself. Then sure, then I can just, you know, give it an answer. And in that sense, yes, it can help.

• 2:09:03Speaker 01

  Like, like we discussed, if you say, oh, then my meaning of life is to create or to grow. Then, then yes, then I think they can help. But how do you know that that is really the meaning of life or the meaning of your life? It's.

• 2:09:21Speaker 01

  Like there's no way for you to really answer the question.

• 2:09:23Lex Fridman?

  For sure, but something about that certainty is liberating. So if it might be an illusion, you don't, you might not really know. You might be just convincing yourself falsely. But being sure that that's the meaning. The, there's something. There's something liberating in that, in that. There's something freeing in knowing this is your purpose. So you can fully give yourself to that without, you know, for a long time, you know, I, I thought like, isn't it all right? Like, why? What's, how do we even know what's good and what's evil? Like, isn't everything just relative? Like, how do we know? You know, uh, the, the question of meaning is ultimately the question of why do anything? Why is anything good or bad?

• 2:10:07Lex Fridman?

  Why is anything valuable or so on?

• 2:10:09Speaker 01

  Exactly.

• 2:10:10Lex Fridman?

  But the moment, then you start to. I think, just like you said, I, I think it's a really useful question to ask. But if you ask it for too long and too aggressively.

• 2:10:25Speaker 01

  It may not be so productive.

• 2:10:26Lex Fridman?

  It may not be productive and not just for traditionally societally defined success, but also for happiness. It seems like asking the question about the meaning of life is like a trap. It's a, we're destined to be asking, we're destined to look up to the stars and ask these big why questions we'll never be able to answer. But we shouldn't get lost in them. I think that's probably the, that's at least the lesson I picked up so far. On that, on that topic.

• 2:10:54Speaker 01

  Oh, let me just add one more thing. So it's interesting. So actually, so sometimes, yes, it can help you to focus. So when I, when I shifted my focus more from security to AI and machine learning. At the time, the, actually, one of the main reasons that I, I did that was because at the time. I thought my meaning, the meaning of my life and the purpose of my life is to build intelligent machines.

• 2:11:31Lex Fridman?

  And that's, and then your inner voice said that this is the right, this is the right journey to take to build intelligent machines and that you're actually fully realize you took a really legitimate big step to become one of the world-class researchers to actually make it, to actually go down that journey. Yeah, that's profound. That's profound. I don't think there's a better way to end a conversation than talking for a while about the meaning of life. Dawn, it's a huge honor to talk to you. Thank you so much for talking today.

• 2:12:05Speaker 01

  Thank you, thank you.

• 2:12:07Lex Fridman?

  Thanks for listening to this conversation with Dawn Song, and thank you to our presenting sponsor Cash App. Please consider supporting the podcast by downloading Cash App and using code LEXSPodcast. If you enjoy this podcast, subscribe on YouTube, review it with five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at @LexFriedman. And now, let me leave you with some words about hacking from the great Steve Wozniak. A lot of hacking is playing with other people, you know, getting them to do strange things. Thank you for listening and hope to see you next time.