Staving off disaster through AI safety research

While covering Applied Machine Learning Days in Switzerland, Chris met El Mahdi El Mhamdi by chance, and was fascinated with his work doing AI safety research at EPFL. El Mahdi agreed to come on the show to share his research into the vulnerabilities in machine learning that bad actors can take advantage of. We cover everything from poisoned data sets and hacked machines to AI-generated propaganda and fake news, so grab your James Bond 007 kit from Q Branch, and join us for this important conversation on the dark side of artificial intelligence. Sponsors: Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogRollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: El Mahdi El Mhamdi – Website https://scholar.google.ch/citations?user=kNA-WLQAAAAJ, X https://x.com/L_badikhoChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: El Mahdi El Mhamdi on LinkedIn https://www.linkedin.com/in/mahdielmhamdiGoogle Scholar https://scholar.google.ch/citations?user=kNA-WLQAAAAJPersonal blog https://elmahdielmhamdi.comWorld Health Organization | Ten threats to global health in 2019 https://www.who.int/emergencies/ten-threats-to-global-health-in-2019AggregaThor https://github.com/LPD-EPFL/AggregaThor/blob/master/README.mdThe Hidden Vulnerability of Distributed Learning in Byzantium http://proceedings.mlr.press/v80/mhamdi18a.htmlMachine Learning with Adversaries: Byzantine Tolerant Gradient Descent http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdfAsynchronous Byzantine Machine Learning (the case of SGD) https://arxiv.org/abs/1802.07928Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode Cloud Servers. Head to linode.com/changelog. This episode is brought to you by Linode, our cloud server of choice, and we're excited to share their recent launch, dedicated CPU instances. If you have build boxes, CI/CD, video encoding, machine learning, game servers, databases, data mining, or application servers that need to be full duty, 100% CPU all day, every day, then check out Linode's dedicated CPU instances.

• 0:39Speaker 00

  These instances are fully dedicated and shared with no one else. There's no CPU steal or competing for these resources with other Linodes. Pricing is very competitive and starts out at 30 bucks a month. Learn more and get started at linode.com/changelog. Again, linode.com/changelog.

• 1:09Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at changelog.com/community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:34Chris Benson?

  Welcome to the Practical AI Podcast. This is Chris Benson, your co-host, as well as the Chief AI Strategist at Lockheed Martin RMS APA Innovations. This week, you're going to hear one of a series of episodes recorded in late January 2019 at the Applied Machine Learning Days Conference in Lausanne, Switzerland. My co-host, Daniel Whitenack, was going to join me, but had to cancel for personal reasons shortly before the conference. Please forgive the noise of the conference in the background. I recorded right in the midst of the flurry of conference activities. Separately from the podcast, Daniel successfully managed the AI for Good track at Applied Machine Learning Days from America, and I was one of his speakers. Now, without further delay, I hope you enjoy the interview.

• 2:15Chris Benson?

  My guest today is Elmadi Elmehandi and he is a PhD student who's just finishing up here at EPFL in Switzerland and he has been focusing on technical AI safety and robustness in biological systems. Welcome to the show and did I actually say your name correctly?

• 2:34Speaker 02

  That was, that was good.

• 2:36Chris Benson?

  And, and if you could start us off, we've talked a little bit before we started recording, you have a fascinating background. Will you share a bit of that as we start this off with the listeners?

• 2:46Speaker 02

  I've been trained as a physicist. So I did math and physics as a bachelor's in Morocco, then moved to France, Switzerland and Germany. But I, I've been trained as a physicist. I even worked in physics research. I've been a research engineer in physics, physics of condensed matter, like semiconductors for, for like things like photovoltaics, solar cells.

• 3:10Speaker 02

  But then I drifted a bit for about five years before coming back for a PhD. So I started like, I did research in physics, but then I, and at the same time, with some friends, we co-founded a media platform in Morocco called Menfekinj, which was some sort of like a news aggregate. During the 2011 events, that's some, some people called the Arab Spring.

• 3:36Speaker 02

  And during that period, I was more and more convinced that the web was enabling through those platforms, tools to help people circumvent usual platform, like, sorry, usual intermediate bodies like electoral, political parties, established news organizations to self-organize. But at the same time, there was a harmful effect, which, which we will start being more aware of five years later during the Yeah, the last events in the US, for example.

• 4:09Chris Benson?

  And would that be misrepresentation of events like fake news and that kind of thing are you referring to?

• 4:14Speaker 02

  True. And so back then, 2011, 2012, there was another thing that caught my attention, which was that whenever we put a lot of effort, me and my colleagues at Mefekinj, would put a lot of effort in doing a deep investigative work on some very relevant public issue and then publish it, the readership would be very low compared to a three-minute video by some activists who just like self-record himself or herself with a basic camera and like starts speaking in a very simple words

• 4:53Speaker 02

  and it will take off. And back then, 2012, say, I kind of stopped being very involved in Mefekinj. I was still working in physics, by the way. But I thought that the video platforms will play an even increasing role as the bandwidth and access to heavy content like video will be democratized. And I said, "Yeah, okay, so the video sharing seems to be more powerful than text sharing on the web." I think this can help a lot into this, like, something that also I care about, which is science education.

• 5:32Speaker 02

  So as much as, as much as politically, like, as much as videos on political issues would have more spread than text, I thought that videos on science would have for like a general audience for kids, to make kids motivated about science or like just to tutor people. I was not aware of the Khan Academy back then. Someone showed me Khan Academy after I did my first videos. And so I said, good, okay, I will start a video project that would just do tutoring in physics and maths, something I know, something I was kind of good at doing, which is tutor people in maths and physics. So I started a tutoring project in maths and physics while working in condensed matter

• 6:17Speaker 02

  physics back then. But then a professor here in computer science, Rashid Gharawi, convinced me to join efforts and to do that full time with him here in Lausanne. So we got initial funding from Google and from the faculty. Then the faculty, of course, like the faculty took over the funding and we got enough funding to do it full time. And I started learning about computer science as a fundamental science and was realizing how epistemologically relevant computability and like concepts like decidability were to understand.

• 6:56Chris Benson?

  And about when was that, just for the timeline, from 2011, 2012, this would have been moved on another year or so?

• 7:03Speaker 02

  This transition would happen in 2013. So in 2013, I left my job as a physicist engineer, came to Lausanne, like by the mid, like June, I think, June 2013, I left my job as a research engineer in physics and came to Lausanne to fully start this tutoring project that became an official tutoring platform of EPFL. So bachelor students, we had like a very good reception through, like, it's not like the kind of YouTube channel that would go popular because it's on very specific and technical topics and it's in French. Like, so most of the content is in French because EPFL is a French-speaking university. So the audience was not huge, it's a small-sized audience, but there was a high quality.

• 7:45Speaker 02

  For example, we had a high retention rate compared to, say, MOOCs, like MOOC platforms, they had a, they had a 8% or 7% retention rate. We had something close to 70% retention rate because it was a tutoring. So we were addressing questions bachelor students of EPFL would struggle on before an exam, like how to compute this derivative of this physicist stuff, whatever object.

• 8:10Chris Benson?

  So that, that really does sound quite a lot like Khan Academy and what you were doing. Obviously, you're doing it in French and doing it for the students here, but, so we can kind of think of it in that kind of context and where you're going. So, and so where did that lead you?

• 8:25Speaker 02

  Uh, it led me to, because, because, because I was funded by the computer science department, it led me to learn more about computing. So back then, when I was trained as a physicist, I viewed computer science as this engineering thing where you debug Java and C++ code I didn't like, really. But I was not aware and I was not educated on this fundamental science of computing. And little by little, I started like educating myself. Uh, I started learning, learning, I started learning, learning, that's so meta. So I started reading about learning theory, the work of Leslie Valiant, for example, the work of Wapnick, Shervonenkis, and also like the fundamental CS part, like Turing.

• 9:09Speaker 02

  And I kind of buy into a few calls, for example, from Leslie Valiant to make computing a natural science. I think, I think it's a very powerful epistemological tool to, to understand natural phenomena in terms of like, I, I like to call computing as a, the science of the feasible. Like what can be done, what can be done like complexity theory, what can be done in an amount of time with an amount of resource. And I, I, I, I'd like to view learning theory as the science of the learnable. What can be learnt, given an amount of time and an amount of data points and amount of samples. Uh, and I love that, so I wrote a proposal to start a PhD, uh, trying to understand biological processes with, uh, computability tools.

• 9:54Speaker 02

  Not, not in the, uh, not as a computer scientist collaborating with biologists and like coding stuff for them, but bringing, not bringing the engineering part of CS, bringing the epistemological part of CS that view complex systems through complexity theory, uh, resource, et cetera. And, um, the, the, the, the main, the main, the main guiding line was robustness. So could we explain robustness in biological processes with computational tools? So could we explain, for example, why an ant colony? Is robust to randomly killing some of the ants up to certain level,

• 10:39Speaker 02

  without having a central authority allocating tasks and telling ants, "Oh, by the way, we had a certain amount of foragers that died. Yeah, those of you who were doing, I don't know, nursing should switch to foraging." And we know, we know, like, myrmecologists, biologists who study ants, know that there is no central authority doing that. Like, it's self-organized, and it's robust, it's fault-tolerant. The brain also is a very good example of a robust structure, where there's no central authority telling neurons what to do, or, to a certain extent, it's very distributed and robust, and, like, it can tolerate the loss of some of the nodes. So, that was the starting line. Let's understand the fault tolerance of biological processes

• 11:24Speaker 02

  with tools from algorithmic theory and those kind of tools. And this is in the Distributed Computing Group. So that was like a very physics-y. So that was something that could bring the physicist in me again to liking, like, doing research, five years after I left my masters. But little by little, I was, just my awareness on on more applied aspects of machine learning will grow. So I was, like, I told, I was trying to understand, like, the fault tolerance in neural networks. So how does error propagate in a neural network

• 12:07Speaker 02

  when some of the neurons are removed? This is, today, this is not a practical problem, because neural networks, like, neurons in neural networks do not fail. A neural network is simulated in a machine, so the unit of failure is a whole machine, not a single neuron. This will become a problem when we will have neuromorphic hardware, if you heard about this. Could you, could you define what that is specifically? Neuromorphic hardware is a class of hardware that is itself built as a neural net. So the hardware itself contains... Pieces that behave like a neuron and pieces that behave like a synapse. While today, we just simulate neural networks as a software.

• 12:48Chris Benson?

  So would it be fair to say then that because you are implementing hardware in the form of a neural network, that you can have, just like any other machine out there, you can have parts of the machine fail and therefore, unlike today where it's just software and you either have it's all working or it's not, you can have parts of that hardware in the form of a neural network fail and therefore, it's a new problem for us to solve, which is why you were saying it's not practical? Is that, am I understanding you correctly?

• 13:13Speaker 02

  It's not a really new problem. It was a very popular problem in the 90s and 80s before the last AI winter because people were expecting neuromorphic hardware to arise next day. So people, like you find a lot of papers in the 80s, 90s about fault tolerance in neural nets and they will talk about VLSI circuits, very large. So, and, but then neuromorphic hardware didn't happen and we simulate neural nets on, on, on, on machines and people stopped caring about this problem. But, but yeah, I find it a very good physics, I find it a very good problem for someone who thinks like a physicist, like me.

• 13:56Speaker 02

  So I, I, I cared about it even though there's no neuromorphic hardware room in, room in the air today. But little by little, people who are relevant in machine learning would tell me, yo, look, we don't care about this yet. Though, yeah, if you could understand how error impacts learning in distributed frameworks, like when we train machine learning systems over a set of machines, that might be relevant today. So I switched a bit of interest. I, I, I published a paper on, like I approved some bounds on error propagation in neural nets. The mathematical modeling I did there was also useful to study biomolecular networks with some friends from the Johns Hopkins Medical School because it turns out that biomolecular

• 14:37Speaker 02

  networks are just weighted graphs of nonlinear nodes, just like neural nets.

• 14:42Chris Benson?

  Oh no, that's pretty cool. I had never thought of it that way. Okay, so that's the, that, so that, I know that you, I was, I was gonna ask you that, was that you had talked about that factness that you were dealing with robustness in biological systems with the technical AI safety and is that the crossover there? Is that, are we getting to that or am I, am I jumping in?

• 15:01Speaker 02

  Not yet, but the glue is already there. The glue is fault tolerance. So there are like two. Hemispheres in my PhD. One hemisphere was doing robustness in biological systems. And one hemisphere was doing technical AI safety. They don't seem to be related, but they are actually truthful tolerance. So I cared about fault tolerance. What happens in a complex system when some nodes are knocked out? Or like are misbehaving or are lying to the group?

• 15:28Chris Benson?

  Oh, okay there. So you've gotten to the correct side. I know that as we were talking when we first met and you started talking about that. I can't wait to hear how this goes in because it's fascinating how you've pulled together multiple fields that may not be obviously related upfront, but through fault tolerance. And then you were making comments earlier about how this affects things like fake news and falsified information that goes forward. So take us there.

• 15:57Speaker 02

  Let's go to the more the technical AI safety part of my research. I like to tell this, like when I say that I work in, when I say to my friends, oh, now like for the past two years, I switched a bit interest. I'm caring about technical AI safety. They would go like this. Oh, yeah, isn't this about killer robots and rogue self-driving cars and things we'll have in the far future?

• 16:22Speaker 02

  And I think partly because the media was always showing those kind of motivation when they talk about AI safety. I always like to tell them that there are killer robots already about us. They're very dumb and primitive and doing very basic machine learning and they are called recommender systems.

• 16:40Chris Benson?

  No, that's great, but you'll have to, you'll have to kind of explain what you mean by that because that's a little bit of a shocker when you hear that.

• 16:47Speaker 02

  So imagine a young couple of parents who just had a kid. And then they go to a search engine and type medical advice on vaccines for young kids. And then they got an initial piece of content that tells them that this is harmful, it can cause autism, and then their kids can die. And this is really a conspiracy by Big Pharma to make us. Just buy their products. And then the platform recommend them another video telling them similar stuff. And another one and another one and another one. And actually, that could also happen to people who didn't even search for that.

• 17:31Speaker 02

  Just like, they were just like looking for medical advice on some random topic, I don't know, for herpes. And then they end up on a video telling them, oh, there's this big pharma conspiracy, don't take your kids for vaccine.

• 17:42Chris Benson?

  So it's funny that you say that because I actually have friends and even extended family members that that exact use case has applied for them. And we have gotten into debates on the benefit of vaccine. And so I love the fact that, you know, you started in kind of from that academic perspective, but you're now touching on something that affects lives every day by millions of people out there. And it's a very common misconception. So I love the fact that where you're going, keep going, sorry about that.

• 18:16Speaker 02

  Now today, you know, just this year, I think for the first time in maybe several years, I don't know how much, but like for the first time in at least the past five years or so, the World Health Organization listed vaccine hesitancy as a public health issue. This is listed, I'll give you the reference after the, it is listed, so you can give the link to the audience.

• 18:43Chris Benson?

  Yeah, we'll include that in the show notes.

• 18:45Speaker 02

  So the World Health Organization listed vaccine hesitancy in its 2019 report on the measure, like it's now in the rank of HIV and Ebola. Because there is a surge of anti-vaccine resentment and there is a surge of vaccine preventable diseases. Some estimations I remember from that report.

• 19:08Speaker 02

  Maybe check if I like my, I might miss some details, but for example, there is a surge of thirty percent in measles in developed countries. I'm not talking about, I'm not talking about like countries that solved, that used to solve that. Measles thirty percent. There are some other reports who speak about about a thousand six hundred deaths in the US per year from vaccine preventable diseases. That's three per day, like more than terrorism. So this is, this is a less reliable report, but like the World Health Organization One is talking about a thirty percent surge of measles, and that's a vaccine preventable disease. And the resentment is growing. There were also studies on the people's opinion on vaccines in France

• 19:53Speaker 02

  today and ten years ago, and they consistently show a growth of this resentment. So this is clearly a public health issue. And we can say with very, with confidence that, yeah, poison the machine learning already kills. People think about killer robots. I'd like to tell them, oh, let's just first care for, right? Let's care about poisoned recommender systems and probably what we will do to solve that might probably help in preventing something in the long term. People tend to think about killer robots in the long term and far future stuff we shouldn't worry about too much. I'd like to reply, I'd always like to reply that, no, no, we should care about killer

• 20:38Speaker 02

  recommender systems that are pushing parents into not vaccinating their kids. There are surges of cases like measles, not only in the US, in Switzerland here, there was, there was a case last year, there was an outbreak in the primary school, I think, or kindergarten. So in the Morsch, I think you can, you can search for that, in the, in this region, Lausanne region. And this is a serious problem that is literally already killing some people. I think new generations who didn't witnessed the past, like my generation didn't see what does a non-vaccinated past look like. I'm from Morocco, I grew up in Morocco until I was 21. My aunt had polio, she was handicapped for life. She was born in the 50s and she was not vaccinated back then. So I could see what a non-vaccinated past looked like.

• 21:23Speaker 02

  I think it was even uglier than what I could see because I just saw the survivors. And I think that my generation in the West is not aware of how lucky we are today. And the commander's systems today, as they maximize watch time. So, yeah, the problem is that like when we do, when we maximize for some metric, we tend to like screw stuff in other metrics. And maybe maximizing watch time is now leading to what we do today. So what's, what's, what's, how could we turn that into formalizable scientific questions? If you, if you look at machine running today, if you, if you look at how it is done,

• 22:04Speaker 02

  you would find that fundamentally, there is a computing, there is, there is an averaging mechanism. So when you do gradient descent, that's just a protocol to update parameters.

• 22:19Speaker 02

  That's just a protocol to update, update parameters.

• 22:24Speaker 02

  Okay, you do it thanks to some data points. So you leverage some data points, you compute gradients using those data points, and then you aggregate those gradients. And how it is done today, it's mostly with averaging those gradients or variance of averaging. If you ask a sociologist about averaging,

• 22:44Speaker 02

  like would you do averaging to do socioeconomics of a region? Any reasonable sociologist would tell you, please don't take the average. As a funny illustration, maybe it's not really funny, but it's a bit sad.

• 22:59Speaker 02

  Always in my talks, I ask people, who thinks that the GDP per capita in the, in Finland, Denmark and Sweden is higher than the GDP per capita in the US? Most people in the room raised their hand because they think that the GDP per capita in Sweden, Finland and Denmark is higher. Actually, that's the opposite. It's slightly higher in the US.

• 23:21Chris Benson?

  So I know that I was one of those people you're referring to. I would have said the other way around. It's interesting. I did not realize that.

• 23:27Speaker 02

  Yeah, you have even more striking cases. Like you can take the GDP per capita in Germany and the US or something like that. And you will find that the one in the US is way higher, I think. But for sure, like Denmark, Finland and Sweden have GDPs per capita according to the last OECD or like CIA report. Slightly lower than the one in the US, but no one is fool enough to say that the typical Swedish citizen has a poorest life or a comparable life to a typical US citizen.

• 24:05Speaker 02

  Unfortunately, the typical US citizen tends to have less access to public education, healthcare, etc. Why? Because averaging is not robust. If you take the average and you have a bunch of over rich billionaire and several homeless people, yeah, the average might be good. I come from a country where this is also like, I think like when I meet a Moroccan and an Algerian, so we have a neighboring country called Algeria. And if you ask any educated Moroccan or Algerian in Europe, where do you think the

• 24:46Speaker 02

  median, now this time, the median income, access to healthcare or whatever is higher? They would tend to say Morocco. Because there's like, because there's like this big outline in cities like Rabat and Casablanca where you see fancy constructions and very good cars on the road and think that, yeah, this country seems to be a bit richer than Algeria, but it turns out that's not the case. The median Algerian has a better life than the median Moroccan. But Morocco has a bunch of outliers that think of themselves as a middle class while they are not.

• 25:25Speaker 02

  So sociologists, like short story, sociologists were aware of the weakness of averaging from at least 19th century. If you read Emil Durkheim or yeah, like if you read the, oh no, sorry, Weber. The first data scientists were probably sociologists. And they were aware of this problem. And they will tell you, yeah, like, take the salaries, rank them, take the one that splits the distribution into halves. That could be a better way to evaluate a country than taking the average. So if you do that.

• 25:57Chris Benson?

  I see what you're saying on that. And I was gonna ask you how that, how that the weakness of averages were kind of tying back in to the use case that you're addressing there.

• 26:06Speaker 02

  So now, yeah, the naive idea is, yeah, let's port that into machine learning. Let's take median gradients instead of average gradients. So people behave on a social network. Their behavior create gradients. What's happening today is that this social network will use the average gradients to update the model. If there is a minority of hyperactive, hypermotivated extremists, they might screw the recommender system.

• 26:35Chris Benson?

  So this is, so to tie this back in, I mean, this is exactly what we're seeing day in and day out, you know, with, with the impact of social media in a negative way on our lives. So it's fascinating as you've kind of come in through this academic path that you've taken, but you've, you've landed squarely in the middle of a gigantic problem that we're facing around the world. I know as a US citizen, we are having a lot of a political conversation right now around exactly this. So what are the implications of this?

• 27:05Speaker 02

  The implications might be, for example, what's happened last year with the crisis actor conspiracy. So I don't know if you remember, there was this very sad shooting in Florida in the Parkland and that high school. And a few survivors of that shooting, David Hogg, Emma Gonzalez and others, they raised to prominence with their campaign promoting more safety measures and gun control measures that will like protect high schools from shootings. And there was a video claiming that those kids are not real survivors from the shooting. They were crisis actors.

• 27:44Speaker 02

  Crisis actors used to promote gun control on television. And this video went on the front page of YouTube.

• 27:55Chris Benson?

  So basically, you're talking about an instance of pure fake news in terms of you're having a bad actor that is creating a fiction to serve their end, just to serve their end, it had no basis in reality.

• 28:10Speaker 02

  But it doesn't end on the video and being featured. And so if you went to youtube.com that day, you would find this video in the US. That was the featured video on the front page. But it didn't end there. Those kids received death threats because people believed the video. The video spread. It was, it became very popular and the spread was done. Even though YouTube apologized, so YouTube apologized, of course, later. And they fixed the problem. But it was too late, like the harm was done. The kids received death threats and what I imagine, you are surviving a shooting and then you receive death threats.

• 28:52Speaker 02

  Because people massively saw a video saying that you are a crisis actor going to the television to promote a political ideology of gun control.

• 29:03Chris Benson?

  So is your research into robustness and stuff, how is your research, how can it be applied to these real life situations that we're all trying to figure out right now?

• 29:17Chris Benson?

  How would you, what are your solutions?

• 29:18Speaker 02

  Of course, real life solutions are very complex. I'm not claiming that we have bulletproof solutions to complex real life problems, but we could at least fix the obvious real life problems. And the obvious real life problems is that recommender systems should stop averaging gradients, for example. I'm not claiming that this is pure poisoning what happened to YouTube. I don't know, I don't know what happened exactly to YouTube. But I would say a first fix would to stop stopping taking the average. Maybe if YouTube already fixed that, or maybe YouTube, maybe that's another problem that I was not aware of.

• 29:59Speaker 02

  But let's say, let's say there is a situation where you average people's behavior. And a first fix would to stop averaging because you would be vulnerable to extremist groups.

• 30:11Chris Benson?

  So would it be fair, earlier you mentioned median, would that be a better selection? So fundamentally, the approach that we're taking in machine learning in terms of the choices we're making as we're putting our algorithms together for a given use case or solution. In some cases, maybe we're making, we're kind of following the herd. And we're doing what other people have done on other projects. But in the case that we're talking about, it's not serving us well because you can have extreme ends of that distribution that are able to take advantage of it.

• 30:43Speaker 02

  Most importantly, spotting those extreme ends today is becoming harder and harder. If you talk, I talked to bankers and insurance companies. They're very good at doing fraud detection. And they typically would do it with tools like PCA. I don't know how much details I should go into this podcast, but this is a method that detects like big tendencies in a data set. The problem with that, so it's very good to spot outliers. But the cost of doing it grows quadratically as the data set is big. So it prevents you from leveraging high dimensional

• 31:26Speaker 02

  big data, as we like to say today. So it narrows down the scope of your tool to simple linear regression, logistic regression. You can't do it, you can't do those kind of fraud detection mechanism on something as massive as a video platform. So we need something that scales most linearly with the dimension of the model, of the data. And finding something that behaves like a median in high dimension is a hard problem.

• 32:03Speaker 02

  So the the the technical solution we've been working on, me and my colleagues, since I jumped on this problem two years ago or so, like I took a break from the biological robustness track, I'm I'm getting back to it now, but I took a break for two years and I fully worked on this poisoning resilience and another AI safety question called safe interoperability with some friends. But yeah, on the poisoning side, we've been trying to find alternatives to the median because in high dimensions, you can't, as I said, you can't rank, like you rank salaries and then you spot the salaries that split the salaries into two halves. Half the population earns less than 3,000, half the population earns more than 3,000,

• 32:48Speaker 02

  3,000 is the median, fine. How do you do that for vectors? So for multi-dimensional data, you can't rank vectors, you can't say, oh, this is smaller than this one, this one. Imagine like you have, you have a million spreadsheets, each spreadsheet containing a million cells.

• 33:06Speaker 02

  You can't rank them. So you you want to find the median spreadsheets, that's that's more or less what we're trying to do in a practical manner, so fast. And so that's what we've been doing, we've derived a series of algorithms that behave like a median and that's like provides guarantees that it's it is bounded in between a majority of points, etc. And we proved, so. So we've been also promoting the fact that security measures to always like have a rigorous proven, whenever we found like a bug, we have to like go back and modify, but it it's

• 33:46Speaker 02

  not, it's very good to, security measures should not be supported only by empirical evidence because you can never simulate all the possible attacks. So we we always tried to prove that this protocol called gradient descent will always converge despite the the existence of a fraction of of poisoners. So we we had the first paper on. That in NURIPS 2017. Uh, I'll give references and if you want to.

• 34:19Chris Benson?

  Yeah, we'll definitely include those in the show notes. Is, I guess, is it, is it fair to say these higher order algorithms that you're talking about, is this a way of kind of maybe evolving gradient descent? Or maybe replacing it in such a way that, that we start having real tools to deal with poisoning? And with, with fake news instances and such as that?

• 34:38Speaker 02

  Yeah, so, so talking about tools, I, I've been so, my, my work has been more on like, on, I was the guy who like would find an algorithm and prove that this algorithm satisfies this requirement. But then I've, I've been trying also to, to work with my colleagues and quarters who are more uh, on the engineering side to port this uh, on, on, on tools as soon as possible. And um, we have, so I said, yeah, we had this first paper in NURIPS. Then we published follow-ups in ICML, two follow-ups in ICML 2018, one in asynchronous settings and one in very high dimensional settings. But now we have a fourth um, work where we took TensorFlow,

• 35:24Speaker 02

  uh, like this famous Google uh, framework to do machine learning. We took TensorFlow and we replaced every averaging in the gradient aggregation parts of it with other algorithms uh, been promoting for the past two years. And my friend, my colleagues, Sebastian, Sonny and George, uh, they, they made it work on TensorFlow. And not only that, as also as a side bonus, they also made TensorFlow work communicating with UDP. So, so now, not only like TensorFlow, like the, the version of TensorFlow we'll publish on GitHub this week, uh, is Byzantine resilient, so it tolerates poisoning gradients up to, up to a certain fraction.

• 36:08Speaker 02

  But it also can communicate over UDP, which is an unreliable communication protocol, uh, instead of the previous one who, which required TCP IP. Because you could not afford losing packages, et cetera. So as, as, as like, as a bonus, now you can communicate over a. Faster but less reliable communication channel. That's not really like it doesn't have to do only with the medium stuff. They also did some some technical changes.

• 36:33Chris Benson?

  So if you were an engineer out there and you'd listened to this and wanted to to take advantage of that. So because I had a sense that that's where you were going in terms of of the research. You've you've now kind of have your own approach to gradient descent. Do you foresee that ever being included with TensorFlow? Or do you think it's a is it is it is the usage of of what the output of the work? These tools that you've created. Do you think it'll be common enough for dealing with things like poisoning? And we're dealing with, you know, bad actors trying to take advantage of the data set. Do you think we're gonna we're gonna gradually evolve into using these types of of updated algorithms? To replace the average based stuff?

• 37:16Chris Benson?

  Or do you think it's always going to be a little bit more a specialized thing?

• 37:20Speaker 02

  I like I don't know if you know Stuart Russell, this famous professor at Berkeley. He's a very Stuart Russell. Yeah, so Stuart Russell is one of the pioneers of of modern AI. He wrote that textbook AI, a modern approach with Peter Norvig. And I like one of his arguments. We met in a conference weeks ago in Puerto Rico in this beneficial AGI. Beneficial, sorry. And beneficial AI conference in Puerto Rico by the Future of Life Institute. And I like one of his arguments for AI safety where he said, if you talk to civil engineering people,

• 38:02Speaker 02

  you will never find someone talking about bridges and someone else talking about safe bridges, which are bridges that do not fall apart after three hours. So not falling apart after three hours of deployment is part of the definition of a bridge.

• 38:22Speaker 02

  I think that what the feeling I had from talking to attendants of Applied Machine Learning Days is we are going slowly towards this good direction where most of the people involved in machine learning research are more and more aware that not falling apart after a few hours of production is part of the definition of a bridge. And I think we will stop talking about safe AI and AIs. Like, it was just, it will just, it should become part of the definition.

• 38:51Chris Benson?

  Yeah, so it sounds like it's a foundational thing that we probably should have been thinking about ahead of time, but it will become the de facto standard. It's essentially going to, the success of safety AI essentially eclipses itself. It just becomes AI and the tools we use.

• 39:06Speaker 02

  So, and then now coming back to your question, is poisoning so like, maybe I'm rephrasing exactly, not exactly what you said, but is poisoning really solvable like that? The bad news, there is always a bad news in computing. People, people tend to forget that computer science was founded by an impossibility theorem. Turing, before proving what algorithms could do, he started by proving what algorithms could never do, the halting problem. You could never find an algorithm that audits algorithms and says whether this algorithm would terminate or not.

• 39:48Speaker 02

  Okay, so algorithmic science, computer science, started out of an impossibility result. We have to really remember that. And we are a field of science, I like that, we are a field of science where impossibility results are foundational because they narrow down the scope of what you can do. You cannot do this, so you can only do what is within this scope on the left. Good. Distributed computing, so the field and part of, partially, also has strong impossibility results. You can't solve consensus, you can't agree if a fraction of the nodes is malicious

• 40:31Speaker 02

  and exceeding a certain fraction. So, for example, if we want to agree on a common decision and more than 51% of the group are malicious, we will not agree on the safest choice. This is trivial. There are similar theorems in game theory, by the way, like the Arrow-Gibbert theorem, like impossibility theorems for democracy and social choice. We also have impossibility results for distributed machine learning. Or you can just think of it like a machine learning, gradient-based machine learning. Those are not new. We were just like, I'm not like, I'm not claiming that we were behind that.

• 41:16Speaker 02

  We just renewed the interest in them. They were proven in particular in 85 by a Belgian guy called Peter Rusev, mathematician. So, and the community of robust statistics. You could actually prove that if you have a group of estimators, so a group of random variables, and, sorry, not a group of estimators, a group of random variables following some distribution, and an estimator could not guess the mean of those random variables if more than a half of them are adversarial. And then he coined this thing called the breakdown point. We call it Byzantine fault tolerance in distributed computing because it has to do with a thought experiment

• 42:00Speaker 02

  called the Byzantine generals problem, that we don't really need to go there. It's just an agreement problem between three generals surrounding a city. So if one of them is corrupt, they can't agree whether to attack or not. So if you have N generals surrounding a city and the city only needs to corrupt a third, it doesn't need to corrupt everyone. If it corrupts only a third of the generals, the generals could not agree on a common decision. And the same, like you cannot make gradient descent work if a certain fraction is not reliable. So if most people are promoting anti-vaccine...

• 42:34Speaker 02

  Of course, no solution will work. I'm not claiming that we have a complete proof.

• 42:38Chris Benson?

  So there's limitations. In other words, there's success to be had, but there's also some limitations that if certain circumstances like that many are working against you, you won't be able to overcome that.

• 42:49Speaker 02

  But then, people on those big platforms, I think are smart enough to realize that, and they are realizing that. I saw a very good press release from YouTube last week, where they said that they will actively now try to work to prevent phony medical advice to be recommended on YouTube. So this is not about censorship, it's just about not recommending. So they are actively looking at the problem, and I believe they have enough smart people to think about that. And what I'm working on now, as a follow-up of what I've mentioned before, are situations where you don't have a majority of reliable notes,

• 43:31Speaker 02

  but you have a minority of experts. It's some sort of epistocracy. So you give the power to those who know. So imagine you have the John Hopkins Medical School YouTube account, the Pasteur Institute in French medical account, and the YouTube account. And then you have the Hospital of Lausanne, etc., and they're producing content on, say, vaccine. But then you have a majority, a majority of poisoners, of anti-vaxers, and you might want to do something in the PageRank style. So some sort of like a PageRank gradient descent, where you follow the experts?

• 44:11Chris Benson?

  Got you. So you're basically taking, you want to take advantage of their expertise, which is a way of countering the fact that you have a majority of poisoners in there. So it sounds like you're almost taking a couple of tools and making a composite out of it. As we start to finish up, is how can practitioners out there start to take advantage of these results that you found and the research that you've done to help better the situation we find ourselves in now, where we have so much poisoning going on with so many people trying to, in search of an answer? What are some practical tips that you can offer?

• 44:46Speaker 02

  Uh, I would say, um, start by reading the literature. There's like literature on poisoning has been there before I even start doing machine learning. There was like people who started looking at that since at least 2004. And people who had made significant progress in 2012, 13. And so, yeah, there is a good literature to be read. Uh, they could also like, we will release a GitHub repo with our, the code based on the algorithms I've been promoting before. So like the, my colleagues will release that on GitHub. So they could take it, play with it, find bugs, uh, potential bugs in it, find new vulnerabilities we didn't see. So because the space of vulnerabilities is technically limited.

• 45:29Speaker 02

  So you can always find new vulnerabilities or a new threat model for which our, because you always make a threat model. Uh, maybe we overlooked another threat model and they can make progress on that. So I would also advise, um, taking data sets that might, uh, give you a sense of what a recommender system does. And try to poison it just to understand, uh, how easy it is. And maybe you might find the vulnerability because now we enter in an era where you don't need to be, uh, like a classic hacker. You don't need to penetrate. You don't need to do a penetration in the servers and the system to poison a recommender system.

• 46:10Speaker 02

  You just need to behave, like, comment, dislike, post. So maybe there are still much more vulnerabilities that could be allowing people to just behave and look legit and poison. I don't know, make a, make a movie platform recommend a suicidal, uh, content to a depressed, uh, user. So this, this is something we don't want to have. And I would bet that those things do not need hacking inside the servers. And I don't know, finding a zero day and switching the code. I think there are like, because, because of high dimensionality, we have a paper called the hidden vulnerability of distributed learning in Byzantium. And the hidden vulnerability is basically high dimension.

• 46:55Speaker 02

  Today, as we are making machine learning powerful, we are learning more and more high-dimensional models. And this high dimensionality is give a lot of leeway, a lot of margin to attackers. So the bad news is that as machine learning is going to be high-dimensional and powerful, it is also becoming very wide in the amount of leeway it gives to attackers. So I think, yeah, a good starting point would like try to play with those algorithms and find eventual vulnerabilities we overlooked. And yeah, if you are a practitioner and you don't care much about the theory. Yeah, if you're a theoretician, I would also be very happy to hear about what we might have missed in the theoretical analysis.

• 47:39Speaker 02

  So maybe, maybe there's a bug in our proof and I'll be happy to, I'll be happy to learn that and, and work on fixing that. But if you are a practitioner and you don't care much about the theory, I would say download the GitHub repo of my colleagues and try to improve it and try to apply it on public data sets that are more relevant for recommender systems and, and maybe for other stuff, not only recommender systems. So I, I just like, yeah, for something like to conclude on, I, I've been overusing recommender systems here because I think this is the, this is the most pressing example of killer robots we have. Yeah, today, people are not killed by, like, not massively killed by self-driving cars. They're more like killed with hate speech and anti-vaccine. But of course, like poisoning will become a problem also for self-driving cars.

• 48:23Speaker 02

  If you poison the traffic sign and then you make self-driving cars learn a new relevant model, they might like, you might, you might start

• 48:34Speaker 02

  leading them into unsafe behavior. So, but, but the idea of poisoning resilience is very broad. So it doesn't apply only to recommender systems. You can think of your own problem and your own motivation and try, and try to, to improve on that.

• 48:48Chris Benson?

  That's fantastic. We'll certainly include the GitHub repo in the show notes, but I'll tell you what, you concluded, that was a strong conclusion. I mean, if there's anything that makes me realize how relevant what you're talking about is, even beyond social media, is the fact that we have all these, we have now cars and trucks and other vehicles and other IoT devices that may be mobile that could be poisoned along the way. And that itself can present a physical danger separate from that. So it's amazing how relevant what you're working on is going to be to our future. Thank you very much for coming on the show and I really appreciate you taking the time late in the conference to do this.

• 49:29Speaker 02

  Thank you. You're welcome.

• 49:32Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before our users do here at changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers. Head to linode.com/changelog, check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email

• 50:17Speaker 00

  keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.