Federated Learning 📱

Federated learning is increasingly practical for machine learning developers because of the challenges we face with model and data privacy. In this fully connected episode, Chris and Daniel dive into the topic and dissect the ideas behind federated learning, practicalities of implementing decentralized training, and current uses of the technique. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipLaunchDarkly https://launchdarkly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Ship fast. Rest easy. Deploy code at any time, even if a feature isn’t ready to be released to your users. Wrap code in feature flags to get the safety to test new features and infrastructure in prod without impacting the wrong end users. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Learning: Google Federated Learning comic https://federated.withgoogle.com/Federated Learning: A Step by Step Implementation in Tensorflow https://towardsdatascience.com/federated-learning-a-step-by-step-implementation-in-tensorflow-aac568283399Frameworks/ open source projects: TensorFlow Federated https://www.tensorflow.org/federatedIntel Open Federated Learning https://github.com/intel/openflPyGrid https://github.com/OpenMined/PyGridFlower https://github.com/adap/flowerExample uses of Federated Learning: Federated Learning for Mobile Keyboard Prediction https://research.google/pubs/pub47586/Your voice & audio data stays private while Google Assistant improves https://support.google.com/assistant/answer/10176224?hl=enFacebook is rebuilding its ads to know a lot less about you https://www.theverge.com/2021/8/11/22619639/facebook-plans-privacy-focused-advertising-revampFederated learning for predicting clinical outcomes in patients with COVID-19 https://www.nature.com/articles/s41591-021-01506-3Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  Federated learning is not new. This is not a new topic, but we're seeing what I think is a real shift right now into trying to find a better solution into being able to have a central model that is decentrally trained. I think both the need is there. Other approaches are effective in some ways, but also have some deficiencies. And the technology from an implementation standpoint has finally arrived with federated learning. So we're starting to see a lot of different implementation paths at this point from vendors and various frameworks.

• 0:35Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at Rudderstack. And we're calling all data engineers to check out Rudderstack Cloud and start building smart customer data pipelines. Rudderstack is warehouse-first, no more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. Rudderstack can be easily integrated into your existing development processes.

• 1:18Speaker 01

  And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free, up to 500,000 events, and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 1:53Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at kingdawg.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:19Daniel Whitenack?

  Welcome to another fully connected episode of the Practical AI Podcast. This is where Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI topics and news and then we'll dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitek. I'm a data scientist with SIL International and I'm joined as always by my co-host, Chris Benson, who is a strategist at Lockheed Martin. How you doing, Chris?

• 2:53Chris Benson?

  Doing very well, Daniel. Looking forward to having a good conversation between the two of us and no guests today. So we're just gonna have to have to do something ourselves here, man.

• 3:01Daniel Whitenack?

  No guests. Yeah. And it is, you know, whenever we do these fully connected episodes, we try to take a look at, you know, what's trending in the AI world.

• 3:11Chris Benson?

  Yep.

• 3:11Daniel Whitenack?

  One of those things is privacy and security concerns as related to AI systems. I don't know, before we jump into the topic for today, which impacts those areas. So we're gonna talk today about federated learning.

• 3:27Chris Benson?

  Uh huh.

• 3:27Daniel Whitenack?

  Which is a sort of recent trend that we're seeing. Before we jump into that, do you want to say anything about, I know you've done some more deep thinking about some of the kind of ethics concerns and bias and privacy type things. What are you seeing in recent days as related to that and how are companies, you know, are companies taking that seriously? What are companies thinking about in regard to that?

• 3:53Chris Benson?

  Well, I think what I've observed is that, and it's a little bit of a mixture of all these things. So that it's kind of a, there's a whole bunch of influences that are affecting the way companies are thinking. And a lot of those are legal. It's, you know, where is your data located? Moving across national boundaries, even though it's, you know, electronic is a big deal because of the laws. Things like the war on terror, the global war on terror created sets of laws to support that. And that's now having unexpected consequences. On how different countries want to share data down the road here, you know, in 2021, even though we're kind of moving past that era, we have a mix mash of different laws in

• 4:34Chris Benson?

  different countries, some of which have a little bit more thought and strength behind them, some of which don't. And here in the US where we are, there are some laws that other countries are very wary of, of having their data here in the US based on, on what that is. And so I've seen there are so many factors now that are playing into this and many of, some of them you would expect and some of them were kind of unintentional. But it's leaving us in a moment where, you know, going back to the topic, we've seen federated learning on the rise lately as a possible solution to some of these issues, or at least, you know, a good way to tackle it, the best option available right now.

• 5:13Daniel Whitenack?

  Yeah, I think everyone that's working in this space is encountering issues around ethics and privacy. In my space, in the NLP space recently, one of the AI ethics pioneers, Margaret Mitchell, she joined the Hugging Face team, which is of course, the sort of darling of the NLP community right now and is developing sort of a five-year plan around open source AI at Hugging Face and the ethics around that and what privacy and bias and all of those concerns are. And we're starting to see more, of course, around the data side as well. And people, as they're releasing their models and they're putting their results on like leaderboards and such, there's a sort of general call to provide like model cards and such with those,

• 5:58Daniel Whitenack?

  like describing a bit more about the model, what data was trained on and some, you know, more details about the statistics around the data, bias in the data, all of that, which kind of goes along with releasing these models. So it's definitely, it's definitely a trend. Yeah, transparency. And, you know, on the data side, speaking of that, that kind of gets us into our topic today because most of the time, at least in my experience, most of the time when I've been tackling an AI problem, my first thought is, Let's aggregate all of the data together. You know, do whatever preprocessing and such on it that we need to do.

• 6:41Daniel Whitenack?

  And, you know, in an ideal world, if we're acting responsibly, also analyze that data set for biases. Make sure that we're not, you know, violating any privacy concerns. All of those things on the front end in terms of the data prep. But there is a different approach to this that we're seeing more and more being talked about in the sort of trends of the AI world. And that is federated learning.

• 7:08Chris Benson?

  Mm hmm.

• 7:08Daniel Whitenack?

  So that's what we're going to talk about today. I'm not an expert, Chris. Are you an expert in federated learning?

• 7:14Chris Benson?

  I am definitely not an expert in federated learning, but I'm coming across it a lot right now. I'm in a lot of different contexts, so it's definitely something that's become part of my world.

• 7:23Daniel Whitenack?

  So for our listeners, you're listening to two non-experts in federated learning, trying to get a grip on what federated learning is. We've done a bit of looking and can discuss some of that today. So hopefully, as you, as you listen, you learn a little bit of that as well. And of course, we'll provide some, some learning resources about that. But federated learning has to do with the kind of main idea behind it is that we want to train a centralized model on decentralized data. Now, that's kind of interesting. So we still want a centralized model. There are other paradigms out there that are sort of privacy preserving sort of ways of going about machine learning.

• 8:08Daniel Whitenack?

  One of those is, hey, we could get our model, train it on some centralized data and then port it to mobile devices. And then sort of update or fine tune it on the device using device data.

• 8:21Chris Benson?

  Mm hmm.

• 8:21Daniel Whitenack?

  So you could do that with like TensorFlow Lite or like integrations with JavaScript or Swift and other things on mobile devices. However, that's not what we're talking about when we're talking about federated learning. It's related, but in that case, there's no centralized model, right? In that case, you're sort of porting a parent model out to a whole bunch. Of client devices and then maybe doing some learning on the device. So updating the model on the device. But all of those changes that are local never get ported back into a centralized model. So in that case, there are some advantages to that. I don't know. Before we jump into how federated learning is different.

• 9:04Daniel Whitenack?

  From your perspective, Chris, what are some of the advantages of that kind of model where you're porting? Or that sort of framework where you're porting models to devices and kind of updating them on devices? Any thoughts?

• 9:16Chris Benson?

  Sure. Well, for one thing, it's more mature now. It's something we've been doing for a little while in terms of having those different models. There's a history, there's a track record of that at this point. But it's also proven itself to be insufficient for a lot of use cases. So at this point, I think it's interesting because we've seen this topic evolve over time. Federated learning is not new. As we are recording this today, in late 2021, this is not a new topic, but it's really come into its own. I think for a long time it was a discussion. It had limited implementation capability. As I was looking around at different things for today's episode, there's talk of federated learning in 2016, 2017, where people are talking about the way forward into that.

• 10:01Chris Benson?

  But we haven't in those initial years with some exceptions that were really kind of edge cases. You didn't see it on the rise. You saw these other approaches that you just described there. We're seeing what I think is a real shift right now this year into trying to find a better solution into being able to have a central model that is decentrally trained.

• 10:24Daniel Whitenack?

  Yeah.

• 10:25Chris Benson?

  And how it does that. So I think this is natural evolution. I think both the need is there. Other approaches are effective in some ways, but also have some deficiencies. And the technology from an implementation standpoint has finally arrived with federated learning. So we're starting to see a lot of different implementation paths at this point from vendors and various frameworks.

• 10:48Daniel Whitenack?

  Yeah, so that previously adopted way of going about things, which is still valuable. So having the model on the device, maybe updating the model on the device, but never communicating any model changes back to a central model. That has been useful and maybe one advantage is its privacy. So if you're using user data to update the model, that data actually never leaves the device. It stays on the device with the model. The model is updated. However, all the other devices out in the world that are updating their models, they never get the benefit of the model updates that are on your own device. So if we think of something like, you know, speech recognition

• 11:30Daniel Whitenack?

  or interfaces on phones, if my device is learning how to better recognize my accent of English or whatever it is, and it fine tunes on the device, that's great. But then all of those other people out there that have a similar accent of English to me, I don't know how many of them there are, you know, they're not getting the benefit of those updates. And so they have to do their own training. So it's actually, in some cases, it's a lot of duplicated effort as well, potentially, where people aren't gaining the value out of other people's model updates. But it is, does have an advantage for privacy. So now that we're talking about federated learning, we're also talking about doing certain things and certain training on client devices.

• 12:13Daniel Whitenack?

  So for example, phones, or it could be people's own computers or their tablets or IoT devices or whatever it might be.

• 12:23Chris Benson?

  Yeah, I mean, edge devices. And I think that's important at this point to call it out because you're kind of going there anyway. And that is that the rise of federated learning in a practical sense is also happening concurrent to the rise of edge computing in a practical sense. And it's tremendously scaled and widely available.

• 12:42Daniel Whitenack?

  Yeah, and so that is now widely available. But we also have learned how to do sort of some training on edge devices. And we have the knowledge that, hey, if we gathered a whole bunch of data together centrally, or had a central model, it would probably be better than training all of these child models, potentially. And so federated learning, I think, tries to take the approach of the best of both worlds. Yeah. So doing some training and operations locally on client devices while still having a centralized model, which can benefit all users. So it does that in a way that preserves privacy. Mm hmm.

• 13:24Daniel Whitenack?

  And distributes the sort of training and compute out to the client devices. And so in that sense, you have this sort of decentralized compute and centralized model. So this is definitely a very interesting approach. So that's what we mean by federated learning and when we say a centralized model on decentralized data. Now, the question is sort of how that works. And like you say, I think there's been a whole bunch of effort in this direction. But before we get there, I want to emphasize what you're talking about. I think this has been a topic for, that has been in our minds for some time.

• 14:07Daniel Whitenack?

  I remember, I think it was 2017, there was a blog post by Google, which you can still read on their research or AI blog. And it had some really cool pictures about phones doing some of the training and communicating things back to a central server. But to be honest, I didn't hear a lot about it in that sort of interim time. Like, I heard it every once in a while. But now, you know, part of the reason why we're doing this episode is we're hearing about it a lot more. I'm curious, from your perspective, do you think that's mostly driven by the privacy

• 14:43Daniel Whitenack?

  concerns or mostly driven by sort of the desire to, you know, have decentralized compute versus like trying to always have like a big farm of GPUs? There's an answer that I want to give.

• 15:00Chris Benson?

  But I don't actually believe it. The answer I want to give is that there are such concern for privacy issues out there, you know, in the corporate world that that is driving it. I don't, however, you know, this is pure opinion, but I don't, however, think that that's the, that that is the, the driving factor. I think that, I think it is primarily legal constraints and logistics personally with large organizations that are trying to put products and services into the field and to, you know, get those deployed. And they are constrained by, by those various, uh, both legal aspects and technical aspects,

• 15:41Chris Benson?

  such as networking and such, is that being able to, even if you can move data around, if you are working on a large model that is, you know, on highly scaled data that would be trained on trying to get that data in the right place at the right time, especially if you have ongoing training, can be really challenging. And given that federated learning is, is, and we'll get into the details later, but basically pushing weights and biases around as opposed to all the data makes it logistically much, much, much better in that sense. So I, that's what my gut is. And that's what conversations and, and presentations that I see are largely geared around logistics and legalities.

• 16:34Speaker 03

  Change log plus plus is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and help sustain our production of practical AI into the future. Simply follow the change log plus plus link in your show notes or point your favorite web browser to changelog.com slash plus plus. Once again, that's changelog.com slash plus plus.

• 17:21Daniel Whitenack?

  Okay, Chris, well, this is Practical AI, so let's get into some of the practicalities around federated learning.

• 17:29Chris Benson?

  Yep.

• 17:30Daniel Whitenack?

  First of all, at least based on my understanding, this sort of architecture of federated learning differs from a sort of typical AI training architecture in that there is a centralized server or set of servers, you know, maybe in the cloud, maybe on premise, it doesn't matter.

• 17:48Chris Benson?

  Yep.

• 17:48Daniel Whitenack?

  But this is centralized, maybe a larger server, like what we would normally think about doing, training or having as a cloud server. Sometimes that's called a curator, and that coordinates all the training activities with all of the clients. And then, of course, there's a bunch of clients, which are edge devices, and these could be hundreds of devices, thousands of devices, millions of devices, if we're thinking about phones. And that central curator server coordinates the training of a model with all of these edge devices. Now, you talked a little bit about what's communicated back and forth. Do you want to go into a little bit more detail

• 18:33Daniel Whitenack?

  based on your understanding there?

• 18:35Chris Benson?

  Sure, you know, and we can dive into the detail of this, but kind of the high level is that you have that model on your central server that you're talking about.

• 18:43Daniel Whitenack?

  And model, we're meaning like a neural network, for example.

• 18:46Chris Benson?

  A neural network, correct. Thanks for clarifying that. And you are going to put that model out to your client. We're doing the opposite of what we've historically done, where we've pulled the data to where our model is going to be trained. And now we are pushing the model out to be trained where the data is. And there does have to be a capability that at that point, you have to have hardware and software on the client that can do training at some level. And so it changes the architecture in that sense. So you're pushing the model in the beginning. You're pushing the model with its initial values, the weights and biases and such, and it's going to. Train based on the data set that's there and there's different ways, which we can dive into later.

• 19:30Chris Benson?

  There are different ways of evaluating whether or not the data that is available on a particular client supports the training process. So there are, there are some gateways, if you will, that you can evaluate the data with and you do training on the device and you know, and as we know, you know, we keep talking about our phones being kind of the classical example of this. All of our phones these days are getting these capabilities for doing that kind of training. You know, they have the chips on them now. And so you're doing that, you get a result within a particular accuracy range and you're passing, then you're passing the resultant weights back up to the server. And so that centralized server is receiving those from all of those hundreds, thousands

• 20:14Chris Benson?

  or millions of client devices connected to it. And it has to do a form of aggregation on all of those model weights coming back in, which is referred to as federated averaging. And we can dive into what that means as we go, but then it averages those out and it measures that. And then it does it again for the next iteration. So without going all the way back through it, you're, you're going to keep going through that process, that cycling over and over again, until your centralized model is yielding the level of accuracy that you're, you're desiring. And then at that point, then you are able to finally deploy that model, those weights back out. And you can run that as a, as a production model in all those clients.

• 20:57Daniel Whitenack?

  So yeah, I, great description. And from my mind, I always try to put that sort of description and pair it with, with some examples. So I'm imagining like on my phone, I don't know about you, but sometimes like when I'm walking or something like that, I don't like type a text message. I just click the speech recognition thing. And then awkwardly, while everyone else watches me on the sidewalk, I talk into my phone and it records my voice. But I always look at what was recognized from my voice before I hit send usually. And then I like correct it, right? Cause sometimes it didn't get a correct word or something like that.

• 21:35Chris Benson?

  I'm impressed because I'll see people doing this a lot and they just hit send and go, and everyone just expects the thing to be off. But anyway, go ahead.

• 21:43Daniel Whitenack?

  Yeah. Yeah. So let's say that I did that, you know, a hundred times or something like that. And so I have. You know, my voice, I have what was recognized, then I have what should have been recognized because I have my correction. And I don't actually know if this is how it works in practice. I'm not a, I'm not on the voice team at Google or anything like that.

• 22:04Chris Benson?

  You should be, Daniel.

• 22:06Daniel Whitenack?

  You know, it would be fun if anyone wants to fly me out for like a speech visiting scholar position at Google. And, you know, I'm, I'm open to that.

• 22:14Chris Benson?

  Okay, Google, OpenAI, you guys have heard it, right there, Daniel Whiteknight.

• 22:18Daniel Whitenack?

  So let's say I have that set, it's a small set, right? It's not enough to train a full speech recognition model, but let's say that Google then, you know, they have their centralized English speech recognition model. And they then send that model, an updated version of that up to my phone. And my phone then, in this federated learning scheme, would use the data that's just on my phone. So the data hasn't been transferred back up to Google.

• 22:46Chris Benson?

  Correct.

• 22:47Daniel Whitenack?

  It uses that audio and the text from my phone, does a training, a retraining of that model, or a fine tuning of that model based on the data that I've seen and, and looks. So now I'm going to have updated weights and biases or updated parameters from that model, right, on my phone. And then my phone can send not the data, but the weights and biases, the parameters of that model, the updated ones, or maybe just the deltas, the changes, back up to the centralized server or curation server. So if thousands or millions of devices do the same thing, they're going to all be sending their updated weights back.

• 23:29Chris Benson?

  Which are evaluated before, they're not just lumped in, that averaging process scores them based on that. And it's really, this approach is really cool in that you're getting the benefit of the average weights and values across, all the data sets across all clients. So you're getting a training benefit as though you have access to everything there is, while having only that limited data set that you're there, you know, which then gets scored as that goes through the process. It's pretty cool when you think about it.

• 23:59Daniel Whitenack?

  Yeah, and I think it's important to emphasize that this is a practical reality now. So I mean, people are still doing research in this, no doubt. This is an active research topic. Sure. But there are practical ways to go about that that have been developed. So we'll list out a few of those a little bit later in the episode. But I'm just looking at PyGrid, which is one of these that's been released. And in that, there's, just to give you a sense of what this might look like, there's a couple of Flask-based applications. So Flask is a Python framework that allows you to build web applications, like APIs, REST APIs, that sort of thing.

• 24:42Daniel Whitenack?

  And so there is a Flask application that is like centralized. I think they call it the network and it manages and monitors and controls routing instructions to various domains, which are, in my understanding, hosted on workers, so PyGrid workers. And that domain is another Flask-based application that receives instructions and executes a worker application on the device to do ephemeral updates to the model and communicate those back. So the device receives a request to train a model. The device will request to train a model. So the device actually has to sort of opt in to the training bit, which makes sense. And then the model and some sort of parameters about the training plan

• 25:27Daniel Whitenack?

  will be sent to the device. The training will take place on that device with the private data. Once the training is completed, in this case with this framework, it's the delta or the diff of the parameters, the original state of the model is communicated back up to the server. And then that's averaged, like you say, into the centralized model. So that's a sort of what they call the model-centric federated learning. Type of technique. There are other kind of versions or flavors of federated learning that might include some like communication of privacy preserving data. But I think the one that we've been mostly emphasizing here is the model centric version, which is what we're talking about here.

• 26:12Chris Benson?

  Correct.

• 26:13Daniel Whitenack?

  The data stays on the device. I mean, that seems practical to me in the sense that I've worked with flask applications before and I've done some AI training a few times. So it seems like something I could work on, although I haven't worked with phones much myself. So maybe that part's a little bit scary to me and like how that actually works. I've never developed a mobile app.

• 26:36Chris Benson?

  I have, but I haven't done it from a deep learning standpoint. So.

• 26:40Daniel Whitenack?

  Yeah, yeah. So I don't know. I don't know all of the mobile application development pieces that you'd have to tie in. I know that there's some like JavaScript and Kotlin and Swift libraries for these frameworks that will allow you to kind of build and support that worker capability on the device.

• 27:00Chris Benson?

  Yep.

• 27:01Daniel Whitenack?

  One thing that came up in some of the, some of the information that I was reading was like, doesn't this like just suck away all the battery of the device? Like, what are the implications for the device user? So like if I'm other than like, you know, because it's maybe useful to talk about the disadvantages for the device user rather than just the advantages, like they get a cool new model, which is good, but you know, you kind of, what was that program where like, it was like citizen science type thing where you could, you could register your computer with like a science lab and they would run like astronomical calculations, like in a decentralized way on your computer?

• 27:45Chris Benson?

  Yeah, there was the SETI program that was doing that with, with computers. That was like the earliest one. That I can remember. That's way back now and there'd be other sense.

• 27:54Daniel Whitenack?

  I mean, it's in the same vein, but that's going to drain your computer power, right?

• 27:58Chris Benson?

  It can, but I think our conversation right now is also leaning toward that phone assumption. And it's not always a phone assumption. It can be a larger device. Your edge device might be a, might be a mobile platform. And when I say mobile, I mean, like with wheels or wings or, or rockets or something else.

• 28:15Daniel Whitenack?

  Oh, right. Like a car or.

• 28:17Chris Benson?

  It could be a car. Exactly. And so it may be going forward and, and I'm, this is completely pure speculation, but if you're going to do a lot of federated learning and doing that processing, maybe there's another battery that's in that car that's, you know, that's there to run your training hardware that's there. And so it kind of depends if it's the phone, yeah, that's probably going to start sucking battery down if you're doing any substantial amount of training. But if you're in literally a vehicle that it's tremendously benefiting from that over time as use cases get found. And I think we're still, and this goes out, we can get into use cases in a few minutes, but I think that we're still at a point where we are exploring use cases and where does federated learning give us a strategic advantage to implement

• 29:03Chris Benson?

  for that? And, and there may be cases out there where in doing that, then you simply architect in the ability to do on platform learning out there on the edge, so to speak, to accomplish this.

• 29:15Daniel Whitenack?

  Before we jump into frameworks and some of the use cases that we've seen out there, Chris, one of the things that I think is worth noting as related to this federated learning topic is related to security. So I remember very clearly in my mind, back when I did go to conferences in person, I think it was a ODSC conference or something like that. I saw Jim Klukar, who was with Immuda at the time. Shout out Jim. I don't know if you listen, but great guy, but he talked about privacy in it in his talk and he showed some examples where I think it was facial recognition or I'm pretty sure it was facial recognition. Anyway, you could take a model because our models are so big.

• 30:00Daniel Whitenack?

  Now, right? And there's so much encoded into our models and you could actually from a prediction sort of work back to the original data that was used. So you could reconstruct people's faces in the training set, right? Just from the model parameters. And so in theory, you could imagine sort of reconstructing the data that's on client devices from what's sent to the central curator server or coordinator server. So that's one thing to note here about the aggregation. And I think this is probably what you were getting to when you were talking about the aggregation. There is a method of securing that aggregation. So there's two things. There's encryption when

• 30:44Daniel Whitenack?

  the data is sent back to the central server, but then there's a way to securely aggregate that. And that has to do with differential privacy. So yeah, any, I don't know if that's what you were meaning when you're getting at like averaging some of those results back together.

• 31:00Chris Benson?

  I do. And I think that, I mean, that's one of those areas that there's going to be a lot more research on in terms of being able to do that because right now there are some federated averaging algorithms that are in use. And the most basic one is just called federated averaging, but there are others that are being built on top of that. And I think that's going to be one of those areas that people are having to explore is data scientists on the research side are going to have to explore is can you get back to the data? Because and there's so much that's going to depend on on where that research goes because, you know, going back to our example of national boundaries is that you have laws protecting citizens, rightly so, but they vary across national boundaries and therefore for you to have a model that is A, for you to be able to participate in federated learning in

• 31:43Chris Benson?

  that capacity and to be able to deploy a subsequent model across national boundaries. That is one of those areas that we need to ensure that that even though the data itself will reside on a client and maybe across the national boundary, that you cannot recreate that coming back across. So I'm expecting to see a lot more on that in the years ahead.

• 32:02Daniel Whitenack?

  Yeah, and I agree with that. And you know, differential privacy and this sort of aggregation could be a topic in and of itself. And maybe we'll have a follow-up episode on that. Just a few sort of very brief phrases about differential privacy is that, so if we're thinking about phones and them contributing their data or edge devices, it limits how much any single contribution from a phone can contribute to the overall changes in the model. And in that way, the model isn't sort of overly skewed or memorizing results from a single device, which could lead to sort of reconstruction of rare private data. And also, that noise

• 32:43Daniel Whitenack?

  is added to that sort of rare data. So that, all of those ideas are kind of, if you want to learn more about that, you might look up more on differential privacy.

• 32:54Chris Benson?

  Yeah, I'm speculating that the more diversity you have in your data set, you know, will help protect you from that as well, because it is an averaging function. And so if your core data that you're training on is close to the average, if the data that's coming back from your, all of your client devices is very similar, that obviously is something that would have to be addressed in that sense.

• 33:15Daniel Whitenack?

  So when I was looking up how I might go about implementing some federated learning, I was curious as to the state of the various frameworks and tools that you can use to actually achieve, achieve this process. I was pleasantly surprised with, you know, of course, I haven't run a real experiment with millions of devices. Maybe that is in my future. It'll definitely be a fun experiment. But it looks like, for example, one of these frameworks is TensorFlow federated. So TensorFlow has an open source framework for doing this computation on decentralized data. And some of what you have to do to your model

• 33:59Daniel Whitenack?

  to enable that is kind of wrap some of your model definitions in classes and helper functions that are provided by the TensorFlow federated framework. But you do get to Sort of preserve your, you know, your Charis model, which you love. You know, you have your Charis model and you kind of wrap things around it to use it in this federated way, which seems like, seems like a nice approach. You don't have to throw out everything that you did with Charis that you love. You can kind of wrap it and, and use it from, from there. There's a bunch of other organizations and large organizations that are really contributing to open source frameworks.

• 34:37Daniel Whitenack?

  Intel has their open federated learning framework and Facebook was involved in PySift and PyGrid, which I mentioned before. PyGrid was the sort of methodology that I talked through before. And then there's some, some other ones too, like Flower I saw, which is a friendly federated learning framework, which is nice and lots of Fs in there.

• 35:04Daniel Whitenack?

  That looked really cool and, and there's other ones too. I'm sorry if I'm leaving out your favorite one for those listening out there, but.

• 35:10Chris Benson?

  I guess at this point, you know, I think we're really to a point where I'm really curious to see if anyone in our audience is, is actually doing this. I know that the teams at my employer are now into federated learning. And as we've done, just as we've, we love to hear from folks about who is engaging in it. Have you guys done any federated at the nonprofit that you work at yet? Or have you not yet had a need to?

• 35:33Daniel Whitenack?

  We haven't, although I do wonder about that because, you know, one of our use cases is translation. So we have over a thousand translation projects going on.

• 35:45Chris Benson?

  Yeah.

• 35:45Daniel Whitenack?

  Around the world and, and part of what I'm working on is augmented quality assessment types of tools for those translations. And those will, that very much fits in this framework where there's a centralized set of models that are maybe used on all of these client devices. And could be improved by data that can be gathered on those client devices. But also, these are people working all around the world on their own translation stuff that oftentimes includes like their own copyright restrictions. For example, where they, they might not be able to share that translation data in certain contexts

• 36:25Daniel Whitenack?

  or otherwise. So there's rights holders and copyright information associated with all those translations. So it's definitely got me thinking along those lines.

• 36:34Chris Benson?

  That's a fascinating use case right there. You could ask the user's permission to be able to do it because as you pointed out earlier, you're also using some of the, the power, you know, that's available. And if you're going to do this federally learning, then you're going to be training on the device. And if their device is capable of doing that, then you're presumably draining the battery faster. But you also have these other ancillary issues that the end user may or may not know. And so have you put any thought into how you might address that? Is there any way of evaluating that?

• 37:01Daniel Whitenack?

  To be honest, I don't know. My first thought is that this can be a little bit tricky because for so long, people have been exposed to messages that pop up on their device that say, "Share data with us and we'll make your experience better." Right? That's been a sort of common thing when you accept terms and services, or when you like, when always you get a new phone, right? It's like, or a new phone service. You're like, "Share data back to us about like your network usage and we can make the network better for everybody." Right? And that's really ingrained in people's minds. And so if you try to like, put some messaging in an app around that,

• 37:43Daniel Whitenack?

  that's probably what people are going to assume at first, like, oh, they're collecting our data. I don't want to share my data. Right? So it's very interesting, like, how and how much do you share with the end user? And what is the phrasing around that to help them understand what is actually happening? Yes, you are sharing something with a centralized server, but in- A differentially private way and it may reduce, you know, it may suck away some of your battery or, you know, run certain things on your device that you weren't running before. And so maybe there's battery issues and other things. But I think that there's a sort of a lot of assumptions that we'll have to overcome as we do that. Because people are so used to the fact of like, oh, everybody's just gathering my data.

• 38:28Daniel Whitenack?

  Right. And when I see one of these pop ups, it's asking me for data and I don't want to share my data.

• 38:33Chris Benson?

  It makes for a fascinating consideration to have to try to, to overcome and mitigate. I work in the defense industry and governments that are, that are interested, they kind of control their entire environment, you know? It may be contested by another government, so to speak. But at the end of the day, you know, any given government that might be interested in that is, you know, they are running both the central server and they also own those endpoints, those, those edge devices, whatever those are. And so there is a potentially much less to have to consider from a user rights standpoint and a privacy standpoint. So it becomes just a logistical thing to some degree. Fascinating consideration, and I think that your use case is much closer to what most end users would have to deal with.

• 39:18Chris Benson?

  They're going to have customers and they're going to have user communities that they're serving. So.

• 39:23Daniel Whitenack?

  Yeah. And there have been examples of successful uses of this across industry where people have, you know, at least started navigating these concerns. Google, of course, I mentioned they were investigating this even a few years ago or more. And they've shown various actual real world applications of this in mobile keyboard sort of development and autocomplete prediction type stuff. Voice and audio data being used to improve things like Google Assistant. Also, you know, other tech giants you might expect are investigating this as well. Facebook is sort of rebuilding, in my understanding, at least from public articles,

• 40:06Daniel Whitenack?

  rebuilding some of their ad infrastructure and models to do things in a more decentralized way with federated learning. One that I think is really cool. Which is dominating the actual applications that I've seen are related to healthcare. I don't know if you've seen some of these, Chris, but one I was reading in a nature article from Harvard Medical School where they actually were predicting sort of clinical outcomes of patients with COVID using federated learning. So, you know, they had something running presumably on patient devices, or at least clinic devices, and, you know, preserve the privacy around patients' actual health data while also

• 40:46Daniel Whitenack?

  maybe providing some predicted outcomes to doctors to help them, you know, augment maybe their treatments and that sort of thing. So yeah, that was a really interesting example that I ran across.

• 40:58Chris Benson?

  Another one that I've run across is predictive maintenance. You have all of these different types, both in the civilian world and the military world, et cetera. All these vehicles out there, all these devices, factory machines, and they do have their own data set there. And from just even even if privacy is not a concern, just logistically, being able to to benefit from that diversity of things out there that you can then train on and train toward that is, I think, I think it lends itself very naturally to federated learning.

• 41:27Daniel Whitenack?

  So towards the end of each of these fully connected episodes, we always like to leave you with a few different learning resources. One of the things we will do is include in the show notes links to all of these different federated learning frameworks and open source projects that we've mentioned, like TensorFlow federated and the Intel Open Federated Learning and others. So that you can go there and actually get hands on and try out a few things. However, one of the things that I think is really great, you know, if this is a new topic to you and you just want to think about it a little bit more and its implications. Google put out this federated learning comic, which is really good at sort of leading you through both the motivation of federated learning, how it works,

• 42:12Daniel Whitenack?

  maybe some concerns or questions that come out of that. And you'll see some of the themes that we talked about in this episode represented in that comic, which is a great way, sort of visual and fun way to get introduced to. Federated learning and we'll include the link in our show notes. What about you, Chris? What's some of the stuff that you were looking at?

• 42:32Chris Benson?

  You know, one of the websites that we have talked about with various learning resources over many episodes is one called Towards Data Science and Towards Data Science had a good tutorial for stepping into federated learning with TensorFlow. It's called Federated Learning: A Step-by-Step Implementation in TensorFlow. It was in April 10th of 2020, so about a year and a half ago. And it's a really good introduction into the basics of it and doing kind of a toy network to try it out. So that was a good one. And then I'll mention, this may be almost funny to say that, but go to the Federated Learning page on Wikipedia. It has quite a lot of information there. It's probably not where I would start. If you're not familiar with Federated Learning, it's not

• 43:17Chris Benson?

  the first place, but after you've read a few other things and maybe gone through some of the other resources, it may throw some terms at you that you missed in certain areas and you can say, oh, what is this? What is this? So I thought I'd mention those two. The Towards Data Science is a good beginner. And then after you've gotten a little bit under your belt, Wikipedia.

• 43:35Daniel Whitenack?

  Awesome. Well, we hope that our listeners will explore this. As Chris said, we're interested to hear how you are exploring this topic. Connect with us in our Slack community. You can find that at changelog.com/community. We'd love to hear what you're working on and what you would like us to be talking about on the podcast. We really do value your input. You can also find us on LinkedIn or on Twitter as well. So keep in contact. Let us know what you're learning about as related to federated learning and other things. And thanks for the conversation today, Chris. It was fun.

• 44:10Chris Benson?

  I enjoyed it. Thanks a lot, Daniel.

• 44:15Speaker 03

  That's our show. Thanks for listening. For more like this, check out our master feed. It is all Changelog podcasts in one easy to consume place. Let your podcast app snack everything we produce and then pick and choose which ones to listen to. Subscribe today at changelog.com/master or just search for Changelog Master in your podcast app of choice, you'll find it. Special thanks to Brake Master Cylinder for providing our music and to our longtime sponsors, Facely, Launch Darkly, and Linode. That's all for this week. We'll talk to you again next time.