Friendly federated learning 🌼

This episode is a follow up to our recent Fully Connected show discussing federated learning https://practicalai.fm/153. In that previous discussion, we mentioned Flower https://flower.dev/ (a “friendly” federated learning framework). Well, one of the creators of Flower, Daniel Beutel, agreed to join us on the show to discuss the project (and federated learning more broadly)! The result is a really interesting and motivating discussion of ML, privacy, distributed training, and open source AI. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. Me, Myself, and AI https://sloanreview.mit.edu/audio-series/me-myself-and-ai/ – A podcast on artificial intelligence and business produced by MIT Sloan Management Review and Boston Consulting Group. Each episode, Sam Ransbotham and Sheervin Khodabandeh talk to AI leaders from organizations like Nasdaq, Spotify, Starbucks, and IKEA. Me, Myself, and AI is available wherever you get your podcasts. Just search Me, Myself, and AI. Featuring: Daniel Beutel – GitHub https://github.com/danieljanes, X https://x.com/daniel_janesChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Our FC episode on federated learning https://practicalai.fm/153Flower https://flower.dev/Flower Blog https://flower.dev/blogFlower Conference (with videos) https://flower.dev/conf/flower-summit-2021Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  There are a ton of things that get me excited. From a practical perspective, one initiative that we are participating in is called MedPerf. The real world impact and real world improvements that we're going to see from this can be very profound. Because it's about medical AI and getting better performance estimates in medical AI is actually a very fundamental challenge. So that's something I'm quite keen on contributing to.

• 0:27Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/Changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse-first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. RudderStack can be easily integrated into your existing development processes.

• 1:11Speaker 01

  And because they're open source, you can see all their code. So you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 1:45Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at kingdaw.com/community and follow us on Twitter, we are at Practical AI FM.

• 2:11Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International. And I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How was your Thanksgiving, Chris? It was US Thanksgiving. For those listeners that aren't in the US, might not be aware.

• 2:32Chris Benson?

  It was very good. Nice family stuff, flew around the plane, things like that. And now we're into the holiday season and, you know, looking forward to, uh, to see what, uh, what kind of machine learning gifts are under the tree this year.

• 2:46Daniel Whitenack?

  Yes, well, in the spirit of distributing machine learning to all the boys and girls, maybe not by Santa, but, but a couple of weeks ago, you and I had a conversation about federated learning. Now, neither you or I is an expert in that area or a practitioner in that area, although I think it was a good conversation. But today, we're privileged to have Daniel Boitelle with us, who is one of the creators of Flower, which is one of the, the open source federated learning frameworks that we talked about. He's a co-founder at ADAP and a visiting researcher at University of Cambridge. Welcome, Daniel.

• 3:28Speaker 00

  Thanks. Thanks for having me.

• 3:30Daniel Whitenack?

  Yeah, well, as you heard, Chris and I were talking about federated learning without being experts in federated learning. So maybe to follow up on that conversation and maybe for people that didn't hear that conversation, could you just give us a sketch of what federated learning is? And then we can take it from there.

• 3:51Speaker 00

  Yeah, of course. I'm happy to. So federated learning is a way to train models across multiple data sets. That's the, the very easy take on it. So you might be wondering, how does this work? The way you do it in federated learning, and let's just start off by giving an example. Let's say we have, for example, a group of hospitals. They have some in-house data, but due to They cannot share this data and they cannot put this data in the cloud. And they can't use the usual machine learning workflow where you basically collect all of the data in a central repository. And then train your model on it. So that's not an option for them. Um. So they might be interested in using federated learning.

• 4:35Speaker 00

  And how would a federated learning setup then then work in such a scenario? So the way it works is that you have your plain old machine learning model, say it's a neural network, like for example a CNN. That does some kind of image classification. Maybe you want to look at radiology images, for example. And you would initialize this model in a central place, let's call this the central server. And the central server would, after initializing the model, send this model out to all of the participating hospitals. So it would send the uninitialized model, but there there are other variants of it, just to say this for the sake of completeness. But in our initial example, just to to explain the very basic version of it.

• 5:17Speaker 00

  They would send out the initialized model, so model that hasn't done anything yet. The model would then be trained locally within each hospital on the data that is available locally. So each hospital obviously has a different data set. They would train the model not not until convergence, but they would only train it for a little while. So let's say they would train it for for one or two epochs. And after they trained the model for one or two epochs, they would send the updated model parameters or the gradients that they accumulated. Back to the central server. So that way they don't have to share the data. The data stays where it originated. The data always stays within each participating hospital.

• 5:57Speaker 00

  And the central server would only get the refined model parameters. So the model parameters that have been trained for one or two epochs. It would get that from all of the participating hospitals. And what the central server then does is it aggregates those parameters. In the simplest version, it just doesn't weighted average over these parameters. What I just described is a way of initializing the model, sending it out, training it locally, collecting the updated parameters. And then aggregating the parameters. That is one single round of federated. And then what you usually do is you perform these rounds over and over again until the model converges. And the interesting part about it is why organizations actually do this is they get access to a lot more data than they had before.

• 6:42Speaker 00

  So we've probably all had this experience, especially in practical AI projects that oftentimes there is just not enough data. And having more data beats any fancy model architecture. So in this case, federated learning solves this data access problem. They can collaborate on the model training without having to share the underlying training data. Yeah, that's the gist of it.

• 7:03Chris Benson?

  That's a good explanation. It was much better than the one we were trying a few weeks ago.

• 7:08Daniel Whitenack?

  Yeah, we should link this episode to that one because it took us half an hour to get there.

• 7:13Chris Benson?

  We just need to voice over what he just said to what we said. Yeah, totally.

• 7:18Daniel Whitenack?

  Yeah.

• 7:18Speaker 00

  I mean, I left out a ton of detail, right?

• 7:21Chris Benson?

  I get it, but we can ask you questions and find out what some of that is and looking forward to that. So as a starter, it's very clear, given the data is distributed in terms of where it's located. And given laws and regulations and other such things that may constrain the training process with privacy concerns and stuff. It's very clear what the advantage is in federated learning. What also might be considered some disadvantages or maybe another way of asking it is when you do consolidate the model, after you've done the federated learning and stuff, what is the delta in a trained model versus if you had not done that?

• 8:00Chris Benson?

  If you had been able to aggregate kind of in the traditional way, all the data into one spot and train it in the traditional way we've done before federated learning. What's the difference in what you get as an output, you know, or is there much of one?

• 8:12Speaker 00

  Yeah, there is a difference. The biggest difference, I shall say, is obviously in convergence time because you have these rounds of communication. And also the averaging process has some, some impact there. Often as researchers, we make these comparisons between centralized learning and federated aspects of it. The interesting bit is that this comparison is somewhat artificial because it's not something that one would face in reality very often. It's either federated learning or nothing. We've seen this in the past, right? If we look a little bit at the journey that machine learning and deep learning now now took is the summer around twenty twelve, we really realized that by making these models bigger, we suddenly get better accuracy.

• 8:57Speaker 00

  So there was this this ImageNet moment and then a a couple of other moments like this um afterwards. And we saw that we can achieve ever greater accuracy and and other performance metrics with these models. And the thing is, we always, when we when we read a research paper, for example, and when we look at these recent advances, it's often quite fascinating and it's often in the context of web-scale companies like Google or Facebook, who have these massive amounts of data in-house. But then, often in practice, there's this realization that, okay, I'm I read about this cool technique, I'm trying to apply it to my problem, and suddenly I don't get the amazing results that I expected to have. So the question is, what what happened, right?

• 9:41Speaker 00

  And in many cases, the answer is really that the amount of data and the diversity that that you have in your local data set is just not enough. And the interesting thing, and and the thing that got us very interested in federated learning was this realization that for many of those cases, you might not have a large data set on your own, but there are a lot of others just like you who are facing the same challenge and who might want to train the same model, but they also have some data, but not enough data for a very good model. I mean, we could obviously solve this if we could put all of this data in a in a just in a single destination, in a single cloud account, and then train a model on it. But that's something that just doesn't happen.

• 10:26Speaker 00

  It doesn't happen for regulatory reasons. It doesn't happen for confidentiality reasons. For example, corporations, they have a lot of financial data and they might want to have models that predict predict certain aspects about these data. But again, it's it's it's a thing of confidentiality. It's something they would never share. And the types of use cases that federated learning gets used for, sometimes we are surprised ourselves and where exactly these companies are hesitant to share. Share data. For example, there was one case where a couple of manufacturing companies, they are all operating the same manufacturing machine and they want to train a model that does predictive maintenance, basically for this machine to predict whether this machine is likely going to fail.

• 11:09Speaker 00

  So whether they need to do some manual maintenance or something like that. And one would think that this is a case where they could just collaborate and they could just put all of their machine sensory data in a cloud account and train a predictive maintenance model. No, they don't. Why don't they collaborate on this? Well, the reason is that the data that they have from running these machines could allow others to see how often they run these machines, which could allow others to draw some conclusions about how many parts they are producing, which is highly confidential. So even in those seemingly easy cases, in reality, it's not that easy.

• 11:43Chris Benson?

  So that's almost a perfect lead in for what I wanted to ask next is that federated learning, it sounds like, offers different business models from maybe some of the things we've done in the past, or even among competitors, directly cooperating. So have you seen this start to happen yet? Or maybe consortiums come into being and they may include direct competitors who are all in the same line of business that want to protect their data so that they don't give away competitive intelligence and federated learning through a consortium or some other structure similar to that might be a way to everyone benefit from that and get the new model without giving away the secret sauce, so to speak. Do you expect to see more of that kind of thing?

• 12:26Speaker 00

  We are sort of seeing this. The way it usually starts out is that organizations who are maybe not direct competitors, maybe they are somewhat in the same space, but they're not like the toughest competitors. They start to get together.

• 12:41Chris Benson?

  Yeah.

• 12:41Speaker 00

  In some cases, it can even be the sub-organizations of a larger enterprise, for example, because they are also often facing these restrictions for sharing data. But then we also see that sometimes even real, really strong competitors, they get together because they see something else as a threat to their business model and they see that this is a way to collaborate. Without sharing this, as you call it, the secret sauce. And the interesting bit is that the way I described federated learning in the beginning, this is really end-to-end federated learning, where you initialize the model just globally. And then you train the model end-to-end with all participating parties. This is not the only model that's possible, right? I want to describe one, which I think is quite interesting,

• 13:26Speaker 00

  especially for this case where you have sort of competing organizations collaborating. And it's one where you train a certain part of the model in a federated fashion across multiple data sets. And then other parts of the model, you just train it yourself on your local data. So this is pretty interesting, because you can, in such a federation, you can, for example, train the entire backbone of a model. But then the last few layers, the head of the model, you don't train this in a federated fashion. You leave that up to each of the participating organizations to do it themselves. So everyone ends up with a similar, yet different model. And everyone has something where they say, okay, we benefit from this federation, but we are not giving away everything.

• 14:08Speaker 00

  The one important thing to mention, though, is that there are different types of federated learning. So you can roughly categorize it into two different types. One is this cross-stylate type that we just talked about, where different organizations collaborate with each other. The other type that we often see also in scientific literature is the cross-device setting, where you would usually typically you would have one organization. For example, think about Google or Apple, for example. And this organization would have access to a large number of devices. For example, mobile devices, like an Android phones or iOS phones. And the goal in this case is also to train a model, to train a model across all of these devices.

• 14:51Speaker 00

  And these devices, they hold data that is also where you wouldn't want to upload this data to the cloud. So this is the cross-device setting, where a single organization trains these models without access to the underlying training data.

• 15:23Speaker 01

  This episode is brought to you by me, myself, and AI. It's a podcast on artificial intelligence and business. It's produced by our friends at MIT Slow Management Review and Boston Consulting Group. The question is, why do only 10% of companies succeed with artificial intelligence? That's the question they aim to answer with this podcast. Here's Google Cloud's Will Granis on an unusual AI challenge.

• 15:45Speaker 04

  When I think about what AI is, I find the algorithms mathematically fascinating, but I find the use of the algorithms far more fascinating because from a technical perspective, we're finding correlations in extremely high dimensional nonlinear spaces. It's statistics at scale in some sense, right? We're finding these correlations between A and B. And those algorithms are really interesting and I'm still teaching those now and they're fun. But what's more interesting to me is what do those correlations mean for the people?

• 16:14Speaker 01

  All right, me, myself, and AI is a collaboration between MIT Sloan Management Review and Boston Consulting Group. It's available wherever you get your podcasts. Just search me, myself, and AI.

• 16:44Daniel Whitenack?

  So Daniel, I think we've mostly talked about some of the kind of data centric motivations for federated learning, or maybe privacy focus, or whatever it is, competitive type of advantages. But I'm also thinking of like the devices on which the actual training is happening. So like if I'm thinking of the centralized model, I'm thinking of like, oh, I'm going to spin up like a pod of GPUs and a really expensive pod of GPUs and do like all my training there and get my data there somehow. So am I correct that you could have some sort of infrastructure savings with this where

• 17:25Daniel Whitenack?

  the actual computation is happening on those edge devices and you're doing a smaller amount of aggregation and updating of the model centrally? Could you talk to that a little bit and what people have seen and how they look at infrastructure in that way?

• 17:40Speaker 00

  Yes, that's a very interesting question. The answer is, as almost always in engineering, it depends. So as you noted correctly, in the centralized setting, you have a pretty well-defined stack that there's not a lot that changes from one setup to another. You usually have some kind of x86 processor and then you have a usually you have a Nvidia GPU attached to that. You have Linux running on that machine. And then the biggest choice you have is whether to use TensorFlow PyTorch or Jax nowadays. In the federated setting, that's quite different. In the federated setting, you can have anything as a client, starting from even a tiny embedded device. There's research going on in that direction.

• 18:24Speaker 00

  Then you can have something like an Apple Watch or a mobile phone, or you can have some bigger device like a tablet or a laptop. You can have your standard x86 server that I just described, or you can even have a much larger compute cluster. If you're in the cross-silo setting where you have a ton of data and one of these organizations has massive in-house infrastructure, you can have a HPC cluster as a client. So this is obviously quite interesting and also challenging from an infrastructure and just a software perspective. In some cases, you can actually, and there is some recent research, for example, from a group in Cambridge I'm involved with, about the CO2 impact of these workloads, comparing, for example, the CO2 impact,

• 19:09Speaker 00

  and this obviously quite related to your question, about the CO2 impact of federated workloads versus central workloads. And the interesting bit is that it's not, you can't say it in general. Actually, it's quite an interesting thing because I originally expected federated learning to do much worse because you have these communication rounds and it takes longer to converge. So obviously it must have a higher CO2 impact. It turned out that that's not necessarily the case because in some situations, the reason is, once you hear it, it gets quite obvious. But it was surprising to me. In the central setting, you have the major impact on the CO2 emissions is the cooling. So you have active cooling of your

• 19:50Speaker 00

  GPU clusters in the federated setting, you don't necessarily have cooling. You have additional cost for communication. But then if you have a mobile edge device, these edge devices, they are usually passively cooled. So they are running the workload and they produce the result without ever needing energy for cooling. So that can be quite good, but obviously it depends a lot on the workload, the type of model you train, the number of communication rounds you do and then and other aspects. In terms of infrastructure cost, this sort of answers this question as well because you can have in some cases, if you have, for example, the cross-device setting, then obviously if you're not the one operating these devices, then you don't have to pay for the energy that that goes into training.

• 20:34Speaker 00

  Usually when companies do this, they're very careful about it. They do it in a very careful way, so they wait until the device is plugged in and the device is connected to wifi, until it's fully charged and idle, and only then they do the federated learning to not impact user experience or not to not drain the battery or things like that. In the cross-silo setting, there's, I wouldn't say that there's much of a difference. In terms of infrastructure, you need, well, each company needs the infrastructure they would need anyways. And then you need one additional server, so that's pretty similar. Especially in the cross-silo setting where you often have large models. You do have a lot of network bandwidth that you need, so that's something that you should consider.

• 21:16Daniel Whitenack?

  You talked a little bit about the training time. You talked a little bit about what's happening on the device. I think what's happening in the back of my mind is I'm thinking like, okay, I've got all of these devices and there's sort of various axes along which things could change, right? I I could have like the computational power of that edge device or the client. And then I've got also like the number of samples that are available for training on that device. I'm thinking if, and maybe you could speak to this, so I'm thinking like in the scenario of like a low-power edge device or a phone, like I'm going to have very few samples, which might be a sort of quick update on that device of the model and communicate the parameters back.

• 21:58Daniel Whitenack?

  Whereas like as the kind of amount of data that you have on the client is larger, you sort of need more computational power, at least more time to do the update. Is that kind of how that trade-off happens in practice?

• 22:12Speaker 00

  Yes, absolutely. I mean, yeah, obviously, if you have more data on a device, you need more time to train the model on the data. But this is actually also a very interesting aspect, not just in terms of practical things like communication bandwidth and so on. But it's also quite interesting from a more fundamental perspective, namely in the, both in the cross-device setting and in the cross-silo setting. Usually, the data in these partitions, as we like to call them, the data in these partitions is coming from different distributions. So it's what we like to call it non-ID data. And this actually has an impact on the learning process.

• 22:50Speaker 00

  There are certain scenarios, which are very rare in a practical setting, but there are scenarios actually where the data distribution within each partition can be so different that it's just not possible that these workloads converge. And this is something where a lot of research is going on, on how to make federated learning more robust towards such scenarios. And yeah, the practical aspects of it are also quite interesting because if you have multiple clients in the same workload, one of these clients just has very few data examples and another client has tons of data examples. For example, we all know that one type of person that takes very few photos when they're on vacation. And we all know that other type of person who takes a ton of photos when they're on vacation.

• 23:35Speaker 00

  So this is a very practical example for different amounts of data on each device. In such a scenario, when you instruct a client to do, for example, one epoch on their data, then obviously this one client will be, the update will be coming back much, much faster. So what you want to have in your entire system is some robustness towards clients who either take a very long time because they have so much data or towards even real stragglers who, I don't know, maybe the device is suddenly getting busy with other things, which delays the update coming back. So this is something where your software infrastructure needs to be able to handle these kinds of cases.

• 24:15Speaker 00

  And it's also something where you need appropriate ways of handling it on the server side. So. The one obvious or the one that the easiest thing to do is obviously to discard those clients that are taking a long time, that are strugglers. But then there are more clever ways to approach this, for example, to let this client know, hey, your time is running out. We are about to close the round. Why don't you submit your partial update? But then your server side and the aggregation logic, it needs to be able to handle those partial updates coming from clients.

• 24:46Daniel Whitenack?

  So Daniel, you've talked a little bit about like certain client devices being strugglers from one perspective. But I'm curious in terms of how the federated learning community is thinking about things like bias in data. So like if I am a data scientist in like a central location, I'm seeing maybe updates to my model. But I'm not seeing the data that is producing those updates to the weights and biases of my model. So if there's bias in terms of those in client devices, like maybe ninety seven percent of my client devices are being operated by males. And I I have some gender bias in the data that's coming back.

• 25:29Daniel Whitenack?

  Are there ways that the community is thinking about that and ways to address that sort of, I guess, maybe, maybe there's a term for it. I'm thinking of it like client bias. Yeah. Any thoughts there?

• 25:39Speaker 00

  Yes, absolutely. It's a very good question and it's a very important question. There are different ways to think about it. One way is or one approach that one topic that the community thinks about a lot is how to address that from an algorithmic perspective. So there are approaches, for example, Qfair, federated learning, that tackle this from an algorithmic perspective. So when you collect updates, you can do this in a certain way and you can try, for example. There are many different approaches, but one thing you could do is you could try to address that, for example, through the averaging process. It's a weighted averaging. So there are ways to influence this. Another perspective is more from a more intuitive and more practical perspective

• 26:20Speaker 00

  in the sense that you can think of federated learning as a way compared to centralized learning to actually overcome bias because you can, or not overcome it completely, but that's. Not what I mean, but help to overcome it in the sense that you can suddenly get access to more training data and hopefully more representative training data. And then you can make better decisions about how to train your model and what kind of pieces of data to include in your training process, how to sample these data examples that you have on the clients and a lot of those related questions.

• 27:05Speaker 01

  Earlier in the show, you heard a teaser from our friends behind the podcast, Me, Myself and AI. MIT Sloan Management Review and Boston Consulting Group came together to produce this awesome podcast. And every episode, host Sam and Sherwin talk to leaders that are engaged in the theory and the practice of AI.

• 27:22Speaker 05

  I remember one project we had, we were training a chatbot. And it turned out we used raw, you know, logs, all privacy assured and everything, but we used these logs that a customer had provided because they wanted to see if we could build a better model. And it turns out that the chat agent wasn't exactly speaking the way we'd want another human being to speak to us. And why? Because people get pretty upset when they're talking to customer support. And the language that they use isn't necessarily a language I think we would use with each other,

• 27:55Speaker 05

  you know, on this podcast.

• 27:56Speaker 01

  All right. Me, Myself and AI is a collaboration between MIT Sloan Management Review and Boston Consulting Group. It's available wherever you get your podcasts. Just search Me, Myself and AI.

• 28:32Daniel Whitenack?

  Well, Daniel, this is practical AI. So we definitely should get into the practicalities of how federated learning can be implemented. And I think you're probably one of the better people to speak to that because you've been heavily involved and one of the creators of the Flower framework. So maybe just to start out our discussion around that, could you talk about the backstory of Flower, the motivation behind it and what it is?

• 28:59Speaker 00

  Yeah, absolutely. Trying to do my best. So when we started out on this journey, we obviously, we got excited about federated learning for the reasons I was stating earlier. So we were actually in real industry projects. We were facing these challenges where we saw that the data these organizations had in-house was just not enough. But we saw that there were other organizations who had similar challenges and we saw the potential to build collaborative approaches there. And the only way to do this, these kinds of collaborations would be, the only way that would be feasible would be if the underlying trade accounting would not have to be shared. This was sort of the setting that we were in when we first looked at federated learning.

• 29:40Speaker 00

  And at the time, we obviously looked at our solutions, but there wasn't really a solution that was a really good fit for our requirements. One of our requirements was obviously because we were coming from this, we were looking at these practical problems, was that we could build a system that we can then, at a later point, move into production. So obviously, you would start out with some prototyping and see if you could get such a workload to converge. But then at a later point, if you cannot move this into production, then why would you invest in this, right? So this was one of our hard requirements. And then for moving a federated learning or federated analytics workload into production, there are a ton of associated challenges with that. I was hinting at this large heterogeneity that we see on the client side.

• 30:25Speaker 00

  So being able to integrate with a embedded device, a mobile device, server, HPC cluster, this is something we thought was high on our priority list. So at the time, We didn't really see any solution that that was a fit to the requirements that we had. We sort of had to shift our focus a little bit away from building this one particular system that we had in mind. And we we shifted the focus away to first building the infrastructure that we had in mind for it. Out of that, we built a prototype for that. And then out of that prototype, we gathered a lot of learnings, obviously. And eventually, at the beginning of last year, Tanner, my co-founder and I, we said, um, okay, let's start a company and build this infrastructure

• 31:06Speaker 00

  to bring these advances that we see and this huge potential to make this really accessible for others to use as well. The flow framework is probably obvious by now that that one of the reasons the flow framework is there is that we want to enable everyone to build such workflows because there's a lot of details going on under the hood that are not easy to implement. And if you just want to do federated learning, it would obviously be a a huge hurdle for others to first build this infrastructure before they then can build their actual workload. We wanted to make this easy. We want to make it easy to start in research and then gradually enhance these workloads and move them to to production eventually and then to operate them in production.

• 31:48Speaker 00

  This is also something we we haven't quite seen in other frameworks. Other frameworks that we've seen are are usually focused on one thing. For example, focused on being a good simulation engine, but then you can't take these workloads and move them into production. And the other opportunity that we saw, and this is part of this user journey, making it easy to start, prototype something, is the opportunity to be compatible with all of the um machinery frameworks that we are seeing out there. So we see huge excitement about TensorFlow and PyTorch, obviously. Um, those are the dominating frameworks, I should say. Now there's a lot of excitement about JAX by many people. And there are these um other frameworks which are also relevant, sometimes relevant for very specific cases.

• 32:30Speaker 00

  And the opportunity that we saw is, well, sort of based around this story, you have an existing machinery project. What's the minimal amount of code changes that you that you have to do in order to federate this thing? And we have code examples on that where you can. Take an existing workload and then federate it in in less than twenty lines of code, which is actually, I still find it amazing, given the amount of things that are going on under the hood.

• 32:55Daniel Whitenack?

  Yeah, and you mentioned supporting all of these different frameworks, which does seem like a big task. And I'm kind of looking through the Flower usage examples and the documentation. And I also love just, you know, I mean, you explicitly say it's a friendly framework, which I think is great. You talked about accessibility. You've got a very friendly Flower logo. And so, yeah, I think it, you know, it puts up an inviting front for people, which I think is cool, because it is a, it can be, like you said, a very overwhelming, complicated thing to get into. You were talking about supporting these different frameworks and maybe you could give a sense of like, it seems like a big task to support all of those in this way.

• 33:36Daniel Whitenack?

  And I see that the main kind of way in which you wrap things with Flower is like creating this class, Python class, maybe that wraps certain methods. And within those, you can define your own sort of TensorFlow or PyTorch or whatever ways to fit or get parameters of a model or whatever it is. Did you purposely create that structure because you had this vision of supporting the multiple frameworks? And am I representing that accurately?

• 34:05Speaker 00

  Yes, absolutely. We call it Flower, the Friendly Federating Framework, exactly for that reason. So we want to be friendly in many different dimensions, actually. We want to be friendly when it comes to different machine learning frameworks. We want to be friendly when it comes to different device types. We want to be friendly when it comes to different transport mechanisms. So we, we actually have this, not something that's upfront on the website, but we have different transport mechanisms built in and you can swap these out, actually. So building and support for different frameworks, this was something that we intended to do from the very beginning. And there's different layers to this that are important or at least interesting to understand. So one layer is the client class that you just described. So when you build your client in Python, then you would create a subclass of client.

• 34:50Speaker 00

  So flower dot client dot client, it's what the class is called, or a sub- Class of Flower dot client dot NumPy client, which is even easier to implement. And you basically just need to add these few lines of code that then call into your existing machine learning pipelines. Which is on the one hand, a simple concept, but on the other hand, a very powerful concept. Because it allows you, when you implement these classes, it allows you to call arbitrary Python libraries. So for example, one good and one important example for that is the support for differential privacy. We sometimes get requests, hey, does Flower come with differential privacy built in? And actually, the answer is, we don't have to.

• 35:32Speaker 00

  Because you can, for example, for a PyTorch based workload, you can use this library called Opacos, which is, gives you a sort of a differential private SGD optimizer that you can plug into your workload. And then you can just use it. And the amazing thing about this is that with the Flower framework, it doesn't even have to change. If there's a new library coming out, a new approach coming out for what you can do on the client side, you can just integrate it with arbitrary code. The other layer that is maybe interesting to understand, maybe not so much for researchers who do most of their day to day work in Python, but to others who want to maybe more deeply integrate this in a, in the automotive setting or something similar to that. They wouldn't want to use Python for their on-device processing.

• 36:17Speaker 00

  So they would use, they would want to use a different language, for example, C to do that. In the automotive world, there's this C dialect called Mr. C that you have to use for safety purposes. For example, it prevents you from using recursion and other things like that, things that are being considered unsafe in the automotive world. And in those scenarios, you can still integrate your device with Flower by directly handling the events that are coming from the server. So in the end, Flower has been designed in a way where the client side is actually rather easy to implement. And if you have something that is running on C or C++, all you would have to do is, you would have to establish a connection to the server. The server would then occasionally select this client.

• 37:02Speaker 00

  And when it selects the client, it sends it a message. You on the client side, you have to handle this message. You can do your processing, it doesn't have to be any of the well-known machine learning frameworks. You can hand code the type of model that you have. And then you send back a message containing your update, for example, the gradients that you collected.

• 37:19Daniel Whitenack?

  That's awesome. I love that sort of client agnostic focus. It's cool.

• 37:24Chris Benson?

  One of the things I was curious about, because as a practitioner, I'm kind of in and out and I'll do other things in my job. And when I'm coming back in, I'm having to kind of go, how did I do that before and stuff? And one of the things that I've noticed in the industry is that the barriers to be able to access or utilize machine learning are getting lower. And there's a lot of tools around usability coming out. What does the story look like for Flower and maybe for Federated Learning at large as you have more users out there of various technical capability and maybe gradually, you know, having that technical requirement going lower and lower as the tooling gets better? How will Federated Learning fit into that world where more users with less specific skill in this area are

• 38:07Chris Benson?

  accessing these tools and creating models of various types? What does that look like there?

• 38:12Speaker 00

  That's a great question. So I'm sometimes saying that we've been, or maybe we still are, I'm not perfectly sure on that, in a pre-Tensorflow era when it comes to Federated Learning. It was the case for a long time that if you wanted to build a Federated Learning workload, you usually had research scientist type of person start out to prototype this, make a simulation of it. And if that converges, then you could actually, you could make the decision that you want to have this in production, but then you would basically start from scratch and you would implement it in quote unquote "real" system with, I don't know, Java or C++ or something like that. So you had to build these systems by hand. There's a, for

• 38:55Speaker 00

  example, there's a blog post that compares Federated Learning frameworks. And before Flower was around, the conclusion was really, if you want to build this workload, a Federated Learning system, and you want to build it in really a production environment, then your best option is to just build it from scratch by hand. They've recently updated this blog post to say that for their scenario, they choose to use Flower for that. Obviously, I'm happy about that, but it's still not a super, super polished experience. So Flower makes it a lot easier to start out on that journey, but it's still a couple of moving pieces that you should sort of understand to make in front of Decisions about how to configure your workload, for example.

• 39:37Speaker 00

  That's something that is obviously one of our priorities to make this even easier, to make it even less likely that if you are not an expert on this, that you are configuring something, building something that might not be a good choice in production. So one of the things that we take very seriously is that we build in the right defaults. So one of the defaults, for example, that the Flower Framework is following, that is for certain types of workloads, it's the go-to recommendation, is that the Flower Framework, when it gets updates from clients, it does not persist these updates in any way. So there's these individual updates from clients, and they could allow you to peek into it and to draw some, at least some minor conclusions about the client's dataset.

• 40:21Speaker 00

  And therefore, the recommendation is to receive these updates, only keep them in memory and only for the minimum amount of time absolutely necessary. So once you aggregate it with other updates, you can safely discard it. And another very related thing is that, for example, the server does not log any client-specific metrics by default. So those are things that we are trying to build in, that if you just start the server with all defaults, that it makes something that takes a sensible approach. But then obviously, there are more advanced users and they want to customize it. So the perspective is make defaults sort of safe, as safe as we can, and then allow more advanced users to customize these workloads.

• 41:04Daniel Whitenack?

  So as we close out here, I'm interested to hear about, you know, what is like one or a couple of things that like really excites you about the future of Flower and maybe its its applications within the wider context of federated learning? What's the one thing or the couple of things that really get you excited about where this is headed? Or maybe sort of within the roadmap of Flower?

• 41:30Speaker 00

  There are a ton of things that get me excited, both from a research perspective, but also from a practical perspective. From a research perspective, we just launched a preview of a new feature that we are calling the Virtual Client Engine. The Virtual Client Engine is something that, well, it manages clients as virtual clients, so those clients, they don't actually exist in memory. And what this gives you, it sounds pretty trivial, but what this gives you is amazing scalability for your research workloads. So we did a survey of research papers and looked at what the scale of these workloads is in the research of those experiments.

• 42:09Speaker 00

  And really, the vast majority of papers, they used up to a hundred clients and also up to a hundred clients doing work concurrently, so training concurrently, for example. So you can have a large client pool, you can have a client pool of, I don't know, 10,000 clients, but then they would have only a hundred of them participating in the same round. And this is something that is likely due to resource constraints, because those workloads can get very heavy. And the systems that we read about from industry, they are at a vastly different scale. So they have millions, or tens of millions, or even hundreds of millions of clients in such a workload. And this is quite interesting and also quite an important challenge to address, because

• 42:54Speaker 00

  obviously, we want to have research that eventually translates to the real world to practical setting. And if the scale in research is a very different scale from the practical settings, it's less likely that the research that we are conducting will translate into the practical setting. So the Virtual Client Manager is one thing where we demonstrated on quite average hardware, actually. We ran a workload with 15 million clients in it and a thousand of these clients training concurrently, and this worked super well. So I'm quite excited about that one, and especially quite excited to see what the community is going to do with that. That's from a research perspective, and we have a couple of things in the pipeline that we are going to announce over the coming months. Also, from a practical perspective, that's maybe even more exciting.

• 43:40Speaker 00

  In terms of the real outcomes that we're going to see from that, one initiative that we are, for example, participating in is called MEDPERF, which is a hosted by ML Commons, which is sort of the organization that emerged out of MLPerf. So MEDPERF is a way to use federated evaluation to get a better understanding of the performance of medical AI models. It also requires federated infrastructure. We put the paper in archive, the entire MEDPERF group put the paper in archive a couple of weeks ago. Super interesting read, highly recommended. And this is something where you can really see that the real world impact and real world improvements

• 44:20Speaker 00

  that we're going to see from this can be very profound because it's about medical AI and medical, getting better performance estimates in medical AI is actually a very fundamental challenge. And once we have these better estimates, it is much safer to roll out medical AI models much faster. And apart from MEDPERF, there are also a couple of other initiatives in the medical AI space and the drug discovery space that I'm very excited about. Because any advance our infrastructure will help in generating can have a very profound impact on society as a whole. So that's something I'm quite keen on contributing to.

• 44:59Daniel Whitenack?

  Well, Daniel, I'm super excited about all of the things that you've mentioned in terms of things on the roadmap of research with flower or practical uses of flower and federated learning. And really appreciate you joining us and talking us through everything on the podcast. Appreciate it. And we'll include some show notes in our show hosting for flower and all the wonderful things that you've talked about. But yeah, thank you so much for joining and looking forward to keeping tabs on flower.

• 45:29Speaker 00

  Thanks for having me.

• 45:33Speaker 06

  That's our show. Thanks for listening. For more like this, check out our master feed. It is all Changelog podcasts in one easy to consume place. Let your podcast app snag everything we produce and then pick and choose which ones to listen to. Subscribe today at changelog.com/master or just search for Changelog Master in your podcast app of choice, you'll find it. Special thanks to Brake Master Cylinder for providing our music and to our longtime sponsors, Facily, Launch Darkly, and Linode. That's all for this week. We'll talk to you again next time.