How Critical is it for a Data Scientist to Adapt Federated Machine Learning? | by Poornachandra Sarang | Oct, 2022
Opinion
Federated Machine Learning: A New ML Baby
Google introduced the term Federated Learning in 2016 to mark the beginning of a new machine learning approach in the ML paradigm. Federated learning resolves many shortcomings of the centralized and distributed training approaches. Without the use of federated learning, we would not have seen the highly improved on-device machine learning model like “Hey Google” in Google Assistant. To understand federated learning and its importance in today’s IoT world, let me first describe the shortcomings of the existing models.
The notion of machine learning started with centralized training. You upload the entire dataset to a centralized server, train the machine learning model using a central server, do evaluation/testing, and finally deploy it on a central server. The clients are then required to communicate to a remotely deployed model for inference on their data. This approach has severe shortcomings. You need to upload the entire data to the server, which these days could be in terabytes even for a medium-sized business. This results in high communication overheads between participants and the cloud server. Uploading the data on the cloud server also compromises data privacy, leading to the violation of GDPR (General Data Protection Regulation). On the other hand, the advantage of this approach is that participants are free from computation responsibilities, which require high resources.
The distributed training uses specifically designed machine learning algorithms to address the computational problems in the complex algorithms that operate on large datasets. These algorithms are efficient and scalable, compared to algorithms used in centralized training. Multiple participants perform independent training. A central server aggregates these trainings to create an ultimate model. We do convergence testing on a central server and in case of poor accuracy, we repeat the entire process until we achieve acceptable accuracy. You see that the entire approach requires massive cloud resources. We deploy the last model on the cloud and the clients have to connect to this remote model for inference on their data. This training approach still does not address the data privacy issue.
So, here comes the federated learning.
Federated learning takes a training approach that differs a lot from the conventional training that I described earlier. In this method, you typically start by creating a generic machine-learning model on a centralized server. This initial model acts as a baseline for all the participants involved in further training. The participants download the base model from the central server. Each participant improves this model by using its local data. As a small focused update, participants send changes to the server. The communication uses homomorphic encryption or any other secured transmission protocols, ensuring data privacy. We do not send the user data to the server, we only send the model updates (weights in neural networks).
The server then aggregates the learnings from all the participants to improve the base model. We immediately discard the updates to ensure absolute data privacy. The server sends the improved model to the participants for further personalization. We repeatedly run this process to improve the model. On each iteration, the server model gets better and more personalized to each participant.
Now, let us try to analyze how this approach overcomes the shortcomings of the earlier models.
As you see from the working of federated learning, all the training data remains on your (participant) device. We store no individual updates in the cloud. In fact, Google uses a Secure Aggregation protocol for their federated learning. It uses cryptographic techniques that allow the coordinating server to decrypt the average update of 100s and 1000s of users. The protocol does not allow for the inspection of individual updates. This allows for total privacy for the participants.
Obviously, the process itself allows for creating smarter models which are dynamically updated. As we share the updated model with the users continuously, you will see the personalization while using your device, for example, a mobile phone. Low-latency and high-throughput are required for the traditional and distributed algorithms. The nature of federated learning does not cause these requirements. The power consumption and computing resources required to aggregate the model on the server too is substantially reduced.
Just to mention a disadvantage, in federal learning, the data distribution is highly uneven. Note that millions of devices process their own individual data. These devices have high-latency and low-throughput connections. Not only this, the devices are available only intermittently for training. To address these issues, Google developed the Federated Averaging algorithm, and so do others in this space. Another challenge is to deploy this technology to millions of heterogeneous devices. You will need a miniature version of TensorFlow or any such machine learning framework on each device.
Tech-giants have successfully resolved these issues, and there are many practical use-cases of this technology which I will discuss next.
As I mentioned earlier, Google Assistant uses federated learning to continuously improve “Hey Google”. The on-device training ensures data privacy by not sharing the personalized voice messages to the cloud. Yahoo. Baidu, Google, and DuckDuckGo all use federated search indexes for more accurate results. Apple also uses federated machine learning to improve the models on their mobile devices, keeping your data private. Microsoft Research has introduced FLUTE (Federated Learning Utilities and Tools for Experimentation) — a framework for running large-scale offline federated learning simulations. Google’s Gboard on Android uses FL for next-word predictor.
Following on the steps of these tech-giants, data scientists can now apply federated learning to many potential industries.
For example, healthcare and health insurance industries can surely benefit from federated learning. In these industries, data privacy requirements are more stringent. Federated learning protects sensitive data, provides better data diversity — which usually is observed in these industries. The data comes from hospitals, health monitoring devices and so on. The data diversity can help diagnose rare diseases and also provide early detection.
Think of an e-commerce company that will benefit by having a better product purchase predictor if they had the customer spending capacity, patterns from the bank. Certainly, banks and e-commerce are two verticals who will never share their data. However, using federated learning, both can train their own ML models based on their preferred features; We can then aggregate the model parameters by a third party curator, which would ultimately benefit both consumers.
For autonomous vehicles, federated learning can provide a better and safer experience for passengers. The model can learn with real-time data on the traffic and road conditions.
There are a number of more industries like FinTech (Finance and Technology), Insurance and, in general, IoT, which can benefit from FL. Data scientists need to dig into these areas to find more business for themselves.
There are several federated learning platforms available for you to start your journey on FL. Google provides a federated learning platform based on Tensorflow — it is called Tensorflow federated. Intel has released an open-source framework for federated learning, called openfl. Then there is IBM FL platform. NVIDIA Clara powers federated learning. There are many more for you to Google out.
As you have seen, federated learning has overcome many shortcomings of traditional and distributed machine learning. Importantly, it provides user data protection, which is the demand of today’s consumers of technology. FL can easily handle data diversity. It allows for real time continual learning and is less strenuous on hardware resources. FL has opened many new model development opportunities across domains. So, what are you waiting for?
Pooja Gramopadhye — Copy editing
Opinion
Federated Machine Learning: A New ML Baby
Google introduced the term Federated Learning in 2016 to mark the beginning of a new machine learning approach in the ML paradigm. Federated learning resolves many shortcomings of the centralized and distributed training approaches. Without the use of federated learning, we would not have seen the highly improved on-device machine learning model like “Hey Google” in Google Assistant. To understand federated learning and its importance in today’s IoT world, let me first describe the shortcomings of the existing models.
The notion of machine learning started with centralized training. You upload the entire dataset to a centralized server, train the machine learning model using a central server, do evaluation/testing, and finally deploy it on a central server. The clients are then required to communicate to a remotely deployed model for inference on their data. This approach has severe shortcomings. You need to upload the entire data to the server, which these days could be in terabytes even for a medium-sized business. This results in high communication overheads between participants and the cloud server. Uploading the data on the cloud server also compromises data privacy, leading to the violation of GDPR (General Data Protection Regulation). On the other hand, the advantage of this approach is that participants are free from computation responsibilities, which require high resources.
The distributed training uses specifically designed machine learning algorithms to address the computational problems in the complex algorithms that operate on large datasets. These algorithms are efficient and scalable, compared to algorithms used in centralized training. Multiple participants perform independent training. A central server aggregates these trainings to create an ultimate model. We do convergence testing on a central server and in case of poor accuracy, we repeat the entire process until we achieve acceptable accuracy. You see that the entire approach requires massive cloud resources. We deploy the last model on the cloud and the clients have to connect to this remote model for inference on their data. This training approach still does not address the data privacy issue.
So, here comes the federated learning.
Federated learning takes a training approach that differs a lot from the conventional training that I described earlier. In this method, you typically start by creating a generic machine-learning model on a centralized server. This initial model acts as a baseline for all the participants involved in further training. The participants download the base model from the central server. Each participant improves this model by using its local data. As a small focused update, participants send changes to the server. The communication uses homomorphic encryption or any other secured transmission protocols, ensuring data privacy. We do not send the user data to the server, we only send the model updates (weights in neural networks).
The server then aggregates the learnings from all the participants to improve the base model. We immediately discard the updates to ensure absolute data privacy. The server sends the improved model to the participants for further personalization. We repeatedly run this process to improve the model. On each iteration, the server model gets better and more personalized to each participant.
Now, let us try to analyze how this approach overcomes the shortcomings of the earlier models.
As you see from the working of federated learning, all the training data remains on your (participant) device. We store no individual updates in the cloud. In fact, Google uses a Secure Aggregation protocol for their federated learning. It uses cryptographic techniques that allow the coordinating server to decrypt the average update of 100s and 1000s of users. The protocol does not allow for the inspection of individual updates. This allows for total privacy for the participants.
Obviously, the process itself allows for creating smarter models which are dynamically updated. As we share the updated model with the users continuously, you will see the personalization while using your device, for example, a mobile phone. Low-latency and high-throughput are required for the traditional and distributed algorithms. The nature of federated learning does not cause these requirements. The power consumption and computing resources required to aggregate the model on the server too is substantially reduced.
Just to mention a disadvantage, in federal learning, the data distribution is highly uneven. Note that millions of devices process their own individual data. These devices have high-latency and low-throughput connections. Not only this, the devices are available only intermittently for training. To address these issues, Google developed the Federated Averaging algorithm, and so do others in this space. Another challenge is to deploy this technology to millions of heterogeneous devices. You will need a miniature version of TensorFlow or any such machine learning framework on each device.
Tech-giants have successfully resolved these issues, and there are many practical use-cases of this technology which I will discuss next.
As I mentioned earlier, Google Assistant uses federated learning to continuously improve “Hey Google”. The on-device training ensures data privacy by not sharing the personalized voice messages to the cloud. Yahoo. Baidu, Google, and DuckDuckGo all use federated search indexes for more accurate results. Apple also uses federated machine learning to improve the models on their mobile devices, keeping your data private. Microsoft Research has introduced FLUTE (Federated Learning Utilities and Tools for Experimentation) — a framework for running large-scale offline federated learning simulations. Google’s Gboard on Android uses FL for next-word predictor.
Following on the steps of these tech-giants, data scientists can now apply federated learning to many potential industries.
For example, healthcare and health insurance industries can surely benefit from federated learning. In these industries, data privacy requirements are more stringent. Federated learning protects sensitive data, provides better data diversity — which usually is observed in these industries. The data comes from hospitals, health monitoring devices and so on. The data diversity can help diagnose rare diseases and also provide early detection.
Think of an e-commerce company that will benefit by having a better product purchase predictor if they had the customer spending capacity, patterns from the bank. Certainly, banks and e-commerce are two verticals who will never share their data. However, using federated learning, both can train their own ML models based on their preferred features; We can then aggregate the model parameters by a third party curator, which would ultimately benefit both consumers.
For autonomous vehicles, federated learning can provide a better and safer experience for passengers. The model can learn with real-time data on the traffic and road conditions.
There are a number of more industries like FinTech (Finance and Technology), Insurance and, in general, IoT, which can benefit from FL. Data scientists need to dig into these areas to find more business for themselves.
There are several federated learning platforms available for you to start your journey on FL. Google provides a federated learning platform based on Tensorflow — it is called Tensorflow federated. Intel has released an open-source framework for federated learning, called openfl. Then there is IBM FL platform. NVIDIA Clara powers federated learning. There are many more for you to Google out.
As you have seen, federated learning has overcome many shortcomings of traditional and distributed machine learning. Importantly, it provides user data protection, which is the demand of today’s consumers of technology. FL can easily handle data diversity. It allows for real time continual learning and is less strenuous on hardware resources. FL has opened many new model development opportunities across domains. So, what are you waiting for?
Pooja Gramopadhye — Copy editing
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.