Federated Learning (FL) is the training of an AI model. Instead of sending all sensitive data to a central location, Florida keeps the data intact and only shares model updates. This preserves privacy and allows AI to get closer to where data is generated.
However, spreading calculations and data across many devices presents new security challenges. Attackers participate in the training process and subtly influence it, degrading to accuracy, biased output, or hidden background within the model.
The project will investigate how such attacks can be detected and mitigated in Florida. To do this, we have built a multi-node simulator that allows researchers and industry experts to replicate attacks and test defenses more efficiently.
Why is this important?
- Non-technical examples: Think of books on shared recipes that many restaurant chefs have contributed to. Each chef updates some recipes with their own improvements. An unfair chef can intentionally add the wrong ingredients to interfere with the dish, or quietly insert a special flavor that he knows how to fix. Without anyone carefully checking the recipe, all future diners in every restaurant could end up being ruined or manipulated.
- Technical example: FL displays the same concepts as data addiction (manipulation of training examples) and model addiction (changes in weight updates). These attacks are especially harmful when the federal is subject to non-IID data distribution, imbalanced data partitions, or delayed participation in clients. Modern defenses such as multi-crom, trimmed averages, division and conquests can still fail in certain scenarios.
Building a multi-node FL attack simulator
To assess the resilience of federated learning to real-world threats, we constructed a multi-node attack simulator on top of the Scaleout Systems Fedn framework. This simulator allows you to replicate attack, test defense, and scale experiments with hundreds or even thousands of clients in a controlled environment.
Important features:
- Flexible deployment: Distribute FL jobs using Kubernetes, Helm, and Docker.
- Realistic data settings: Supports IID/non-IID label distributions, imbalanced data partitions, and delayed clients.
- Attack Injection: Includes general addiction attack implementations (label flipping, little is enough) to make new attacks easy to define.
- Defense Benchmark: It integrates existing aggregation strategies (FEDAVG, trimmed averages, multikrum, division, conquest) to allow experimentation and testing of various defensive strategies and aggregation rules.
- Scalable experiments: Simulation parameters such as client count, malicious sharing, and participation patterns can be adjusted from a single configuration file.
Using FEDN's architecture means that simulations benefit from robust training orchestration, client management, and allow visual monitoring via the studio web interface.
It is also important to note that the FEDN framework supports server functionality. This feature allows you to implement new aggregation strategies and evaluate them using an attack simulator.
To start with your first example project using FEDN, here is the QuickStart guide:
The FEDN Framework is free for all academic and research projects, as well as industrial testing and exams.
Attack simulators are available and can be used as open source software.
Attacks we studied
- Label Flip (Data Addicted) – Malicious clients flip labels on local datasets, such as changing “cat” to “dog” to reduce accuracy.
- Little is enough (model addiction) – Attackers make small, targeted adjustments to model updates and shift the output of the global model to their own goals. In this paper, we applied a small attack every three rounds.
Beyond Attacks – Understanding Unintended Influences
Although this study focuses on intentional attacks, it is equally valuable in understanding the effects of marginal contributions caused by false breaches or device malfunctions in large coalitions.
In our recipe example, even honest chefs can accidentally use the wrong ingredients because their oven is broken or the scale is inaccurate. The mistakes are not intentional, but many contributors change the shared recipe in ways that can be harmful if repeated.
In a cross-device or fleet learning setup, thousands or millions of uneven devices can contribute to shared models, failed sensors, outdated configurations, or unstable connections, and can degrade the performance of the model in a manner similar to malicious attacks. Studying attack resilience also reveals how to robust aggregation rules for such unintended noise.
Mitigation strategies explained
In FL, the aggregation rules determine how to combine model updates from the client. Robust aggregation rules aim to reduce the impact of outliers, even when caused by malicious attacks or faulty devices. The strategies we tested were:
- FEDAVG (baseline) – Simply average all updates without filtering. It is extremely vulnerable to attacks.
- Trimmed average (trmean) – Sort each parameter across the client and discard the highest and lowest values before averaging. It reduces extreme outliers, but can miss subtle attacks.
- Multicrum – Score each update by how close it is to the nearest neighbor in the parameter space, keeping only the least total distance. Very sensitive to the number of selected updates (k).
- EE trimmed average (newly developed) – Epsilon – An adaptive version of Trmean that uses Greedy's scheduling to determine when to test different client subsets. Resilient due to client behavior changes, slow arrivals, and non-IID distributions.
The tables and plots presented in this post were originally designed by the scale-out team.
experiment
In 180 experiments, we evaluated different aggregation strategies under different attack types, malicious client ratios, and data distribution. For more information, see the full paper here.
The above table shows one of a series of experiments using label flipping attacks using non-IID label distributions and partially unbalanced data partitions. The table will be displayed Test accuracy and Test loss AUCis calculated for all participating clients. The results of each aggregation strategy are displayed in two rows, corresponding to two rows Slow policy (Participating from the fifth round or malicious client or from the fifth round). The column separates the results with three malicious rates, resulting in six experimental structures per aggregation strategy. The best results for each configuration are displayed Bold.
The table shows a relatively uniform response across all defence strategies, while the individual plots offer a completely different view. In FL, the federals can reach a certain level of accuracy, but examining client participation is equally important. In particular, the client was successfully trained and was rejected for malicious intent. The following plot shows the participation of clients under various defence strategies.
Use 20% malicious clients under label flipping attacks against non-IID, partially unbalanced data, cropped averages (Figure 1) maintained overall accuracy but never completely blocked clients from contributing. Although coordinate trimming reduced the impact of malicious updates, rather than excluding the entire client, it filters out individually filtered parameters, allowing both benign and malicious participants to remain in aggregation through training.
Malicious clients join 30% late and non-IID, disproportionate data, multicrum (Figure 2) I accidentally selected a malicious update from round 5 and later. Updates with high data heterogeneity appear to be less similar, with malicious updates ranked as one of the most central and can last for a third of the models aggregated for the rest of the training.
Why an adaptive aggregation strategy is needed?
In general, it depends on static thresholds to determine which client updates to be included in the aggregation of the new global model. This highlights the shortcomings of current aggregation strategies. This makes you vulnerable to overdue clients, non-IID data distributions, or data volume imbalances between clients. These insights led us to develop the EE-Trimmed Average (EE-TRMEAN).
EE-TRMEAN: Epsilon's Greedy Aggregation Strategy
EE-TRMEAN is built on classic trimmed averages, but adds a layer of exploitation vs. exploitation to select clients, Epsilon greedy.
- Exploration stage: All clients are allowed to contribute, and the usual trimmed average aggregation round is performed.
- Exploitation stage: Untrimmed clients are included in the exploitation stage through an average score system based on previous rounds they participated in.
- The switch between the two phases is controlled by an epsilon ready policy with a collapsed epsilon and an alpha lamp.
Each client scores based on whether the parameters withstand trimming in each round. Over time, the algorithms will increase the advantages of the best scoring clients, sometimes exploring others to detect behavioral changes. This adaptive approach allows EE-TRMEAN to be more resilient when data heterogeneity and malicious activity are high.
In a label flipping scenario with 20% malicious clients and non-IID, partially unbalanced data, and late benign joiners in EE-TRMEAN (Figure 3) The exploration stage and exploitation stage are alternated. Enables all clients and selectively blocks low-score clients. Although we occasionally excluded benign clients due to data heterogeneity (much better than known strategies), we successfully identified and minimized the contributions of malicious clients during training. This simple yet powerful change will improve client contributions. The literature reports that as long as the majority of clients are honest, the accuracy of the model remains reliable.
