A defense method against multi-label poisoning attacks in federated learning

To evaluate the proposed method, we have conducted experiments with the FL framework and analyzed the experimental results.

Experiment setup

We evaluated the performance of the proposed method on the following four commonly used public datasets, including MNIST, FashionMNIST, NSL-KDD, and CICIDS-2017. The purpose of selecting different datasets is to simulate various application scenarios, such as image classification or text classification. Among the datasets we selected, MNIST and FashionMNIST are primarily focused on image data, while NSL-KDD and CICIDS-2017 are predominantly text-based datasets.

MNIST The MNIST dataset contains grayscale images of handwritten digits (0–9), totaling 70,000 images, each with a resolution of \(28 \times 28\) pixels. With this dataset, we trained a two-layer convolutional neural network (CNN) classification model with two fully connected layers using 60,000 images and tested it on the remaining 10,000 images.

FashionMNIST FashionMNIST is a benchmark dataset similar in format to MNIST, containing grayscale images of 10 fashion item categories, with 7000 images per class and a total of 70,000 samples. While the data structure mirrors MNIST (\(28 \times 28\) grayscale images, 10 classes), FashionMNIST introduces greater semantic complexity and higher intra-class variance due to the nature of fashion objects. Including FashionMNIST allows us to evaluate whether the proposed defense method remains effective under more challenging and realistic visual classification tasks. The same CNN model structure is used for both MNIST and FashionMNIST to ensure consistency in experimental comparison.

NSL-KDD NSL-KDD is a widely used dataset for research on network intrusion detection. This dataset contains various network traffic features, categorized into normal traffic and attack traffic, with a total of 41 features, including classification labels (normal or attack type). With this dataset, we trained a single-layer LSTM (Long Short-Term Memory) classification model.

CICIDS-2017 CICIDS-2017 is a large-scale network traffic dataset containing nearly 3 million entries, simulating normal traffic and various attack behaviors in a real-world network environment. The dataset contains over 80 features and five major types of attacks. With this dataset, we trained a two-layer LSTM (Long Short-Term Memory ) classification model.

To evaluate the robustness of the proposed method, we randomly and uniformly distributed the training data to m participants. These participants used the cross-entropy loss function and the Stochastic Gradient Descent (SGD) optimizer to train their local models. The settings of the hyperparameters are summarized in Table 1.  

The malicious node attack strategy is set as follows. Before training their local models, malicious participants flip the source labels of the training samples to the target labels. For example, in MNIST, malicious participants flip the label 1–7; in FashionMNIST, they flip the label “coat” to “bag”; in NSL-KDD, they flip the label “DoS” to “normal”; and for CICIDS-2017, malicious participants flip the label “port scan” to “web attack”. Each malicious participant can only flip one type of label, and different malicious participants can perform different attack operations. Additionally, we assume that malicious participants execute attacks in every iteration of the FL training process.

Comparative methods To validate the performance of our method, we compared it with some baseline methods and state-of-the-art methods, as follows.

FedAvg This is a standard FL approach without security measures. It averages the local models returned by participants and updates them as the global model.

Krum Selects the ‘closest’ update among all received updates to serve as the global model update.

Median For the parameters at each model location, the median of the update values from all participants at that location is calculated, and these medians are used as the new parameter values for the global model.

Trimmed Mean The update values of participants for each parameter location are sorted, and then the smallest and largest portions of the updates are discarded. The remaining updates are averaged to obtain the new value for that parameter.

FLDetector FLDetector uses a prediction model based on historical updates to estimate expected behavior of each participant. A client is flagged as malicious if its updates consistently deviate from the prediction over several rounds.

FLAME²⁶ FLAME clusters local models and calculates client trust scores using adaptive gradient clipping and reputation mechanisms. Clients with low scores are excluded from aggregation.

LFighter The aggregation server dynamically extracts gradients from the local updates of nodes, then performs clustering analysis on the extracted gradients, and filters out potential malicious updates before model aggregation. Lfighter is one of the state-of-the-art defense methods.

Results

Impact of the number of attacked classes

To evaluate the robustness of the proposed method under varying levels of attack complexity, we examine its performance against different numbers of attacked classes (\(class\_n\)).This setup reflects scenarios where malicious participants flip labels across a growing number of output classes, ranging from simple (single-label) to complex (multi-label) attacks.

Firstly, we need to determine the value of the number of attacked classes \(class\_n\). Hence, we experimentally assessed the impact of \(class\_n\) on the performance of the global model. The results are presented in Table 2.

As illustrated in Fig. 2, In the absence of defense mechanisms, we observe that an increase in the number of attacked classes generally reduces the overall impact of poisoning. This is particularly visible in the NSL-KDD and CICIDS-2017 datasets. For example, in NSL-KDD, model accuracy improves from 82.25% (\(class\_n\)=1) to 94.50% (\(class\_n\)=5). This trend suggests that distributed attacks dilute the per-class impact, thereby reducing global degradation. Similarly, in FashionMNIST and CICIDS-2017, accuracy also improves as \(class\_n\) increases, indicating that distributed attacks can reduce the adversary’s control over targeted misclassifications.

Table 3 summarize the global model’s classification accuracy across \(class\_n\) = 1 to \(class\_n\) = 5 on four datasets. Defense mechanisms should ideally maintain strong performance regardless of the number of attacked labels. Since robust aggregators (namely Krum, Median, and Trimmed Mean) are designed to enhance robustness rather than detect poisoned attackers, they perform poorly against poisoning attacks and, in some cases, their performance is inferior to FedAvg. FLDetector and FLAME exhibit competitive performance under low to moderate attack settings (e.g., \(class\_n\)=2). However, both methods begin to degrade under higher multi-label scenarios (\(class\_n\) >2) and in the CICIDS-2017 dataset. FLDetector is sensitive to attack diversity due to its reliance on historical behavior, while FLAME’s trust scoring may misclassify adaptive attackers. On the other hand, LFighter exhibits good defensive performance against single-label flipping attacks but fails to effectively defend against multi-label flipping attacks. This is because LFighter only considers the gradients of source and target class neurons, i.e., the gradients related to the attack. As the number of attacked labels increases, LFighter cannot effectively identify the source and target classes.

Our method, in contrast, demonstrates consistently high robustness and accuracy as the number of attacked classes increases. In the MNIST dataset, our method maintains accuracies of 0.9834, 0.9833, 0.9826, 0.9831, and 0.9833 for \(class\_n\) = 1 to \(class\_n\) = 5, respectively indicating stable performance across different levels of attack complexity. Notably, LFighter slightly outperforms our method when \(class\_n\) = 1 and \(class\_n\) = 3, but this is reversed when \(class\_n\) = 4 and \(class\_n\) = 5, where our method regains superiority.

This slight underperformance at \(class\_n\) = 1 can be attributed to the intersection strategy used to identify benign clients. In moderate attack scenarios (e.g., \(class\_n\)=3), benign participants’ gradients may still cluster closely with malicious ones, causing some benign updates to be wrongly excluded. However, as the number of attacked labels increases, the gradients of malicious clients become more dispersed across multiple output neurons, making their deviation from benign participants more noticeable and thus more effectively filtered by our method.

This performance trend is even more prominent in the NSL-KDD and CICIDS-2017 datasets. For instance, in NSL-KDD, our method achieves 96.01%, 95.99%, and 95.98% accuracy for \(class\_n\) = 2, \(class\_n\) = 3, and \(class\_n\) = 5, respectively, consistently outperforming all baselines including LFighter. In CICIDS-2017, our method also demonstrates impressive stability, maintaining 94.46–94.47% accuracy across various \(class\_n\) values, while the baseline methods significant drops in accuracy as attack complexity increases.

These findings support a key insight: The proposed method benefits from higher gradient dispersion introduced by multi-label attacks, which enhances the separability between benign and malicious updates during clustering.

In contrast, single-label attacks may affect only a narrow subset of the gradient space (e.g., connections related to a specific output neuron), making it difficult to distinguish malicious patterns from normal variations-especially when attackers attempt to mimic benign behaviors.

Impact of Federal Learning scale

To evaluate the scalability of our defense under varying federated learning (FL) sizes, we vary the number of participating clients from 10 to 50 and examine model performance under both single-label and multi-label label-flipping attacks. Results are shown for the NSL-KDD and CICIDS-2017 datasets in Fig. 3.

In the single-label setting, as show in Fig. 3a and c, we observe that FedAvg consistently exhibits the worst performance across all FL sizes, particularly when the number of participants is small. For example, on NSL-KDD, it achieves only 72.16% accuracy with 10 clients, improving slightly to 86.35% with 50 clients, due to averaging over a larger number of benign updates. However, this naive averaging remains highly vulnerable.

FLDetector shows strong early performance (e.g., 93.79% at 10 participants) but gradually declines as the number of clients increases, suggesting that its anomaly detection becomes less effective in larger systems with more diverse updates. A similar trend is observed for FLAME, which peaks at smaller scales (e.g., 95.88% at 10 participants) but loses effectiveness beyond 30 clients possibly due to increased variance and difficulty maintaining accurate reputation scoring at scale. LFighter maintains relatively strong performance across all FL sizes, although we observe a mild degradation from 96.98% at 10 clients to 94.98% at 50 clients in NSL-KDD. This may be due to the challenge of detecting subtle logit deviations when more clients are aggregated.In contrast, our method remains highly robust and stable, maintaining accuracy above 95% across all settings.

Under the more challenging multi-label attack setting, as show in Fig. 3b and d, the weaknesses of baseline defenses are more pronounced. In NSL-KDD, FedAvg degrades from 95.53 (10 clients) to 91.16% (50 clients), and FLDetector drops from 93.79 to 88.71%, showing that as scale increases, the attacks become harder to detect using history based or rule based defenses. FLAME shows similar degradation, while LFighter maintains moderate performance, slightly increasing at scale (e.g., 91.35% at 50 participants), possibly due to reduced per-client impact in larger groups. However, all baselines remain consistently weaker than our approach.

In contrast, our method shows strong and stable performance across all scales, maintaining above 94.4% accuracy even in the most challenging settings. The clustering-based filtering remains effective regardless of the number of participating clients, thanks to its reliance on inter-class consistency and density metrics, rather than historical or statistical heuristics.

Impact of attacker ratio

To evaluate the resilience of different defense methods under varying adversarial intensities, we conducted experiments on the NSL-KDD and CICIDS-2017 datasets with attacker ratios of 10%, 20%, 30%, and 40%. Both single-label and multi-label label-flipping attacks were considered. The results are summarized in Fig. 4.

In the single-label attack setting, as show in Fig. 4a and c, we observe that FedAvg suffers severe accuracy degradation as the attacker ratio increases. For example, on NSL-KDD, its accuracy drops from 94.56% at 10% to 82.25% at 40%, and similarly on CICIDS-2017 from 93.1 to 79.12%. This is expected since FedAvg aggregates all client updates equally, without any mechanism to filter or assess the trustworthiness of the inputs-making it highly vulnerable to poisoning when attackers form a significant proportion of participants. FLDetector, which relies on predicting client behavior from historical gradients, performs well at low attacker ratios (e.g., 95.36% at 10% in NSL-KDD), but its accuracy declines at higher ratios (e.g., 91.17% at 40%). This degradation stems from its dependence on the assumption that early rounds are attack-free. Once malicious clients participate early or dominate gradient history, the method’s predictive capability becomes unreliable, leading to undetected poisoned updates.

FLAME, leveraging gradient clipping and client reputation, shows relatively strong performance at lower ratios (e.g., 94.58% at 10% in NSL-KDD and 94.45% in CICIDS-2017). However, its accuracy drops as the attacker ratio increases-down to 92.44% and 92.87% at 40%, respectively. This is partly because its reputation mechanism can be fooled by adaptive attackers who mimic benign gradient behavior, and because benign clients with noisy gradients might be unfairly penalized. LFighter maintains strong accuracy at all ratios in the single-label setting (e.g., 95.98–96.02% in NSL-KDD), due to its model-based detection strategy focusing on logit consistency. However, its performance plateaus and slightly drops under higher attacker ratios, especially in CICIDS-2017, where overlapping gradient characteristics between benign and malicious clients can reduce detection sensitivity. In contrast, our method consistently performs best across all settings-achieving 95.98–96.02% in NSL-KDD and maintaining 94.46% in CICIDS-2017, even at 40% attacker ratio.

Under multi-label flipping attacks,as show in Fig. 4b and d, the trend becomes even clearer. FedAvg degrades rapidly, with accuracy on NSL-KDD falling from 94.79 to 91.76%, and CICIDS-2017 from 92.54 to 87.47%, again reflecting its inability to reject poisoned inputs.FLDetector and FLAME show sharper performance drops here than in the single-label case. For example, FLDetector drops from 94.88 to 90.01% on NSL-KDD and from 92.37 to 85.63% on CICIDS-2017. These declines highlight the limitations of relying on past gradient behavior or per-client scoring when attacks affect multiple classes and cause broader gradient variation.LFighter also shows weakened robustness in this setting, with accuracy dropping to 89.43% and 85.31% at 40% attacker ratio in NSL-KDD and CICIDS-2017, respectively. This can be attributed to its assumption of a fixed label-flip direction, which becomes ineffective under complex, dispersed attacks.

By contrast, our method not only remains stable but shows slight performance improvement at moderate attacker ratios, reaching 96.02% at 30% on NSL-KDD. This likely results from increased gradient divergence introduced by multi-label attacks, which enhances clustering separability. Even at 40% attacker ratio, our method maintains 96.01% and 94.46% on NSL-KDD and CICIDS-2017, respectively-significantly higher than all baselines.

Convergence

Figure 5 illustrates the convergence performance of our method in the absence of attacks on both the MNIST and NSL-KDD datasets. As depicted, our method has minimal negative impact on model convergence for both the image dataset (MNIST) and the text dataset (NSL-KDD). The convergence curves of our method closely align with those observed without attacks, demonstrating similar convergence speed and stability. This indicates that our defense mechanism does not compromise model training efficiency or convergence performance. Instead, it effectively maintains the training speed of the model while ensuring robust security.

Distance metric ablation study

To evaluate the impact of different distance metrics on the effectiveness of our clustering-based defense, we conducted an ablation study comparing cosine similarity, Euclidean distance, and the silhouette score as clustering criteria.

In our original method, we used average pairwise cosine similarity to measure the internal density of each cluster and distinguish between benign and malicious groups. To ensure consistency and fair comparison, we applied the same class-wise clustering pipeline while replacing the distance metric used in K-means and in density evaluation with:

Cosine similarity the original distance measure used in our method, which evaluates the directional alignment between gradient vectors and is robust to scale differences.

Euclidean distance the standard \(L_2\) norm, sensitive to both direction and magnitude.

Silhouette score a cluster quality metric that incorporates both intra-cluster cohesion and inter-cluster separation.

We used the FashionMNIST dataset under a multi-label flipping attack scenario with 30 clients, 40% of which were malicious. For each setting, we computed the average classification accuracy over five independent runs and recorded the precision of benign client identification.

The experimental results are illustrated in Fig. 6, showing the impact of different distance metrics on model accuracy and benign client identification.Cosine similarity yielded the most stable and robust clustering performance across classes. It consistently produced compact benign clusters, leading to high accuracy and low false positive rate in trusted client selection.Euclidean distance performed comparably in some cases but was more sensitive to gradient magnitude variations, especially when the scale of updates differed between clients.Silhouette score, although informative, was computationally more expensive and introduced instability in estimating the number of clusters, particularly when the benign group was small or under attack.

These results support our original design choice. Cosine similarity is well-suited to federated learning settings where update direction is more informative than magnitude. It achieves a good balance between clustering quality, computational cost, and robustness to poisoning.

Effectiveness of the intersection step

To evaluate the contribution of the intersection step in our clustering-based defense, we conducted an ablation study comparing the full method with a variant that omits the intersection across classes.

In the proposed method, each class is clustered separately, and only the clients that are consistently identified as benign across all classes are retained for global aggregation-this enforces cross-class consistency and is referred to as the intersection strategy.

To isolate its effect, we implemented a simplified version where we retain clients that are identified as benign in any one class. In this version, a client only needs to appear in a benign cluster for a single class to be included in aggregation, regardless of its behavior in other classes. Table 4 presents the results of this comparison on the FashionMNIST dataset under a multi-label label-flipping attack with 40% malicious clients.

The results show that removing the intersection step leads to a notable drop in both accuracy and benign client identification precision. Without the intersection, clients with inconsistent behavior across classes may still be retained, increasing the likelihood of poisoned updates contaminating the aggregation.

In contrast, the intersection enforces a stricter filtering criterion: only clients that are consistently clustered with benign peers in every class are trusted. This improves the resilience of our method under multi-label attacks, where a client may behave normally in one class but maliciously in others.

Computational complexity and scalability analysis

Although our method is effective in filtering out malicious clients, it incurs additional computational overhead due to the clustering operations performed on the server.

Specifically, for each communication round, the server performs K-means clustering with Gap Statistic analysis on the output-layer gradients submitted by all clients, separately for each class. If there are C classes and m clients, the server conducts C independent clustering processes per round. The computational complexity of K-means is approximately O(tkmn), where t is the number of iterations, k is the number of clusters, and n is the dimension of the gradient vector. Since the server only clusters output-layer gradients (which are low-dimensional), and the number of classes C is fixed and typically small (e.g., 10), the total cost remains acceptable for moderate-scale federated learning deployments.

Importantly, our method does not introduce any additional communication cost, as all clustering operations are conducted on the server side using the gradients that are already transmitted during the standard FL training process. No extra messages or gradient dimensions are exchanged between the server and clients.

Therefore, while the method introduces some time overhead due to per-class clustering and intersection operations, it remains communication-efficient and scalable for practical federated learning applications.

Limitations

While the proposed method demonstrates strong robustness against multi-label label-flipping attacks across different datasets and attacker ratios, several important limitations should be acknowledged.

First, the method assumes that all clients use the same model architecture, i.e., model homogeneity. This is a common assumption in existing federated learning defense studies, as it simplifies gradient representation and comparison. However, real-world FL systems often involve model heterogeneity due to varying hardware capacities or application-specific model requirements. Extending our method to heterogeneous model settings will require more general gradient representations or alternative similarity metrics to enable effective comparison across different model structures.

Second, the method assumes that client data is independently and identically distributed (IID). However, data heterogeneity (Non-IID data) is the norm in practical federated learning. Clients may possess data from only a subset of classes, or their data distributions may differ significantly in feature space. In such cases, even benign clients may exhibit divergent gradient behaviors, reducing the effectiveness of clustering-based defense methods. Future work may consider incorporating more robust feature selection strategies or leveraging multi-round history to improve modeling of benign behavior under Non-IID settings.

Third, the method is designed under the assumption that the proportion of malicious clients does not exceed 40% of the total participants. When the attacker ratio exceeds 50%, the clustering mechanism may be dominated by adversarial updates, leading to two problematic outcomes: (1) benign clients form small or sparse clusters and are mistakenly excluded; (2) attackers form large, dense clusters that are erroneously retained. These scenarios violate the core assumption that benign updates are more consistent and compact than malicious ones.

Therefore, while our method performs well under moderate adversarial intensity, IID data, and homogeneous models, its effectiveness may degrade in more extreme or realistic FL environments.