Pairwise statistical comparison of multiple algorithms

In machine learning, it is often necessary to statistically compare the overall performance of two algorithms (e.g., a proposed algorithm and their respective baselines compared to each other) based on multiple benchmark datasets. In this case, the algorithm we propose is usually called a control algorithm. However, in some cases it is also important to perform pairwise statistical comparisons of multiple algorithms without using a control algorithm.

To perform pairwise statistical comparisons, a research team led by Min-Ling ZHANG published new research results on December 15, 2025. Frontiers of computer science Co-published by Higher Education Press and Springer Nature.

The researchers proposed that average rank-based strategies (such as a combination of the Friedman test and the Nemeni posttest) can yield test results that are inconsistent with common sense. Additionally, we designed a new presentation plot to report the results of pairwise statistical comparisons.

In their study, they analyze that critical difference (CD) values ​​are often greater than 1 in daily machine learning research, and that combining the Friedman test and the Nemeny posttest usually yields test results that contradict common sense. They recommend strategies that are not based on average rankings for statistical comparisons (such as the Wilcoxon signed rank test). To present the experimental results of pairwise statistical comparisons, they further designed a new presentation plot based on regular polygons. Specifically, each vertex corresponds to one algorithm. If one algorithm achieves statistically better performance than another, use the arrow to connect from the better algorithm to the other. Otherwise, they are connected by dotted lines.

Future work can focus on applying the proposed presentation plot to more pairwise statistical comparisons in related studies.