- Benign Overfitting in Adversarially Robust Linear Classification (arXiv)
Author: Jinghui Chen, Yuan Cao, Quanquan Gu
Abstract: “Benign overfitting”, in which classifiers memorize noisy training data yet still achieve good generalization performance, has received a great deal of attention in the machine learning community. A body of work provides theoretical justifications in overparameterized linear regression, classification, and kernel methods to explain this surprising phenomenon. However, it is not clear whether benign overfitting still occurs in the presence of adversarial examples, i.e. examples with small intentional perturbations to fool the classifier. In this paper, we show that benign overfitting does indeed occur in adversarial training, a principled approach to defending against adversarial examples. In detail, we prove risk bounds for adversarially trained linear classifiers for mixtures of sub-Gaussian data under adversarial perturbations of ℓp. Our results show that under modest perturbations, an adversarially trained linear classifier can achieve near-optimal standard and adversarial risks despite overfitting on noisy training data. suggests. Numerical experiments validate theoretical findings.
2. Stupid crowds support benign overfitting (arXiv)
Authors: Niladri S. Chatterji, Philip M. Long
Abstract: We prove a lower bound on the excess risk of sparse interpolation procedures for linear regression with Gaussian data in an overparameterized domain. Apply this result to get the lower bound of the basis pursuit (minimum ℓ1-norm interpolation). This is sparse ground truth. Our analysis reveals the advantage of an effect similar to “wisdom of crowds”. However, here, the harm caused by fitting noise is ameliorated by distributing the noise in multiple directions. So the reduction in variance comes from stupid crowds.
