Linear discriminant regularized regression
Author: Xin Bing, Bingqing Li, Marten Wegkamp
Abstract: Linear discriminant analysis (LDA) is an important classification approach. The simple linear format is easy to interpret and can handle multiple classes of responses. It is closely related to other classical multivariate statistical methods such as Fisher's discriminant analysis, canonical correlation analysis, and linear regression. In this paper, we strengthen the relationship with multivariate response regression by characterizing an explicit relationship between the discriminant direction and the regression coefficient matrix. This critical characterization leads to a new regression-based multiclass classification procedure that is flexible enough to incorporate existing structured, regularized, and even nonparametric regression techniques. Moreover, our new formulation is generally easier to analyze compared to existing regression-based LDA procedures. In particular, we provide full theoretical guarantees for the use of the widely used ℓ1 regularization, which has not yet been well analyzed in the LDA context. Our theoretical findings are supported by extensive simulation studies and real data analysis.