Dynamics of Regularized Regression Part 2 (Machine Learning 2024) | Written by Monodeep Mukherjee | May 2024

Machine Learning


Monodeep Mukherjee
Photo by insung yuon on Unsplash

Linear discriminant regularized regression

Author: Xin Bing, Bingqing Li, Marten Wegkamp

Abstract: Linear discriminant analysis (LDA) is an important classification approach. The simple linear format is easy to interpret and can handle multiple classes of responses. It is closely related to other classical multivariate statistical methods such as Fisher's discriminant analysis, canonical correlation analysis, and linear regression. In this paper, we strengthen the relationship with multivariate response regression by characterizing an explicit relationship between the discriminant direction and the regression coefficient matrix. This critical characterization leads to a new regression-based multiclass classification procedure that is flexible enough to incorporate existing structured, regularized, and even nonparametric regression techniques. Moreover, our new formulation is generally easier to analyze compared to existing regression-based LDA procedures. In particular, we provide full theoretical guarantees for the use of the widely used ℓ1 regularization, which has not yet been well analyzed in the LDA context. Our theoretical findings are supported by extensive simulation studies and real data analysis.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *