The role of regularization in classification of high-dimensional noisy Gaussian mixture

Francesca Mignacco (1) , Florent Krzakala (2) , Yue M. Lu (3) , Lenka Zdeborová (1)

1 IPHT - Institut de Physique Théorique - UMR CNRS 3681
2 Systèmes Désordonnés et Applications
3 Harvard University

We consider a high-dimensional mixture of two Gaussians in the noisy regime where even an oracle knowing the centers of the clusters misclassifies a small but finite fraction of the points. We provide a rigorous analysis of the generalization error of regularized convex classifiers, including ridge, hinge and logistic regression, in the high-dimensional limit where the number n of samples and their dimension d go to infinity while their ratio is fixed to $\alpha$ = $n/d$. We discuss surprising effects of the regularization that in some cases allows to reach the Bayes-optimal performances. We also illustrate the interpolation peak at low regularization, and analyze the role of the respective sizes of the two clusters.

Mots clés: Disordered Systems and Neural Networks; Statistics Theory; Machine Learning

Domaines: Physique [physics]

Francesca Mignacco, Florent Krzakala, Yue M. Lu, Lenka Zdeborová. The role of regularization in classification of high-dimensional noisy Gaussian mixture. International Conference on Machine Learning, Jun 2020, Virtual event, France. pp.6874-6883. ⟨cea-02529853⟩