Optimal errors and phase transitions in high-dimensional generalized linear models

Jean Barbier (1, 2) , Florent Krzakala (2) , Nicolas Macris (3) , Léo Miolane (4) , Lenka Zdeborová (5)

1 ICTP - Abdus Salam International Center for Theoretical Physics
2 Systèmes Désordonnés et Applications
3 EPFL - Ecole Polytechnique Fédérale de Lausanne
4 DYOGENE - Dynamics of Geometric Networks
5 IPHT - Institut de Physique Théorique - UMR CNRS 3681

Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix israndom, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or “free entropy”) fromwhich we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the socalled replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalizedapproximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurposealgorithms

Domaines: Théorie de l'information et codage [math.IT]; Intelligence artificielle [cs.AI]; Apprentissage [cs.LG]; Physique mathématique [math-ph]

Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proceedings of the National Academy of Sciences of the United States of America, 2019, 116 (12), pp.5451-5460. ⟨10.1073/pnas.1802705116⟩. ⟨cea-01614258⟩