Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Generalisation error in learning with random features and the hidden manifold model

Abstract : We study generalised linear regression and classi cation for a synthetically generated dataset encompassing di erent problems of interest, such as learning with random features, neural networks in the lazy training regime, and the hidden manifold model. We consider the high-dimensional regime and using the replica method from statistical physics, we provide a closed-form expression for the asymptotic general-isation performance in these problems, valid in both the under-and over-parametrised regimes and for a broad choice of generalised linear model loss functions. In particular, we show how to obtain analytically the so-called double descent behaviour for logistic regression with a peak at the interpolation threshold, we illustrate the superiority of orthogonal against random Gaussian projections in learning with random features, and discuss the role played by correlations in the data generated by the hidden manifold model. Beyond the interest in these particular problems, the theoretical formalism introduced in this manuscript provides a path to further extensions to more complex tasks.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

Cited literature [62 references]  Display  Hide  Download

https://hal-cea.archives-ouvertes.fr/cea-02529798
Contributor : Emmanuelle de Laborderie <>
Submitted on : Thursday, April 2, 2020 - 3:30:40 PM
Last modification on : Thursday, October 29, 2020 - 10:44:01 PM

File

2002.09339.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : cea-02529798, version 1
  • ARXIV : 2002.09339

Citation

Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mezard, Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. 2020. ⟨cea-02529798⟩

Share

Metrics

Record views

43

Files downloads

30