Nearly Optimal Learning Using Sparse Deep ReLU Networks in Regularized Empirical Risk Minimization With Lipschitz Loss.
Saved in:
| Title: | Nearly Optimal Learning Using Sparse Deep ReLU Networks in Regularized Empirical Risk Minimization With Lipschitz Loss. |
|---|---|
| Authors: | Huang, Ke (AUTHOR), Liu, Mingming (AUTHOR), Ma, Shujie (AUTHOR) |
| Source: | Neural Computation. Apr2025, Vol. 37 Issue 4, p815-870. 56p. |
| Subjects: | Feedforward neural networks, Sobolev spaces, Sample size (Statistics), Logarithms, Polynomials |
| Abstract: | We propose a sparse deep ReLU network (SDRN) estimator of the regression function obtained from regularized empirical risk minimization with a Lipschitz loss function. Our framework can be applied to a variety of regression and classification problems. We establish novel nonasymptotic excess risk bounds for our SDRN estimator when the regression function belongs to a Sobolev space with mixed derivatives. We obtain a new, nearly optimal, risk rate in the sense that the SDRN estimator can achieve nearly the same optimal minimax convergence rate as one-dimensional nonparametric regression with the dimension involved in a logarithm term only when the feature dimension is fixed. The estimator has a slightly slower rate when the dimension grows with the sample size. We show that the depth of the SDRN estimator grows with the sample size in logarithmic order, and the total number of nodes and weights grows in polynomial order of the sample size to have the nearly optimal risk rate. The proposed SDRN can go deeper with fewer parameters to well estimate the regression and overcome the overfitting problem encountered by conventional feedforward neural networks. [ABSTRACT FROM AUTHOR] |
| Copyright of Neural Computation is the property of MIT Press and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Psychology and Behavioral Sciences Collection |
| FullText | Links: – Type: pdflink Text: Availability: 0 |
|---|---|
| Header | DbId: pbh DbLabel: Psychology and Behavioral Sciences Collection An: 183763471 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Nearly Optimal Learning Using Sparse Deep ReLU Networks in Regularized Empirical Risk Minimization With Lipschitz Loss. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Huang%2C+Ke%22">Huang, Ke</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Liu%2C+Mingming%22">Liu, Mingming</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Ma%2C+Shujie%22">Ma, Shujie</searchLink> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Neural+Computation%22">Neural Computation</searchLink>. Apr2025, Vol. 37 Issue 4, p815-870. 56p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Feedforward+neural+networks%22">Feedforward neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Sobolev+spaces%22">Sobolev spaces</searchLink><br /><searchLink fieldCode="DE" term="%22Sample+size+%28Statistics%29%22">Sample size (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Logarithms%22">Logarithms</searchLink><br /><searchLink fieldCode="DE" term="%22Polynomials%22">Polynomials</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: We propose a sparse deep ReLU network (SDRN) estimator of the regression function obtained from regularized empirical risk minimization with a Lipschitz loss function. Our framework can be applied to a variety of regression and classification problems. We establish novel nonasymptotic excess risk bounds for our SDRN estimator when the regression function belongs to a Sobolev space with mixed derivatives. We obtain a new, nearly optimal, risk rate in the sense that the SDRN estimator can achieve nearly the same optimal minimax convergence rate as one-dimensional nonparametric regression with the dimension involved in a logarithm term only when the feature dimension is fixed. The estimator has a slightly slower rate when the dimension grows with the sample size. We show that the depth of the SDRN estimator grows with the sample size in logarithmic order, and the total number of nodes and weights grows in polynomial order of the sample size to have the nearly optimal risk rate. The proposed SDRN can go deeper with fewer parameters to well estimate the regression and overcome the overfitting problem encountered by conventional feedforward neural networks. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Neural Computation is the property of MIT Press and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=pbh&AN=183763471 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1162/neco_a_01742 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 56 StartPage: 815 Subjects: – SubjectFull: Feedforward neural networks Type: general – SubjectFull: Sobolev spaces Type: general – SubjectFull: Sample size (Statistics) Type: general – SubjectFull: Logarithms Type: general – SubjectFull: Polynomials Type: general Titles: – TitleFull: Nearly Optimal Learning Using Sparse Deep ReLU Networks in Regularized Empirical Risk Minimization With Lipschitz Loss. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Huang, Ke – PersonEntity: Name: NameFull: Liu, Mingming – PersonEntity: Name: NameFull: Ma, Shujie IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 04 Text: Apr2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 08997667 Numbering: – Type: volume Value: 37 – Type: issue Value: 4 Titles: – TitleFull: Neural Computation Type: main |
| ResultId | 1 |
