Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class

preprint2022arXivOpen access

Abstract

In this paper, we construct neural networks with ReLU, sine and 2x2^x as activation functions. For general continuous ff defined on [0,1]d[0,1]^d with continuity modulus ωf(⋅)ω_f(\cdot), we construct ReLU-sine-2x2^x networks that enjoy an approximation rate O(ωf(d)⋅2−M+ωf(dN))\mathcal{O}(ω_f(\sqrt{d})\cdot2^{-M}+ω_{f}\left(\frac{\sqrt{d}}{N}\right)), where M,N∈N+M,N\in \mathbb{N}^{+} denote the hyperparameters related to widths of the networks. As a consequence, we can construct ReLU-sine-2x2^x network with the depth 55 and width max⁡{⌈2d3/2(3με)1/α⌉,2⌈log⁡23μdα/22ε⌉+2}\max\left\{\left\lceil2d^{3/2}\left(\frac{3μ}ε\right)^{1/α}\right\rceil,2\left\lceil\log_2\frac{3μd^{α/2}}{2ε}\right\rceil+2\right\} that approximates f∈Hμα([0,1]d)f\in \mathcal{H}_μ^α([0,1]^d) within a given tolerance ε>0ε>0 measured in LpL^p norm p∈[1,∞)p\in[1,\infty), where Hμα([0,1]d)\mathcal{H}_μ^α([0,1]^d) denotes the Hölder continuous function class defined on [0,1]d[0,1]^d with order α∈(0,1]α\in (0,1] and constant μ>0μ> 0. Therefore, the ReLU-sine-2x2^x networks overcome the curse of dimensionality on Hμα([0,1]d)\mathcal{H}_μ^α([0,1]^d). In addition to its supper expressive power, functions implemented by ReLU-sine-2x2^x networks are (generalized) differentiable, enabling us to apply SGD to train.

Explore this paper’s authors, topics and related work.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

Reviews 0

Write a reviewWrite

No reviews yet.

Discussion 0

Add a commentWrite

No comments yet.