Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in W^{2,\infty}((a,b)^d) can be approximated with arbitrary accuracy, measured in the W^{1,\infty}-norm, by a fixed-size neural network using the Elementary Universal Activation Function (EUAF). To extend this result to W^{s,\infty}((a,b)^d) for s\in\mathbb{N}, we introduce a smooth activation DUAF_{\infty} from the family of Differentiable Universal Activation Functions (DUAF_n). We prove that any function in W^{s,\infty}((a,b)^d) can be approximated with arbitrary accuracy in the W^{s-1,\infty}-norm by a fixed-size DUAF_{\infty}-activated network. We further construct sigmoidal variants \widetilde{DUAF}_n and show that, for every 1\leq s\leq n, fixed-size \widetilde{DUAF}_n-activated networks still approximate any f\in W^{s,\infty}((a,b)^d) with arbitrary accuracy in the W^{s-1,\infty}-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.