Universal approximators

A theoretical question that Daisy always liked to discuss was the ability of machine learning algorithms to approximate any function.

She called them “universal approximators.” To classify, they had to approximate any measurable or continuous function up to any desired accuracy.

Which of the following would you consider universal approximators?

1. Neural Networks

2. Support Vector Machines using a Gaussian kernel

3. Linear Regression

4. Boosted Decision Trees

Expand to see the answer

1, 2, 4

Except for linear regression, all the other three choices are universal approximators.

Thanks to the Universal approximation theorem, neural networks are a well-known member of this category. The theorem states that, when using non-linear activation functions, we can turn a two-layer neural network into a universal function approximator.

The authors of “A Note on the Universal Approximation Capability of Support Vector Machines” show that support vector machines with standard kernels can also approximate any measurable or continuous function.

Finally, the “Universal approximation” chapter of Machine Learning Refined covers trees as one of the standard types of universal approximations.

Recommended reading

• “A Note on the Universal Approximation Capability of Support Vector Machines” is a paper showing that SVMs with standard kernels can approximate any measurable or continuous function.
• Chapter 12.5 “Universal approximation” of Machine Learning Refined covers kernels, neural networks, and trees as universal approximations.