Stone-Weierstrass's Approximation Theorem

Contents

Weierstrass’ Approximation Theorem is a fundamental result in mathematical analysis, named after the German mathematician Karl Weierstrass. The theorem, first proven by Weierstrass in 1885, states that every continuous function defined on a closed interval can be uniformly approximated by polynomials.

Theorem (Weierstrass approximation theorem)
Suppose

f(x)

is a continuous real-valued function defined on the real interval

[a, b]

. For every

\varepsilon > 0

, there exists a polynomial

P(x)

such that for all

x \in [a, b]

, we have:

\begin{equation*}|f(x) - p(x)| < \varepsilon,\end{equation*}

or equivalently, the supremum norm

\begin{equation*}\| f - p \| = \max _{0 \leq x \leq 1} |f(x) - p(x) | < \varepsilon.\end{equation*}

In fact, the approximating polynomials can be defined explicitly

\begin{equation*}B_n(p)=\sum_{k=0}^n f\left(\frac{k}{n}\right) C_n^k p^k q^{n-k},\end{equation*}

which are called Bernstein polynomials after mathematician Sergei Natanovich Bernstein.

The video shows how Bernstein polynomials actually approximate continuous function

f(x)

Motivation

The key motivation comes from a fundamental question in mathematics: Can we approximate complicated continuous functions using simpler functions? Specifically, can we get arbitrarily close(by choosing

\varepsilon

) to any continuous function using just polynomials?

This question is deeply practical. In real-world applications, we often encounter complex continuous functions that are difficult to work with directly. For example:

Functions defined by experimental data
Functions without nice analytical forms
Functions that are hard to integrate or differentiate

If we could approximate these using polynomials, which are much easier to manipulate mathematically, we'd have a powerful tool. We could:

Calculate approximate integrals
Find approximate derivatives
Evaluate the function numerically with good precision
Store the function in a computer using finite information (polynomial coefficients)

Weierstrass answered this question definitively: Yes, we can approximate any continuous function on a closed interval

[a,b]

with polynomials to any desired degree of accuracy. The

\varepsilon

can be made as small as we want by using a polynomial of sufficiently high degree.

This is quite remarkable when you think about it. Functions can be wildly complicated - they could oscillate infinitely many times, have sharp turns, or be defined by some complex physical process. Yet simple polynomials, which are just sums of powers of

x

, can capture all this complexity arbitrarily well.

The theorem also connects to broader ideas in mathematics:

It shows that polynomials are "dense" in the space of continuous functions
It's a foundational result in approximation theory
Evaluate the function numerically with good precision
It justifies the use of polynomial interpolation in numerical methods

This is why the theorem is considered one of the cornerstones of analysis - it bridges the gap between the theoretical world of continuous functions and the practical world of polynomial approximations.

Proof of Weierstrass's Approximation Theorem on $[0,1]$

We provide the elegant and simple Bernstein proof of the Weierstrass approximation theorem, which provides the explicit formula for approximating polynomials.

Definition of Bernstein Polynomials

X_1, \ldots, X_n

is a sequence of independent Bernoulli random variables such that

\forall i \in \{1, \dots, n\}

\begin{equation*}X_i= \begin{cases} 1, & \text{with probability } p, \\ 0, & \text{with probability } 1-p .\end{cases}\end{equation*}

and consider the sum

S_n=X_1+\cdots+X_n

. Let's define the Bernstein polynomial

\begin{equation*}B_n(p) := \mathrm{E} f\left(\frac{S_n}{n}\right) = \sum_{k=0}^n f\left(\frac{k}{n}\right) C_n^k p^k (1-p)^{n-k}\end{equation*}

Properties of Sum of Bernoulli random variables

1. Total probability (normalization):

\begin{equation*}\sum_{k=0}^n C_n^k p^k (1-p)^{n-k} = 1 ,\end{equation*}

2. Chebyshev's inequality:

\begin{equation*}P \left\{ | \frac{S_n}{n} - p | > \delta \right\} \leq \frac{Var(\frac{S_n}{n})}{\delta^2} = \frac{p(1-p)}{n \delta^2}\end{equation*}

Approximation with Bernstein Polynomials

Since the continuous function

f=f(p)

on closed interval

[0,1]

is uniformly continuous, i.e. for every

\varepsilon>0

we can find

\delta>0

such that

|f(x)-f(y)| \leq \varepsilon

whenever

|x-y| \leq \delta

.
It is also known that the continuous function is bounded on closed interval, i.e.

|f(x)| \leq

M < \infty

. Let's consider difference

\forall p \in [0,1]

\begin{equation*}| f(p) - B_n(p) | = | \sum_{k=0}^n\left[f(p)-f\left(\frac{k}{n}\right)\right] C_n^k p^k q^{n-k} | \leq\end{equation*}

Let's split the above sum into two parts:

1
$\{k: \: |\frac{k}{n}-p| \leq \delta\}$ , such sum can be estimated using the uniform continuity property
2
$\{k: \: |\frac{k}{n}-p| > \delta\}$ , such sum can be estimated with probability $P( | \frac{S_n}{n} - p | > \delta )$

So we get

\begin{align*}\leq & \sum_{\{k:|(k / n)-p| \leq \delta\}}\left|f(p)-f\left(\frac{k}{n}\right)\right| C_n^k p^k q^{n-k} + \sum_{\{k:|(k / n)-p|>\delta\}}\left|f(p)-f\left(\frac{k}{n}\right)\right| C_n^k p^k q^{n-k} \\& \leq \varepsilon + 2 M \sum_{\{k:|(k ; n)-p|>\delta\}} C_n^k p^k q^{n-k} \\& = \varepsilon + 2M \: P\left\{ | \frac{S_n}{n} - p | > \delta \right\} \\& \leq \varepsilon+\frac{2 M}{n \delta^2}=\varepsilon+\frac{2 M}{n \delta^2} .\end{align*}

Hence

\begin{equation*}\lim _{n \rightarrow \infty} \max _{0 \leq p \leq 1}\left|f(p)-B_n(p)\right|=0,\end{equation*}

which is the conclusion of Weierstrass's theorem.

\Box

References

[Shiryaev]
Shiryaev, Albert N. Probability-2. Springer, 2010.

Stone-Weierstrass's Approximation Theorem

Proof of Weierstrass's Approximation Theorem on [0,1][0,1][0,1]

Motivation

Motivation

Proof of Weierstrass's Approximation Theorem on [0,1][0,1][0,1]

Proof of Weierstrass's Approximation Theorem on [0,1][0,1][0,1]

Definition of Bernstein Polynomials

Properties of Sum of Bernoulli random variables

Approximation with Bernstein Polynomials

References

References

Proof of Weierstrass's Approximation Theorem on $[0,1]$

Proof of Weierstrass's Approximation Theorem on $[0,1]$