Representer theorem

• Hard deadline on October 16

• Minimum ${\Vert \cdot \Vert }_2$ solution
• Regularized least squares

• Extension to infinite dimension
• Representer theorem

We can adapt the regularization approach to the situation of a finite dimension Hilbert space $\mathcal{F}$ $$\underset{f\in \mathcal{F}}{min}\sum _{i=1}^n(y_i-f({\mathbf{x}}_i){)}^2+\lambda {\Vert f\Vert }_{\mathcal{F}}^2$$

• We are penalizing the norm of the entire function $f$

Using a basis for the space ${\{{\psi }_i\}}_{i=1}^d$ , and constructing $\mathbf{\Psi }$ as earlier, we obtain $$\underset{\mathit{\theta }\in {\mathbb{R}}^d}{min}{\Vert \mathbf{y}-\mathbf{\Psi }\mathit{\theta }\Vert }_2^2+\lambda {\mathit{\theta }}^{⊺}\mathbf{G}\mathit{\theta }$$ with $\mathbf{G}$ the Gram matrix for the basis.

If ${\mathbf{\Psi }}^{⊺}\mathbf{\Psi }+\lambda \mathbf{G}$ is invertible, we find the solution as $${\mathit{\theta }}^{\ast }=({\mathbf{\Psi }}^{⊺}\mathbf{\Psi }+\lambda \mathbf{G}{)}^{-1}{\mathbf{\Psi }}^{⊺}\mathbf{y}$$ and we can reconstruct the function as $f(\mathbf{x})=\sum _{i=1}^d{\theta }_i^{\ast }{\psi }_i(\mathbf{x})$.

If $\mathbf{G}$ is well conditioned, the resulting function is not too sensitive to the choice of the basis

In ${\mathbb{R}}^d$, the problem $\underset{\mathit{\theta }\in {\mathbb{R}}^d}{min}{\Vert \mathbf{y}-\mathbf{X}\mathit{\theta }\Vert }_2^2+\lambda {\Vert \mathit{\theta }\Vert }_2^2$ has a solution $${\mathit{\theta }}^{\ast }={\mathbf{X}}^{⊺}\mathit{\alpha }\mathsf{\text{ with }}\mathit{\alpha }=(\mathbf{X}{\mathbf{X}}^{⊺}+\lambda \mathbf{I}{)}^{-1}\mathbf{y}$$ $\mathbf{X}{\mathbf{X}}^{⊺}\in {\mathbb{R}}^{n\times n}$ is dimension independent! We will be able to extend this to infinite dimensional Hilbert spaces!

Let $\mathcal{F}$ be a Hilbert space and let $f\in \mathcal{F}$ be the function we are trying to estimate

• We will estimate $f\in \mathcal{F}$ using noisy observations ${⟨f,x_i⟩}_{}$ with ${\{x_i\}}_{i=1}^n$ elements of $\mathcal{F}$

• This is the equivalent of saying $\mathbf{y}=\mathbf{A}\mathbf{x}+\mathbf{n}$ in finite dimension

We will see that the situation of the representer theorem happens in Reproducing Kernel Hilber Space (RKHS)