Reproducing Kernel Hilbert Spaces

Logistics

• Drop date: October 30, 2021

• My office hours tomorrow

  • Tuesdays 8am-9am on BlueJeans (https://bluejeans.com/205357142)
  • Come prepared!

• Midterm 2:

  • Moved to Monday November 8, 2021 (gives you weekend to prepare)
  • Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

What’s on the agenda for today?

• Last time:
  • Functional on Hilbert spaces
• Today:
  • Reproducing Kernel Hilbert Spaces
• Reading: Romberg, lecture notes 10/11

Last time: Riesz representation theorem

• Let $F:\mathcal{F}\to \mathbb{R}$ be a continuous linear functional on a (possible infinite dimensional) separable Hilbert space $\mathcal{F}$.

  Then there exists $c\in \mathcal{F}$ such that $F(x)={⟨x,c⟩}_{}$ for every $x\in \mathcal{F}$

• If ${\{{\psi }_n\}}_{n\ge 1}$ is an orthobasis for $\mathcal{H}$, then we can construct $c$ above as $$c\triangleq \sum _{n=1}^{\mathrm{\infty }}F({\psi }_n){\psi }_n$$

Reproducing Kernel Hilbert Spaces

• An RKHS is a Hilbert space $\mathcal{H}$ of real-valued functions $f:{\mathbb{R}}^d\to \mathbb{R}$ in which the sampling operation ${\mathcal{S}}_{\mathit{\tau }}:\mathcal{H}\to \mathbb{R}:f\mapsto f(\mathit{\tau })$ is continuous for every $\mathit{\tau }\in {\mathbb{R}}^d$.

  In other words, for each $\mathit{\tau }\in {\mathbb{R}}^d$, there exists $k_{\mathit{\tau }}\in \mathcal{H}$ s.t. $$f(\mathit{\tau })={{⟨f,k_{\mathit{\tau }}⟩}_{}}_{\mathcal{H}}\text{ for all }f\in \mathcal{H}$$

• The kernel of an RKHS is $$k:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}:(\mathbf{t},\mathit{\tau })\mapsto k_{\mathit{\tau }}(\mathbf{t})$$ where $k_{\mathit{\tau }}$ is the element of $\mathcal{H}$ that defines the sampling at $\mathit{\tau }$.

• A (separable) Hilbert space with orthobasis ${\{{\psi }_n\}}_{n\ge 1}$ is an RKHS with kernel $k(\mathbf{t},\mathit{\tau })=\sum _{n=1}^{\mathrm{\infty }}{\psi }_n(\mathit{\tau }){\psi }_n(\mathbf{t})$ iff $\mathrm{\forall }\mathit{\tau }\in {\mathbb{R}}^d$ $\sum _{n=1}^{\mathrm{\infty }}{\left|{\psi }_n(\tau )\right|}^2<\mathrm{\infty }$

RKHS and non orthogonal basis

• If ${\{{\varphi }_n\}}_{n\ge 1}$ is a Riesz basis for $\mathcal{H}$, we know that every $x\in \mathcal{H}$ can be written $$x=\sum _{n\ge 1}{\alpha }_n{\varphi }_n\mathsf{\text{ with }}{\alpha }_n\triangleq {⟨x,{\tilde{\varphi }}_n⟩}_{}$$ where ${\{{\tilde{\varphi }}_n\}}_{n\ge 1}$ is the dual basis.

• A (separable) Hilbert space with Riesz basis ${\{{\varphi }_n\}}_{n\ge 1}$ is an RKHS with kernel $$k(\mathbf{t},\mathit{\tau })=\sum _{n=1}^{\mathrm{\infty }}{\varphi }_n(\mathit{\tau }){\tilde{\varphi }}_n(\mathbf{t})$$ iff $\mathrm{\forall }\mathit{\tau }\in {\mathbb{R}}^d$ $\sum _{n=1}^{\mathrm{\infty }}{\left|{\varphi }_n(\tau )\right|}^2<\mathrm{\infty }$

Examples

• Finite dimensional Hilbert space

• Space of $L$th order polynomial splines on the real line

• Remark

  • RKHS are more easily characterized by their kernel
  • Often, we try to avoid an explicit description of the the elements in the space

Kernel regression

• Regression problem: given $n$ pairs $({\mathbf{x}}_i,y_i)\in {\mathbb{R}}^d\times \mathbb{R}$, solve $$\underset{f\in \mathcal{F}}{min}\sum _{i=1}^n{\left|y_i-f({\mathbf{x}}_i)\right|}^2+\lambda {\Vert f\Vert }_{\mathcal{F}}^2$$

• If we restrict $\mathcal{F}$ to be an RKHS, the problem becomes $$\underset{f\in \mathcal{F}}{min}\sum _{i=1}^n{\left|y_i-{{⟨f,x_i⟩}_{}}_{\mathcal{F}}\right|}^2+\lambda {\Vert f\Vert }_{\mathcal{F}}^2$$

  where $x_i\triangleq k_{{\mathbf{x}}_i}$ provides the mapping between ${\mathbb{R}}^d$ and $\mathcal{F}$ $$x_i:{\mathbf{R}}^d\to \mathbb{R}:\mathbf{t}\mapsto k_{{\mathbf{x}}_i}(\mathbf{t})=k({\mathbf{x}}_i,\mathbf{t})$$

• The solution is given by $$\hat{f}=\sum _{i=1}^n{\hat{\alpha }}_i{x}_i\mathsf{\text{ with }}\hat{\mathit{\alpha }}\triangleq (\mathbf{K}+\lambda \mathbf{I}{)}^{-1}\mathbf{y}$$ and $\mathbf{K}\triangleq [K_{i,j}{]}_{1\le i,j\le n}$ with $K_{i,j}={⟨x_i,x_j⟩}_{}$

Kernel regression

• Kernel magic
  1. $K_{ij}={⟨x_i,x_j⟩}_{}={⟨k_{{\mathbf{x}}_i},k_{{\mathbf{x}}_j}⟩}_{}=k_{{\mathbf{x}}_i}({\mathbf{x}}_j)=k({\mathbf{x}}_i,{\mathbf{x}}_j)$
  2. $\hat{f}(\mathbf{x})={⟨\hat{f},k_{\mathbf{x}}⟩}_{}=\sum _{i=1}^n\widehat{{\alpha }_i}k({\mathbf{x}}_i,\mathbf{x})$
• Remarks
  • We solved an infinite dimensional problem using an $n\times n$ system of equations and linear algebra
  • Our solution and the evaluation only depend on the kernel; we never need to work directly in $\mathcal{F}$
• Question: can we skip $\mathcal{F}$ entirely? how do we find “good” kernels?

Aronszjan’s theorem

• An inner product kernel is a mapping $k:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}$ for which there exists a Hilbert space $\mathcal{H}$ and a mapping $\mathrm{\Phi }:{\mathbb{R}}^d\to \mathcal{H}$ such that $$\mathrm{\forall }\mathbf{u},\mathbf{v}\in {\mathbb{R}}^{d}k(\mathbf{u},\mathbf{v})=⟨\mathrm{\Phi }(\mathbf{u}),\mathrm{\Phi }(\mathbf{v}){⟩}_{\mathcal{H}}$$

• A function $k:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}$ is a positive semidefinite kernel if

  • $k$ is symmetric, i.e., $k(\mathbf{u},\mathbf{v})=k(\mathbf{v},\mathbf{u})$
  • for all $\{{\mathbf{x}}_i{\}}_{i=1}^N$, the Gram matrix $\mathbf{K}$ is positive semidefinite, i.e., $${\mathbf{x}}^{⊺}\mathbf{K}\mathbf{x}\ge 0\text{ with }\mathbf{K}=[K_{i,j}]\text{ and }{K}_{i,j}\triangleq k({\mathbf{x}}_i,{\mathbf{x}}_j)$$

• A function $k:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}$ is an inner product kernel if and only if $k$ is a positive semidefinite kernel.

Examples

• Regression using linear and quadratic functions in ${\mathbb{R}}^d$
  
• Regression using Radial Basis Functions
• Examples of kernels
  • Homogeneous polynomial kernel: $k(\mathbf{u},\mathbf{v})=({\mathbf{u}}^{⊺}\mathbf{v}{)}^m$ with $m\in {\mathbb{N}}^{\ast }$
  • Inhomogenous polynomial kernel: $k(\mathbf{u},\mathbf{v})=({\mathbf{u}}^{⊺}\mathbf{v}+c{)}^m$ with $c>0$, $m\in {\mathbb{N}}^{\ast }$
  • Radial basis function (RBF) kernel: $k(\mathbf{u},\mathbf{v})=\exp (-\frac{{\Vert \mathbf{u}-\mathbf{v}\Vert }_{}^2}{2{\sigma }^2})$ with ${\sigma }^2>0$