Gaussian Processes

Matthieu Bloch

Tuesday November 22, 2022

Today in ECE 6555

Announcements
- 4 lectures left (including today)
- One more homework (optional, I didn't realize how close the end of the semester was…)
- One final exam take-home
Last time
- Building up to RKHS
Today
- Representer theorem
Questions?

A space is separable if it contains a countable dense subset.
Separability is the key property to deal with sequences instead of collections
Any separable Hilbert space has an orthonormal basis.
Most useful Hilbert spaces are separable! We won't worry about non-separable Hilbert spaces
Key take away for separable Hilbert spaces
- \(x=\sum_{i=1}^\infty\dotp{x}{v_i}v_i\) is perfectly well defined for an orthonormal basis
- Parseval's identity tell us that \(\norm{x}^2=\sum_{k\geq 1}\abs{\dotp{x}{v_k}}^2\).
- It looks like we don't need to even worry about the nature of \(\calH\) and only think about coefficients \(\dotp{x}{v_i}\)
Any separable Hilbert space is isomorphic to \(\ell_2\)

A functional \(F:\calF\to\bbR\) associates real-valued number to an element of a Hilbert space \(\calF\)
Notation can be tricky when the Hilbert space is a space of functions: \(F\) can act on a function \(f\in\calF\)

A functional \(F:\calF\to\bbR\) is continuous at \(x\in\calF\) if \[ \forall \epsilon>0\exists\delta>0\textsf{ such that } \norm[\calF]{x-y}\leq \delta\Rightarrow \abs{F(x)-F(y)}\leq\epsilon \] If this is true for every \(x\in\calF\), \(F\) is continuous.
1. All norms are continuous functionals
2. \(F:\calF\to\bbR:x\mapsto\dotp{x}{c}\) for some \(c\in\calF\) is continuous

A functional \(F\) is linear if \(\forall a,b\in\bbR\) \(\forall x,y\in\calF\) \(F(ax+by) = aF(x)+bF(y)\).
Continuous linear functions are much more constrained than one would imagine
A linear functional \(F:\calF\to\bbR\) is bounded if there exists \(M>0\) such that \[ \forall x\in\calF\quad\abs{F(x)}\leq M\norm[\calF]{x} \]
A linear functional on a Hilbert space that is countinuous at \(0\) is bounded.
For a linear functional \(F:\calF\to\bbR\), the following statements are equivalent:
1. \(F\) is continuous at 0
2. \(F\) is continuous at some point \(x\in\calF\)
3. \(F\) is continuous everywhere on \(\calF\)
4. \(F\) is uniformly continuous everywhere on \(\calF\)

Let \(F:\calF\to\bbR\) be a linear functional on an \(n\)-dimensional Hilbert space \(\calF\).

Then there exists \(c\in\calF\) such that \(F(x)=\dotp{x}{c}\) for every \(x\in\calF\)
Linear functional over finite dimensional Hilbert spaces are continuous!
This is not true in infinite dimension
Let \(F:\calF\to\bbR\) be a continuous linear functional on a (possible infinite dimensional) separable Hilbert space \(\calF\).

Then there exists \(c\in\calF\) such that \(F(x)=\dotp{x}{c}\) for every \(x\in\calF\)
If \(\set{\psi_n}_{n\geq 1}\) is an orthobasis for \(\calF\), then we can construct \(c\) above as \[ c\eqdef \sum_{n=1}^\infty F(\psi_n)\psi_n \]

An RKHS is a Hilbert space \(\calH\) of real-valued functions \(f:\bbR^d\to\bbR\) in which the sampling operation \(\calS_\bftau:\calH\to\bbR:f\mapsto f(\bftau)\) is continuous for every \(\bftau\in\bbR^d\).

In other words, for each \(\bftau\in\bbR^d\), there exists \(k_\bftau\in\calH\) s.t. \[ f(\bftau) = {\dotp{f}{k_\bftau}}_\calH\text{ for all } f\in\calH \]
The kernel of an RKHS is \[ k:\bbR^d\times\bbR^d\to\bbR:(\bft,\bftau)\mapsto k_{\bftau}(\bft) \] where \(k_\bftau\) is the element of \(\calH\) that defines the sampling at \(\bftau\).
A (separable) Hilbert space with orthobasis \(\set{\psi_n}_{n\geq 1}\) is an RKHS iff \(\forall \bftau\in\bbR^d\) \(\sum_{n=1}^\infty\abs{\psi_{n}(\tau)}^2<\infty\)

If \(\calH\) is an RKHS, then \[ \min_{f\in\calF}\sum_{i=1}^n\abs{y_i-f(\vecx_i)}^2+\lambda\norm[\calH]{f} \] has solution \[ f = \sum_{i=1}^n\alpha_i k_{\vecx_i}\textsf{ with } \bfalpha = (\matK+\lambda\matI)^{-1}\vecy\qquad \matK=\mat{c}{k(\vecx_i,\vecx_j)}_{1\leq i,j\leq n} \]