Reproducing Kernel Hilbert Spaces

Logistics

Drop date: October 30, 2021
More office hours
- Tuesdays 8am-9am on BlueJeans (https://bluejeans.com/205357142)
- Come prepared!
Midterm 2: initially scheduled for Wednesday November 3, 2021
- Moved to Monday November 8, 2021 (gives you weekend to prepare)
- Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

In what follows, $F$ is a Hilbert space with scalar field $R$
A functional $F : F \to R$ associates real-valued number to an element of a Hilbert space $F$
Notation can be tricky when the Hilbert space is a space of functions: $F$ can act on a function $f \in F$
Examples
A functional $F : F \to R$ is continuous at $x \in F$ if $\forall ϵ > 0 \exists δ > 0 such that {‖ x - y ‖}_{F} \leq δ \Rightarrow | F (x) - F (y) | \leq ϵ \forall x, y \in F$ If this is true for every $x \in F$ , $F$ is continuous.
Warning: I wasn’t careful enough last time in the definition of continuity
1. All norms are continuous functionals
2. $F : F \to R : x \mapsto {⟨ x, c ⟩}_{}$ for some $c \in F$ is continuous

A functional $F$ is linear if $\forall a, b \in R$ $\forall x, y \in F$ $F (a x + b y) = a F (x) + b F (y)$ .
Continuous linear functions are much more constrained than one would imagine
A linear functional $F : F \to R$ is bounded if there exists $M > 0$ such that $\forall x \in F | F (x) | \leq M {‖ x ‖}_{F}$
A linear functional on a Hilbert space that is countinuous at $0$ is bounded.
For a linear functional $F : F \to R$ , the following statements are equivalent:
1. $F$ is continuous at 0
2. $F$ is continuous at some point $x \in F$
3. $F$ is continuous everywhere on $F$
4. $F$ is uniformly continuous everywhere on $F$

Let $F : F \to R$ be a linear functional on an $n$ -dimensional Hilbert space $F$ .

Then there exists $c \in F$ such that $F (x) = {⟨ x, c ⟩}_{}$ for every $x \in F$
Linear functional over finite dimensional Hilbert spaces are continuous!
This is not true in infinite dimension
Let $F : F \to R$ be a continuous linear functional on a (possible infinite dimensional) separable Hilbert space $F$ .
Then there exists $c \in F$ such that $F (x) = {⟨ x, c ⟩}_{}$ for every $x \in F$
If ${ψ_{n}}_{n \geq 1}$ is an orthobasis for $H$ , then we can construct $c$ above as $c ≜ \sum_{n = 1}^{\infty} F (ψ_{n}) ψ_{n}$

An RKHS is a Hilbert space $H$ of real-valued functions $f : R^{d} \to R$ in which the sampling operation $S_{τ} : H \to R : f \mapsto f (τ)$ is continuous for every $τ \in R^{d}$ .

In other words, for each $τ \in R^{d}$ , there exists $k_{τ} \in H$ s.t. $f (τ) = {⟨ f, k_{τ} ⟩}_{}_{H} for all f \in H$
The kernel of an RKHS is $k : R^{d} \times R^{d} \to R : (t, τ) \mapsto k_{τ} (t)$ where $k_{τ}$ is the element of $H$ that defines the sampling at $τ$ .
A (separable) Hilbert space with orthobasis ${ψ_{n}}_{n \geq 1}$ is an RKHS iff $\forall τ \in R^{d}$ $\sum_{n = 1}^{\infty} {| ψ_{n} (τ) |}^{2} < \infty$

If ${ϕ_{n}}_{n \geq 1}$ is a Riesz basis for $H$ , we know that every $x \in H$ can be written $x = \sum_{n \geq 1} α_{n} ϕ_{n} with α_{n} ≜ {⟨ x, {\tilde{ϕ}}_{n} ⟩}_{}$ where ${{\tilde{ϕ}}_{n}}_{n \geq 1}$ is the dual basis.
A (separable) Hilbert space with Riesz basis ${ϕ_{n}}_{n \geq 1}$ is an RKHS with kernel $k (t, τ) = \sum_{n = 1}^{\infty} ϕ_{n} (τ) {\tilde{ϕ}}_{n} (t)$ iff $\forall τ \in R^{d}$ $\sum_{n = 1}^{\infty} {| ϕ_{n} (τ) |}^{2} < \infty$