# Reproducing Kernel Hilbert Spaces

Wednesday October 20, 2021

## Logistics

• Drop date: October 30, 2021

• More office hours

• Tuesdays 8am-9am on BlueJeans (https://bluejeans.com/205357142)
• Come prepared!
• Midterm 2: initially scheduled for Wednesday November 3, 2021

• Moved to Monday November 8, 2021 (gives you weekend to prepare)
• Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

## What’s on the agenda for today?

• Last time:
• Motivation for RKHS
• Functional on Hilbert spaces
• Today:
• Reproducing Kernel Hilbert Spaces
• Reading: Romberg, lecture notes 10

## Linear functionals on Hilbert spaces

• In what follows, $\calF$ is a Hilbert space with scalar field $\bbR$

• A functional $F:\calF\to\bbR$ associates real-valued number to an element of a Hilbert space $\calF$

• Notation can be tricky when the Hilbert space is a space of functions: $F$ can act on a function $f\in\calF$

• Examples

• A functional $F:\calF\to\bbR$ is continuous at $x\in\calF$ if $\forall \epsilon>0\exists\delta>0\textsf{ such that } \norm[\calF]{x-y}\leq \delta\Rightarrow \abs{F(x)-F(y)}\leq\epsilon\qquad \forall x,y\in\calF$ If this is true for every $x\in\calF$, $F$ is continuous.

• Warning: I wasn’t careful enough last time in the definition of continuity

1. All norms are continuous functionals
2. $F:\calF\to\bbR:x\mapsto\dotp{x}{c}$ for some $c\in\calF$ is continuous

## Continuous linear functionals on Hilbert spaces

• A functional $F$ is linear if $\forall a,b\in\bbR$ $\forall x,y\in\calF$ $F(ax+by) = aF(x)+bF(y)$.
• Continuous linear functions are much more constrained than one would imagine

• A linear functional $F:\calF\to\bbR$ is bounded if there exists $M>0$ such that $\forall x\in\calF\quad\abs{F(x)}\leq M\norm[\calF]{x}$
• A linear functional on a Hilbert space that is countinuous at $0$ is bounded.
• For a linear functional $F:\calF\to\bbR$, the following statements are equivalent:

1. $F$ is continuous at 0
2. $F$ is continuous at some point $x\in\calF$
3. $F$ is continuous everywhere on $\calF$
4. $F$ is uniformly continuous everywhere on $\calF$

## Representation of (continuous) linear functionals

• Let $F:\calF\to\bbR$ be a linear functional on an $n$-dimensional Hilbert space $\calF$.

Then there exists $c\in\calF$ such that $F(x)=\dotp{x}{c}$ for every $x\in\calF$

• Linear functional over finite dimensional Hilbert spaces are continuous!

• This is not true in infinite dimension

• Let $F:\calF\to\bbR$ be a continuous linear functional on a (possible infinite dimensional) separable Hilbert space $\calF$.

Then there exists $c\in\calF$ such that $F(x)=\dotp{x}{c}$ for every $x\in\calF$
• If $\set{\psi_n}_{n\geq 1}$ is an orthobasis for $\calH$, then we can construct $c$ above as $c\eqdef \sum_{n=1}^\infty F(\psi_n)\psi_n$

## Reproducing Kernel Hilbert Spaces

• An RKHS is a Hilbert space $\calH$ of real-valued functions $f:\bbR^d\to\bbR$ in which the sampling operation $\calS_\bftau:\calH\to\bbR:f\mapsto f(\bftau)$ is continuous for every $\bftau\in\bbR^d$.

In other words, for each $\bftau\in\bbR^d$, there exists $k_\bftau\in\calH$ s.t. $f(\bftau) = {\dotp{f}{k_\bftau}}_\calH\text{ for all } f\in\calH$

• The kernel of an RKHS is $k:\bbR^d\times\bbR^d\to\bbR:(\bft,\bftau)\mapsto k_{\bftau}(\bft)$ where $k_\bftau$ is the element of $\calH$ that defines the sampling at $\bftau$.

• A (separable) Hilbert space with orthobasis $\set{\psi_n}_{n\geq 1}$ is an RKHS iff $\forall \bftau\in\bbR^d$ $\sum_{n=1}^\infty\abs{\psi_{n}(\tau)}^2<\infty$

## RKHS and non orthogonal basis

• If $\set{\phi_n}_{n\geq 1}$ is a Riesz basis for $\calH$, we know that every $x\in\calH$ can be written $x = \sum_{n\geq 1}\alpha_n\phi_n\textsf{ with } \alpha_n\eqdef\dotp{x}{\smash{\widetilde{\phi}_n}}$ where $\set{\widetilde{\phi}_n}_{n\geq 1}$ is the dual basis.

• A (separable) Hilbert space with Riesz basis $\set{\phi_n}_{n\geq 1}$ is an RKHS with kernel $k(\bft,\bftau) =\sum_{n=1}^\infty \phi_n(\bftau)\widetilde{\phi}_n(\bft)$ iff $\forall \bftau\in\bbR^d$ $\sum_{n=1}^\infty\abs{\phi_{n}(\tau)}^2<\infty$