Gaussian Processes

Today in ECE 6555

• Announcements
  • 5 lectures left (including today)
  • Class on Tuesday November 22, 2022 will be online
• Last time
  • Gaussian processes - prediction
• Today
  • More Gaussian Processes
• Questions?

Last time: Schur complement

• Suppose that $$\mathbf{M}=\left[\begin{array}{cc}{\mathbf{M}}_{11}& {\mathbf{M}}_{12}\\ {\mathbf{M}}_{21}& {\mathbf{M}}_{22}\end{array}\right].$$ If ${\mathbf{M}}_{22}$ invertible, the Schur complement of $\mathbf{M}$ in ${\mathbf{M}}_{22}$ is ${\mathbf{S}}_{22}\triangleq {\mathbf{M}}_{11}-{\mathbf{M}}_{12}{\mathbf{M}}_{22}^{-1}{\mathbf{M}}_{21}$.

  If ${\mathbf{M}}_{11}$ invertible, the Schur complement of $\mathbf{M}$ in ${\mathbf{M}}_{11}$ is ${\mathbf{S}}_{11}\triangleq {\mathbf{M}}_{22}-{\mathbf{M}}_{21}{\mathbf{M}}_{11}^{-1}{\mathbf{M}}_{12}$.

• $${\mathbf{M}}^{-1}=\left[\begin{array}{cc}{\mathbf{S}}_{22}^{-1}& -{\mathbf{S}}_{22}^{-1}{\mathbf{M}}_{12}{\mathbf{M}}_{22}^{-1}\\ -{\mathbf{M}}_{22}^{-1}{\mathbf{M}}_{21}{\mathbf{S}}_{22}^{-1}& {\mathbf{M}}_{22}^{-1}+{\mathbf{M}}_{22}^{-1}{\mathbf{M}}_{21}{\mathbf{S}}_{22}^{-1}{\mathbf{M}}_{12}{\mathbf{M}}_{22}^{-1}\end{array}\right].$$ $${\mathbf{M}}^{-1}=\left[\begin{array}{cc}{\mathbf{M}}_{11}^{-1}+{\mathbf{M}}_{11}^{-1}{\mathbf{M}}_{12}{\mathbf{S}}_{11}^{-1}{\mathbf{M}}_{21}{\mathbf{M}}_{11}^{-1}& -{\mathbf{M}}_{11}^{-1}{\mathbf{M}}_{12}{\mathbf{S}}_{11}^{-1}\\ -{\mathbf{S}}_{11}^{-1}{\mathbf{M}}_{21}{\mathbf{M}}_{11}^{-1}& {\mathbf{S}}_{11}^{-1}\end{array}\right].$$

Last time: Predicting unknown function values

• $n$ observations $X\triangleq ({\mathbf{x}}_i,f_i)$, $n^{\ast }$ test points $X^{\ast }\triangleq ({\mathbf{x}}_j^{\ast },f_j^{\ast })$
• The joint distribution is $$\left[\begin{array}{c}\mathbf{f}\\ {\mathbf{f}}^{\ast }\end{array}\right]\sim \mathcal{N}(\mathbf{0},\left[\begin{array}{cc}\mathbf{K}(X,X)& \mathbf{K}(X,X^{\ast })\\ \mathbf{K}(X^{\ast },X)& \mathbf{K}(X^{\ast }{X}^{\ast })\end{array}\right])$$
• Assume zero mean

• The distribution of ${\mathbf{f}}^{\ast }$ conditioned on $X$, $X^{\ast }$ and $\mathbf{f}$ is $$\mathcal{N}(\mathbf{K}(X^{\ast },X)\mathbf{K}(X,X{)}^{-1}\mathbf{f},\mathbf{K}(X^{\ast },X^{\ast })-\mathbf{K}(X^{\ast },X)\mathbf{K}(X,X{)}^{-1}\mathbf{K}(X,X^{\ast }))$$ Hence we can estimate ${\mathbf{f}}^{\ast }$ as $\mathbf{K}(X^{\ast },X)\mathbf{K}(X,X{)}^{-1}\mathbf{f}$ and we can quantify our uncertainty.

Predicting unknown function values with noise

• Often, we only observe $y=f(x)+\epsilon$ with $\epsilon \sim \mathcal{N}(0,{\sigma }^2)$ (i.i.d. across different measurements)

  • We only observe $\mathbf{y}=\mathbf{f}+\mathit{\epsilon }$ with $\mathit{\epsilon }\sim \mathcal{N}(0,{\sigma }^2\mathbf{I})$ so that

• $$\left[\begin{array}{c}\mathbf{y}\\ {\mathbf{f}}^{\ast }\end{array}\right]\sim \mathcal{N}(\mathbf{0},\left[\begin{array}{cc}\mathbf{K}(X,X)+{\sigma }^2\mathbf{I}& \mathbf{K}(X,X^{\ast })\\ \mathbf{K}(X^{\ast },X)& \mathbf{K}(X^{\ast }{X}^{\ast })\end{array}\right])$$

• The distribution of ${\mathbf{f}}^{\ast }$ conditioned on $X$, $X^{\ast }$ and $\mathbf{y}$ is $$\mathcal{N}(\mathbf{K}(X^{\ast },X)(\mathbf{K}(X,X)+{\sigma }^2\mathbf{I}{)}^{-1}\mathbf{y},\mathbf{K}(X^{\ast },X^{\ast })-\mathbf{K}(X^{\ast },X)(\mathbf{K}(X,X)+{\sigma }^2\mathbf{I}{)}^{-1}\mathbf{K}(X,X^{\ast }))$$ Hence we can estimate ${\mathbf{f}}^{\ast }$ as $\mathbf{K}(X^{\ast },X)(\mathbf{K}(X,X)+{\sigma }^2\mathbf{I}{)}^{-1}\mathbf{y}$ and we can quantify our uncertainty.

• Small simplification: single test point $x^{\ast }$ $$f^{\ast }={\mathbf{k}}_{\ast }^T(\mathbf{K}+{\sigma }^2\mathbf{I}{)}^{-1}\mathbf{y}{\sigma }_{f^{\ast }}=k(x^{\ast },x^{\ast })-{\mathbf{k}}_{\ast }^T(\mathbf{K}+{\sigma }^2\mathbf{I}{)}^{-1}{\mathbf{k}}_{\ast }$$

  • Note that we can view this as a linear prediction $f^{\ast }=\sum _{i=1}^n{\alpha }_{i}k(x_i,x^{\ast })$

Remarks

• Non zero-mean Gaussian processes
  • Predict ${\mathbf{f}}^{\ast }$ as
  $$\mathbf{m}(X^{\ast })+\mathbf{K}(X^{\ast },X)(\mathbf{K}(X,X)+{\sigma }^2\mathbf{I}{)}^{-1}(\mathbf{y}-\mathbf{m}(X)$$
  • Variance is still $$\mathbf{K}(X^{\ast },X^{\ast })-\mathbf{K}(X^{\ast },X)(\mathbf{K}(X,X)+{\sigma }^2\mathbf{I}{)}^{-1}\mathbf{K}(X,X^{\ast })$$
• Do we need the probabilistic modeling?
  • Recall our two approaches to the Kalman filter
  • We can think of GP modeling completely deterministically using an appropriate space of functions $$\underset{f\in \mathcal{F}}{min}\sum _{i=1}^n{\left|y_i-f({\mathbf{x}}_i)\right|}^2+\lambda {\Vert f\Vert }_{\mathcal{H}}$$
  • What this is space of function? Reproducing Kernel Hilbert Space

Vector spaces

• A vector space $\mathcal{V}$ over a field $\mathbb{K}$ consists of a set $\mathcal{V}$ of vectors, a closed addition rule $+$ and a closed scalar multiplication $\cdot$ such that 8 axioms are satisfied:

  1. $\mathrm{\forall }x,y\in \mathcal{V}$ $x+y=y+x$ (commutativity)
  2. $\mathrm{\forall }x,y,z\in \mathcal{V}$ $x+(y+z)=(x+y)+z$ (associativity)
  3. $\mathrm{\exists }0\in \mathcal{V}$ such that $\mathrm{\forall }x\in \mathcal{V}$ $x+0=x$ (identity element)
  4. $\mathrm{\forall }x\in \mathcal{V}$ $\mathrm{\exists }y\in \mathcal{V}$ such that $x+y=0$ (inverse element)
  5. $\mathrm{\forall }x\in \mathcal{V}$ $1\cdot x=x$
  6. $\mathrm{\forall }\alpha ,\beta \in \mathbb{K}$ $\mathrm{\forall }x\in \mathcal{V}$ $\alpha \cdot (\beta \mathbf{x})=(\alpha \cdot \beta )\cdot \mathbf{x}$ (associativity)
  7. $\mathrm{\forall }\alpha ,\beta \in \mathbb{K}$ $\mathrm{\forall }x\in \mathcal{V}$ $(\alpha +\beta )x=\alpha x+\beta x$ (distributivity)
  8. $\mathrm{\forall }\alpha \in \mathbb{K}$ $\mathrm{\forall }x,y\in \mathcal{V}$ $\alpha (x+y)=\alpha x+\alpha y$ (distributivity)
• • $0\in \mathcal{V}$ is unique
  • Every $x\in \mathcal{V}$ has a unique inverse
  • $0\cdot x=0$
  • The inverse of $x\in \mathcal{V}$ is $(-1)\cdot x\triangleq -x$

Subspaces and span

• A subset $\mathcal{W}\ne \mathrm{\varnothing }$ of a vector space $\mathcal{V}$ is a vector subspace if $\mathrm{\forall }x,y\in \mathcal{W}\mathrm{\forall }\lambda ,\mu \in \mathbb{K}$ $\lambda x+\mu y\in \mathcal{W}$

• If $\mathcal{W}$ is a vector subspace of a vector space $\mathcal{V}$, $\mathcal{W}$ is a vector space.

• Let ${\{v_i\}}_{i=1}^n$ be a set of vectors in a vector space $\mathcal{V}$.

• For ${\{a_i\}}_{i=1}^n\in {\mathbb{K}}^n$, $\sum _{i=1}^n{a}_i{v}_i$ is called a linear combination of the vectors ${\{v_i\}}_{i=1}^n$.

• The span of the vectors ${\{v_i\}}_{i=1}^n$ is the set $$\text{span}({\{v_i\}}_{i=1}^n)\triangleq \{\sum _{i=1}^n{a}_i{v}_i:{\{a_i\}}_{i=1}^n\in {\mathbb{K}}^n\}$$

• The span of the vectors ${\{v_i\}}_{i=1}^n\in {\mathcal{V}}^n$ is a vector subspace of $\mathcal{V}$.

Linear independence

• Let ${\{v_i\}}_{i=1}^n$ be a set of vectors in a vector space $\mathcal{V}$

• ${\{v_i\}}_{i=1}^n$ is linearly independent (or the vectors ${\{v_i\}}_{i=1}^n$ are linearly independent ) if (and only if) $$\sum _{i=1}^n{a}_i{v}_i=0\Rightarrow \mathrm{\forall }i\in [1;n]a_i=0$$ Otherwise the set is (or the vectors are) linearly dependent.

• Any set of linearly dependent vectors contains a subset of linearly independent vectors with the same span.

Bases

• A basis of vector subspace $\mathcal{W}$ of a vector space $\mathcal{V}$ is a countable set of vectors $\mathcal{B}$ such that:

  1. $\text{span}(\mathcal{B})=\mathcal{W}$
  2. $\mathcal{B}$ is linearly independent

• If a non vector space $\mathcal{V}\ne \{0\}$ as a finite basis with $n\in {\mathbb{N}}^{\ast }$ elements, $n$ is called the dimension of $\mathcal{V}$, denoted $\dim \mathcal{V}$. If the basis has an infinite number of elements, the dimension is infinite

• Any two bases for the same finite dimensional vector space contain the same number of elements.

• You should be somewhat familiar with bases (at least in ${\mathbb{R}}^n$):
  • the representation of a vector on a basis is unique
  • every subspace has a basis
  • having a basis reduces the operations on vectors to operations on their components
• Things sort of work in infinite dimensions, but we have to be bit more careful

Norm and inner product

• The properties of vector space seen thus far provide an algebraic structure

• We are missing a topological structure to measure length and distance

• A norm on a vector space $\mathcal{V}$ over $\mathbb{R}$ is a function ${\Vert \cdot \Vert }_{}:\mathcal{V}\to \mathbb{R}$ that satisfies:

  • Positive definiteness: $\mathrm{\forall }x\in \mathcal{V}$ ${\Vert x\Vert }_{}\ge 0$ with equality iff $x=0$
  • Homogeneity: $\mathrm{\forall }x\in \mathcal{V}$ $\mathrm{\forall }\alpha \in \mathbb{R}$ ${\Vert \alpha x\Vert }_{}=\left|\alpha \right|{\Vert x\Vert }_{}$
  • Subadditivity: $\mathrm{\forall }x,y\in \mathcal{V}$ ${\Vert x+y\Vert }_{}\le {\Vert x\Vert }_{}+{\Vert y\Vert }_{}$

• ${\Vert x\Vert }_{}$ measures a length, ${\Vert x-y\Vert }_{}$ measures a distance

• In addition to a topological and algebraic strucure, what if we want to do geometry?

• An inner product space over $\mathbb{R}$ is a vector space $\mathcal{V}$ equipped with a positive definite symmetric bilinear form ${⟨\cdot ,\cdot ⟩}_{}:\mathcal{V}\times \mathcal{V}\to \mathbb{R}$ called an inner product

• An inner product space is also called a pre-Hilbert space

Induced norm

• In an inner product space, an inner product induces a norm ${\Vert x\Vert }_{}\triangleq \sqrt{{⟨x,x⟩}_{}}$

• A norm ${\Vert \cdot \Vert }_{}$ is induced by an inner product on $\mathcal{V}$ iff $\mathrm{\forall }x,y\in \mathcal{V}$ ${\Vert x\Vert }_{}^2+{\Vert y\Vert }_{}^2=\frac{1}{2}({\Vert x+y\Vert }_{}^2+{\Vert x-y\Vert }_{}^2)$ If this is the case, the inner product is given by the polarization identity $${⟨x,y⟩}_{}=\frac{1}{2}({\Vert x\Vert }_{}^2+{\Vert y\Vert }_{}^2-{\Vert x-y\Vert }_{}^2)$$

• Induced norm have some nice additional properties

• An induced norm satisfies $\mathrm{\forall }x,y\in \mathcal{V}$ ${\Vert x+y\Vert }_{}\le {\Vert x\Vert }_{}+{\Vert y\Vert }_{}$

• An inner product satisfies $\mathrm{\forall }x,y\in \mathcal{V}$ ${⟨x,y⟩}_{}^2\le {⟨x,x⟩}_{}{⟨y,y⟩}_{}$

Orthogonality

• In the following $\mathcal{V}$ is an inner product space with induced norm ${\Vert \cdot \Vert }_{}$

• The angle between two non-zero vectors $x,y\in \mathcal{V}$ is $$\cos \theta \triangleq \frac{{⟨x,y⟩}_{}}{{\Vert x\Vert }_{}{\Vert y\Vert }_{}}$$

• Two vectors $x,y\in \mathcal{V}$ are orthogonal if ${⟨x,y⟩}_{}=0$. We write $x\perp y$ for simplicity.

  A vector $x\in \mathcal{V}$ is orthogonal to a set $\mathcal{S}\subset \mathcal{V}$ if $\mathrm{\forall }s\in \mathcal{S}$ ${⟨x,s⟩}_{}=0$. We write $x\perp \mathcal{S}$ for simplicity.

• If $x\perp y$ then ${\Vert x+y\Vert }_{}^2={\Vert x\Vert }_{}^2+{\Vert y\Vert }_{}^2$

• Inner product spaces have almost all the properties of ${\mathbb{R}}^n$

Hilbert spaces

• In infinite dimensions, things are a little bit tricky. What does the following mean? $$x(t)=\sum _{n=1}^{\mathrm{\infty }}{\alpha }_n{\psi }_n(t)$$

• We need to define a notion of convergence, e.g., $$\underset{N\to \mathrm{\infty }}{lim}{\Vert x(t)-\sum _{n=1}^N{\alpha }_n{\psi }_n(t)\Vert }_{}=0$$

• Problems can still arise if "points are missing"; we avoid this by introducing the notion of completeness

• A inner product space $\mathcal{V}$ is complete if every Cauchy sequence converges, i.e., for every ${\{x_i\}}_{i\ge 1}$ in $\mathcal{V}$ $$\underset{min(m,n)\to \mathrm{\infty }}{lim}{\Vert x_m-x_n\Vert }_{}=0\Rightarrow \underset{n\to \mathrm{\infty }}{lim}{x}_n=x^{\ast }\in \mathcal{V}.$$

• We won't worry too much about proving that spaces are complete

• A complete normed vector space is a Banach space

Orthogonality principle

• Let $\mathcal{H}$ be a Hilbert space with induced norm ${⟨\cdot ,\cdot ⟩}_{}$ and induced norm ${\Vert \cdot \Vert }_{}$ ; let $\mathcal{T}$ be subspace of $\mathcal{H}$

• For $x\in \mathcal{H}$, what is the closest point of $\hat{x}\in \mathcal{T}$? How do we solve $\underset{y\in \mathcal{T}}{min}{\Vert x-y\Vert }_{}$?

• This problem has a unique solution given by the orthogonality principle

• Let $\mathcal{X}$ be a pre-Hilbert space, $\mathcal{T}$ be a subspace of $\mathcal{X}$, and $x\in \mathcal{X}$.

  If there exists a vector $m^{\ast }\in \mathcal{T}$ such that $\mathrm{\forall }m\in \mathcal{T}$ ${\Vert x-m^{\ast }\Vert }_{}\le {\Vert x-m\Vert }_{}$, then $m^{\ast }$ is unique.

  $m^{\ast }\in \mathcal{T}$ is a unique minimizer if and only if the error $x-m^{\ast }$ be orthogonal to $\mathcal{T}$.

• This doesn't say that $m^{\ast }$ exists!

• Let $\mathcal{H}$ be a Hilbert space, $\mathcal{T}$ be a closed subspace of $\mathcal{X}$, and $x\in \mathcal{X}$.

  There exists a unique vector $m^{\ast }\in \mathcal{T}$ such that $\mathrm{\forall }m\in \mathcal{T}$ ${\Vert x-m^{\ast }\Vert }_{}\le {\Vert x-m\Vert }_{}$.

  $m^{\ast }\in \mathcal{T}$ is a unique minimizer if and only if the error $x-m^{\ast }$ be orthogonal to $\mathcal{T}$

Orthobases

• A collection of vectors ${\{v_i\}}_{i=1}^n$ in a finite dimensional Hilbert space $\mathcal{H}$ is an orthobasis if 1) $\text{span}({\{v_i\}}_{i=1}^n)=\mathcal{H}$; 2) $\mathrm{\forall }i\ne j\in [1;n]v_i\perp v_j$; 3) $\mathrm{\forall }i\in [1;n]{\Vert v_i\Vert }_{}=1$.

• If the last condition is not met, this is just called an orthogonal basis

• Orthobases are useful because we can write $x=\sum _{i=1}^n{⟨x,v_i⟩}_{}{v}_i$ (what happens in a non-orthonormal basis?)

• We would like to extend this idea to infinite dimension and happily write $x=\sum _{i=1}^{\mathrm{\infty }}{⟨x,v_i⟩}_{}{v}_i$

  • We have to be a bit careful
  • With a little bit of machinery, this works: separability + completeness

Separable space

• A space is separable if it contains a countable dense subset.

• Separability is the key property to deal with sequences instead of collections

• Any separable Hilbert space has an orthonormal basis.

• Most useful Hilbert spaces are separable! We won't worry about non-separable Hilbert spaces

• Key take away for separable Hilbert spaces

  • $x=\sum _{i=1}^{\mathrm{\infty }}{⟨x,v_i⟩}_{}{v}_i$ is perfectly well defined for an orthonormal basis
  • Parseval's identity tell us that ${\Vert x\Vert }_{}^2=\sum _{k\ge 1}{\left|{⟨x,v_k⟩}_{}\right|}^2$.
  • It looks like we don't need to even worry about the nature of $\mathcal{H}$ and only think about coefficients ${⟨x,v_i⟩}_{}$

• Any separable Hilbert space is isomorphic to ${\ell }_2$

Linear functionals on Hilbert spaces

• In what follows, $\mathcal{F}$ is a Hilbert space with scalar field $\mathbb{R}$

• A functional $F:\mathcal{F}\to \mathbb{R}$ associates real-valued number to an element of a Hilbert space $\mathcal{F}$

• Notation can be tricky when the Hilbert space is a space of functions: $F$ can act on a function $f\in \mathcal{F}$

• A functional $F:\mathcal{F}\to \mathbb{R}$ is continuous at $x\in \mathcal{F}$ if $$\mathrm{\forall }ϵ>0\mathrm{\exists }\delta >0\mathsf{\text{ such that }}{\Vert x-y\Vert }_{\mathcal{F}}\le \delta \Rightarrow \left|F(x)-F(y)\right|\le ϵ$$ If this is true for every $x\in \mathcal{F}$, $F$ is continuous.

• 1. All norms are continuous functionals
  2. $F:\mathcal{F}\to \mathbb{R}:x\mapsto {⟨x,c⟩}_{}$ for some $c\in \mathcal{F}$ is continuous

Continuous linear functionals on Hilbert spaces

• A functional $F$ is linear if $\mathrm{\forall }a,b\in \mathbb{R}$ $\mathrm{\forall }x,y\in \mathcal{F}$ $F(ax+by)=aF(x)+bF(y)$.

• Continuous linear functions are much more constrained than one would imagine

• A linear functional $F:\mathcal{F}\to \mathbb{R}$ is bounded if there exists $M>0$ such that $$\mathrm{\forall }x\in \mathcal{F}\left|F(x)\right|\le M{\Vert x\Vert }_{\mathcal{F}}$$

• A linear functional on a Hilbert space that is countinuous at $0$ is bounded.

• For a linear functional $F:\mathcal{F}\to \mathbb{R}$, the following statements are equivalent:

  1. $F$ is continuous at 0
  2. $F$ is continuous at some point $x\in \mathcal{F}$
  3. $F$ is continuous everywhere on $\mathcal{F}$
  4. $F$ is uniformly continuous everywhere on $\mathcal{F}$

Representing (continuous) linear functionals

• Let $F:\mathcal{F}\to \mathbb{R}$ be a linear functional on an $n$-dimensional Hilbert space $\mathcal{F}$.

  Then there exists $c\in \mathcal{F}$ such that $F(x)={⟨x,c⟩}_{}$ for every $x\in \mathcal{F}$

• Linear functional over finite dimensional Hilbert spaces are continuous!

• This is not true in infinite dimension

• Let $F:\mathcal{F}\to \mathbb{R}$ be a continuous linear functional on a (possible infinite dimensional) separable Hilbert space $\mathcal{F}$.

  Then there exists $c\in \mathcal{F}$ such that $F(x)={⟨x,c⟩}_{}$ for every $x\in \mathcal{F}$

• If ${\{{\psi }_n\}}_{n\ge 1}$ is an orthobasis for $\mathcal{F}$, then we can construct $c$ above as $$c\triangleq \sum _{n=1}^{\mathrm{\infty }}F({\psi }_n){\psi }_n$$

Reproducing Kernel Hilbert Spaces

• An RKHS is a Hilbert space $\mathcal{H}$ of real-valued functions $f:{\mathbb{R}}^d\to \mathbb{R}$ in which the sampling operation ${\mathcal{S}}_{\mathit{\tau }}:\mathcal{H}\to \mathbb{R}:f\mapsto f(\mathit{\tau })$ is continuous for every $\mathit{\tau }\in {\mathbb{R}}^d$.

  In other words, for each $\mathit{\tau }\in {\mathbb{R}}^d$, there exists $k_{\mathit{\tau }}\in \mathcal{H}$ s.t. $$f(\mathit{\tau })={{⟨f,k_{\mathit{\tau }}⟩}_{}}_{\mathcal{H}}\text{ for all }f\in \mathcal{H}$$

• The kernel of an RKHS is $$k:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}:(\mathbf{t},\mathit{\tau })\mapsto k_{\mathit{\tau }}(\mathbf{t})$$ where $k_{\mathit{\tau }}$ is the element of $\mathcal{H}$ that defines the sampling at $\mathit{\tau }$.

• A (separable) Hilbert space with orthobasis ${\{{\psi }_n\}}_{n\ge 1}$ is an RKHS iff $\mathrm{\forall }\mathit{\tau }\in {\mathbb{R}}^d$ $\sum _{n=1}^{\mathrm{\infty }}{\left|{\psi }_n(\tau )\right|}^2<\mathrm{\infty }$

Representer theorem

• An RKHS is just the right space to solve our problem

• If $\mathcal{H}$ is an RKHS, then $$\underset{f\in \mathcal{F}}{min}\sum _{i=1}^n{\left|y_i-f({\mathbf{x}}_i)\right|}^2+\lambda {\Vert f\Vert }_{\mathcal{H}}$$ has solution $$f=\sum _{i=1}^n{\alpha }_i{k}_{{\mathbf{x}}_i}\mathsf{\text{ with }}\mathit{\alpha }=(\mathbf{K}+\lambda \mathbf{I}{)}^{-1}\mathbf{y}\mathbf{K}={\left[\begin{array}{c}k({\mathbf{x}}_i,{\mathbf{x}}_j)\end{array}\right]}_{1\le i,j\le n}$$