Dr. Matthieu R Bloch

Wednesday October 27, 2021

**Drop date:**October 30, 2021**My office hours on Tuesdays**- 8am-9am on BlueJeans (https://bluejeans.com/205357142)
- This was great!

**Midterm 2:**- Moved to
**Monday November 8, 2021**(gives you weekend to prepare) - Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on
**regression**

- Moved to

**Last time**:- RKHS

**Today:**- Final thoughts on RKHS
- Symmetric matrices: more linear algebra (
*objective:*further understand least-square problems)

**Reading:**Romberg, lecture notes 10/11/12

An

*inner product kernel*is a mapping \(k:\bbR^d\times\bbR^d\to\bbR\) for which there exists a Hilbert space \(\calH\) and a mapping \(\Phi:\bbR^d\to\calH\) such that \[\forall \bfu,\bfv\in\bbR^d\quad k(\bfu,\bfv)=\langle\Phi(\bfu),\Phi(\bfv)\rangle_\calH\]- A function \(k:\bbR^d\times\bbR^d\to\bbR\) is a
*positive semidefinite*kernel if- \(k\) is symmetric, i.e., \(k(\bfu,\bfv)=k(\bfv,\bfu)\)
- for all \(\{\bfx_i\}_{i=1}^N\), the
*Gram matrix*\(\bfK\) is positive semidefinite, i.e., \[\bfx^\intercal\bfK\bfx\geq 0\text{ with }\bfK=[K_{i,j}]\text{ and }K_{i,j}\eqdef k(\bfx_i,\bfx_j)\]

A function \(k:\bbR^d\times\bbR^d\to\bbR\) is an inner product kernel if and only if \(k\) is a positive semidefinite kernel.

- Regression using linear and quadratic functions in \(\bbR^d\)

- Regression using Radial Basis Functions
**Examples of kernels**- Homogeneous polynomial kernel: \(k(\bfu,\bfv) = (\bfu^\intercal\bfv)^m\) with \(m\in\bbN^*\)
- Inhomogenous polynomial kernel: \(k(\bfu,\bfv) = (\bfu^\intercal\bfv+c)^m\) with \(c>0\), \(m\in\bbN^*\)
- Radial basis function (RBF) kernel: \(k(\bfu,\bfv) = \exp\left(-\frac{\norm{\bfu-\bfv}^2}{2\sigma^2}\right)\) with \(\sigma^2>0\)

Least square problems involved the normal equations \(\bfX^\intercal\bfX \bftheta=\bfX^\intercal\bfy\)

This is a system of symmetric equations \(\bfA\bfx=\bfy\) with \(\bfA^\intercal=\bfA\)

- Ultimately we will talk about the non-symmetric/non square case

A real-valued matrix \(\bfA\) is symmetric if \(\bfA^\intercal=\bfA\) A complex-valued matrix \(\bfA\) is Hermitian if \(\bfA^\dagger=\bfA\) (also written \(\bfA^H=\bfA\))

Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an

*eigenvalue*associated to the*eigenvector*\(\bfv\).If \(\lambda\) is an eigenvalue, there are infinitely many eigenvectors associated to it

Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an

*eigenvalue*associated to the*eigenvector*\(\bfv\).

Consider the canonical basis \(\set{e_i}_{i=1}^n\) for \(\bbR^n\); every vector can be viewed as a vector of coefficients \(\set{\alpha_i}_{i=1}^n\), \[ \bfx = \sum_{i=1}^n \alpha_i e_i = \mat{cccc}{\alpha_1&\alpha_2&\cdots&\alpha_n}^\intercal \]

How do we find the representation of \(\bfx\) in another basis \(\set{v_i}_{i=1}^n\)? Write \(e_i=\sum_{j=1}^n\beta_{ij}v_j\)

Regroup the coefficients \[ \bfx = \cdots + \left(\sum_{i=1}^n\beta_{ij}\alpha_i\right) v_j + \cdots \]

In matrix form \[ \bfx_{\text{new}} = \mat{cccc}{\beta_{11}&\beta_{21}&\cdots&\beta_{n1}\\ \beta_{12}&\beta_{22}&\cdots&\beta_{n2}\\\vdots&\vdots&\vdots&\vdots\\\beta_{1n}&\beta_{2n}&\cdots&\beta_{nn}}\bfx \]

A change of basis matrix \(\matP\) is full rank (basis vectors are linearly independent)

Any full rank matrix \(\matP\) can be viewed as a change of basis

\(\matP^{-1}\) takes you back to the original basis

**Warning**: the columns of \(\bfP\) describe the*old*coordinates as a function of the*new*onesIf \(\matA,\bfB\in\bbR^{n\times n}\) then \(\bfB\) is similar to \(\bfA\) if there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfB=\bfP^{-1}\bfA\bfP\)

*Intuition*: similar matrices are the same up to a change of basis\(\matA\in\bbR^{n\times n}\) is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfD=\bfP^{-1}\bfA\bfP\) with \(\matD\) diagonal

Not all matrices are diagonalizable!

- Every complex matrix \(\matA\) has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
- Every matrix \(\matA^\in\bbR^{n\times n}\) is unitarily similar to an upper triangular matrix, i.e., \[ \bfA = \bfV\boldsymbol{\Delta}\bfV^\dagger \] with \(\boldsymbol{\Delta}\) upper triangular and \(\bfV^\dagger=\bfV^{-1}\)
- Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.