# Symmetric matrices

Wednesday October 27, 2021

## Logistics

• Drop date: October 30, 2021

• My office hours on Tuesdays

• 8am-9am on BlueJeans (https://bluejeans.com/205357142)
• This was great!
• Midterm 2:

• Moved to Monday November 8, 2021 (gives you weekend to prepare)
• Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

## What’s on the agenda for today?

• Last time:
• RKHS
• Today:
• Final thoughts on RKHS
• Symmetric matrices: more linear algebra (objective: further understand least-square problems)
• Reading: Romberg, lecture notes 10/11/12

## Last time: Aronszjan’s theorem

• An inner product kernel is a mapping $k:\bbR^d\times\bbR^d\to\bbR$ for which there exists a Hilbert space $\calH$ and a mapping $\Phi:\bbR^d\to\calH$ such that $\forall \bfu,\bfv\in\bbR^d\quad k(\bfu,\bfv)=\langle\Phi(\bfu),\Phi(\bfv)\rangle_\calH$

• A function $k:\bbR^d\times\bbR^d\to\bbR$ is a positive semidefinite kernel if
• $k$ is symmetric, i.e., $k(\bfu,\bfv)=k(\bfv,\bfu)$
• for all $\{\bfx_i\}_{i=1}^N$, the Gram matrix $\bfK$ is positive semidefinite, i.e., $\bfx^\intercal\bfK\bfx\geq 0\text{ with }\bfK=[K_{i,j}]\text{ and }K_{i,j}\eqdef k(\bfx_i,\bfx_j)$
• A function $k:\bbR^d\times\bbR^d\to\bbR$ is an inner product kernel if and only if $k$ is a positive semidefinite kernel.

## Examples of RKHS

• Regression using linear and quadratic functions in $\bbR^d$
• Regression using Radial Basis Functions
• Examples of kernels

• Homogeneous polynomial kernel: $k(\bfu,\bfv) = (\bfu^\intercal\bfv)^m$ with $m\in\bbN^*$
• Inhomogenous polynomial kernel: $k(\bfu,\bfv) = (\bfu^\intercal\bfv+c)^m$ with $c>0$, $m\in\bbN^*$
• Radial basis function (RBF) kernel: $k(\bfu,\bfv) = \exp\left(-\frac{\norm{\bfu-\bfv}^2}{2\sigma^2}\right)$ with $\sigma^2>0$

## Systems of symmetric equations

• Least square problems involved the normal equations $\bfX^\intercal\bfX \bftheta=\bfX^\intercal\bfy$

• This is a system of symmetric equations $\bfA\bfx=\bfy$ with $\bfA^\intercal=\bfA$

• Ultimately we will talk about the non-symmetric/non square case
• A real-valued matrix $\bfA$ is symmetric if $\bfA^\intercal=\bfA$ A complex-valued matrix $\bfA$ is Hermitian if $\bfA^\dagger=\bfA$ (also written $\bfA^H=\bfA$)

• Given a matrix $\matA\in\bbC^{n\times n}$, if a vector $\vecv\in\bbC^n$ satisfies $\matA\bfv=\lambda\bfv$ for some $\lambda\in\bbC$, then $\lambda$ is an eigenvalue associated to the eigenvector $\bfv$.

• If $\lambda$ is an eigenvalue, there are infinitely many eigenvectors associated to it

• Given a matrix $\matA\in\bbC^{n\times n}$, if a vector $\vecv\in\bbC^n$ satisfies $\matA\bfv=\lambda\bfv$ for some $\lambda\in\bbC$, then $\lambda$ is an eigenvalue associated to the eigenvector $\bfv$.

## Change of basis

• Consider the canonical basis $\set{e_i}_{i=1}^n$ for $\bbR^n$; every vector can be viewed as a vector of coefficients $\set{\alpha_i}_{i=1}^n$, $\bfx = \sum_{i=1}^n \alpha_i e_i = \mat{cccc}{\alpha_1&\alpha_2&\cdots&\alpha_n}^\intercal$

• How do we find the representation of $\bfx$ in another basis $\set{v_i}_{i=1}^n$? Write $e_i=\sum_{j=1}^n\beta_{ij}v_j$

• Regroup the coefficients $\bfx = \cdots + \left(\sum_{i=1}^n\beta_{ij}\alpha_i\right) v_j + \cdots$

• In matrix form $\bfx_{\text{new}} = \mat{cccc}{\beta_{11}&\beta_{21}&\cdots&\beta_{n1}\\ \beta_{12}&\beta_{22}&\cdots&\beta_{n2}\\\vdots&\vdots&\vdots&\vdots\\\beta_{1n}&\beta_{2n}&\cdots&\beta_{nn}}\bfx$

## Similarity

• A change of basis matrix $\matP$ is full rank (basis vectors are linearly independent)

• Any full rank matrix $\matP$ can be viewed as a change of basis

• $\matP^{-1}$ takes you back to the original basis

• Warning: the columns of $\bfP$ describe the old coordinates as a function of the new ones

• If $\matA,\bfB\in\bbR^{n\times n}$ then $\bfB$ is similar to $\bfA$ if there exists an invertible matrix $\bfS\in\bbR^{n\times n}$ such that $\bfB=\bfP^{-1}\bfA\bfP$

• Intuition: similar matrices are the same up to a change of basis

• $\matA\in\bbR^{n\times n}$ is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix $\bfS\in\bbR^{n\times n}$ such that $\bfD=\bfP^{-1}\bfA\bfP$ with $\matD$ diagonal

• Not all matrices are diagonalizable!

## Spectral theorem

• Every complex matrix $\matA$ has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
• Every matrix $\matA^\in\bbR^{n\times n}$ is unitarily similar to an upper triangular matrix, i.e., $\bfA = \bfV\boldsymbol{\Delta}\bfV^\dagger$ with $\boldsymbol{\Delta}$ upper triangular and $\bfV^\dagger=\bfV^{-1}$
• Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.