Symmetric matrices

Logistics

Drop date: October 30, 2021
My office hours on Tuesdays
- 8am-9am on BlueJeans (https://bluejeans.com/205357142)
- This was great!
Midterm 2:
- Moved to Monday November 8, 2021 (gives you weekend to prepare)
- Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

Last time:
- RKHS
Today:
- Final thoughts on RKHS
- Symmetric matrices: more linear algebra (objective: further understand least-square problems)
Reading: Romberg, lecture notes 10/11/12

An inner product kernel is a mapping $k : R^{d} \times R^{d} \to R$ for which there exists a Hilbert space $H$ and a mapping $Φ : R^{d} \to H$ such that $\forall u, v \in R^{d} k (u, v) = ⟨ Φ (u), Φ (v) ⟩_{H}$
A function $k : R^{d} \times R^{d} \to R$ is a positive semidefinite kernel if
- $k$ is symmetric, i.e., $k (u, v) = k (v, u)$
- for all ${x_{i}}_{i = 1}^{N}$ , the Gram matrix $K$ is positive semidefinite, i.e., $x^{⊺} K x \geq 0 with K = [K_{i, j}] and K_{i, j} ≜ k (x_{i}, x_{j})$
A function $k : R^{d} \times R^{d} \to R$ is an inner product kernel if and only if $k$ is a positive semidefinite kernel.

Regression using linear and quadratic functions in $R^{d}$
Regression using Radial Basis Functions
Examples of kernels
- Homogeneous polynomial kernel: $k (u, v) = (u^{⊺} v)^{m}$ with $m \in N^{*}$
- Inhomogenous polynomial kernel: $k (u, v) = (u^{⊺} v + c)^{m}$ with $c > 0$ , $m \in N^{*}$
- Radial basis function (RBF) kernel: $k (u, v) = \exp (- \frac{{‖ u - v ‖}_{}^{2}}{2 σ^{2}})$ with $σ^{2} > 0$

Least square problems involved the normal equations $X^{⊺} X θ = X^{⊺} y$
This is a system of symmetric equations $A x = y$ with $A^{⊺} = A$
- Ultimately we will talk about the non-symmetric/non square case
A real-valued matrix $A$ is symmetric if $A^{⊺} = A$ A complex-valued matrix $A$ is Hermitian if $A^{†} = A$ (also written $A^{H} = A$ )
Given a matrix $A \in C^{n \times n}$ , if a vector $v \in C^{n}$ satisfies $A v = λ v$ for some $λ \in C$ , then $λ$ is an eigenvalue associated to the eigenvector $v$ .
If $λ$ is an eigenvalue, there are infinitely many eigenvectors associated to it
Given a matrix $A \in C^{n \times n}$ , if a vector $v \in C^{n}$ satisfies $A v = λ v$ for some $λ \in C$ , then $λ$ is an eigenvalue associated to the eigenvector $v$ .

Consider the canonical basis ${e_{i}}_{i = 1}^{n}$ for $R^{n}$ ; every vector can be viewed as a vector of coefficients ${α_{i}}_{i = 1}^{n}$ , $x = \sum_{i = 1}^{n} α_{i} e_{i} = {[\begin{array}{cccc} α_{1} & α_{2} & \dots & α_{n} \end{array}]}^{⊺}$
How do we find the representation of $x$ in another basis ${v_{i}}_{i = 1}^{n}$ ? Write $e_{i} = \sum_{j = 1}^{n} β_{i j} v_{j}$
Regroup the coefficients $x = \dots + (\sum_{i = 1}^{n} β_{i j} α_{i}) v_{j} + \dots$
In matrix form $x_{new} = [\begin{array}{cccc} β_{11} & β_{21} & \dots & β_{n 1} \\ β_{12} & β_{22} & \dots & β_{n 2} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ β_{1 n} & β_{2 n} & \dots & β_{n n} \end{array}] x$

A change of basis matrix $P$ is full rank (basis vectors are linearly independent)
Any full rank matrix $P$ can be viewed as a change of basis
$P^{- 1}$ takes you back to the original basis
Warning: the columns of $P$ describe the old coordinates as a function of the new ones
If $A, B \in R^{n \times n}$ then $B$ is similar to $A$ if there exists an invertible matrix $S \in R^{n \times n}$ such that $B = P^{- 1} A P$
Intuition: similar matrices are the same up to a change of basis
$A \in R^{n \times n}$ is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix $S \in R^{n \times n}$ such that $D = P^{- 1} A P$ with $D$ diagonal
Not all matrices are diagonalizable!

Every complex matrix $A$ has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
Every matrix $A^{\in} R^{n \times n}$ is unitarily similar to an upper triangular matrix, i.e., $A = V Δ V^{†}$ with $Δ$ upper triangular and $V^{†} = V^{- 1}$
Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.