Symmetric matrices

Dr. Matthieu R Bloch

Wednesday October 27, 2021

Logistics

  • Drop date: October 30, 2021

  • My office hours on Tuesdays

    • 8am-9am on BlueJeans (https://bluejeans.com/205357142)
    • This was great!

  • Midterm 2:

    • Moved to Monday November 8, 2021 (gives you weekend to prepare)
    • Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

What’s on the agenda for today?

Image
Toddlers can do it!
  • Last time:
    • RKHS
  • Today:
    • Final thoughts on RKHS
    • Symmetric matrices: more linear algebra (objective: further understand least-square problems)
  • Reading: Romberg, lecture notes 10/11/12

Last time: Aronszjan’s theorem

  • An inner product kernel is a mapping \(k:\bbR^d\times\bbR^d\to\bbR\) for which there exists a Hilbert space \(\calH\) and a mapping \(\Phi:\bbR^d\to\calH\) such that \[\forall \bfu,\bfv\in\bbR^d\quad k(\bfu,\bfv)=\langle\Phi(\bfu),\Phi(\bfv)\rangle_\calH\]

  • A function \(k:\bbR^d\times\bbR^d\to\bbR\) is a positive semidefinite kernel if
    • \(k\) is symmetric, i.e., \(k(\bfu,\bfv)=k(\bfv,\bfu)\)
    • for all \(\{\bfx_i\}_{i=1}^N\), the Gram matrix \(\bfK\) is positive semidefinite, i.e., \[\bfx^\intercal\bfK\bfx\geq 0\text{ with }\bfK=[K_{i,j}]\text{ and }K_{i,j}\eqdef k(\bfx_i,\bfx_j)\]

  • A function \(k:\bbR^d\times\bbR^d\to\bbR\) is an inner product kernel if and only if \(k\) is a positive semidefinite kernel.

Examples of RKHS

  • Regression using linear and quadratic functions in \(\bbR^d\)
  • Regression using Radial Basis Functions

  • Examples of kernels

    • Homogeneous polynomial kernel: \(k(\bfu,\bfv) = (\bfu^\intercal\bfv)^m\) with \(m\in\bbN^*\)
    • Inhomogenous polynomial kernel: \(k(\bfu,\bfv) = (\bfu^\intercal\bfv+c)^m\) with \(c>0\), \(m\in\bbN^*\)
    • Radial basis function (RBF) kernel: \(k(\bfu,\bfv) = \exp\left(-\frac{\norm{\bfu-\bfv}^2}{2\sigma^2}\right)\) with \(\sigma^2>0\)

Systems of symmetric equations

  • Least square problems involved the normal equations \(\bfX^\intercal\bfX \bftheta=\bfX^\intercal\bfy\)

  • This is a system of symmetric equations \(\bfA\bfx=\bfy\) with \(\bfA^\intercal=\bfA\)

    • Ultimately we will talk about the non-symmetric/non square case

  • A real-valued matrix \(\bfA\) is symmetric if \(\bfA^\intercal=\bfA\) A complex-valued matrix \(\bfA\) is Hermitian if \(\bfA^\dagger=\bfA\) (also written \(\bfA^H=\bfA\))

  • Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an eigenvalue associated to the eigenvector \(\bfv\).

  • If \(\lambda\) is an eigenvalue, there are infinitely many eigenvectors associated to it

  • Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an eigenvalue associated to the eigenvector \(\bfv\).

Change of basis

  • Consider the canonical basis \(\set{e_i}_{i=1}^n\) for \(\bbR^n\); every vector can be viewed as a vector of coefficients \(\set{\alpha_i}_{i=1}^n\), \[ \bfx = \sum_{i=1}^n \alpha_i e_i = \mat{cccc}{\alpha_1&\alpha_2&\cdots&\alpha_n}^\intercal \]

  • How do we find the representation of \(\bfx\) in another basis \(\set{v_i}_{i=1}^n\)? Write \(e_i=\sum_{j=1}^n\beta_{ij}v_j\)

  • Regroup the coefficients \[ \bfx = \cdots + \left(\sum_{i=1}^n\beta_{ij}\alpha_i\right) v_j + \cdots \]

  • In matrix form \[ \bfx_{\text{new}} = \mat{cccc}{\beta_{11}&\beta_{21}&\cdots&\beta_{n1}\\ \beta_{12}&\beta_{22}&\cdots&\beta_{n2}\\\vdots&\vdots&\vdots&\vdots\\\beta_{1n}&\beta_{2n}&\cdots&\beta_{nn}}\bfx \]

Similarity

  • A change of basis matrix \(\matP\) is full rank (basis vectors are linearly independent)

  • Any full rank matrix \(\matP\) can be viewed as a change of basis

  • \(\matP^{-1}\) takes you back to the original basis

  • Warning: the columns of \(\bfP\) describe the old coordinates as a function of the new ones

  • If \(\matA,\bfB\in\bbR^{n\times n}\) then \(\bfB\) is similar to \(\bfA\) if there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfB=\bfP^{-1}\bfA\bfP\)

  • Intuition: similar matrices are the same up to a change of basis

  • \(\matA\in\bbR^{n\times n}\) is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfD=\bfP^{-1}\bfA\bfP\) with \(\matD\) diagonal

  • Not all matrices are diagonalizable!

Spectral theorem

  • Every complex matrix \(\matA\) has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
  • Every matrix \(\matA^\in\bbR^{n\times n}\) is unitarily similar to an upper triangular matrix, i.e., \[ \bfA = \bfV\boldsymbol{\Delta}\bfV^\dagger \] with \(\boldsymbol{\Delta}\) upper triangular and \(\bfV^\dagger=\bfV^{-1}\)
  • Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.