Symmetric matrices

Dr. Matthieu R Bloch

Monday, November 1, 2021

Logistics

  • Grades

    • Midterm 1 was long… future exams will be better calibrated
    • I will curve to get GPA similar to past semesters

  • My office hours on Tuesdays

    • 8am-9am on BlueJeans (https://bluejeans.com/205357142)
    • Tomorrow (Tuesday November 02, 2021) will focus on Midterm 1 solution
    • I’ll try to record the session

  • Midterm 2:

    • Moved to Monday November 8, 2021 (gives you weekend to prepare)
    • Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

What’s on the agenda for today?

Image
Toddlers can do it!
  • Last time:
    • Symmetric matrices: more linear algebra
    • Objective: further understand least-square problems
  • Reading: lecture notes 12

Systems of symmetric equations

  • Least square problems involved the normal equations \(\bfX^\intercal\bfX \bftheta=\bfX^\intercal\bfy\)

  • This is a system of symmetric equations \(\bfA\bfx=\bfy\) with \(\bfA^\intercal=\bfA\)

    • Ultimately we will talk about the non-symmetric/non square case

  • A real-valued matrix \(\bfA\) is symmetric if \(\bfA^\intercal=\bfA\)

    A complex-valued matrix \(\bfA\) is Hermitian if \(\bfA^\dagger=\bfA\) (also written \(\bfA^H=\bfA\))

  • Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an eigenvalue associated to the eigenvector \(\bfv\).

  • If \(\lambda\) is an eigenvalue, there are infinitely many eigenvectors associated to it

  • Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an eigenvalue associated to the eigenvector \(\bfv\).

Change of basis

  • Consider the canonical basis \(\set{e_i}_{i=1}^n\) for \(\bbR^n\); every vector can be viewed as a vector of coefficients \(\set{\alpha_i}_{i=1}^n\), \[ \bfx = \sum_{i=1}^n \alpha_i e_i = \mat{cccc}{\alpha_1&\alpha_2&\cdots&\alpha_n}^\intercal \]

  • How do we find the representation of \(\bfx\) in another basis \(\set{v_i}_{i=1}^n\)? Write \(e_i=\sum_{j=1}^n\beta_{ij}v_j\)

  • Regroup the coefficients \[ \bfx = \cdots + \left(\sum_{i=1}^n\beta_{ij}\alpha_i\right) v_j + \cdots \]

  • In matrix form \[ \bfx_{\text{new}} = \mat{cccc}{\beta_{11}&\beta_{21}&\cdots&\beta_{n1}\\ \beta_{12}&\beta_{22}&\cdots&\beta_{n2}\\\vdots&\vdots&\vdots&\vdots\\\beta_{1n}&\beta_{2n}&\cdots&\beta_{nn}}\bfx \]

Similarity

  • A change of basis matrix \(\matP\) is full rank (basis vectors are linearly independent)

  • Any full rank matrix \(\matP\) can be viewed as a change of basis

  • \(\matP^{-1}\) takes you back to the original basis

  • Warning: the columns of \(\bfP\) describe the old coordinates as a function of the new ones

  • If \(\matA,\bfB\in\bbR^{n\times n}\) then \(\bfB\) is similar to \(\bfA\) if there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfB=\bfP^{-1}\bfA\bfP\)

  • Intuition: similar matrices are the same up to a change of basis

  • \(\matA\in\bbR^{n\times n}\) is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfD=\bfP^{-1}\bfA\bfP\) with \(\matD\) diagonal

  • Not all matrices are diagonalizable!

Spectral theorem

  • Every complex matrix \(\matA\) has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
  • Every matrix \(\matA\in\bbC^{n\times n}\) is unitarily similar to an upper triangular matrix, i.e., \[ \bfA = \bfV\boldsymbol{\Delta}\bfV^\dagger \] with \(\boldsymbol{\Delta}\) upper triangular and \(\bfV^\dagger=\bfV^{-1}\).
  • Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.

  • Note that if \(\matA = \matV\matD\matV^\dagger\) then \[ \matA = \sum_{i=1}^n\lambda_i \vecv_i\vecv_i^\dagger \]

  • How about real-valued matrices \(\matA\in\bbR^{n\times n}\)

Symmetric positive definite matrices

  • A symmetric matrice \(\matA\) is positive definite if it has positive eigenvalues, i.e., \[ \forall i\in\set{1,\cdots,n}\quad\lambda_i>0. \] A symmetric matrice \(\matA\) is positive semidefinite if it has nonnegative eigenvalues, i.e., \[ \forall i\in\set{1,\cdots,n}\quad\lambda_i\geq 0. \]

  • Convention: \(\lambda_1\geq \lambda_2\geq \cdots \geq \lambda_n\)

  • Variational form of extreme eigenvalues for symmetric positive definite matrices \(\bfA\) \[ \lambda_1 &= \max_{\vecx\in\bbR^n:\norm[2]{\bfx}=1}\vecx^\intercal \matA\vecx = \max_{\vecx\in\bbR^n}\frac{\vecx^\intercal \matA\vecx}{\norm[2]{\vecx}^2}\\ \lambda_n &= \min_{\vecx\in\bbR^n:\norm[2]{\bfx}=1}\vecx^\intercal \matA\vecx = \min_{\vecx\in\bbR^n}\frac{\vecx^\intercal \matA\vecx}{\norm[2]{\vecx}^2}\\ \]

  • For any analytic function \(f\), we have \[ f(\vecA) = \sum_{i=1}^n f(\lambda_i)\vecv_i\vecv)i^\intercal \]

System of symmetric definite equations

  • Consider the system \(\vecy=\matA\vecx\) with \(\matA\) symmetric positive definite

  • Let \(\set{\vecv_i}\) be the eigenvectors of \(\matA\). \[ \vecx = \sum_{i=1}^n\frac{1}{\lambda_i}\dotp{\vecy}{\vecv_i}\vecv_i \]

  • Assume that there exists some observation error \(\vecy=\matA\vecx+\vece\)

    • \(\vece\) is unknown
    • we try to reconstruct \(\vecx\) as \(\widetilde{\vecx}\) by applying \(\matA^{-1}\)
  • \[ \frac{1}{\lambda_1}^2\norm[2]{\vece}^2\leq \norm[2]{\vecx-\tilde{\vecx}}\leq \frac{1}{\lambda_n}^2\norm[2]{\vece}^2. \]