# Symmetric matrices

Monday, November 1, 2021

## Logistics

• Midterm 1 was long… future exams will be better calibrated
• I will curve to get GPA similar to past semesters
• My office hours on Tuesdays

• 8am-9am on BlueJeans (https://bluejeans.com/205357142)
• Tomorrow (Tuesday November 02, 2021) will focus on Midterm 1 solution
• I’ll try to record the session
• Midterm 2:

• Moved to Monday November 8, 2021 (gives you weekend to prepare)
• Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

## What’s on the agenda for today?

• Last time:
• Symmetric matrices: more linear algebra
• Objective: further understand least-square problems

## Systems of symmetric equations

• Least square problems involved the normal equations $\bfX^\intercal\bfX \bftheta=\bfX^\intercal\bfy$

• This is a system of symmetric equations $\bfA\bfx=\bfy$ with $\bfA^\intercal=\bfA$

• Ultimately we will talk about the non-symmetric/non square case
• A real-valued matrix $\bfA$ is symmetric if $\bfA^\intercal=\bfA$

A complex-valued matrix $\bfA$ is Hermitian if $\bfA^\dagger=\bfA$ (also written $\bfA^H=\bfA$)

• Given a matrix $\matA\in\bbC^{n\times n}$, if a vector $\vecv\in\bbC^n$ satisfies $\matA\bfv=\lambda\bfv$ for some $\lambda\in\bbC$, then $\lambda$ is an eigenvalue associated to the eigenvector $\bfv$.

• If $\lambda$ is an eigenvalue, there are infinitely many eigenvectors associated to it

• Given a matrix $\matA\in\bbC^{n\times n}$, if a vector $\vecv\in\bbC^n$ satisfies $\matA\bfv=\lambda\bfv$ for some $\lambda\in\bbC$, then $\lambda$ is an eigenvalue associated to the eigenvector $\bfv$.

## Change of basis

• Consider the canonical basis $\set{e_i}_{i=1}^n$ for $\bbR^n$; every vector can be viewed as a vector of coefficients $\set{\alpha_i}_{i=1}^n$, $\bfx = \sum_{i=1}^n \alpha_i e_i = \mat{cccc}{\alpha_1&\alpha_2&\cdots&\alpha_n}^\intercal$

• How do we find the representation of $\bfx$ in another basis $\set{v_i}_{i=1}^n$? Write $e_i=\sum_{j=1}^n\beta_{ij}v_j$

• Regroup the coefficients $\bfx = \cdots + \left(\sum_{i=1}^n\beta_{ij}\alpha_i\right) v_j + \cdots$

• In matrix form $\bfx_{\text{new}} = \mat{cccc}{\beta_{11}&\beta_{21}&\cdots&\beta_{n1}\\ \beta_{12}&\beta_{22}&\cdots&\beta_{n2}\\\vdots&\vdots&\vdots&\vdots\\\beta_{1n}&\beta_{2n}&\cdots&\beta_{nn}}\bfx$

## Similarity

• A change of basis matrix $\matP$ is full rank (basis vectors are linearly independent)

• Any full rank matrix $\matP$ can be viewed as a change of basis

• $\matP^{-1}$ takes you back to the original basis

• Warning: the columns of $\bfP$ describe the old coordinates as a function of the new ones

• If $\matA,\bfB\in\bbR^{n\times n}$ then $\bfB$ is similar to $\bfA$ if there exists an invertible matrix $\bfS\in\bbR^{n\times n}$ such that $\bfB=\bfP^{-1}\bfA\bfP$

• Intuition: similar matrices are the same up to a change of basis

• $\matA\in\bbR^{n\times n}$ is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix $\bfS\in\bbR^{n\times n}$ such that $\bfD=\bfP^{-1}\bfA\bfP$ with $\matD$ diagonal

• Not all matrices are diagonalizable!

## Spectral theorem

• Every complex matrix $\matA$ has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
• Every matrix $\matA\in\bbC^{n\times n}$ is unitarily similar to an upper triangular matrix, i.e., $\bfA = \bfV\boldsymbol{\Delta}\bfV^\dagger$ with $\boldsymbol{\Delta}$ upper triangular and $\bfV^\dagger=\bfV^{-1}$.
• Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.
• Note that if $\matA = \matV\matD\matV^\dagger$ then $\matA = \sum_{i=1}^n\lambda_i \vecv_i\vecv_i^\dagger$

• How about real-valued matrices $\matA\in\bbR^{n\times n}$

## Symmetric positive definite matrices

• A symmetric matrice $\matA$ is positive definite if it has positive eigenvalues, i.e., $\forall i\in\set{1,\cdots,n}\quad\lambda_i>0.$ A symmetric matrice $\matA$ is positive semidefinite if it has nonnegative eigenvalues, i.e., $\forall i\in\set{1,\cdots,n}\quad\lambda_i\geq 0.$

• Convention: $\lambda_1\geq \lambda_2\geq \cdots \geq \lambda_n$

• Variational form of extreme eigenvalues for symmetric positive definite matrices $\bfA$ $\lambda_1 &= \max_{\vecx\in\bbR^n:\norm{\bfx}=1}\vecx^\intercal \matA\vecx = \max_{\vecx\in\bbR^n}\frac{\vecx^\intercal \matA\vecx}{\norm{\vecx}^2}\\ \lambda_n &= \min_{\vecx\in\bbR^n:\norm{\bfx}=1}\vecx^\intercal \matA\vecx = \min_{\vecx\in\bbR^n}\frac{\vecx^\intercal \matA\vecx}{\norm{\vecx}^2}\\$

• For any analytic function $f$, we have $f(\vecA) = \sum_{i=1}^n f(\lambda_i)\vecv_i\vecv)i^\intercal$

## System of symmetric definite equations

• Consider the system $\vecy=\matA\vecx$ with $\matA$ symmetric positive definite

• Let $\set{\vecv_i}$ be the eigenvectors of $\matA$. $\vecx = \sum_{i=1}^n\frac{1}{\lambda_i}\dotp{\vecy}{\vecv_i}\vecv_i$

• Assume that there exists some observation error $\vecy=\matA\vecx+\vece$

• $\vece$ is unknown
• we try to reconstruct $\vecx$ as $\widetilde{\vecx}$ by applying $\matA^{-1}$
• $\frac{1}{\lambda_1}^2\norm{\vece}^2\leq \norm{\vecx-\tilde{\vecx}}\leq \frac{1}{\lambda_n}^2\norm{\vece}^2.$