Dr. Matthieu R Bloch

Monday, November 1, 2021

**Grades**- Midterm 1 was long… future exams will be better calibrated
- I will curve to get GPA similar to past semesters

**My office hours on Tuesdays**- 8am-9am on BlueJeans (https://bluejeans.com/205357142)
- Tomorrow (Tuesday November 02, 2021) will focus on Midterm 1 solution
- I’ll try to record the session

**Midterm 2:**- Moved to
**Monday November 8, 2021**(gives you weekend to prepare) - Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on
**regression**

- Moved to

**Last time**:- Symmetric matrices: more linear algebra
*Objective:*further understand least-square problems

**Reading:**lecture notes 12

Least square problems involved the normal equations \(\bfX^\intercal\bfX \bftheta=\bfX^\intercal\bfy\)

This is a system of symmetric equations \(\bfA\bfx=\bfy\) with \(\bfA^\intercal=\bfA\)

- Ultimately we will talk about the non-symmetric/non square case

A real-valued matrix \(\bfA\) is symmetric if \(\bfA^\intercal=\bfA\)

A complex-valued matrix \(\bfA\) is Hermitian if \(\bfA^\dagger=\bfA\) (also written \(\bfA^H=\bfA\))

Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an

*eigenvalue*associated to the*eigenvector*\(\bfv\).If \(\lambda\) is an eigenvalue, there are infinitely many eigenvectors associated to it

Given a matrix \(\matA\in\bbC^{n\times n}\), if a vector \(\vecv\in\bbC^n\) satisfies \(\matA\bfv=\lambda\bfv\) for some \(\lambda\in\bbC\), then \(\lambda\) is an

*eigenvalue*associated to the*eigenvector*\(\bfv\).

Consider the canonical basis \(\set{e_i}_{i=1}^n\) for \(\bbR^n\); every vector can be viewed as a vector of coefficients \(\set{\alpha_i}_{i=1}^n\), \[ \bfx = \sum_{i=1}^n \alpha_i e_i = \mat{cccc}{\alpha_1&\alpha_2&\cdots&\alpha_n}^\intercal \]

How do we find the representation of \(\bfx\) in another basis \(\set{v_i}_{i=1}^n\)? Write \(e_i=\sum_{j=1}^n\beta_{ij}v_j\)

Regroup the coefficients \[ \bfx = \cdots + \left(\sum_{i=1}^n\beta_{ij}\alpha_i\right) v_j + \cdots \]

In matrix form \[ \bfx_{\text{new}} = \mat{cccc}{\beta_{11}&\beta_{21}&\cdots&\beta_{n1}\\ \beta_{12}&\beta_{22}&\cdots&\beta_{n2}\\\vdots&\vdots&\vdots&\vdots\\\beta_{1n}&\beta_{2n}&\cdots&\beta_{nn}}\bfx \]

A change of basis matrix \(\matP\) is full rank (basis vectors are linearly independent)

Any full rank matrix \(\matP\) can be viewed as a change of basis

\(\matP^{-1}\) takes you back to the original basis

**Warning**: the columns of \(\bfP\) describe the*old*coordinates as a function of the*new*onesIf \(\matA,\bfB\in\bbR^{n\times n}\) then \(\bfB\) is similar to \(\bfA\) if there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfB=\bfP^{-1}\bfA\bfP\)

*Intuition*: similar matrices are the same up to a change of basis\(\matA\in\bbR^{n\times n}\) is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix \(\bfS\in\bbR^{n\times n}\) such that \(\bfD=\bfP^{-1}\bfA\bfP\) with \(\matD\) diagonal

Not all matrices are diagonalizable!

- Every complex matrix \(\matA\) has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
- Every matrix \(\matA\in\bbC^{n\times n}\) is unitarily similar to an upper triangular matrix, i.e., \[ \bfA = \bfV\boldsymbol{\Delta}\bfV^\dagger \] with \(\boldsymbol{\Delta}\) upper triangular and \(\bfV^\dagger=\bfV^{-1}\).
- Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.
Note that if \(\matA = \matV\matD\matV^\dagger\) then \[ \matA = \sum_{i=1}^n\lambda_i \vecv_i\vecv_i^\dagger \]

How about real-valued matrices \(\matA\in\bbR^{n\times n}\)

A symmetric matrice \(\matA\) is positive definite if it has positive eigenvalues, i.e., \[ \forall i\in\set{1,\cdots,n}\quad\lambda_i>0. \] A symmetric matrice \(\matA\) is positive semidefinite if it has nonnegative eigenvalues, i.e., \[ \forall i\in\set{1,\cdots,n}\quad\lambda_i\geq 0. \]

**Convention:**\(\lambda_1\geq \lambda_2\geq \cdots \geq \lambda_n\)**Variational form of extreme eigenvalues**for symmetric positive definite matrices \(\bfA\) \[ \lambda_1 &= \max_{\vecx\in\bbR^n:\norm[2]{\bfx}=1}\vecx^\intercal \matA\vecx = \max_{\vecx\in\bbR^n}\frac{\vecx^\intercal \matA\vecx}{\norm[2]{\vecx}^2}\\ \lambda_n &= \min_{\vecx\in\bbR^n:\norm[2]{\bfx}=1}\vecx^\intercal \matA\vecx = \min_{\vecx\in\bbR^n}\frac{\vecx^\intercal \matA\vecx}{\norm[2]{\vecx}^2}\\ \]For any analytic function \(f\), we have \[ f(\vecA) = \sum_{i=1}^n f(\lambda_i)\vecv_i\vecv)i^\intercal \]

Consider the system \(\vecy=\matA\vecx\) with \(\matA\) symmetric positive definite

Let \(\set{\vecv_i}\) be the eigenvectors of \(\matA\). \[ \vecx = \sum_{i=1}^n\frac{1}{\lambda_i}\dotp{\vecy}{\vecv_i}\vecv_i \]

Assume that there exists some observation error \(\vecy=\matA\vecx+\vece\)

- \(\vece\) is unknown
- we try to reconstruct \(\vecx\) as \(\widetilde{\vecx}\) by applying \(\matA^{-1}\)

- \[ \frac{1}{\lambda_1}^2\norm[2]{\vece}^2\leq \norm[2]{\vecx-\tilde{\vecx}}\leq \frac{1}{\lambda_n}^2\norm[2]{\vece}^2. \]