Symmetric matrices

Logistics

Grades
- Midterm 1 was long… future exams will be better calibrated
- I will curve to get GPA similar to past semesters
My office hours on Tuesdays
- 8am-9am on BlueJeans (https://bluejeans.com/205357142)
- Tomorrow (Tuesday November 02, 2021) will focus on Midterm 1 solution
- I’ll try to record the session
Midterm 2:
- Moved to Monday November 8, 2021 (gives you weekend to prepare)
- Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

Last time:
- Symmetric matrices: more linear algebra
- Objective: further understand least-square problems
Reading: lecture notes 12

Least square problems involved the normal equations $X^{⊺} X θ = X^{⊺} y$
This is a system of symmetric equations $A x = y$ with $A^{⊺} = A$
- Ultimately we will talk about the non-symmetric/non square case
A real-valued matrix $A$ is symmetric if $A^{⊺} = A$

A complex-valued matrix $A$ is Hermitian if $A^{†} = A$ (also written $A^{H} = A$ )
Given a matrix $A \in C^{n \times n}$ , if a vector $v \in C^{n}$ satisfies $A v = λ v$ for some $λ \in C$ , then $λ$ is an eigenvalue associated to the eigenvector $v$ .
If $λ$ is an eigenvalue, there are infinitely many eigenvectors associated to it
Given a matrix $A \in C^{n \times n}$ , if a vector $v \in C^{n}$ satisfies $A v = λ v$ for some $λ \in C$ , then $λ$ is an eigenvalue associated to the eigenvector $v$ .

Consider the canonical basis ${e_{i}}_{i = 1}^{n}$ for $R^{n}$ ; every vector can be viewed as a vector of coefficients ${α_{i}}_{i = 1}^{n}$ , $x = \sum_{i = 1}^{n} α_{i} e_{i} = {[\begin{array}{cccc} α_{1} & α_{2} & \dots & α_{n} \end{array}]}^{⊺}$
How do we find the representation of $x$ in another basis ${v_{i}}_{i = 1}^{n}$ ? Write $e_{i} = \sum_{j = 1}^{n} β_{i j} v_{j}$
Regroup the coefficients $x = \dots + (\sum_{i = 1}^{n} β_{i j} α_{i}) v_{j} + \dots$
In matrix form $x_{new} = [\begin{array}{cccc} β_{11} & β_{21} & \dots & β_{n 1} \\ β_{12} & β_{22} & \dots & β_{n 2} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ β_{1 n} & β_{2 n} & \dots & β_{n n} \end{array}] x$

A change of basis matrix $P$ is full rank (basis vectors are linearly independent)
Any full rank matrix $P$ can be viewed as a change of basis
$P^{- 1}$ takes you back to the original basis
Warning: the columns of $P$ describe the old coordinates as a function of the new ones
If $A, B \in R^{n \times n}$ then $B$ is similar to $A$ if there exists an invertible matrix $S \in R^{n \times n}$ such that $B = P^{- 1} A P$
Intuition: similar matrices are the same up to a change of basis
$A \in R^{n \times n}$ is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix $S \in R^{n \times n}$ such that $D = P^{- 1} A P$ with $D$ diagonal
Not all matrices are diagonalizable!

Every complex matrix $A$ has at least one complex eigenvector and every real symmetrix matrix has real eigenvalues and at least one real eigenvector.
Every matrix $A \in C^{n \times n}$ is unitarily similar to an upper triangular matrix, i.e., $A = V Δ V^{†}$ with $Δ$ upper triangular and $V^{†} = V^{- 1}$ .
Every hermitian matrix is unitarily similar to a real-valued diagonal matrix.
Note that if $A = V D V^{†}$ then $A = \sum_{i = 1}^{n} λ_{i} v_{i} v_{i}^{†}$
How about real-valued matrices $A \in R^{n \times n}$

A symmetric matrice $A$ is positive definite if it has positive eigenvalues, i.e., $\forall i \in {1, \dots, n} λ_{i} > 0.$ A symmetric matrice $A$ is positive semidefinite if it has nonnegative eigenvalues, i.e., $\forall i \in {1, \dots, n} λ_{i} \geq 0.$
Convention: $λ_{1} \geq λ_{2} \geq \dots \geq λ_{n}$
Variational form of extreme eigenvalues for symmetric positive definite matrices $A$ $Misplaced &$
For any analytic function $f$ , we have $f (\vecA) = \sum_{i = 1}^{n} f (λ_{i}) v_{i} v) i^{⊺}$

Consider the system $y = A x$ with $A$ symmetric positive definite
Let ${v_{i}}$ be the eigenvectors of $A$ . $x = \sum_{i = 1}^{n} \frac{1}{λ_{i}} {⟨ y, v_{i} ⟩}_{} v_{i}$
Assume that there exists some observation error $y = A x + e$
- $e$ is unknown
- we try to reconstruct $x$ as $\tilde{x}$ by applying $A^{- 1}$
${\frac{1}{λ_{1}}}^{2} {‖ e ‖}_{2}^{2} \leq {‖ x - \tilde{x} ‖}_{2} \leq {\frac{1}{λ_{n}}}^{2} {‖ e ‖}_{2}^{2} .$