Singular value decomposition

Logistics

• General announcements

  • Let me know if you have concerns about your grades (disagreements, etc.)
  • Be careful with honor code

• Office hours on Friday November 05, 2021

  • 8am-9:30am on BlueJeans (BlueJeans)
  • Focus on Midterm 2 preparation

• Midterm 2:

  • Moved to Monday November 8, 2021 (gives you weekend to prepare)
  • Coverage: everything since Midterm 1 (dont’ forget the fundamentals though), emphasis on regression

What’s on the agenda for today?

• Last time:

  • Symmetric matrices and spectral theorem
  • Objective: further understand least-square problems

• Today: singular value decomposition

• Reading: lecture notes 12/13

Spectral theorem

Symmetric positive definite matrices

• A symmetric matrice $\mathbf{A}$ is positive definite if it has positive eigenvalues, i.e., $\mathrm{\forall }i\in \{1,\cdots ,n\}{\lambda }_i>0$.

  A symmetric matrice $\mathbf{A}$ is positive semidefinite if it has nonnegative eigenvalues, i.e., $\mathrm{\forall }i\in \{1,\cdots ,n\}{\lambda }_i\ge 0$.

• Convention: ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_n$

• Variational form of extreme eigenvalues for symmetric positive definite matrices $\mathbf{A}$ $$\begin{array}{rl}{\lambda }_1& =\underset{\mathbf{x}\in {\mathbb{R}}^n:{\Vert \mathbf{x}\Vert }_2=1}{max}{\mathbf{x}}^{⊺}\mathbf{A}\mathbf{x}=\underset{\mathbf{x}\in {\mathbb{R}}^n}{max}\frac{{\mathbf{x}}^{⊺}\mathbf{A}\mathbf{x}}{{\Vert \mathbf{x}\Vert }_2^2}\\ {\lambda }_n& =\underset{\mathbf{x}\in {\mathbb{R}}^n:{\Vert \mathbf{x}\Vert }_2=1}{min}{\mathbf{x}}^{⊺}\mathbf{A}\mathbf{x}=\underset{\mathbf{x}\in {\mathbb{R}}^n}{min}\frac{{\mathbf{x}}^{⊺}\mathbf{A}\mathbf{x}}{{\Vert \mathbf{x}\Vert }_2^2}\end{array}$$

• For any analytic function $f$, we have $$f(\mathbf{A})=\sum _{i=1}^{n}f({\lambda }_i){\mathbf{v}}_i{\mathbf{v}}_i^{⊺}$$

System of symmetric definite equations

• Consider the system $\mathbf{y}=\mathbf{A}\mathbf{x}$ with $\mathbf{A}$ symmetric positive definite

• Let $\{{\mathbf{v}}_i\}$ be the eigenvectors of $\mathbf{A}$. $$\mathbf{x}=\sum _{i=1}^n\frac{1}{{\lambda }_i}{⟨\mathbf{y},{\mathbf{v}}_i⟩}_{}{\mathbf{v}}_i$$

• Assume some observation error $\mathbf{y}=\mathbf{A}\mathbf{x}+\mathbf{e}$, with $\mathbf{e}$ unknown, and we reconstruct $\mathbf{x}$ as $\tilde{\mathbf{x}}$ by applying ${\mathbf{A}}^{-1}$

• $$\frac{1}{{\lambda }_1^2}{\Vert \mathbf{e}\Vert }_2^2\le {\Vert \mathbf{x}-\tilde{\mathbf{x}}\Vert }_2\le \frac{1}{{\lambda }_n^2}{\Vert \mathbf{e}\Vert }_2^2.$$
• If $\mathbf{e}\sim \mathcal{N}(\mathbf{0},{\sigma }^2\mathbf{I})$, then $${\mathbb{E}}_{}\left[{\Vert \mathbf{x}-\tilde{\mathbf{x}}\Vert }_2\right]={\sigma }^2\sum _{i=1}^n\frac{1}{{\lambda }_i^2}$$

Singular value decomposition

Important properties of the SVD

• The columns of $\mathbf{V}$ ${\{{\mathbf{v}}_i\}}_{i=1}^r$ are eigenvectors of the psd matrix ${\mathbf{A}}^{⊺}\mathbf{A}$. $\{{\sigma }_i:1\le i\le n\text{ and }{\sigma }_i\ne 0\}$ are the square roots of the non-zero eigenvalues of ${\mathbf{A}}^{⊺}\mathbf{A}$.

• The columns of $\mathbf{U}$ ${\{{\mathbf{u}}_i\}}_{i=1}^r$ are eigenvectors of the psd matrix $\mathbf{A}{\mathbf{A}}^{⊺}$. $\{{\sigma }_i:1\le i\le n\text{ and }{\sigma }_i\ne 0\}$ are the square roots of the non-zero eigenvalues of $\mathbf{A}{\mathbf{A}}^{⊺}$.

• The columns of $\mathbf{V}$ form an orthobasis for $\text{row}(\mathbf{A})$

• The columns of $\mathbf{U}$ form an orthobasis for $\text{col}(\mathbf{A})$

• Equivalent form of the SVD: $\mathbf{A}=\tilde{\mathbf{U}}\tilde{\mathbf{\Sigma }}{\tilde{\mathbf{V}}}^T$ where

  • $\tilde{\mathbf{U}}\in {\mathbb{R}}^{m\times m}$ is orthonormal
  • $\tilde{\mathbf{V}}\in {\mathbb{R}}^{n\times n}$ is orthonormal
  • $\tilde{\mathbf{\Sigma }}\in {\mathbb{R}}^{m\times n}$ is $$\tilde{\mathbf{\Sigma }}\triangleq \left[\begin{array}{cc}\mathbf{\Sigma }& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right]$$

SVD and least-squares

• When we cannot solve $\mathbf{y}=\mathbf{A}\mathbf{x}$, we solve instead $$\underset{\mathbf{x}\in {\mathbb{R}}^n}{min}{\Vert \mathbf{x}\Vert }_2^2\text{ such that }{\mathbf{A}}^{⊺}\mathbf{A}\mathbf{x}={\mathbf{A}}^{⊺}\mathbf{y}$$

  • This allows us to pick the minimum norm solution among potentially infinitely many solutions of the normal equations.

• Recall: when $\mathbf{A}\in {\mathbb{R}}^{m\times n}$ is of rank $n$, then $\mathbf{x}={\mathbf{A}}^{⊺}(\mathbf{A}{\mathbf{A}}^{⊺}{)}^{-1}\mathbf{y}$

• The solution of $$\underset{\mathbf{x}\in {\mathbb{R}}^n}{min}{\Vert \mathbf{x}\Vert }_2^2\text{ such that }{\mathbf{A}}^{⊺}\mathbf{A}\mathbf{x}={\mathbf{A}}^{⊺}\mathbf{y}$$ is $$\hat{\mathbf{x}}=\mathbf{V}{\mathbf{\Sigma }}^{-1}{\mathbf{U}}^{⊺}\mathbf{y}$$ where $\mathbf{A}=\mathbf{U}\mathbf{\Sigma }{\mathbf{V}}^T$ is the SVD of $\mathbf{A}$.

Pseudo inverse

• ${\mathbf{A}}^{+}=\mathbf{V}{\mathbf{\Sigma }}^{-1}{\mathbf{U}}^{⊺}$ is called the pseudo-inverse, Lanczos inverse, or Moore-Penrose inverse of $\mathbf{A}=\mathbf{U}\mathbf{\Sigma }{\mathbf{V}}^T$.

• If $\mathbf{A}$ is square invertible then ${\mathbf{A}}^{+}=\mathbf{A}$

• If $m\ge n$ (tall and skinny matrix) of rank $n$ then ${\mathbf{A}}^{+}=({\mathbf{A}}^{⊺}\mathbf{A}{)}^{-1}{\mathbf{A}}^{⊺}$

• If $m\ge m$ (short and fat matrix) of rank $m$ then ${\mathbf{A}}^{+}={\mathbf{A}}^{⊺}(\mathbf{A}{\mathbf{A}}^{⊺}{)}^{-1}$

• Note ${\mathbf{A}}^{+}$ is as “close” to an inverse of $\mathbf{A}$ as possible