Regression

Assignment 4 assigned Tuesday, October 5, 2021

• Includes a (small) programming component

• Due October 14, 2021 (soft deadline, hard deadline on October 16)

Last time: Least-square regression

Today

• Solving linear least-square regression

• Extension to infinite dimension

Reading: Romberg, lecture notes 8

Facts: for any matrix $\mathbf{A}\in {\mathbb{R}}^{m\times n}$

• $\ker {\mathbf{A}}^{⊺}\mathbf{A}=\ker \mathbf{A}$

• $\text{col}({\mathbf{A}}^{⊺}\mathbf{A})=\text{row}(\mathbf{A})$

• $\text{row}(\mathbf{A})$ and $\ker \mathbf{A}$ are orthogonal complements

We can say a lot more about the normal equations

1. There is always a solution
2. If $\mathsf{\text{rank}}(\mathbf{X})=d$, there is a unique solution: $({\mathbf{A}}^{⊺}\mathbf{A}{)}^{-1}{\mathbf{A}}^{⊺}\mathbf{y}$
3. if $\mathsf{\text{rank}}(\mathbf{X})<d$ there are infinitely many non-trivial solution
4. if $\mathsf{\text{rank}}(\mathbf{X})=n$, there exists a solution ${\mathit{\theta }}^{\ast }$ for which $\mathbf{y}=\mathbf{X}{\mathit{\theta }}^{\ast }$

In machine learning, there are often infinitely many solutions

One reasonable to choose a solution among infinitely many is the minimum energy principle $$\underset{\mathit{\theta }\in {\mathbb{R}}^d}{min}{\Vert \mathit{\theta }\Vert }_2^2\text{ such that }{\mathbf{X}}^{⊺}\mathbf{X}\mathit{\theta }={\mathbf{X}}^{⊺}\mathbf{y}$$

• We will see the solution is always unique using the SVD

For now, assume that $\mathsf{\text{rank}}(\mathbf{X})=d$, so that the problem becomes $$\underset{\mathit{\theta }\in {\mathbb{R}}^d}{min}{\Vert \mathit{\theta }\Vert }_2^2\text{ such that }\mathbf{X}\mathit{\theta }=\mathbf{y}$$

• There are infinitely many solution if $\ker \mathbf{X}$ is non trivial
• The space of solution is unbounded!
• Even if $\ker \mathbf{X}=\{0\}$, the system can be poorly conditioned

• This problem always has a unique solution