Deterministic Least Squares
Dr. Matthieu R Bloch
Thursday, September 1, 2022
Today in ECE 6555
- Don’t forget
- Problem set 1 posted and due Thursday September 8, 2022 on
Gradescope
- Check out the self assessment (solutions posted)
- Announcements
- Mathematics of ECE workshops (second session on linear algebra on
Friday September 02, 2022)
- Last time
- Normal equations and geometric approach
- Today’s plan
- Geometry in action
- Deterministic least squares
- Why? Geometric intuition is incredibly powerful and will
carry over to more complex settings
- Questions?
Geometry in action
Consider a full rank matrix \(\matH\in\bbR^{m\times n}\) with \(m\gg n\)
Let \(\hat{\vecx}^{(n)}\) be the
least-square solution of \(\vecy\approx
\matH\vecx\)
Assume we get one more input (preserving full rank) so that we
want to solve \[
\vecy\approx \left[\begin{array}{cc}\matH
&\vech_{n+1}\end{array}\right]
\left[\begin{array}{c}\vecx\\x_{n+1}\end{array}\right]
\]
- Can we do this efficiently without recomputing everything?
Variations on a theme
There are many variations on the least square minimization
problem
Weighted least squares \[
J(\vecx)\eqdef \norm[\matW]{\vecy-\matH\vecx}^2\eqdef
(\vecy-\matH\vecx)^T\matW(\vecy-\matH\vecx)
\] for some symmetric positive definite matrix \(\matW\)
Regularized least squares \[
J(\vecx)\eqdef (\vecy-\matH\vecx)^T\matW(\vecy-\matH\vecx) +
(\vecx-\vecx_0)^T\mathbf{\Pi}(\vecx-\vecx_0)
\] for some symmetric positive definite matrices \(\matW\) and \(\mathbf{\Pi}\)
Recursive least squares
- Suppose at step \(i-1\) we have
solved the LS problem \(\matH_{i-1}\approx
\vecy_{i-1}\) with \[
\matH_{i-1}=\left[\begin{array}{ccc}-&\vech_0^T&-\\&\vdots&\\-&\vech_{i-1}^T&-\end{array}\right]\qquad
\vecy_{i-1}=\left[\begin{array}{c}y_0\\\vdots\\y_{i-1}\end{array}\right]
\]
- This is different (and more interesting) than the order recursive
least squares
- We now want to solve \(\matH_{i}\approx
\vecy_{i}\) with \[
\matH_{i}=\left[\begin{array}{ccc}&\matH_{i-1}&\\-&\vech_i^T&-\end{array}\right]\qquad
\vecy_{i-1}=\left[\begin{array}{c}\vecy_{i-1}\\y_{i}\end{array}\right]
\]
- Can we do this efficiently?