Deterministic Least Squares

Dr. Matthieu R Bloch

Thursday, September 1, 2022

Today in ECE 6555

  • Don’t forget
    1. Problem set 1 posted and due Thursday September 8, 2022 on Gradescope
    2. Check out the self assessment (solutions posted)
  • Announcements
    • Mathematics of ECE workshops (second session on linear algebra on Friday September 02, 2022)
  • Last time
    • Normal equations and geometric approach
  • Today’s plan
    • Geometry in action
    • Deterministic least squares
    • Why? Geometric intuition is incredibly powerful and will carry over to more complex settings
  • Questions?

Geometry in action

    • Consider a full rank matrix \(\matH\in\bbR^{m\times n}\) with \(m\gg n\)

    • Let \(\hat{\vecx}^{(n)}\) be the least-square solution of \(\vecy\approx \matH\vecx\)

    • Assume we get one more input (preserving full rank) so that we want to solve \[ \vecy\approx \left[\begin{array}{cc}\matH &\vech_{n+1}\end{array}\right] \left[\begin{array}{c}\vecx\\x_{n+1}\end{array}\right] \]

      • Can we do this efficiently without recomputing everything?

Variations on a theme

  • There are many variations on the least square minimization problem

  • Weighted least squares \[ J(\vecx)\eqdef \norm[\matW]{\vecy-\matH\vecx}^2\eqdef (\vecy-\matH\vecx)^T\matW(\vecy-\matH\vecx) \] for some symmetric positive definite matrix \(\matW\)

  • Regularized least squares \[ J(\vecx)\eqdef (\vecy-\matH\vecx)^T\matW(\vecy-\matH\vecx) + (\vecx-\vecx_0)^T\mathbf{\Pi}(\vecx-\vecx_0) \] for some symmetric positive definite matrices \(\matW\) and \(\mathbf{\Pi}\)

Recursive least squares

    • Suppose at step \(i-1\) we have solved the LS problem \(\matH_{i-1}\approx \vecy_{i-1}\) with \[ \matH_{i-1}=\left[\begin{array}{ccc}-&\vech_0^T&-\\&\vdots&\\-&\vech_{i-1}^T&-\end{array}\right]\qquad \vecy_{i-1}=\left[\begin{array}{c}y_0\\\vdots\\y_{i-1}\end{array}\right] \]
    • This is different (and more interesting) than the order recursive least squares
    • We now want to solve \(\matH_{i}\approx \vecy_{i}\) with \[ \matH_{i}=\left[\begin{array}{ccc}&\matH_{i-1}&\\-&\vech_i^T&-\end{array}\right]\qquad \vecy_{i-1}=\left[\begin{array}{c}\vecy_{i-1}\\y_{i}\end{array}\right] \]
    • Can we do this efficiently?