Linear Models

Today in ECE 6555

• Don't forget
  • Problem set 2 due Thursday September 22, 2022 on Gradescope
• Announcements
  • Mathematics of ECE workshops website
  • Recordings available on mediaspace
• Last time
  • Linear estimators
  • Geometric perspective on stochastic least squares
• Today's plan
  • Linear models
  • Gauss Markov theorem
  • Sensor fusion
• Questions?

Linear models

• We can't say much more about LMSE without knowing more

• Fortunately, many engineering problems problems impose more structure between \(\vecy\) and \(\vecx\)

• A linear model is one in which \(\vecy=\matH\vecx+\vecv\) where:

  • \(\matH\in\bbR^{m\times n}\)
  • \(\vecv\) is zero mean an uncorrelated with \(\vecx\)

• The LLSE of \(\vecx\) given \(\vecy\) in a linear mode (assuming \(\matR_\vecx\) and \(\matR_\vecv\) non singular) is \(\hat{\vecx}=\matK_0\vecy\) with \[ \matK_0=\matR_\vecx\matH^T(\matH\matR_\vecx\matH^\intercal + \matR_\vecv)^{-1} = (\matR_\vecx^{-1}+\matH^T\matR_\vecv^{-1}\matH)^{-1}\matH^T\matR_\vecv^{-1} \] \[ \matP_\vecx = \matR_\vecx -\matR_\vecx\matH^T(\matH\matR_\vecx\matH^T+\matR_\vecv)^{-1}\matH\matR_\vecx = (\matR_\vecx^{-1}+\matH^T\matR_\vecv^{-1}\matH)^{-1} \]

• In particular note that \(\matP_{\vecx}^{-1}\hat{\vecx} = \matH^T\matR_\vecv^{-1}\vecy\) (independent of \(\matR_\vecx\))

Gauss-Markov Theorem

• Sometimes we are interested in characterizing a deterministic quantity \(\vecx\)

• Consider the linear model \(\vecy=\matH\vecx+\vecv\) where

  • \(\matH\) is full column rank
  • \(\vecx\in\bbR^n\) is a deterministic parameter
  • \(\vecv\in\bbR^m\) is zero mean with covariance matrix \(\matR_\vecv=\matI\)

  Then \(\hat{\vecx}_\infty\eqdef (\matH^T\matH)^{-1}\matH^T\vecy\) is the optimal unbiased llmse of \(\vecx\)

• Notes:

  • This looks like the LLSE for linear models with \(\matR_\vecx = \alpha \matI\) and \(\alpha\to\infty\)
  • What if \(\matR_\vecv\succ 0\) but \(\matR_{\vecv}\neq \matI\)?
  • We can view the result of the Gauss-Markov theorem as the solution of the optimization \[ \min_{\matK} \matK\matK^T\textsf{ s.t. }\matK\matH=\matI \]

Combining estimators

• Consider a zero mean random variable \(\vecx\in\bbR^n\) (\(\matR_\vecx\succ 0\)) observed through \(\vecy_1\) and \(\vecy_2\) according to \[ \vecy_1 = \matH_1+\vecv_1\qquad \vecy_2 = \matH_2+\vecv_2 \] with \((\vecx,\vecv_1,\vecv_2)\) mutually independent and zero mean.

  • Let \(\hat{\vecx}_1\) be the LLSE of \(\vecx\) from \(\vecy_1\) with error covariance matrix \(\matP_1\)
  • Let \(\hat{\vecx}_2\) be the LLSE of \(\vecx\) from \(\vecy_2\) with error covariance matrix \(\matP_2\)

  Then \(\matP^{-1}\vecx = \matP_1^{-1}\hat{\vecx}_1+\matP_2^{-1}\hat{\vecx}_2\) with \(\matP^{-1} = \matP_1^{-1}+\matP_2^{-1}-\matR_\vecx^{-1}\)

Correspondence with deterministic least squares

• Consider the deterministic least square optimization \[ \min_{\vecx} (\vecx-\vecx_0)^T\Pi_0^{-1}(\vecx-\vecx_0) + \norm[W]{\vecy-\matH\vecx}^2 \]

• Consider the linear stochastic least square optimization for the linear model \(\vecy=\matH\vecx+\vecv\)

• Can we draw parallels between the two problems?