Linear Models

Today in ECE 6555

• Don't forget
  • Problem set 2 due Thursday September 22, 2022 on Gradescope
• Announcements
  • Mathematics of ECE workshops website
  • Recordings available on mediaspace
• Last time
  • Linear estimators
  • Geometric perspective on stochastic least squares
• Today's plan
  • Linear models
  • Gauss Markov theorem
  • Sensor fusion
• Questions?

Linear models

• We can't say much more about LMSE without knowing more

• Fortunately, many engineering problems problems impose more structure between $\mathbf{y}$ and $\mathbf{x}$

• A linear model is one in which $\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{v}$ where:

  • $\mathbf{H}\in {\mathbb{R}}^{m\times n}$
  • $\mathbf{v}$ is zero mean an uncorrelated with $\mathbf{x}$

• The LLSE of $\mathbf{x}$ given $\mathbf{y}$ in a linear mode (assuming ${\mathbf{R}}_{\mathbf{x}}$ and ${\mathbf{R}}_{\mathbf{v}}$ non singular) is $\hat{\mathbf{x}}={\mathbf{K}}_0\mathbf{y}$ with $${\mathbf{K}}_0={\mathbf{R}}_{\mathbf{x}}{\mathbf{H}}^T(\mathbf{H}{\mathbf{R}}_{\mathbf{x}}{\mathbf{H}}^{⊺}+{\mathbf{R}}_{\mathbf{v}}{)}^{-1}=({\mathbf{R}}_{\mathbf{x}}^{-1}+{\mathbf{H}}^T{\mathbf{R}}_{\mathbf{v}}^{-1}\mathbf{H}{)}^{-1}{\mathbf{H}}^T{\mathbf{R}}_{\mathbf{v}}^{-1}$$ $${\mathbf{P}}_{\mathbf{x}}={\mathbf{R}}_{\mathbf{x}}-{\mathbf{R}}_{\mathbf{x}}{\mathbf{H}}^T(\mathbf{H}{\mathbf{R}}_{\mathbf{x}}{\mathbf{H}}^T+{\mathbf{R}}_{\mathbf{v}}{)}^{-1}\mathbf{H}{\mathbf{R}}_{\mathbf{x}}=({\mathbf{R}}_{\mathbf{x}}^{-1}+{\mathbf{H}}^T{\mathbf{R}}_{\mathbf{v}}^{-1}\mathbf{H}{)}^{-1}$$

• In particular note that ${\mathbf{P}}_{\mathbf{x}}^{-1}\hat{\mathbf{x}}={\mathbf{H}}^T{\mathbf{R}}_{\mathbf{v}}^{-1}\mathbf{y}$ (independent of ${\mathbf{R}}_{\mathbf{x}}$)

Gauss-Markov Theorem

• Sometimes we are interested in characterizing a deterministic quantity $\mathbf{x}$

• Consider the linear model $\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{v}$ where

  • $\mathbf{H}$ is full column rank
  • $\mathbf{x}\in {\mathbb{R}}^n$ is a deterministic parameter
  • $\mathbf{v}\in {\mathbb{R}}^m$ is zero mean with covariance matrix ${\mathbf{R}}_{\mathbf{v}}=\mathbf{I}$

  Then ${\hat{\mathbf{x}}}_{\mathrm{\infty }}\triangleq ({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{y}$ is the optimal unbiased llmse of $\mathbf{x}$

• Notes:

  • This looks like the LLSE for linear models with ${\mathbf{R}}_{\mathbf{x}}=\alpha \mathbf{I}$ and $\alpha \to \mathrm{\infty }$
  • What if ${\mathbf{R}}_{\mathbf{v}}\succ 0$ but ${\mathbf{R}}_{\mathbf{v}}\ne \mathbf{I}$?
  • We can view the result of the Gauss-Markov theorem as the solution of the optimization $$\underset{\mathbf{K}}{min}\mathbf{K}{\mathbf{K}}^T\mathsf{\text{ s.t. }}\mathbf{K}\mathbf{H}=\mathbf{I}$$

Combining estimators

• Consider a zero mean random variable $\mathbf{x}\in {\mathbb{R}}^n$ (${\mathbf{R}}_{\mathbf{x}}\succ 0$) observed through ${\mathbf{y}}_1$ and ${\mathbf{y}}_2$ according to $${\mathbf{y}}_1={\mathbf{H}}_1+{\mathbf{v}}_1{\mathbf{y}}_2={\mathbf{H}}_2+{\mathbf{v}}_2$$ with $(\mathbf{x},{\mathbf{v}}_1,{\mathbf{v}}_2)$ mutually independent and zero mean.

  • Let ${\hat{\mathbf{x}}}_1$ be the LLSE of $\mathbf{x}$ from ${\mathbf{y}}_1$ with error covariance matrix ${\mathbf{P}}_1$
  • Let ${\hat{\mathbf{x}}}_2$ be the LLSE of $\mathbf{x}$ from ${\mathbf{y}}_2$ with error covariance matrix ${\mathbf{P}}_2$

  Then ${\mathbf{P}}^{-1}\mathbf{x}={\mathbf{P}}_1^{-1}{\hat{\mathbf{x}}}_1+{\mathbf{P}}_2^{-1}{\hat{\mathbf{x}}}_2$ with ${\mathbf{P}}^{-1}={\mathbf{P}}_1^{-1}+{\mathbf{P}}_2^{-1}-{\mathbf{R}}_{\mathbf{x}}^{-1}$

Correspondence with deterministic least squares

• Consider the deterministic least square optimization $$\underset{\mathbf{x}}{min}(\mathbf{x}-{\mathbf{x}}_0{)}^T{\mathrm{\Pi }}_0^{-1}(\mathbf{x}-{\mathbf{x}}_0)+{\Vert \mathbf{y}-\mathbf{H}\mathbf{x}\Vert }_W^2$$

• Consider the linear stochastic least square optimization for the linear model $\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{v}$

• Can we draw parallels between the two problems?