Innovation Processes / State Space Models

Today in ECE 6555

• Don't forget
  • Problem set 2 due Wednesday September 22, 2022 on Gradescope (hard deadline extended)
  • Office hours today at 12pm in TSRB (live and recorded)
  • Make sure you start homework early
  • Don't spend 30 hours on a homework
• Last time
  • Innovation process
• Today's plan
  • Innovation process
  • State space model
• Questions?

Recall: Innovation processes

• Back to normal equations ${\mathbf{K}}_0{\mathbf{R}}_y={\mathbf{R}}_y$
  • A key difficulty we create is that ${\mathbf{R}}_y$ need to be inverted (esp. for causal filtering)
  • Would be easier if ${\mathbf{R}}_y$ were diagonal (which in general it has no reason to do)
• Geometric approach to simplify dealing with normal equations
  • The normal equations are obtained by projecting onto a subspace
  • We are not bound to use ${\{{\mathbf{y}}_i\}}_{i=0}^m$: we can orthogonalize!
  • Gram-Schmidt orthogonalization for random variables $${\mathbf{e}}_0={\mathbf{y}}_0\mathrm{\forall }i\ge 1{\mathbf{e}}_i={\mathbf{y}}_i-\underset{{\hat{\mathbf{y}}}_i}{\underbrace{\sum _{j=0}^{i-1}{⟨{\mathbf{y}}_i,{\mathbf{e}}_j⟩}_{}{\Vert {\mathbf{e}}_j\Vert }_{}^{-2}{\mathbf{e}}_j}}$$

• The random variable ${\mathbf{e}}_i\triangleq {\mathbf{y}}_i-{\hat{\mathbf{y}}}_i$ is called the innovation

• There is an invertible causal relation ship between $\{{\mathbf{y}}_i\}$ and $\{{\mathbf{e}}_i\}$

Applications of innovation processes

• Causal filtering with innovation

• Example: innovations for exponentially correlated process

State Space Models: Motivation

• Consider the linear stochastic differential equations ${\mathbf{y}}_{i+1}-a{\mathbf{y}}_i={\mathbf{u}}_i$ for $i\ge 0$, $a\in \mathbb{R}$

  • Initial conditions ${\mathbf{y}}_0$
  • ${\mathbf{u}}_i$ such that ${⟨{\mathbf{u}}_i,{\mathbf{u}}_j⟩}_{}={\mathbf{Q}}_i{\delta }_{ij}$, ${\Vert {\mathbf{y}}_0\Vert }_{}^2={\mathrm{\Pi }}_0$, ${⟨{\mathbf{u}}_i,{\mathbf{y}}_0⟩}_{}=0$
  • The process may be non-stationary

• $\mathrm{\forall }i\ge j$ we have ${⟨{\mathbf{u}}_i,{\mathbf{y}}_j⟩}_{}=0$

  Let ${\mathrm{\Pi }}_i\triangleq {\Vert {\mathbf{y}}_i\Vert }_{}^2$; then ${\mathrm{\Pi }}_{i+1}=a^2{\mathrm{\Pi }}_i+{\mathbf{Q}}_i$

• In general the process is not stationary

• If ${\mathbf{Q}}_i=\mathbf{Q}$, ${\mathrm{\Pi }}_0=\frac{\mathbf{Q}}{1-a^2}$ with $\left|a\right|<1$, the process is stationary

• Innovations can be computed regardless of stationarity

• State space models allow us quite a bit of generality

• Does the slimplicity extend beyond that simple example?

Higher order models

• An autoregressive process is defined by the stochastic equation $$y_{i+1}=a_{0,i}{y}_i+a_{1,i}{y}_{i-1}+\cdots +a_{n-1,i}{y}_{i-n+1}+u_i\mathrm{\forall }i\ge 0$$ where:

  • $u_i$ is zero mean with ${⟨u_i,u_j⟩}_{}=Q_i{\delta }_{i,j}$ $\mathrm{\forall }i\ge 0$
  • $y_0,\cdots ,y_{-n+1}$ zero mean with know covariance matrix ${\mathrm{\Pi }}_0$

• The autoregressive model can be represented as a simple order 1 recursion of the form $${\mathbf{x}}_{i+1}={\mathbf{F}}_i{\mathbf{x}}_i+{\mathbf{G}}_i{\mathbf{u}}_i{y}_i={\mathbf{H}}_i{\mathbf{x}}_i$$

• An autoregressive moving-average process is defined by the stochastic equation $$y_{i+1}=a_0{y}_i+a_1{y}_{i-1}+\cdots +a_{n-1}{y}_{i-n+1}+b_0{u}_i+\cdots +b_{n-1}{u}_{i-n+1}\mathrm{\forall }i\ge 0$$ where:

  • $u_i$ is zero mean with ${⟨u_i,u_j⟩}_{}=Q_i{\delta }_{i,j}$ $\mathrm{\forall }i\ge 0$
  • $y_0,\cdots ,y_{-n+1}$ zero mean with know covariance matrix ${\mathrm{\Pi }}_0$

• The autoregressive moving-average model can also be represented as a simple order 1 recursion

Standard state space model