Innovation Processes

Today in ECE 6555

• Don't forget
  • Problem set 2 due Thursday September 22, 2022 on Gradescope
  • Hard deadline extended to Monday September 26, 2022
  • Mathematics of ECE workshops (website)
  • Extra office hours today at 2pm in TSRB (live and recorded)
• Last time
  • Stochastic processes: smoothing, causal filtering, prediction
  • Wiener Hopf solution to causal filtering
• Today's plan
  • Clarification of causal filtering proof
  • Innovation process
• Questions?

Causal filtering

• Geometry strikes back…

• For ${\mathbf{R}}_{\mathbf{y}}\succ 0$ decomposed as ${\mathbf{R}}_{\mathbf{y}}=\mathbf{L}\mathbf{D}{\mathbf{L}}^T$ ($\mathbf{L}$ lower triangular) $${\hat{\mathbf{x}}}_f=\mathcal{L}\left[{\mathbf{R}}_{\mathbf{x}\mathbf{y}}{\mathbf{L}}^T{\mathbf{D}}^{-1}\right]{\mathbf{L}}^{-1}\mathbf{y}$$ where $${\hat{\mathbf{x}}}_f\triangleq \left[\begin{array}{c}{\hat{x}}_{0|0}\\ {\hat{x}}_{1|1}\\ ⋮\\ {\hat{x}}_{m|m}\end{array}\right]{\mathbf{R}}_{\mathbf{y}}\triangleq [{\mathbf{R}}_y(i,j)]{\mathbf{R}}_{\mathbf{x}\mathbf{y}}\triangleq [{\mathbf{R}}_{xy}(i,j)]$$ and $\mathcal{L}[\cdot ]$ is the operator that makes a matrix lower triangular.

Smoothing vs. Causal filtering

• Example: Linear model $\mathbf{y}=\mathbf{x}+\mathbf{v}$ with ${\mathbb{E}}_{}\left[\mathbf{x}{\mathbf{x}}^T\right]\triangleq {\mathbf{R}}_x$, ${\mathbb{E}}_{}\left[\mathbf{v}{\mathbf{v}}^T\right]\triangleq {\mathbf{R}}_v$, ${\mathbb{E}}_{}\left[\mathbf{x}{\mathbf{v}}^T\right]\triangleq 0$
  • Can we compare the smoothing and causal filtering filters?

• Let ${\mathbf{K}}_s$ denote the smoothing linear estimator, let ${\mathbf{K}}_f$ denote the causal filtering linear estimator. $${\mathbf{K}}_s={\mathbf{K}}_f+\mathcal{SU}[{\mathbf{R}}_y{L}^{-T}{D}^{-1}]L^{-1}$$ where $\mathcal{SU}$ denote the strict upper triangularization operator.

Innovation processes

• Back to normal equations ${\mathbf{K}}_0{\mathbf{R}}_y={\mathbf{R}}_y$
  • A key difficulty we create is that ${\mathbf{R}}_y$ need to be inverted (esp. for causal filtering)
  • Would be easier if ${\mathbf{R}}_y$ were diagonal (which in general it has no reason to do)
• Geometric approach to simplify dealing with normal equations
  • The normal equations are obtained by projecting onto a subspace
  • We are not bound to use ${\{{\mathbf{y}}_i\}}_{i=0}^m$: we can orthogonalize!
  • Gram-Schmidt orthogonalization for random variables $${\mathbf{e}}_0={\mathbf{y}}_0\mathrm{\forall }i\ge 1{\mathbf{e}}_i={\mathbf{y}}_i-\underset{{\hat{\mathbf{y}}}_i}{\underbrace{\sum _{j=0}^{i-1}{⟨{\mathbf{y}}_i,{\mathbf{e}}_j⟩}_{}{\Vert {\mathbf{e}}_j\Vert }_{}^{-2}{\mathbf{e}}_j}}$$

• The random variable ${\mathbf{e}}_i\triangleq {\mathbf{y}}_i-{\hat{\mathbf{y}}}_i$ is called the innovation

• There is an invertible causal relation ship between $\{{\mathbf{y}}_i\}$ and $\{{\mathbf{e}}_i\}$

Innovation processes

• Algebraic approach to simplify dealing with normal equations

  • ${\mathbf{R}}_y$ has not reason to be diagonal, but can we "whiten" it with a linear operation? $$\mathsf{\text{Find }}\mathbf{A}\mathsf{\text{ such that }}ϵ=\mathbf{A}\mathbf{t}\mathsf{\text{ has covariance }}{\mathbf{R}}_{ϵ}=A{\mathbf{R}}_y{\mathbf{A}}^T=\mathbf{D}$$

  • $\mathbf{A}$ should be non singular to avoid losing information

  • Many solutions unless we impose more constraints

  • Required causal relationship: $\mathbf{A}=\mathbf{L}$, lower triangular, and impose ${\mathbf{R}}_y=\mathbf{L}\mathbf{D}{\mathbf{L}}^T$

  • LDL decomposition is unique for positive semi definite matrices

• Compare geometric and algebraic approaches

Applications of innovation processes

• Estimation with innovation

• Causal filtering with innovation

• Example: innovations for exponentially correlated process