The Mathematics of ECE

Logistics

Housekeeping
- Piazza for course: sign-up
- Slides available here
Today’s class: Probability & Statistics
- Key concepts: conditional probability and independence, random variables, distributions
Next workshop: Wednesday September 14, 2022

The expectation $E_{\cdot} [\cdot]$ of a function $g$ of a random variable $X$ is defined by it’s distribution $E_{X} [g (X)] = \int_{x} f_{X} (x) g (x) d x$ .
- (Technicality: law of the unconscious statistician)
Properties:
- $E_{X, Y} [X + Y] = E_{X} [X] + E_{Y} [Y]$
- $(X, Y) indep. ⟹ E_{X, Y} [X Y] = E_{X} [X] E_{Y} [Y]$
- $E_{Y} [E_{X | Y} [g (X, Y)]] = E_{X, Y} [g (X, Y)]$
  - $E_{Y} [E_{X | Y} [X]] = E_{X} [X]$ (law of total expectation)
The mean $μ_{X}$ of a random variable $X$ is defined as $μ_{X} = E_{} [X]$ .
The variance $σ_{X}^{2}$ of a random variable $X$ is defined as $σ_{X}^{2} = {Var}_{X} (X) = E_{X} [(X - μ_{X})^{2}]$ .
Drill: Prove that the variance may also be represented as $E_{} [X^{2}] - {E_{} [X]}^{2}$ .
Drill: Recall the uniform distribution has pdf $f$ defined on $[0, 1]$ by $f (x) = 1$ . For a uniformly distributed random variable $X$ , compute $μ_{X} = E_{X} [X]$ and ${Var}_{X} (X)$ .

Problem: We want to learn the realization of some random vector $X \in R^{n}$ from a dependent random vector $Y \in R^{m}$ .
- A common model for this situation is $Y = A X + E$ where $A \in R^{m \times n}$ is a constant, known matrix and $E$ is some random noise.
The mean squared error (MSE) of a deterministic estimator $\hat{x} : R^{m} \to R^{n}$ of random variable $X \in R^{n}$ from observation $Y \in R^{m}$ is defined as $MSE (\hat{x}) = E_{X, Y} [| | X - \hat{x} (Y) | |_{2}^{2}]$ The minimum mean square estimator (MMSE) ${\hat{x}}_{MMSE}$ is then defined as ${\hat{x}}_{MMSE} = \underset{\hat{x}}{argmin} MSE (\hat{x})$ with solution ${\hat{x}}_{MMSE} (y) = E_{X | Y} [X | y]$

If one can compute $E_{X | Y} [X | y]$ for arbitrary $y$ , then we have our ideal (in the MMSE sense) estimator.
- Example: $[X; Y]$ is a zero-mean Gaussian random vector with (block) covariance matrix $[\begin{matrix} E_{X} [X X^{T}] & E_{X, Y} [X Y^{T}] \\ E_{X, Y} [Y X^{T}] & E_{Y} [Y Y^{T}] \end{matrix}] = [\begin{matrix} R_{X X} & R_{X Y} \\ R_{X Y}^{T} & R_{Y Y} \end{matrix}]$ then the MMSE is ${\hat{x}}_{M M S E} (y) = R_{X Y} R_{Y Y}^{- 1} y$
- When we are not so lucky, what can we do…

A linear estimator $\hat{x} : R^{m} \to R^{n}$ is one of the form $\hat{x} (y) = K y + b$ The parameters of the linear minimum mean square estimator (LMMSE) ${\hat{x}}_{LMMSE}$ is then defined as $K_{LMMSE}, b_{LMMSE} = \underset{K, b}{argmin} MSE (\hat{x})$ with solution satisfying $K_{LMMSE} R_{Y Y} = R_{X Y}$ $b_{LMMSE} = μ_{X} - K μ_{Y}$ where $R_{X Y} = E_{X, Y} [X Y^{T}]$ and $R_{Y Y} = E_{Y} [Y Y^{T}]$
Drill: Further restrict the linear model above to be of the form $\hat{x} (y) = K y$ (equivalently, set $b = 0$ ). Prove that the $K$ matrix which solves the LMMSE problem in this case indeed satisfies $K R_{Y Y} = R_{X Y}$

The LMMSE problem could have been a linear algebra problem…
- Random variables form their own vector space
- The natural inner product on this vector space is $⟨ X, Y ⟩ = E_{X, Y} [X^{T} Y]$ .
- LLMSE problem becomes a least squares problem, can be solved with orthogonality principle.