The Mathematics of ECE

Probability & Statistics - Lecture 2

Monday, September 12, 2022

Logistics

  • Housekeeping
  • Today’s class: Probability & Statistics
    • Key concepts: conditional probability and independence, random variables, distributions
  • Next workshop: Wednesday September 14, 2022

Expectations

  • The expectation \(\E[\cdot]{\cdot}\) of a function \(g\) of a random variable \(X\) is defined by it’s distribution \(\E[X]{g(X)} = \int_x f_X(x) g(x) dx\).

    • (Technicality: law of the unconscious statistician)

  • Properties:

    • \(\E[X,Y]{X + Y} = \E[X]{X} + \E[Y]{Y}\)
    • \((X, Y) \text{ indep. } \implies \E[X,Y]{X Y} = \E[X]{X} \E[Y]{Y}\)
    • \(\E[Y]{\E[X|Y]{g(X,Y)}} = \E[X,Y]{g(X,Y)}\)
      • \(\E[Y]{\E[X|Y]{X}} = \E[X]{X}\) (law of total expectation)

  • The mean \(\mu_X\) of a random variable \(X\) is defined as \(\mu_X = \E{X}\).

  • The variance \(\sigma^2_X\) of a random variable \(X\) is defined as \(\sigma^2_X = \Var[X]{X} = \E[X]{(X - \mu_X)^2}\).

  • Drill: Prove that the variance may also be represented as \(\E{X^2} - \E{X}^2\).

  • Drill: Recall the uniform distribution has pdf \(f\) defined on \([0,1]\) by \(f(x) = 1\). For a uniformly distributed random variable \(X\), compute \(\mu_X = \E[X]{X}\) and \(\Var[X]{X}\).

Estimation (MMSE)

  • Problem: We want to learn the realization of some random vector \(X \in \bbR^n\) from a dependent random vector \(Y \in \bbR^m\).
    • A common model for this situation is \(Y = A X + E\) where \(A \in \bbR^{m \times n}\) is a constant, known matrix and \(E\) is some random noise.

  • The mean squared error (MSE) of a deterministic estimator \(\hat{x} : \bbR^m \to \bbR^n\) of random variable \(X \in \bbR^n\) from observation \(Y \in \bbR^m\) is defined as \[ \text{MSE}(\hat{x}) = \E[X,Y]{||X - \hat{x}(Y)||_2^2} \] The minimum mean square estimator (MMSE) \(\hat{x}_{\text{MMSE}}\) is then defined as \[ \hat{x}_{\text{MMSE}} = \underset{\hat{x}}{\text{argmin}} \text{MSE}(\hat{x}) \] with solution \[ \hat{x}_{\text{MMSE}}(y) = \E[X|Y]{X | y} \]

Estimation (MMSE)

  • If one can compute \(\E[X|Y]{X | y}\) for arbitrary \(y\), then we have our ideal (in the MMSE sense) estimator.
    • Example: \([X; Y]\) is a zero-mean Gaussian random vector with (block) covariance matrix \[ \begin{bmatrix} \E[X]{X X^T} & \E[X,Y]{X Y^T} \\ \E[X,Y]{Y X^T} & \E[Y]{Y Y^T} \end{bmatrix} = \begin{bmatrix} R_{XX} & R_{XY} \\ R_{XY}^T & R_{YY} \end{bmatrix}\] then the MMSE is \[ \hat{x}_{MMSE}(y) = R_{XY} R_{YY}^{-1} y \]
    • When we are not so lucky, what can we do…

Estimation (LMMSE)

  • A linear estimator \(\hat{x} : \bbR^m \to \bbR^n\) is one of the form \[ \hat{x}(y) = K y + b \] The parameters of the linear minimum mean square estimator (LMMSE) \(\hat{x}_{\text{LMMSE}}\) is then defined as \[ K_{\text{LMMSE}}, b_{\text{LMMSE}} = \underset{K, b}{\text{argmin}} \text{MSE}(\hat{x}) \] with solution satisfying \[ K_{\text{LMMSE}} R_{YY} = R_{XY} \] \[ b_{\text{LMMSE}} = \mu_X - K \mu_Y \] where \(R_{XY} = \E[X,Y]{X Y^T}\) and \(R_{YY} = \E[Y]{Y Y^T}\)

  • Drill: Further restrict the linear model above to be of the form \(\hat{x}(y) = K y\) (equivalently, set \(b = 0\)). Prove that the \(K\) matrix which solves the LMMSE problem in this case indeed satisfies \(K R_{YY} = R_{XY}\)

Estimation (LMMSE)

  • The LMMSE problem could have been a linear algebra problem…
    • Random variables form their own vector space
    • The natural inner product on this vector space is \(\langle X, Y \rangle = \E[X,Y]{X^T Y}\).
    • LLMSE problem becomes a least squares problem, can be solved with orthogonality principle.