Prof. Matthieu Bloch
Monday, November 25, 2024
For a probabilistic model governing the distributions of samples \(\set{x_i}_{i=1}^n\), the likelihood function is \[ L(\theta;x_1,\cdots,x_n)\eqdef p_{X_1\cdots X_n}(x_1,\cdots,x_n;\theta). \] It is often convenient to work with a log-likelihood \(\ell(\theta;x_1,\cdots,x_n)=\log L(\theta;x_1,\cdots,x_n)\).
\[ \hat{\theta}_{\textnormal{MLE}} \eqdef \argmax_{\theta}L(\theta;x_1,\cdots,x_n) \]
An estimator \(\hat{\theta}\) of \(\theta_0\in\calT\) has bias \(\E{\hat{\theta}}-\theta_0\). The estimator is unbiased if the bias is zero for all \(\theta_0\in\calT\)
An estimator \(\hat{\theta}_n\) of \(\theta_0\in\calT\) using \(n\) observations \(x_1,\cdots,x_n\) is consistent if for every \(\epsilon>0\) and \(\delta\in(0;1)\) \[ \lim_{n\to\infty}\P{\abs{\hat{\theta}_n-\theta_0}>\epsilon}\leq \delta. \]