MSE(Mean Squared Error)

$MSE(\hat{\theta})=E(\hat{\theta}-\theta)^2$

Bias

$bias(\hat{\theta})=E(\hat{\theta})-\theta$

• Some facts
  ① $MSE(\hat{\theta})=V(\hat{\theta})+bias^2(\hat{\theta})$
  pf)
  
  ② If $\hat{\theta}$ : unbiased, $MSE(\hat{\theta})=V(\hat{\theta})$

ex1) $X_1,...,X_n \overset{iid}{\sim} N(\mu,\sigma^2)$

MVUE(Minimum Variance Unbiased Estimator)

Let $C_\tau= \{ \hat{\theta}: E(\hat{\theta})=\tau(\theta) \}$

Def)
$\hat{\theta}^*$ is called MVUE if $\forall \hat{\theta} \in C_\tau, V(\hat{\theta}^*) \leq V(\hat{\theta})$
$\hat{\theta}^*$ is called UMVUE if $\forall \theta \in \Omega, \hat{\theta}^*$ is MVUE

Theorem) Cramer-Rao inequality
Let $X_1,...,X_n \overset{iid}{\sim} f(x;\theta), \hat{\theta} \in C_\tau$
If $\frac{d}{d \theta} \int f(x;\theta) dx= \int \frac{d}{d \theta} f(x;\theta) dx$ and $V(\hat{\theta})<\infty$, then
$V(\hat{\theta}) \geq \frac{(\tau'(\theta))^2}{V(\frac{d}{d \theta}log f(\underline{x};\theta))}$
pf)

✔︎ For iid case,
$I_n(\theta)=E[(\frac{d}{d \theta} log f(\underline{x};\theta))^2]=E[(\sum_i\frac{d}{d \theta} log f(x_1; \theta))^2]=nI(\theta)$

Lemma)
Let $s(x;\theta)=\frac{d}{d\theta} log f(x;\theta)$
If $\frac{d}{d \theta} \int \frac{d}{d \theta} f(x;\theta) dx= \int \frac{d^2}{d \theta^2} f(x;\theta) dx,$ then
$V(s(x_1;\theta))=E[(\frac{d}{d\theta} log f(x_1;\theta))^2]=-E(\frac{d^2}{d \theta^2}log f(x_1;\theta))$
pf)

Remark)
If $\hat{\theta}$ is unbiased and $V(\hat{\theta})=\frac{(\tau'(\theta)^2)}{I_n(\theta)}$, then $\hat{\theta}$ is UMVUE

ex1) $X_1,...,X_n \overset{iid}{\sim} b(1,\theta)$

Multivariate case of CR inequality

Let $X_1,...,X_n \overset{iid}{\sim} f(x_1,...,x_n;\theta)$ and $\underline{\theta}=(\theta_1,\theta_2)^T$

• score vector
  $s(\underline{x};\theta_1,\theta_2)=(\frac{\partial}{\partial \theta_1} log f(\underline{x};\theta_1,\theta_2), \frac{\partial}{\partial \theta_2} log f(\underline{x};\theta_1,\theta_2))^T$
• information
  $I(\theta_1,\theta_2)=E(s(\underline{x};\theta_1,\theta_2)s^T(\underline{x};\theta_1,\theta_2))=-E(\frac{d^2}{d \underline{\theta}^2} log f(\underline{x};\underline{\theta}))$
  
  => If $E(\hat{\underline{\theta}})=\underline{\theta}$,
  $V(\hat{\underline{\theta}})=\begin{pmatrix} V(\hat{\theta_1}) & Cov(\hat{\theta_1},\hat{\theta_2}) \\ Cov(\hat{\theta_1},\hat{\theta_2}) & V(\hat{\theta_2}) \\ \end{pmatrix} \geq I^{-1}(\theta_1,\theta_2)$
  
  => If $E(\hat{\underline{\theta}})=(\tau_1(\underline{\theta}),\tau_2(\underline{\theta}))^T,\\ V(\underline{\theta}) \geq A(\underline{\theta}) I^{-1}(\underline{\theta}) A^T(\underline{\theta}) \space where \space A(\underline{\theta})= \begin{pmatrix} \frac{\partial}{\partial \theta_1} \tau_1(\theta_1,\theta_2) & \frac{\partial}{\partial \theta_2} \tau_1(\theta_1,\theta_2) \\ \frac{\partial}{\partial \theta_1} \tau_2(\theta_1,\theta_2)& \frac{\partial}{\partial \theta_2} \tau_2(\theta_1,\theta_2)\\ \end{pmatrix}$

ex1) $X_1,...,X_n \overset{iid}{\sim} N(\mu,\sigma^2)$