P1

• Suppose that $X_0, X_1, X_2,\dots,X_{N-1}$ be a random sample of an exponential random variable $X$ with an unknown parameter $\alpha$, corresponding the mean of $X$
  $f_X(x;\alpha)=\frac{1}{\alpha}e^{-x/\alpha}$
  When $\alpha$ is to be estimated, find the Cramer-Rao lower bound (CRLB)

---

Solution

• Log Likelihood
  $\ln p(x;\alpha)=\sum_{i=0}^{N-1}\log(f_X(X_i;\alpha))\\[0.3cm] =\sum_{i=0}^{N-1}\log\left(\frac{1}{\alpha}e^{-X_i/\alpha}\right)\\[0.3cm] =\sum_{i=0}^{N-1}\left(-\log(\alpha)-\frac{X_i}{\alpha}\right)\\[0.3cm] =-N\log(\alpha)-\frac{1}{\alpha}\sum_{i=0}^{N-1} X_i$
• 1st Derivative of Log Likelihood
  $\frac{\partial\ln p(x;\alpha)}{\partial\alpha}=-\frac{N}{\alpha}+\frac{1}{\alpha^2}\sum_{i=0}^{N-1}X_i\\[0.3cm]$
• Fisher Information Function
  $I(\alpha)=-E\left[\frac{\partial^2 \ln p(x;\alpha)}{\partial\alpha^2}\right]\\[0.3cm] \frac{\partial^2\ln p(x;\alpha)}{\partial\alpha^2}=\frac{N}{\alpha^2}-\frac{2}{\alpha^3}\sum_{i=0}^{N-1} X_i\\[0.3cm]$
• $E[X_i]=\alpha$
  $I(\alpha)=\frac{N}{\alpha^2}\\[0.3cm] \therefore \text{CRLB}=\frac{\alpha^2}{N}$

---

---

P2

• Let $Y_0,Y_1,Y_2,\dots,Y_{N-1}$ be a random sample of a Gaussian random variable of mean $\alpha+\beta x_i$ and variance $1$, where constants $x_0,x_1,\dots,x_{N-1}$ are known, whereas $\alpha$ and $\beta$ are unknown parameters. Please derive the Fisher information matrix for CRLB of $\theta=[\alpha\;\beta]^T$

---

Solution

• PDF

  • mean of $Y_i$ : $\alpha + \beta x_i$
  • variance of $Y_i$ : $1$
    $f_Y(y_i;\alpha,\beta)=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}(y_i-\alpha-\beta x_i)^2\right)\\[0.3cm]$

• Log Liklihood Function

  $\ln p(x;\alpha,\beta)=\sum_{i=0}^{N-1}\log(f_Y(y_i;\alpha,\beta))\\[0.3cm] =-\frac{N}{2}\log(2\pi)-\frac{1}{2}\sum_{i=0}^{N-1}(y_i-\alpha-\beta x_i)^2$

• 1st Derivative of $\alpha$

  $\frac{\partial \ln p(x;\alpha,\beta)}{\partial\alpha}=\sum_{i=0}^{N-1}(y_i-\alpha-\beta x_i)$

• 1st Derivative of $\beta$

  $\frac{\partial \ln p(x;\alpha, \beta)}{\partial\beta}=\sum_{i=0}^{N-1}(y_i-\alpha-\beta x_i)x_i$

• 2nd Derivative of $\alpha$

  $\frac{\partial^2 \ln p(x;\alpha,\beta)}{\partial\alpha^2}=-N$

• 2nd Derivative of $\beta$

  $\frac{\partial^2 \ln p(x;\alpha,\beta)}{\partial\beta^2}=-\sum_{i=0}^{N-1}x_i^2$

• $\alpha$, $\beta$ 2nd Derivative

  $\frac{\partial \ln p(x;\alpha, \beta)}{\partial\alpha\partial\beta}=-\sum_{i=0}^{N-1}x_i$

• Fisher Information Function

  $I(\theta) = \begin{bmatrix} -N & -\sum_{i=0}^{N-1} x_i \\ -\sum_{i=0}^{N-1} x_i & -\sum_{i=0}^{N-1} x_i^2 \end{bmatrix}$

---

---

P3

The data $x[n]=Ar^n +w[n]$ for $n=0,1,\dots,N-1$ are observed, where $w[n]$ is WGN with variance $\sigma^2$ and $r>0$ is known. Find the CRLB for $A$. Show that an efficient estimator exists and find its variance. What happens to the variance as $N\rightarrow\infty$ for various values of $r$?

---

• PDF

  $f(x[n];A)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x[n]-Ar^n)^2}{2\sigma^2}\right)$

• Log Likelihood Function

  $\ln p(x;A)=\sum_{n=0}^{N-1}\log f(x[n];A)\\[0.3cm] =-\frac{N}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{n=0}^{N-1}(x[n]-Ar^n)^2$

• 1st Derivative of Log Likelihood Function

  $\frac{\partial\ln p(x;A)}{\partial A}=\frac{1}{\sigma^2}\sum_{n=0}^{N-1}(x[n]-Ar^n)r^n$

• 2nd Derivative of Log Likelihood Function

  $\frac{\partial^2 \ln p(x;A)}{\partial A^2}=-\frac{1}{\sigma^2}\sum_{n=0}^{N-1}(r^n)^2$

• Fisher Information Function

  $I(A)=\frac{1}{\sigma^2}\sum_{n=0}^{N-1}(r^n)^2$

• CRLB

  $\text{var}(\hat A) \geq \frac{1}{I(A)}=\frac{\sigma^2}{\sum_{n=0}^{N-1}(r^n)^2}$

• Efficient Estimator and Variance

  If 1st derivative of log liklihood function equal to 0 then MLE

  $\hat A = \frac{\sum_{n=0}^{N-1}x[n]r^n}{\sum_{n=0}^{N-1}(r^n)^2}\\[0.3cm] \text{var}(\hat A)=\frac{\sigma^2}{\sum_{n=0}^{N-1}(r^n)^2}=\text{CRLB}$

• What happens if $N\rightarrow\infty$

  $\sum_{n=0}^{N-1}(r^n)^2=\sum_{n=0}^{N-1}r^{2n}=\frac{1-r^{2N}}{1-r^2},\;\text{for }r<1\\[0.3cm] \text{When }N\rightarrow\infty,\;r<1\\[0.3cm] \text{var}(\hat A)\rightarrow\frac{\sigma^2(1-r^n)}{1-r^{2N}}\approx\frac{\sigma^2}{1-r^2}\\[0.3cm] \text{When }r=1\\[0.3cm] \sum_{n=0}^{N-1}(r^n)^2=N\\[0.3cm] \text{When }N\rightarrow\infty,\;r=1\\[0.3cm] \text{Var}(\hat A)\rightarrow\frac{\sigma^2}{N}\rightarrow0$

---

---

P4

Prove that

What conditions on $f_0$ are required for this to hold? Note that

and use the geometrix progression sum formula

---

Solution

1. 코사인의 복소수 표현
   $\cos(\theta)=\text{Re}(e^{j\theta})\\[0.3cm] \sum_{n=0}^{N-1}\cos(4\pi f_0 n+2\phi)=\text{Re}\left(\sum_{n=0}^{N-1}\exp(j(4\pi f_0 n+2\phi))\right)\\[0.3cm] =\text{Re}\left(e^{j2\phi}\sum_{n=0}^{N-1}e^{j4\pi f_0 n}\right)$
   문제는 $\sum_{n=0}^{N-1}e^{j4\pi f_0 n}$의 합을 계산해야함
2. 기하급수 수열의 합 공식 적용
   $\sum_{n=0}^{N-1}r^n=\frac{1-r^N}{1-r},\;r\neq1\\[0.3cm] r=e^{j4\pi f_0}\\[0.3cm] \sum_{n=0}^{N-1}e^{j4\pi f_0n}=\frac{1-e^{j4\pi f_0N}}{1-e^{j4\pi f_0}}$
3. 조건 도출
   $e^{j4\pi f_0 N} = 1$의 경우 $f_0$가 정수여야 함 그럼 기하급수의 합은 0
   $\sum_{n=0}^{N-1}e^{j4\pi f_0 n}=0\;\text{if }f_0\in\text{Integer}\\[0.3cm] \frac{1}{N}\sum_{n=0}^{N-1}\cos(4\pi f_0 n+2\phi)\approx0$

---

---

P5

• We observe two samples of a DC level in correlated Gaussian noise
  $x[0]=A+w[0]\\ x[1]=A+w[1]$
  where $\text{w}=[w[0]\;w[1]]^T$ is zero mean with covariance matrix
  $\mathbf{C} = \sigma^2 \begin{bmatrix} 1 & \rho \\ \rho & 1 \end{bmatrix}$
• The parameter $\rho$ is the correlation coefficient between $w[0]$ and $w[1]$
• Compute the CRLB for $A$ and compare it to the case when $w[n]$ is WGN or $\rho=0$
• Also, explain what happens when $\rho \rightarrow \pm1$
• Finally, comment on the additivity property of the Fisher Information for nonindependent observations

---

• CRLB
  $I(A)=\frac{\partial\mu^T}{\partial A}C^{-1}\frac{\partial\mu}{\partial A}\\[0.3cm] \mu=[A\;A]^T\rightarrow\frac {\partial\mu}{\partial A}=[1\;1]^T\\[0.3cm] C^{-1}=\frac{1}{\sigma^2(1-\rho^2)}\begin{bmatrix} 1 & -\rho \\ -\rho & 1 \end{bmatrix}\\[0.3cm] I(A)=[1\;1]\frac{1}{\sigma^2(1-\rho^2)}\begin{bmatrix} 1 & -\rho \\ -\rho & 1 \end{bmatrix}[1\;1]^T\\[0.3cm] =\frac{1}{\sigma^2(1-\rho^2)}((1)(1)+(-\rho)(1)+(-\rho)(1)+(1)(1))\\[0.3cm] =\frac{1}{\sigma^2(1-\rho^2)}(2-2\rho)\\[0.3cm] =\frac{2(1-\rho)}{\sigma^2(1-\rho^2)}\\[0.3cm] I(A)=\frac{2(1-\rho)}{\sigma^2(1-\rho^2)}\\[0.3cm] \text{Var}(\hat A)\geq\frac{\sigma^2(1-\rho^2)}{2(1-\rho)}=\text{CRLB}$
• $\rho=0$ (WGN)
  $I(A)=\frac{2}{\sigma^2}\\[0.3cm] \text{Var}(\hat A) \geq \frac{\sigma^2}{2}=\text{CRLB}$
  We can't get CRLB
• $\rho \rightarrow \pm 1$
  • If $\rho \rightarrow 1$
    $I(A) \rightarrow 0$
  • If $\rho \rightarrow -1$
    $I(A)=\frac{2(1-(-1))}{\sigma^2(1-(-1)^2)}=\frac{2(2)}{\sigma^2(4)}=\frac{1}{\sigma^2}\\[0.3cm] \text{Var}(\hat A)\geq\sigma^2=\text{CRLB}$
    The Estimator getting worse
• Conclude
  • When $\rho=0$, lowest CRLB
  • When $\rho \rightarrow \pm1$ decrease Fisher information, increase CRLB

---

---

P6

• Consider a generalization of the line fitting problem as described in Problem 4 termed polynomial or curve fitting
• The data model is
  $x[n]=\sum_{k=0}^{p-1}A_kn^k+w[n]$
  for $n=0,1,\dots,N-1$
• As before, $w[n]$ is WGN with variance $\sigma^2$
• It is desired to estimate $\{A_0,A_1,\dots,A_{p-1}\}$
• Find the Fisher Information matrix for this problem

---

Solution

---

---

P7

• It is desired to estimate the total power $P_0$ of a WSS random process, whose PSD is given as
  $P_{xx}(f)=P_0Q(f)$
  where
  $\int_{-\frac{1}{2}}^{\frac{1}{2}}Q(f)df=1$
  and $Q(f)$ is known
• If $N$ observations are available, find the CRLB for the total power using the exact from as well as the asymptotic approximation and compare

---

Solution

• Fisher Information Function
  $I(P_0)=N\cdot \int_{-1/2}^{1/2}\frac{1}{P_{xx}(f)}\left(\frac{\partial P_{xx}(f)}{\partial P_0}\right)^2 df\\[0.3cm] P_{xx}(f)=P_0Q(f)\\[0.3cm] I(P_0)=N\cdot \int_{-1/2}^{1/2}\frac{1}{P^2_0Q(f)}(Q(f))^2df\\[0.3cm] =N\cdot\frac{1}{P^2_0}\int_{-1/2}^{1/2}Q(f)df\\[0.3cm] =N\cdot\frac{1}{P^2_0}\cdot 1=\frac{N}{P^2_0}$
• CRLB
  $\text{Var}(\hat P_0)\geq\frac{1}{I(P_0)}=\frac{P^2_0}{N}=\text{CRLB}$
• Asymptotic approximation
  $\text{Var}(\hat P_0)\approx\frac{P^2_0}{N}$

aa