P1

• Let $(X_0,X_1,\dots,X_{N-1})$ be a random sample of a Bernoulli random variable $X$ with a probability mass function
  $f(x;p)=p^x(1-p)^{1-x}$
  where $x\in\{0,1\}$ and $0\leq p\leq 1$ is unknown
• Find the maximum-likelihood estimator (MLE) of $p$

---

Solution

• Let $\sum X_n=Y$
  $=Y\ln p+(N-Y)\ln(1-p)\\[0.3cm] \frac{\partial \ln p(x;p)}{\partial p}=\frac{Y}{p}+\frac{Y-N}{1-p}\\[0.3cm] \rightarrow\frac{Y}{\hat p}+\frac{Y-N}{1-\hat p}=0\\[0.3cm] \rightarrow \hat p(Y-N-Y)+Y=0\\[0.3cm] \rightarrow \hat p = \frac{Y}{N}=\frac{1}{N}\sum_{n=0}^{N-1}X_n$

---

---

P2

• Let $(X_0,X_1,\dots,X_{N-1})$ be a random sample of a binomial random variable $X$ with parameters $(n,p)$, where $n$ is assumed to be known and $p$ is unknown

1. Determine the maximum-likelihood estimator (MLE) of $p$
2. Show that the MLE of $p$ is unbiased

---

Solution 1

Solution 2

---

---

P3

• We observe $N$ IID samples from the PDFs :
  1. Gaussian
     $p(x;\mu)=\frac{1}{\sqrt{2\pi}}\exp\left[-\frac{1}{2}(x-\mu)^2\right]$
  2. Exponential
     $p(x; \lambda) = \begin{cases} \lambda \exp(-\lambda x) & x > 0 \\ 0 & x < 0 \end{cases}$
• In eaxh case find the MLE of the unknown parameter and be sure to verify that it indeed maximizes the likelihood function
• Do the estimators make sense?

---

Solution

1. $p(x;\mu)=\frac{1}{\sqrt{2\pi}}\exp[-1/2(x-\mu)^2]$
   • Log-Likelihood Function
     $\frac{N}{2}\log\left(\frac{1}{2\pi}\right)-\frac{1}{2}\sum_{i=0}^{N-1}(X_i-\mu)^2\\[0.3cm]$
   • 1st Derivative of Log-Likelihood Function
     $\frac{\partial \ln p(x;\mu)}{\partial\mu}=\sum_{i=0}^{N-1}(X_i-\mu)=Y-N\mu\\[0.3cm] Y=\sum_{i=0}^{N-1}X_i\\[0.3cm] \rightarrow \hat \mu=\frac{Y}{N}$
   • 2nd Derivative of Log-Likelihood Function
     $\frac{\partial^2 \ln p(x;\mu)}{\partial\mu^2}=-N$
   • Conclusion : Second derivative negative ▷ it's maximum
2. $p(x; \lambda) = \begin{cases} \lambda \exp(-\lambda x) & x > 0 \\ 0 & x < 0 \end{cases}$

• Log Likelihood Function
  $\text{likelihood}=\prod_{i=0}^{N-1}\lambda\exp(-\lambda X_i)\\[0.3cm] \text{Log-likelihood}=N\log\lambda-\lambda\sum_{i=0}^{N-1}X_i$
  • 1st Derivative of Log-likelihood Function
    $\frac{\partial \ln p(x;\lambda)}{\partial\lambda}=\frac{N}{\lambda}-\sum_{i=0}^{N-1} X_i \rightarrow\hat\lambda=\frac{N}{\sum_{i=0}^{N-1} X_i}$
  • 2nd Derivative of Log-Likelihood Function
    $\frac{\partial^2 \ln p(x;\lambda)}{\partial\lambda^2}=-\frac{N}{\lambda^2}$
    It's maximum because negative
  • Exponential distribution
    • Mean $=\frac{1}{\lambda}$
  • Conclusion
    $\hat\lambda=\frac{N}{\sum_{i=0}^{N-1}X_i}\\[0.3cm] \text{mean}=\frac{1}{\lambda}=\frac{\sum X_i}{N}=\frac{1}{\hat \lambda}$