Lecture 17: Moment Generating Functions

Intuition of Memoryless Property

• $E(T | T > 20) > E(T)$

  • 20대 이상의 기대 수명(T)은 무작위적인 기대 수명에 비해 큰 것이 사실이다.
  • 모든 사람의 기대 수명이 같지 않고 다 가변성이 있다면 이는 의미가 있다.

• If memoryless, we would have $E(T | T > 20) = 20 + E(T)$

  • 인간에게는 그다지 유용하지 않으나 화학, 물리학 분야에서는 꽤 의미가 있다.
  • 시간이 지나도 decay되지 않는 속성을 가지는 것들에 대하여 적용될 수 있다.

Theorem

• $X$ is a positive continuous r.v. with memoryless property,

  • then $X \sim Expo(\lambda)$ for same $\lambda$

• Proof.

  • Let $F$ be the CDF of $X$, $G(x) = P(X > x) = 1 - F(x)$ = 1 - 지수함수's CDF

    • memoryless property is $G(s+t) = G(s)G(t)$, Solve for $G$

    • $s = t \Rightarrow G(2t) = G(t)^2$, $G(3t) = G(t)^2, ... , G(kt) = G(t)^k$

    • $G(\frac{t}{2}) = G(t)^{1/2}, G(\frac{t}{3}) = G(t)^{1/3}, ... \;, G(\frac{t}{k}) = G(t)^{1/k}$

    • $G(\frac{m}{n} t) = G(t)^{m/n}$, So $G(xt) = G(t)^x$ for all real $x > 0$

    • $t=1 \Rightarrow G(x) = G(1)^x = e^{x lnG(1)} = e^{x -\lambda} = e^{-\lambda x}$

      • $0 < G(1) < 1 → lnG(1)$ is negative→ $-\lambda$ !

Moment Generating Function(MGF)

• Defn.

  • A r.v. $X$ has MGF $M(t) = E(e^{tx})$, as a function of $t$,

    • if this is finite on some $(-a, a)$, $a>0$.

  • Why moment "generating"?

    • $E(e^{tx}) = E(\displaystyle \sum_{0}^{∞} \frac{x^n t^n}{n!}) = \displaystyle \sum_{0}^{∞} \frac{E(x^n) t^n}{n!}$ by taylor series.

    • $E(x^n)$ : nth moment

• Why is MGF important?

  • Let $X$ have MGF $M(t)$

  1. The nth moment, $E(X^n)$, is coef of $\displaystyle \frac{t^n}{n!}$ in Taylor series of M,

     • i.e. $M^{(n)}(0) = E(X^n)$ nth derivative.

  2. MGF determines the distribution, i.e., if $X$, $Y$ have same MGF,

     • then they have same CDF.

  3. If $X$ has MDF $M_X$, $Y$ has MDF $M_Y$, $X$ indep. $Y$, then

     • MGF of $X+Y$ is $E(e^{t(X+Y)} = E(e^{tX})E(e^{tY}) = M_X(t) M_Y(t)$ by indep.

• Examples.

  • $X \sim Bern(p)$, $M(t) = E(e^{tx}) = pe^{t} + q$, $q = 1-p$

  • $X \sim Bin(n, p) \Rightarrow M(t) = (pe^{t} + q)^n$

  • $Z \sim N(0, 1) \Rightarrow M(t) = \displaystyle \frac{1}{\sqrt{2 \pi}} \int_{-∞}^{∞} e^{tz - z^2/2} dz$

    $\displaystyle \frac{e^{t^2/2}}{\sqrt{2 \pi}} \int_{-∞}^{∞} e^{\frac{1}{2}(z-t)^2} dz = \displaystyle \frac{e^{t^2/2}}{\sqrt{2 \pi}} \; • \; \sqrt{2 \pi} = e^{t^2/2}$

Laplace Rule of Succession

• 내일 해가 뜰 확률은 얼마인가?

  • Given $p$ ⇒ We assume conditional indep. $X_1, X_2, ... i.i.d.$ to $Bern(p)$
  • But actually, we don't know about $p$.

• $p$ unknown, Bayesian: treat $p$ as a r.v.

• Let $p \sim Unif(0, 1)$ (사전 확률 prior). Let $S_n = X_1 + ... + X_n$

  • So $S_n | p \sim Bin(n, p)$, $p \sim Unif(0, 1)$
  • Find $p | S_n$ (사후 확률 posterior), and $P(X_{n+1} = 1 | S_n = n)$
    • 지난 n일 동안 태양이 떴을 때, 내일 해가 뜰 확률

• $f(p | S_n = k) = \displaystyle \frac{P(S_n = k | p) f(p)}{P(S_n = k)}$

  • $f(p)$ is prior 1, $P(S_n = k)$ does not depend on $p$

    • $P(S_n = k) = \int_{0}^{1} P(S_n = k | p) f(p) dp$

  • $f(p | S_n = k) ∝ p^k (1-p)^{n-k}$

    • $f(p | S_n = n) = (n+1)p^n$

  • $P(X_{n+1} = 1 | S_n = n) = \int_{0}^{1} (n+1) \; p \; p^n dp = \displaystyle \frac{n+1}{n+2}$

• 출처 : Statistics 110, boostcourse