Lecture 21: Covariance and Correlation

Covariance

• Defn. : need 2 variables

  • $Cov(X, Y) = E((X-EX)(Y-EY)) = E(XY) - E(X)E(Y)$
    • If X is bigger than EX, Y is also bigger than EY.
    • If X is smaller than EX, Y is also : like a + = +*+, -*-
    • Since linearlity, $E(XY) - E(X)E(Y) - E(X)E(Y) + E(X)E(Y).$

• Properties

  1. $Cov(X, Y) = Var(X)$

  2. $Cov(X, Y) = Cov(Y, X)$

  3. $Cov(X, c) = 0$ if c is constant.

  4. $Cov(cX, Y) = c Cov(X, Y) = Cov(X, cY)$

  5. $Cov(X, Y+Z) = Cov(X, Y) + Cov(X, Z)$ ← $X(Y+Z) = XY + XZ$

     • 4 & 5 : bilinearity

  6. $Cov(X+Y, Z+W) = Cov(X, Z) + Cov(X, W) + Cov(Y, Z) + Cov(Y, W).$
     ← $Cov(\displaystyle \sum_{i=1}^{m} a_i X_i, \sum_{j=1}^{m} b_i Y_i = \displaystyle \sum_{i, j} a_i b_i Cov(X_i, Y_j).$
     : Combination of Random variables

  7. $Var(X_1 + X_2) = Var(X_1) + Var(X_2) + 2Cov(X_1, X_2)$
     : 합의 분산은 분산의 합 (독립이라면 공분산이 0, 공분산이 0인 경우에만!)

  8. $Var(X_1 + ... + X_n) = Var(X_1) + ... Var(X_n) + 2 \displaystyle \sum_{i<j} Cov(X_i, X_j)$

• Thm.

  • IF $X$, $Y$ are indep., then theiy're uncorrelated, i.e., $Cor(X, Y) = 0$

    • Converse(역) is false

  • e.g., $Z \sim N(0, 1)$, $X = Z$, $Y = Z^2$

    • $Cov(X, Y) = E(XY) - E(X)E(Y) = E(Z^3) - E(Z)E(Z^2) = 0$

  • But very dependent : $Y$ is a function of X, and $Y$ determines magnitude of X.

    • $Cov(X, Y) = 0$, but they did not independent!
    • Correlation : 선형 관계가 있어야만 상관성 증명 가능

Correlation

• Defn.

  • $Corr(X, Y) = \displaystyle \frac{Cov(X, Y)}{SD(X)SD(Y)} = Cov(\frac{X-EX}{SD(X)}, \frac{Y-EY}{SD(Y)})$
    • After Standard Deviation(표준화), get Covariance
    • 표준화는 단위를 없애는 작업 : 무차원

• Thm.

  • $-1 \le Corr(X, Y) \le 1$ (form of Couchy-Schwarz)

• Proof.

  • WLOG(Without Loss Of Generality) assume $X$, $Y$ are standardized(0~1), let $Corr(X, Y) = \rho.$
  • $Var(X+Y) = Var(X) + Var(Y) + 2Cov(X, Y) = 2 + 2\rho$
  • $Var(X-Y) = Var(X) + Var(Y) - 2Cov(X, Y) = 2 - 2\rho$
    ⇒ $-1 \le \rho \le 1$

• Ex. Cov in a Multinomial

  • $(X_1, ... X_k) \sim Mult(n, \vec{p})$, Find $Cov(X_i, X_j)$ for all i, j.
  1. If $i=j$, $Cov(X_i, X_j) = Var(X_i) = n p_i (1-p_i).$
  2. Now let $i \ne j$, Find $Cov(X_i, X_j) = c$.

  • $Var(X_1+X_2) = n p_1(1-p_1) + n p_2(1-p_2) + 2c$
    $= n(p_1 + p_2) (1-(p_1 + p_2))$

    $\Rightarrow Cov(X_1, X_2) = -n p_1 p_2$

  • General : $Cov(X_i, X_j) = -np_ip_j$, for $i \ne j$.

• Ex. Cov in binomial

  • $X \sim Bin(n, p)$, write as $X = X_1 + ... + X_n$,

    $\begin{cases} X_j \; are \; i.i.d. \; Bern(p).\\ Var \; X_i = EX_j^2 - (EX_j)^2 = p-p^2 = p(1-p) = pq \end{cases}$

  • Let $I_A$ be indicator r.v. of event A.

    • $I_A^2 = I_A$
    • $I_A^3 = I_A$
    • $I_A I_B = I_{A \cap B}$

  ⇒ $Var(X) = npq$ since $Cov(X_i, X_j) = 0$ for $i \ne j$.

• Ex. Cov in Hypergeometric

  • $X \sim HGeom(w, b, n)$

    • $X = X_1 + ... X_n$, $X_j = \begin{cases} 1, {if \; jth \; ball \; is \; white}\\ 0, {otherwise} \end{cases}$

  • Symmetry, $Var(X) = nVar(X_1) + 2 \displaystyle {n \choose k}Cov(X_1, X_2)$

    • equally likely

  • $Cov(X_1, X_2) = E(X_1X_2) - E(X_1)E(X_2) = \displaystyle \frac{w}{w+b} (\frac{w-1}{w+b-1}) - (\frac{w}{w+b})^2$

• 출처 : Statistics 110, boostcourse