VAE 탐구 - 4/6

Tetrapod·2024년 5월 22일
KL-Divergence

VAE 탐구

목록 보기
4/6

이 글에서는 KL-Divergence의 정의를 분포의 평균과 분산으로 전개하고자 한다.
이 글이 VAE 탐구의 Bottom이라고 하자.

Gaussian 분포

mean(평균):μ,  variance(분산):σ2mean(평균) : \mu, \;variance(분산) : \sigma^2

N(μ,σ2)=12πσ2e−(x−μ)22σ2N(\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{\frac{-(x-\mu)^2}{2\sigma^2}}

기댓값

  • μ=E[X]=∫xf(x)dx\mu=E[X]=\int xf(x)dx
  • E[X2]=μ2+σ2E[X^2]=\mu^2+\sigma^2

가우스 적분

∫e−x2dx=π∫0x2e−x2dx=Γ(2)21+1π=(2−1)!4π=π4\int e^{-x^2}dx = \sqrt{\pi}\\ \int_0x^2e^{-x^2}dx=\frac{\Gamma(2)}{2^{1+1}}\sqrt{\pi}=\frac{(2-1)!}{4}\sqrt{\pi}=\frac{\sqrt{\pi}}{4}

위 적분 안의 식은 symmetric 하므로

∫x2e−x2dx=π2\int x^2e^{-x^2}dx=\frac{\sqrt{\pi}}{2}

가우스 적분의 증명은 생략

KL-Divergence distribution

  • DKLD_{KL} 정의
DKL(p∣∣q)=∫p(x)  logp(x)q(x)dx=∫p(x)  log  p(x)dx  −  ∫p(x)  log  q(x)dx\begin{aligned} D_{KL}(p||q)&=\int p(x)\;log\frac{p(x)}{q(x)}dx \\ &=\int p(x)\;log\;p(x)dx \;-\; \int p(x)\;log\;q(x)dx \end{aligned}

∫p(x)  log  p(x)dx\int p(x)\;log \;p(x)dx 계산

  • 가우시안 분포 적용
∫p(x)  log  p(x)dx=∫12πσ2e−(x−μ)22σ2(−(x−μ)22σ2−log2πσ2)dx\begin{aligned} &\int p(x)\;log \;p(x)dx \\ &=\int \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\left(-\frac{(x-\mu)^2}{2\sigma^2}-log\sqrt{2\pi\sigma^2}\right)dx \end{aligned}
  • 치환 : t=x−μ2σt=\frac{x-\mu}{\sqrt{2}\sigma}, dt=12σdxdt=\frac{1}{\sqrt{2}\sigma}dx
∫12πσ2e−t2(−t2−log2πσ2)2σdt\begin{aligned} &\int \frac{1}{\sqrt{2\pi\sigma^2}}e^{-t^2}(-t^2-log\sqrt{2\pi\sigma^2})\sqrt{2}\sigma dt \end{aligned}
  • 식 정리
∫1πe−t2(−t2−log2πσ2)dt=−1π∫e−t2t2dt  −  log(2πσ2)2π∫e−t2dt \begin{aligned} &\int \frac{1}{\sqrt{\pi}}e^{-t^2}(-t^2-log\sqrt{2\pi\sigma^2}) dt \\ &=-\frac{1}{\sqrt{\pi}}\int e^{-t^2}t^2dt\;-\;\frac{log(2\pi\sigma^2)}{2\sqrt{\pi}}\int e^{-t^2}dt\ \end{aligned}
  • 가우스 적분 적용
−12−log(2πσ2)2=−12(1+log(2πσ2))\begin{aligned} &-\frac{1}{2} - \frac{log(2\pi\sigma^2)}{2} \\ &=-\frac{1}{2}\left(1+log(2\pi\sigma^2)\right) \end{aligned}
∴∫p(x)  log  p(x)dx=−12(1+log(2πσ2))\therefore \int p(x)\;log \;p(x)dx = -\frac{1}{2}\left(1+log(2\pi\sigma^2)\right)

−∫p(x)  log  q(x)dx-\int p(x)\;log \;q(x)dx 계산

  • 가우시안 분포 적용
−∫p(x)  log  q(x)dx=−∫p(x)log(12πσ22e−(x−μ2)22σ22)dx=−∫p(x)log12πσ22+∫p(x)(x−μ2)22σ22\begin{aligned} &-\int p(x)\;log \;q(x)dx \\ &=-\int p(x)log\left(\frac{1}{\sqrt{2\pi\sigma_2^2}}e^{-\frac{(x-\mu_2)^2}{2\sigma_2^2}}\right)dx \\ &=-\int p(x)log\frac{1}{\sqrt{2\pi\sigma_2^2}} + \int p(x)\frac{(x-\mu_2)^2}{2\sigma_2^2} \end{aligned}
  • ∫p(x)dx=1\int p(x)dx=1 임을 이용
  • 두번째 항 전개
12log(2πσ22)  +  ∫p(x)x2dx  −  2μ2∫xp(x)dx  +  μ22∫p(x)dx2σ22=12log(2πσ22)  +  E1[X2]  −  2μ2E1[X]  +  μ222σ22\begin{aligned} &\frac{1}{2}log(2\pi\sigma_2^2)\;+\;\frac{\int p(x)x^2dx\;-\;2\mu_2\int xp(x)dx\;+\;\mu_2^2\int p(x)dx}{2\sigma_2^2} \\ &=\frac{1}{2}log(2\pi\sigma_2^2)\;+\;\frac{E_1[X^2]\;-\;2\mu_2E_1[X]\;+\;\mu_2^2}{2\sigma_2^2} \end{aligned}
  • E[X2]=μ2+σ2E[X^2]=\mu^2+\sigma^2
  • E[X]=μE[X]=\mu 임을 이용.
12log(2πσ22)  +  μ12+σ12  −  2μ1μ2  +  μ222σ22=12log(2πσ22)  +  σ12+(μ1−μ2)22σ22\begin{aligned} &\frac{1}{2}log(2\pi\sigma_2^2)\;+\;\frac{\mu_1^2+\sigma_1^2\;-\;2\mu_1\mu_2\;+\;\mu_2^2}{2\sigma_2^2} \\ &=\frac{1}{2}log(2\pi\sigma_2^2)\;+\;\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2} \end{aligned}
∴−∫p(x)  log  q(x)dx=12log(2πσ22)  +  σ12+(μ1−μ2)22σ22\therefore -\int p(x)\;log \;q(x)dx = \frac{1}{2}log(2\pi\sigma_2^2)\;+\;\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}

∫p(x)  log  p(x)dx−∫p(x)  log  q(x)dx\int p(x)\;log \;p(x)dx-\int p(x)\;log \;q(x)dx 계산

∴DKL(p∣∣q)=∫p(x)  log  p(x)dx−∫p(x)  log  q(x)dx=12log(2πσ22)  +  σ12+(μ1−μ2)22σ22−12(1+log(2πσ12))=12log2πσ222πσ12  +  σ12+(μ1−μ2)22σ22−12=−12(logσ12σ22  −  σ12+(μ1−μ2)2σ22+1)\begin{aligned} &\therefore D_{KL}(p||q)=\int p(x)\;log \;p(x)dx-\int p(x)\;log \;q(x)dx \\ &=\frac{1}{2}log(2\pi\sigma_2^2)\;+\;\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2} -\frac{1}{2}\left(1+log(2\pi\sigma_1^2)\right) \\ &=\frac{1}{2}log\frac{2\pi\sigma_2^2}{2\pi\sigma_1^2}\;+\;\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2} -\frac{1}{2} \\ &=-\frac{1}{2}\left(log\frac{\sigma_1^2}{\sigma_2^2}\;-\;\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{\sigma_2^2} +1\right) \end{aligned}

Reference

profile
Tetrapod
이전 포스트

VAE 탐구 - 3/6

다음 포스트

VAE 탐구 - 5/6

0개의 댓글