Binary Classification	Multi-Class Classfication
Output Layer Act. Func	sigmoid	softmax
Cost Function	mean of Cross-Entropy	mean of Cross-Entropy
da (GD)		da = a - y

1. Softmax Function : Output Layer Activation Function

입력값들을 확률로 변환해주는 함수.

$a^{[l]}_i = \frac{e^{z_i^{[l]}}}{\sum_k e^{z_k^{[l]}}}$

▶️ multi-class Classification 문제의 output layer act. func.
cf) hardmax → [0, 1, 0, 0]

2. Loss Function

• Cross-Entropy를 로스 함수로 사용
• maximum likelihood principle에 의해 가능
  • $argmin_{W,b}(-log\hat y_i) \\ = argmax_{W,b}(\hat y_i) \\ = argmax_{W,b}(p(y_i|W, b))$

m개 data 의 cost function은 $E(Loss)$ 이다.

3. Gradient Descent with Softmax

$dz = \frac{dJ}{dz} = \hat y - y$

proof of $dz$