Pytorch Weight Initialization

Weight Initialization

• torch.nn.init.constant_(tensor, val) : 상수로 설정
• torch.nn.init.ones_(tensor) : 1로 설정
• torch.nn.init.zeros_(tensor) : 0으로 설정
• torch.nn.init.eye_(tensor)
• torch.nn.dirac_(tensor, gropus=1)
• torch.nn.uniform(tensor, a=0.0, b=1.0) : a부터 b사이의 값을 균일한 분포로 설정
• torch.nn.init.normal_(tensor, mean=0.0, std=1.0)
• torch.nn.init.orthogonal_(tensor, gain=1)
• torch.nn.sparse_(tensor, sparsity, std=0.01)

Xavier

• torch.nn.init.xavier_uniform_(tensor, gain=1.0)
• torch.nn.init.xavier_normal_(tensor, gain=1.0)

Kaiming

• torch.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')
• torch.nn.init.kaiming_normal_(tensor, a=0, model='fan_in', nonlinearity='leaky_relu')

---

import torch
import torch.nn as nn

layer = nn.Linear(2048, 2, bias=True)

# Normal Distribution
torch.nn.init.normal_(layer.weight)

# Xavier initialization
torch.nn.init.xavier_uniform_(layer.weight)

# Kaiming Initialization
torch.nn.init.kaiming_uniform_(layer.weight)

---

Kaiming

• tanh, sigmoid를 활성화 함수로 사용하는 경우라 가정(선형성)

(Kaiming He) He Initialization

• ReLU를 위해 만들어진 방법

Reference

• https://velog.io/@cha-suyeon/