트랜스포머(Transfomer) 모델

Data 준비

import tensorflow as tf

# IMDB
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.imdb.load_data()
print(x_train.shape, y_train.shape)
# (25000,), (25000,)

x_train = tf.keras.utils.pad_sequences(x_train, maxlen=20)
x_test = tf.keras.utils.pad_sequences(x_test, maxlen=20)
print(x_train.shape, y_train.shape)
# (25000, 20), (25000,)

---

Transfomer 모델

X = tf.keras.Input(shape=[20])
H = tf.keras.layers.Embedding(88585, 28)(X)
H = tf.keras.layers.SimpleRNN(32, return_sequences=True)(H)

# Transformer::self-attentions
H1 = tf.keras.layers.MultiHeadAttention(2, 32)(H, H)
H = tf.keras.layers.BatchNormalization()(H + H1)
# Transformer::feed-forward
H1 = tf.keras.layers.Dense(32, activation='swish')(H)
H = tf.keras.layers.BatchNormalization()(H + H1)

H = tf.keras.layers.GlobalAveragePooling1D()(H)
Y = tf.keras.layers.Dense(1, activation="sigmoid")(H)

model = tf.keras.Model(X, Y)
model.compile(loss="binary_crossentropy", metrics="accuracy")

[중요] BatchNormalization

• 매우 중요함
• Nomalize layer은 딥러닝 모델이 학습이 되지 않을 때 학습이 되도록 돕는다.
• 결과값을 다음 레이어에 입력할 때 스케일을 맞춰주는 역할

---

Attention Layer

• Attention Layer의 핵심 매커니즘

• V value가 중요하다.
• 최종 결과물 => 5, 16

---

Attention Layer 수식 읽기

MultiHeadAttention(1, 2)(Q, K, V)

---

Transformer 모델