Self Supervised Representation Learning in NLP

jkl133·2021년 2월 26일
0

출처: https://amitness.com/2020/05/self-supervised-learning-nlp/
참고: https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html

  • Language Models have existed since the 90’s even before the phrase “self-supervised learning” was termed.
  • At the core of these self-supervised methods lies a framing called “pretext task”
    • pretext task(= auxiliary task, pre-training task): use the data itself to generate labels and use supervised methods to solve unsupervised problems

Self-Supervised Formulations

1. Center Word Prediction

In this formulation, we take a small chunk of the text of a certain window size and our goal is to predict the center word given the surrounding words.

For example, in the below image, we have a window of size of one and so we have one word each on both sides of the center word. Using these neighboring words, we need to predict the center word.

2. Neighbor Word Prediction

In this formulation, we take a span of the text of a certain window size and our goal is to predict the surrounding words given the center word.

3. Neighbor Sentence Prediction

In this formulation, we take three consecutive sentences and design a task in which given the center sentence, we need to generate the previous sentence and the next sentence. It is similar to the previous skip-gram method but applied to sentences instead of words.

profile
재밌는게 재밌는거다

0개의 댓글