(2023ICCV)EfficientViT, (2022ICLR)CosFormer를 읽고 이해한 Linear Attention

• softmax-based similarity function이 갖는 특징
  1. attention matrix are non-negative
  2. non-linear re-weighting scheme
     
     
• ReLU-based linear similarity function은 softmax-based similarity function이 갖는 특징의 대부분을 활용할 수 있음.
  1. attention matrix are non-negative
  2. non-linear, but not yet re-weighting scheme...
     
     
• CosFormer는 ReLU-based linear simlarity function의 2.에서 re-weighting scheme이 없는 것을 보완하기 위해 cos-based re-weighting mechanism을 제안함.
  1. attention matrix are non-negative
  2. non-linear, $cos$-based re-weighting scheme