Introduction

In NLP, Self-Attention-based arcitectures became major model. Transformers'computational efficiecy and scalability make it unprecedented size trains.
On the other hand, before this paper, in vision domain, CNN was dominant.
To use goodness of attention patterns, they split an image into patches and provide the sequence of linear embeddings of these patches as an input. As words tokens, they treated those.
When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size.
Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance an locality, and therefore do not generalize well when trained on insufficient amounts of data.

''inductive bias
''translation equivariance and locality

However, if the models are trained on larger datasets(14M-300M images), they find that large scale training trumps inductive bias.

Related Work

Transformers are SOTA in NLP such as BERT, GPT. However, Just applying self-attetion to image, the model does not scale to realistic input sizes because of the feature that each pixel attends to other every pixels.
(Naive application of self-attention to images would require that each pixel attends to every other
pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes.)
Thus, there were several approximations. The most simmilar way to this paper is using 2x2 small patch. they explored image recognition at larger scales than the standard ImageNet dataset.

Method

The standard transformer receives as input a 1D sequence of token embeddings.
They reshape the the image x ∈ R ^ H×W×C into a sequence of flattened 2D x_p patches whose dimension is N×(P^2·C), where (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each image patch, and N = HW/ P^2 is the resulting number of patches, which also serves as the effective input sequence length for the Transformer.

The Transformer uses constant latent vector size D through all of its layers, so we flatten the patches and map to D dimensions with a trainable linear projection. We refer to the output of this projection as the patch embeddings.
The refer to the output of this projection as the patch embeddings.

Similar to BERT's class token, they prepend a learnable embedding to the sequence of embedded patches(special) whose state at the output of the Transformer encoder serves as the image representation y.

Both during pre-training and fine-tuning, a classfication head is attached to the z. The classficaiton head is implemented by a MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time.

They also add position embeddings to the patch embeddings to retain positional information. They use standard learnable 1D position embeddings, since they have not observed significant performance gains from using more advanced 2D-aware position embeddings.

inductive bias.

Hybrid Architecture

Fine-tuning and higher resolution