Google's machine learning model
“Transformer” can translate and summarize data such as natural language without processing it in chronological order, and is the basis of chat AI that allows natural conversations such as ChatGPT. In addition, ' vision transformers ” is a model that applies Transformer technology to the video field.Software Engineer Dennis Tharp Visually shows how Vision Transformer components work and how data flows.
Vision Transformer Visual Guide | MDTURP
https://blog.mdturp.ch/posts/2024-04-05-visual_guide_to_vision_transformer.html
0: Introduction
First, the Vision Transformer is monitored, as is the behavior of the Transformer. This means that the model is trained on a dataset of images and their corresponding labels.
1: Focus on one piece of data
Select a single piece of data called “Patch Size 1”.
2: Image division
To make an image available to Vision Transformer, divide it into equally sized patches.
3: Flattening the image patch
Convert the patch to a vector with p' = p²*c. Here, p is the size of the patch and c is the number of patches into which the patch is divided.
4: Creating patch embedding vector
(PDF file)
Through linear transformation, the image patches converted to vectors are further converted to patch embedding vectors.
5: Apply to all patches
Converting all patches to patch embedding vectors yields an nxd array. Here, n is the number of image patches and d is the size of the patch embedding vector.
6: Adding classification tokens
To effectively train the model, we add vectors called classification tokens (cls tokens). This is a learnable parameter of the network and is randomly initialized.
7: Adding position embedding vector
Previously, vectors did not have location information associated with them, so we add a learnable, randomly initialized “location embedding vector” to every vector containing a cls token.
8: Transformer input
After the position embedding vectors are added, an array of size (n+1) × d remains. This corresponds to the input to the converter.
9: Assignment to three types of vectors
An array of size (n+1) × d is divided into a “query vector” corresponding to Q, a “key vector” corresponding to K, and a “value vector” corresponding to V.
10: Calculating attention score
To calculate the attention score, all query vectors are multiplied by the key vector.
11: Attention score matrix
Now that we have the attention matrix from the calculation, we apply '.
Softmax' function is applied to all rows so that the sum of all rows is 1.
12: Calculating aggregated context information
Focusing on the first row of the matrix, compute the aggregate context information of the patch embedding vector and use its entirety as the weight of the value vector to obtain the aggregate context information vector of the first patch embedding vector.
13: Apply to all rows
Applying this calculation to the entire attention matrix results in N+1 aggregated context information vectors.
14: Repeat the process
This process is repeated multiple times depending on the number.
heads results in multiple aggregated context information vectors.
15: Mapping to vector of size d
Merge multiple heads and map them to a vector of the same size d as the patch embedding vector.
16: Completing the featured layer
Mapping to a vector produces an embedding of exactly the same size and amount as the input embedding vector.
17: Applying residual connections
The input of the layer that adds the position embedding vector is added to the output of the layer of interest.
18: Calculate remaining connections
Add input and output.
19: Feedforward network
The output generated so far is
Feedforward network with nonlinear activation function.
20: Final result
Performing multiple operations produces an output that is the same size as the input.
21: Repeat the process
Repeat this process multiple times.
22: Identifying classification token output
The final step of Vision Transformer is to identify the output of the classification token.
23: Classification probability prediction
Use another neural network fully connected with the output of the classification token to predict the classification probability of the original image.
24: Vision Transformer Training
Train the vision transformer using: cross entropy error .
