The residual connections
So, Karpathy discutiu que, com o increasing da rede com o uso do bloco com um número específico de uso, a rede observa the vanishing gradientes, o que prejudica o treinamento. Karpathy então introduz o conceito de residual connections ou skip connections. The paper is the one in computer vision context, but it is used in the Transformer architecture. It basically consider the path with the input added in the final transformation for a block, so, in this way, the summation can help in preserving the gradients.
To really consider the study for the paper. I think is another paper really important to understand because is crucial for the training of scaled neural networks.
Really great learning.