Distinct perspectives with multiple attentions
So, Karpathy finishes the part that he discusses the multi-head attention with this perspective regarding the mechanism. It is to provide a strategy so that the tokens can learn distinct perspectives regarding the interaction with the other tokens. Really interesting.
Besides that, great perspective in just a few seconds, man. Karpathy run the code and observe the validation loss. It decreased comparing with just a single self-attention. And that provided a great perspective for research.
Really great learning.