The block for communication followed by computation
So, Karpathy discusses the great principle involved in the Transformer block. Simple like that. Communication followed by computation. The communication is the self-attention part and the computation is the feedforward linear layer.
The block is built that way and the architecture consider this block times some number. So, the communication and computation in iteration. This poses challenges regarding the vanishing of the gradientes. Karpathy stated that the addition for many blocks did not improved the result.
Really great learning.