The generation and optimization in simple neural networks
So, the forward and the generation are not quite the same process. But the generation process uses the forward and uses it without the targets. So, in this case, when generating characters or tokens, you do not consider the loss. And, as the generation process uses the forward, this has to be set in the forward function.
Interesting to discuss about the way for the generation process. It can consider the whole time-step until the considered character to be generated next or it can consider only the previous one.
For the simple model, Karpathy consider a general function and it takes only the last character in the sequence to generate the next character.
Karpathy discusses that the idea is to have a general generation function and compare the way to proceed.
For the simple neural network, it just consider the last character to generate the next character. But the process, he says, would basically be the same, so the general generation function.
For the optimization, Karpathy introduces the use for the AdamW class. He instantiates the object and discusses the learning rate hyperparameter for Adam. If the neural network is small, it can consider a not so small learning rate.
Really great learning.