<aside> 💡
This was not in the lecture, I just investigated more by myself to get a better understanding of Normalization. Hope this material helps.
</aside>
In Deep Learning, Normalization(ì •ê·œí™”) is a technique used to standardize the data being fed into and through a neural network.
The main idea is to transform the features to be on a similar scale. This is done by adjusting the range of the data, often to have a mean of zero and a standard deviation of one.
This process is important because it helps to stabilize and speed up the training of deep neural networks. By ensuring that the inputs to each layer have a consistent distribution, normalization can prevent issues like vanishing or exploding gradients, which can hinder the learning process. Ultimately, this leads to more reliable and faster convergence to a good solution.
What are we exactly normalizing?
It depends on the type. But generally, we are trying to normalize the input of each layer before implementing activation function to it during the forward pass.
