Create normalisation.md

normalisation.md

@@ -0,0 +1,37 @@
+# The Need for Normalization in Deep Learning
+
+Normalization is a crucial aspect of deep learning, addressing a fundamental challenge that arises during neural network training. The challenge is the variation in activation values across layers. When activations from previous layers exhibit high variation, it can lead to training instability. This instability results in slow convergence, vanishing gradients, and makes training deep neural networks challenging.
+
+In this blog, we will explore three key techniques for data normalization in deep learning: Batch Normalization (BN), Layer Normalization (LN), and Root Mean Square (RMS) Normalization. Each of these techniques has its own unique approach to tackling the problem of unstable activations, ultimately bringing stability to the model training process and accelerating convergence.
+
+![image](https://github.com/prasad4fun/ML-Portfolio/assets/12726341/f862e871-d926-49bd-91a6-3b8a735422d1)
+
+
+## Batch Normalization (BN)
+
+- **Batch-wise normalization:** BN calculates the mean and variance across the entire mini-batch, ensuring activations have a mean of 0 and a variance of 1.
+
+- **Effect on training:** BN is highly effective for stabilizing the training process, leading to faster convergence.
+
+- **Dependency on batch size:** BN's effectiveness can be influenced by the batch size but is generally robust.
+
+## Layer Normalization (LN)
+
+- **Example-wise normalization:** LN calculates the mean and variance for each feature (channel) within an individual example.
+
+- **Preservation of individual characteristics:** LN is suitable when you want to preserve the individual characteristics of each example.
+
+- **Training stability:** While LN is effective for various tasks, it may not be as efficient as BN in stabilizing the training process for very deep networks.
+
+## Root Mean Square (RMS) Normalization
+
+![image](https://github.com/prasad4fun/ML-Portfolio/assets/12726341/7a8d57d4-210f-46ec-895b-a010fe6b266a)
+
+
+- **Rescaling invariance:** RMS normalization emphasizes rescaling invariance.
+
+- **RMS statistic:** RMS(a) is calculated for each feature and normalized using a learnable parameter (gamma).
+
+- **Advantages:** RMS normalization requires less computation compared to layer normalization since it avoids the need to compute both mean and variance.
+
+In summary, the choice between BN, LN, and RMS normalization depends on the specific requirements of your deep learning model. These techniques bring stability to the training process, allowing for faster convergence and more effective training of deep neural networks. The best choice ultimately depends on your model and its specific needs.