07_Functions-Activation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3da69403",
   "metadata": {},
   "source": [
    "# Activation Functions\n",
    "\n",
    "Activation functions are mathematical operations applied to the output of a neural network layer. They introduce non-linearity into the network, enabling it to learn complex patterns and relationships in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af73eabc",
   "metadata": {},
   "source": [
    "### Sigmoid Function\n",
    "- Formula: $f(x) = \\frac{1}{1 + e^{-x}}$, the output is in the range (0, 1).\n",
    "- It squashes the input values between 0 and 1, which can be interpreted as probabilities. However, it suffers from the vanishing gradient problem (explained below) and is rarely used in hidden layers nowadays due to its limitations.\n",
    "- Derivative: $f'(x) = f(x) \\cdot (1 - f(x))$\n",
    "\n",
    "### Tanh Function\n",
    "- Formula: $f(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}}$, the output is in the range (-1, 1).\n",
    "- It squashes the input values between -1 and 1, which helps in centering the data around 0. It also suffers from the vanishing gradient problem.\n",
    "- Derivative: $f'(x) = 1 - f(x)^2$\n",
    "\n",
    "### ReLU Function\n",
    "- Formula: $f(x) = max(0, x)$, the output is in the range (0, $\\infty$).\n",
    "- It is the most widely used activation function in hidden layers. It is computationally efficient and helps in mitigating the vanishing gradient problem. However, it suffers from the dying ReLU problem.\n",
    "- Derivative: $f'(x) = 1$ if $x > 0$, else $0$\n",
    "\n",
    "### Leaky ReLU Function\n",
    "- Formula: $f(x) = max(\\alpha x, x)$, where $\\alpha$ is a small positive constant (e.g. 0.01). The output is in the range ($-\\infty$, $\\infty$).\n",
    "- It is similar to ReLU but allows a small, non-zero gradient when the input is negative. This helps alleviate the dying ReLU problem.\n",
    "\n",
    "### Softmax Function\n",
    "- Formula: $f(x_i) = \\frac{e^{x_i}}{\\sum_{j=1}^{n} e^{x_j}}$, the output is in the range (0, 1) and the sum of all outputs is 1.\n",
    "- It is used in the output layer of a neural network for multi-class classification problems. It converts the raw scores into probabilities, making it easier to interpret the output."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bd37378",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "### What is vanishing gradient?\n",
    "\n",
    "As we add more and more hidden layers, back propagation becomes less and less useful in passing information to the lower layers. In effect, as information is passed back, the gradients begin to vanish and become small relative to the weights of the networks.\n",
    "\n",
    "The vanishing gradient problem is particularly pronounced when using activation functions like sigmoid and tanh, which have derivatives that become very small for large or small inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f01dd477",
   "metadata": {},
   "source": [
    "## 1. RELU\n",
    "\n",
    "What are the advantages of ReLU over sigmoid function in deep neural networks?\n",
    "\n",
    "1. **Reduced likelihood of vanishing gradient**: $h$ arises when $a$>0. In this regime the gradient has a constant value. In contrast, the gradient of sigmoids becomes increasingly small as the absolute value of x increases. The constant gradient of ReLUs results in faster learning.\n",
    "2. **Sparsity**: Sparsity arises when $a$≤0. The more such units that exist in a layer the more sparse the resulting representation.\n",
    "3. **Better convergence performance**\n",
    "4. **More computationally efficient**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9563a0b",
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mFailed to start the Kernel. \n",
      "\u001b[1;31mrequest to http://localhost:8888/api/sessions?1675408565110 failed, reason: connect ETIMEDOUT 127.0.0.1:8888. \n",
      "\u001b[1;31mView Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "relu = torch.nn.ReLU()\n",
    "A = torch.randn(5)\n",
    "print(A)\n",
    "ans = relu(A)\n",
    "print(ans)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2693abf8",
   "metadata": {},
   "source": [
    "## 2. Sigmoid and Softmax\n",
    "\n",
    "Sigmoid formula: $f(x) = \\frac{1}{1+e^{-x}}$, and its derivative: $f'(x) = f(x)(1-f(x))$\n",
    "\n",
    "Softmax formula: $f(x) = \\frac{e^x}{\\sum_{i=1}^{n}e^x}$\n",
    "\n",
    "What is the difference between sigmoid and softmax functions?\n",
    "\n",
    "1. Sigmoid function is used in the output layer of a binary classification model. It squashes the output between 0 and 1. The output of sigmoid function is interpreted as the probability of the input belonging to class 1.\n",
    "2. Softmax function is used in the output layer of a multi-class classification model. It squashes the output between 0 and 1. The output of softmax function is interpreted as the probability of the input belonging to each class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2833e8d8",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- [https://stats.stackexchange.com/questions/126238/what-are-the-advantages-of-relu-over-sigmoid-function-in-deep-neural-networks](https://stats.stackexchange.com/questions/126238/what-are-the-advantages-of-relu-over-sigmoid-function-in-deep-neural-networks)\n",
    "- [https://towardsdatascience.com/fantastic-activation-functions-and-when-to-use-them-481fe2bb2bde](https://towardsdatascience.com/fantastic-activation-functions-and-when-to-use-them-481fe2bb2bde)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cb973f3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 ('py310')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "d1a940d2a5a085e4f840c6fa90dace1e4a81d9a7fba180f0fcbb4947149ff9d2"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}