02_networks/NetworkOperations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f6c2d90d",
   "metadata": {},
   "source": [
    "# Network Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8f51351",
   "metadata": {},
   "source": [
    "Let's see how the networks perform operations using the layers defined as modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2e5592a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "from torch import nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ac33f08",
   "metadata": {},
   "source": [
    "In **Linear Layers**, the operation is a matrix multiplication. Given the output $\\mathbf{y}_{i}$ of the $i$th layer, the output of the $(i+1)$th layer is $W_{i+1}\\mathbf{y}_{i}$, where $W_{i+1}$ is a matrix containing the weights of connections between layers $i$ and $i+1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d0ba316f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([30., 40., 50.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "in_features = torch.tensor([1,2,3,4], dtype=torch.float32)\n",
    "weight_matrix = torch.tensor([\n",
    "    [1,2,3,4],\n",
    "    [2,3,4,5],\n",
    "    [3,4,5,6]\n",
    "], dtype=torch.float32)\n",
    "weight_matrix.matmul(in_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e314bbd",
   "metadata": {},
   "source": [
    "The ```torch.tensor``` method has a callable attribute ```matmul()``` that can be used to perform matrix multiplications.\n",
    "\n",
    "The layers constructed using ```torch.nn``` modules have callable functions that perform the corresponding operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bdeb8a91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.0697, -0.1287,  0.4758, -0.0875],\n",
       "        [ 0.1852, -0.2088, -0.0620,  0.1778],\n",
       "        [-0.3900,  0.3577,  0.4638, -0.4139]], requires_grad=True)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense = nn.Linear(in_features=4, out_features=3)\n",
    "dense.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "108e6375",
   "metadata": {},
   "source": [
    "The ```nn.Linear``` module creates contains a weight matrix that is randomly initialised, which is a learnable parameter indicated by the ```requires_grad``` flag. The module is also callable, that provides the matrix multiplication operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cfb5b18e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.6775, 0.2688, 0.2498], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense(in_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ec6be2",
   "metadata": {},
   "source": [
    "The output is a tensor with the ```grad_fn``` attribute that keeps track of the gradient computations in the graph."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0b00da0",
   "metadata": {},
   "source": [
    "We can change the ```weight``` of the ```dense``` layer by using the ```nn.Parameter()``` wrapper class, to then get the same output as the initial construction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ecfa66a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[1., 2., 3., 4.],\n",
       "        [2., 3., 4., 5.],\n",
       "        [3., 4., 5., 6.]], requires_grad=True)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense.weight = nn.Parameter(weight_matrix)\n",
    "dense.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9048626a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([29.7880, 39.9760, 50.1886], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense(in_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81479ef4",
   "metadata": {},
   "source": [
    "The result is not accurate. This is due to the additional bias that the linear layer adds to the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "daf3ddb2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([-0.2120, -0.0240,  0.1886], requires_grad=True)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense.bias"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee23d7b",
   "metadata": {},
   "source": [
    "We can set this off in two ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "df2b86f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([30., 40., 50.], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dense = nn.Linear(in_features=4, out_features=3, bias=False)\n",
    "dense.bias = nn.Parameter(torch.zeros(3))\n",
    "dense(in_features)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}