Node_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Graph Learning Tasks\n",
    "In this tutorial, we are going to demonstrate some basic tasks in graph learning. In general, many of the graph learning problems can fall into the following categories:\n",
    "\n",
    "* Node classification: assign a label to a node.\n",
    "* Link prediction: predict the existence of an edge between two nodes.\n",
    "* Graph classification: assign a label to a graph.\n",
    "\n",
    "Many real-world applications can be formulated as one of these graph problems.\n",
    "* Fraud detection in financial transactions: transactions form a graph, where users are nodes and transactions are edges. In this case, we want to detect malicious users, which is to assign binary labels to users.\n",
    "* Community detection in a social network: a social network is naturally a graph, where nodes are users and edges are interactions between users. We want to predict which community a node belongs to.\n",
    "* Recommendation: users and items form a bipartite graph. They are connected with edges when users purchase items. Given users' purchase history, we want to predict what items a user will purchase in a near future. Thus, recommendation is a link prediction problem.\n",
    "* Drug discovery: a molecule is a graph whose nodes are atoms. We want to predict the property of a molecule. In this case, we want to assign a label to a graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get started\n",
    "\n",
    "DGL can be used with different deep learning frameworks. Currently, DGL can be used with Pytorch and MXNet. Here, we show how DGL works with Pytorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we load DGL, we need to set the DGL backend for one of the deep learning frameworks. Because this tutorial develops models in Pytorch, we have to set the DGL backend to Pytorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import dgl\n",
    "from dgl import DGLGraph\n",
    "\n",
    "# Load Pytorch as backend\n",
    "dgl.load_backend('pytorch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the rest of the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GNN model\n",
    "\n",
    "Typically, GNN is used to compute meaningful node embeddings. With the embeddings, we can perform many downstream tasks.\n",
    "\n",
    "DGL provides two ways of implementing a GNN model:\n",
    "* using the [nn module](https://doc.dgl.ai/features/nn.html), which contains many commonly used GNN modules.\n",
    "* using the message passing interface to implement a GNN model from scratch.\n",
    "\n",
    "For simplicity, we implement the GNN model in the tutorial with the nn module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we use [GraphSage](https://cs.stanford.edu/people/jure/pubs/graphsage-nips17.pdf), one of the first inductive GNN models. GraphSage performs the following computation on every node $v$ in the graph:\n",
    "\n",
    "$$h_{N(v)}^{(l)} \\gets AGGREGATE_k({h_u^{(l-1)}, \\forall u \\in N(v)})$$\n",
    "$$h_v^{(l)} \\gets \\sigma(W^k \\cdot CONCAT(h_v^{(l-1)}, h_{N(v)}^{(l)})),$$\n",
    "\n",
    "where $N(v)$ is the neighborhood of node $v$ and $l$ is the layer Id."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The GraphSage model has multiple layers. In each layer, a vertex accesses its direct neighbors. When we stack $k$ layers in a model, a node $v$ access neighbors within $k$ hops. The output of the GraphSage model is node embeddings that represent the nodes and all information in the k-hop neighborhood.\n",
    "\n",
    "<img src=\"https://github.com/zheng-da/DGL_devday_tutorial/raw/master/GNN.png\" alt=\"drawing\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use DGL's `nn` module to build the GraphSage model. `SAGEConv` implements the operations of `GraphSage` in a layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from dgl.nn.pytorch import conv as dgl_conv\n",
    "\n",
    "class GraphSAGEModel(nn.Module):\n",
    "    def __init__(self,\n",
    "                 in_feats,\n",
    "                 n_hidden,\n",
    "                 out_dim,\n",
    "                 n_layers,\n",
    "                 activation,\n",
    "                 dropout,\n",
    "                 aggregator_type):\n",
    "        super(GraphSAGEModel, self).__init__()\n",
    "        self.layers = nn.ModuleList()\n",
    "\n",
    "        # input layer\n",
    "        self.layers.append(dgl_conv.SAGEConv(in_feats, n_hidden, aggregator_type,\n",
    "                                         feat_drop=dropout, activation=activation))\n",
    "        # hidden layers\n",
    "        for i in range(n_layers - 1):\n",
    "            self.layers.append(dgl_conv.SAGEConv(n_hidden, n_hidden, aggregator_type,\n",
    "                                             feat_drop=dropout, activation=activation))\n",
    "        # output layer\n",
    "        self.layers.append(dgl_conv.SAGEConv(n_hidden, out_dim, aggregator_type,\n",
    "                                         feat_drop=dropout, activation=None))\n",
    "\n",
    "    def forward(self, g, features):\n",
    "        h = features\n",
    "        for layer in self.layers:\n",
    "            h = layer(g, h)\n",
    "        return h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interested readers can check out our [online tutorials](https://doc.dgl.ai/tutorials/models/index.html) to see how to use DGL's message passing interface to implement GNN models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the dataset for the tutorial\n",
    "\n",
    "DGL has a large collection of built-in datasets. Please see [this doc](https://doc.dgl.ai/api/python/data.html) for more information.\n",
    "\n",
    "In this tutorial, we use a citation network called pubmed for demonstration. A node in the citation network is a paper and an edge represents the citation between two papers. This dataset has 19,717 papers and 88,651 citations. Each paper has a sparse bag-of-words feature vector and a class label.\n",
    "\n",
    "All other graph data, such as node features, are stored as NumPy tensors. When we load the tensors, we convert them to Pytorch tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from dgl.data import citegrh\n",
    "\n",
    "# load and preprocess the pubmed dataset\n",
    "data = citegrh.load_pubmed()\n",
    "\n",
    "# sparse bag-of-words features of papers\n",
    "features = torch.FloatTensor(data.features)\n",
    "# the number of input node features\n",
    "in_feats = features.shape[1]\n",
    "# class labels of papers\n",
    "labels = torch.LongTensor(data.labels)\n",
    "# the number of unique classes on the nodes.\n",
    "n_classes = data.num_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "print(n_classes)\n",
    "print(data.labels.shape)\n",
    "print(features[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For small datasets, DGL stores the network structure in a [NetworkX](https://networkx.github.io) object. NetworkX is a very popular Python graph library. It provides comprehensive API for graph manipulation and is very useful for preprocessing small graphs.\n",
    "\n",
    "Then we create a DGLGraph from the grpah dataset and convert it to a read-only DGLGraph, which supports more efficient computation. Currently, DGL sampling API only works on read-only DGLGraphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "g = DGLGraph(data.graph)\n",
    "g.readonly()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Node classification in the semi-supervised setting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us perform node classification in a semi-supervised setting. In this setting, we have the entire graph structure and all node features. We only have labels on some of the nodes. We want to predict the labels on other nodes. Even though some of the nodes do not have labels, they connect with nodes with labels. Thus, we train the model with both labeled nodes and unlabeled nodes. Semi-supervised learning can usually improve performance.\n",
    "\n",
    "<img src=\"https://github.com/zheng-da/DGL_devday_tutorial/raw/master/node_classify1.png\" alt=\"drawing\" width=\"200\"/>\n",
    "\n",
    "This dependency graph shows a better view of how labeled and unlabled nodes are used in the training.\n",
    "<img src=\"https://github.com/zheng-da/DGL_devday_tutorial/raw/master/node_classify2.png\" alt=\"drawing\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we create a 2-layer GraphSage model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# Hyperparameters\n",
    "n_hidden = 64\n",
    "n_layers = 2\n",
    "dropout = 0.5\n",
    "aggregator_type = 'gcn'\n",
    "\n",
    "gconv_model = GraphSAGEModel(in_feats,\n",
    "                             n_hidden,\n",
    "                             n_classes,\n",
    "                             n_layers,\n",
    "                             F.relu,\n",
    "                             dropout,\n",
    "                             aggregator_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create the node classification model based on the GraphSage model. The GraphSage model takes a DGLGraph object and node features as input and computes node embeddings as output. With node embeddings, we use a cross entropy loss to train the node classification model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create the node classification model based on the GraphSage model. The GraphSage model takes a DGLGraph object and node features as input and computes node embeddings as output. With node embeddings, we use a cross entropy loss to train the node classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "class NodeClassification(nn.Module):\n",
    "    def __init__(self, gconv_model, n_hidden, n_classes):\n",
    "        super(NodeClassification, self).__init__()\n",
    "        self.gconv_model = gconv_model\n",
    "        self.loss_fcn = torch.nn.CrossEntropyLoss()\n",
    "\n",
    "    def forward(self, g, features, train_idx):\n",
    "        logits = self.gconv_model(g, features)\n",
    "        return self.loss_fcn(logits[train_idx], labels[train_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After defining a model for node classification, we need to define an evaluation function to evaluate the performance of a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "def NCEvaluate(model, g, features, labels, test_idx):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        # compute embeddings with GNN\n",
    "        logits = model.gconv_model(g, features)\n",
    "        logits = logits[test_idx]\n",
    "        test_labels = labels[test_idx]\n",
    "        _, indices = torch.max(logits, dim=1)\n",
    "        correct = torch.sum(indices == test_labels)\n",
    "        return correct.item() * 1.0 / len(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare data for semi-supervised node classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# the dataset is split into training set, validation set and testing set.\n",
    "train_idx = np.where(data.train_mask > 0)[0]\n",
    "val_idx = np.where(data.val_mask > 0)[0]\n",
    "test_idx = np.where(data.test_mask > 0)[0]\n",
    "\n",
    "print(\"\"\"----Data statistics------'\n",
    "      #Classes %d\n",
    "      #Train samples %d\n",
    "      #Val samples %d\n",
    "      #Test samples %d\"\"\" %\n",
    "          (n_classes,\n",
    "           train_idx.shape[0],\n",
    "           val_idx.shape[0],\n",
    "           test_idx.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After defining the model and evaluation function, we can put everything into the training loop to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Node classification task\n",
    "model = NodeClassification(gconv_model, n_hidden, n_classes)\n",
    "\n",
    "# Training hyperparameters\n",
    "weight_decay = 5e-4\n",
    "n_epochs = 150\n",
    "lr = 1e-3\n",
    "\n",
    "# create the Adam optimizer\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)\n",
    "\n",
    "dur = []\n",
    "for epoch in range(n_epochs):\n",
    "    # Set the model in the training mode.\n",
    "    model.train()\n",
    "    # forward\n",
    "    loss = model(g, features, train_idx)\n",
    "    \n",
    "    # backward\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # validation\n",
    "    acc = NCEvaluate(model, g, features, labels, val_idx)\n",
    "    print(\"Epoch {:05d} | Loss {:.4f} | Accuracy {:.4f}\"\n",
    "          .format(epoch, loss.item(), acc))\n",
    "\n",
    "print()\n",
    "acc = NCEvaluate(model, g, features, labels, test_idx)\n",
    "print(\"Test Accuracy {:.4f}\".format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Take home exercise\n",
    "\n",
    "An interested user can try other GNN models to compute node embeddings and use it for node classification. Please check out the [nn module](https://doc.dgl.ai/features/nn.html) in DGL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PyCharm (DGL_Internal_Workshop_April2020)",
   "language": "python",
   "name": "pycharm-6c9a77b7"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}