Fraud_Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Graph Analytics - Fraud Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/joerg84/Graph_Powered_ML_Workshop/blob/master/Fraud_Detection.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traditional fraud detection solutions view data through a straw, focusing on discrete data points such as specific accounts, individuals, devices or IP addresses. However, today’s sophisticated fraudsters escape detection by forming fraud rings or curious loops composed of stolen and synthetic identities and circuitous back channels. To uncover such fraud rings, it is essential to look beyond individual data points in individual data sources to a broader view of the connection patterns between them that exist in multiple disparate data sources.\n",
    " \n",
    "ArangoDB’s multi-model graph allows you to easily fuse together disparate data and identify complex fraudulent patterns of connections, such as fraud rings, using the ArangoDB Query Language (AQL). The identification of fraud ring patterns require very deep (multi-hop) traversals across the graph.  The query for detecting a fraud ring can be accomplished in six lines of (easy to write and maintain) AQL code and ArangoDB can execute these queries with sub-second response times.\n",
    "\n",
    "Thanks to https://twitter.com/arthurakeen for inspiration!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![fraud_overview](img/fraud_detection_collections.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, setting up our environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!git clone https://github.com/joerg84/Graph_Powered_ML_Workshop.git\n",
    "!rsync -av Graph_Powered_ML_Workshop/ ./ --exclude=.git\n",
    "!pip3 install pyarango"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import json\n",
    "import requests\n",
    "import sys\n",
    "import oasis\n",
    "\n",
    "\n",
    "from pyArango.connection import *\n",
    "from pyArango.collection import Collection, Edges, Field\n",
    "from pyArango.graph import Graph, EdgeDefinition\n",
    "from pyArango.collection import BulkOperation as BulkOperation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, create a temporary database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve tmp credentials from ArangoDB Tutorial Service\n",
    "login = oasis.getTempCredentials()\n",
    "\n",
    "# Connect to the temp database\n",
    "conn = oasis.connect(login)\n",
    "db = conn[login[\"dbName\"]] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"https://\"+login[\"hostname\"]+\":\"+str(login[\"port\"]))\n",
    "print(\"Username: \" + login[\"username\"])\n",
    "print(\"Password: \" + login[\"password\"])\n",
    "print(\"Database: \" + login[\"dbName\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to use to above URL to checkout the UI!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Import "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note the included arangorestore will only work on Linux system, if you want to run this notebook on a different OS please consider using the appropriate arangorestore from the [Download](https://www.arangodb.com/download-major/) area.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./tools/arangorestore -c none --server.endpoint http+ssl://{login[\"hostname\"]}:{login[\"port\"]} --server.username {login[\"username\"]} --server.database {login[\"dbName\"]} --server.password {login[\"password\"]} --default-replication-factor 3  --input-directory \"data/fraud_dump\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Graph "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph we will be using in the following analysis looks as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![graph](img/fraud_graph.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyArango.collection import Collection, Field\n",
    "from pyArango.graph import Graph, EdgeDefinition\n",
    "\n",
    "\n",
    "class account(Collection):\n",
    "    _fields = {\n",
    "        \"Name\": Field()\n",
    "    }\n",
    "    \n",
    "class customer(Collection):\n",
    "    _fields = {\n",
    "        \"Name\": Field()\n",
    "    }\n",
    "    \n",
    "class transaction(Edges): \n",
    "    _fields = {\n",
    "    }\n",
    "\n",
    "class accountHolder(Edges): \n",
    "    _fields = {\n",
    "    }\n",
    "\n",
    "class FraudDetection(Graph) :\n",
    "    _edgeDefinitions = [EdgeDefinition(\"accountHolder\", fromCollections=[\"customer\"], toCollections=[\"account\"]),EdgeDefinition(\"transaction\", fromCollections=[\"account\"], toCollections=[\"account\"])]\n",
    "    _orphanedCollections = []\n",
    "\n",
    "fraudGraph = db.createGraph(\"FraudDetection\")\n",
    "\n",
    "print(\"Collection/Graph Setup done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fraud Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will look at 3 different techniques to identify fraudulant behavior:\n",
    "\n",
    "1. Find long loops (potential fraud rings) from a suspicious account. \n",
    "2. Find all long loops (potential fraud rings)\n",
    "3. Find suspicious accounts (e.g. dormant or orphan accounts)\n",
    "4. Find disaggregation(fanout)/re-aggregation(fan-in) money laundering patterns. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Long Loops\n",
    "\n",
    "This query searches the whole graph for accounts involved in long loops, meaning transactions starting at one account and after a long circle (IN 5..10 OUTBOUND) the money gets back to the account that started the transaction.This looping behavior is important to us because it is a method for attempting to circumvent local laws. In really large datasets with millions of data points this can be a long running query but it should complete quickly with this example dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find loops for account 10000032\n",
    "loop_query = \"\"\"\n",
    "FOR accounts IN account\n",
    "   FOR acct, tx, path IN 5..10 OUTBOUND  'account/10000032'  GRAPH 'FraudDetection'\n",
    "   FILTER tx._to == 'account/10000032'\n",
    "RETURN DISTINCT path\n",
    "\"\"\"\n",
    "\n",
    "queryResult = db.AQLQuery(loop_query, rawResults=True)\n",
    "for result in queryResult:\n",
    "    print(result)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also execute the AQL part of the all the queries directly in the ArangoDB UI (using the above link and login). The result will include a handy graph representation for visual inspection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![result](img/fraud_loop.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find All Suspicious Long Loops\n",
    "In this step you can find all suspicious long loops of an account and its financial transactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find loops for all accounts\n",
    "loop_all_query = \"\"\"\n",
    "WITH transaction, account\n",
    "FOR suspicous_account IN account\n",
    "   FOR acct, tx, path IN 5..10 OUTBOUND suspicous_account._id  GRAPH 'FraudDetection'\n",
    "   PRUNE tx._to == suspicous_account._id\n",
    "   FILTER tx._to == suspicous_account._id\n",
    "RETURN  path\n",
    "\"\"\"\n",
    "\n",
    "queryResult = db.AQLQuery(loop_all_query, rawResults=True)\n",
    "for result in queryResult:\n",
    "    print(result)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find Orphan Accounts\n",
    "\n",
    "Orphan Accounts are accounts which have little or no transactions. These accounts may be set up in advance of money laundering operations. The query below finds accounts with no transactions. \n",
    "\n",
    "Note that the query below is not a graph query but a JOIN operation. You can see that we are nesting FOR loops which is a classical JOIN operation in ArangoDB (well, just without the JOIN keyword)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find orphaned accounts\n",
    "orphaned_query = \"\"\"\n",
    "LET usedResources = UNION_DISTINCT(\n",
    "  FOR relationship IN transaction RETURN relationship._from, \n",
    "  FOR relationship IN transaction RETURN relationship._to\n",
    ") \n",
    "FOR resource IN account \n",
    "  FILTER resource._id NOT IN usedResources \n",
    "  SORT resource.account_type, resource.customer_id \n",
    "  RETURN {\n",
    "  \"customerName\" : DOCUMENT(CONCAT(\"customer/\", resource.customer_id)).Name, \n",
    "  \"customerID\": resource.customer_id,\n",
    "  \"accountID\": resource._id, \n",
    "  \"type\": resource.account_type \n",
    "  }\n",
    "\"\"\"\n",
    "\n",
    "queryResult = db.AQLQuery(orphaned_query, rawResults=True)\n",
    "for result in queryResult:\n",
    "    print(result)\n",
    "    print()   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anti Money Laundering Pattern Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find transaction patterns that contain a disaggregation and re-aggregation of funds pattern.\n",
    "\n",
    " This pattern is characterized by transactions that dis-aggregate funds from a source account to multiple accounts in amounts that are below a reporting threshold, (e.g., just below $10,000) followed by a series of small transactions into 1 or more accounts, followed by re-aggregation\n",
    "    of the small transactions into a destination account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find aml account\n",
    "aml_query = \"\"\"\n",
    "WITH account, transaction\n",
    "LET accountOutDegree = (FOR transaction IN transaction\n",
    "    COLLECT accountOut = transaction._from WITH COUNT INTO outDegree\n",
    "    RETURN {account : accountOut, outDegree : outDegree})\n",
    "LET accountInDegree = (FOR transaction IN transaction\n",
    "    COLLECT accountIn = transaction._to WITH COUNT INTO inDegree\n",
    "    RETURN  {account : accountIn, inDegree : inDegree} )\n",
    "LET accountDegree = (FOR inRecord in accountInDegree\n",
    "   FOR outRecord in accountOutDegree\n",
    "   FILTER inRecord.account == outRecord.account \n",
    "   RETURN MERGE(inRecord, outRecord))\n",
    "LET maxAccount = (FOR maxDegree IN accountOutDegree \n",
    "                    FILTER maxDegree.outDegree == MAX(accountOutDegree[*].outDegree)\n",
    "                    RETURN maxDegree)[0]\n",
    "FOR account, transaction IN 1..4 OUTBOUND maxAccount.account transaction\n",
    "RETURN transaction\n",
    "\"\"\"\n",
    "\n",
    "queryResult = db.AQLQuery(aml_query, rawResults=True)\n",
    "for result in queryResult:\n",
    "    print(result)\n",
    "    print()  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete collections\n",
    "db.dropAllCollections() \n",
    "db.reload()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}