First commit

LICENSE

@@ -1,3 +1,4 @@
+
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
@@ -186,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright 2018 Lancaster University, Queen’s University Belfast, Crown Copyright 2018 GCHQ
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,8 +1,66 @@
-# TENNISON
-TENNISON is an SDN security framework which is soon to appear in JSAC SI on SDN Scalabiltiy Issues.
 
-The TENNISON source code is currently undergoing licensing and is expected to be released to the public in the <b> 15th of October 2018</b>. 
 
-This repository will include TENNISON's source code, user applications, graphical interface, and the developer's guide for working with the system.
+![Tennison Logo](imgs/tennison_logo.png)
+
+TENNISON is a network security framework that harnesses Software Defined Networks.
 
 Please contact Lyndon at [email protected] about any questions on TENNISON.
+
+
+TENNISON's KSPs:
+* Extensibility
+* Holistic view
+* Rapid reaction
+* Transparency and interoperability
+* Kill chain detection support
+* Legacy network support
+
+TENNISON requires multiple components to function correctly. Below shows an
+overview of the system architecture.
+
+![Tennison Logo](imgs/tennison_overview.png)
+
+Please contact Lyndon at [email protected] about any questions on TENNISON.
+
+# License
+Gaffer is licensed under the Apache 2 license and is covered by [Crown Copyright](https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/copyright-and-re-use/crown-copyright/).
+
+
+# Getting started
+**Details on getting started with TENNISON are available in docs/developer_guide.pdf**
+
+---------------
+This repository is laid out as follows:
+
+#### _coordinator/_
+This is the primary component of TENNISON and is where the policy engine is
+located and is what decides what should happen to traffic. For extensibility it
+has southbound and northbound interfaces.
+The *southbound interfaces* are responsible for collecting a range of information
+from networks and hosts. The *northbound interface* provides users/developers
+with the ability to create their own security applications, providing TENNISON
+with rapid reaction capability.
+
+
+#### _onos-tennison-apps/_
+These applications interface with ONOS. They assist in montiroing and
+remediation, providing the primitives to interface with the network.
+
+#### _pig-relay/_
+This is a wrapper for snort that manages it, providing the coordinator with an 
+ability to update rules and also a method of alerting the coordinator on attack
+detection.
+
+
+#### _onos-security-pipeline/_
+This is the lowest level component of the system and sits at ONOS's driver layer
+and is what realises the OpenFlow pipeline.
+It has been created so that security and monitoring rules can be injected before
+any forwarding is applied. This makes the system transparent at the control
+plane meaning that it can work with any routing implementation. 
+
+#### _tools/_
+This directory provides scripts to automate the testing and deployment of
+TENNISON, reducing the learning curve to working with TENNISON. Most of these
+are wrapped in the *"tennison_experimenter"*
+application.

coordinator/.gitignore

@@ -0,0 +1,4 @@
+*__pycache__
+*.log
+*.pyc
+*.out

coordinator/ChangeLog

@@ -0,0 +1,68 @@
+483bd7c was Jamie Bird, 22 hours ago, message: Merged latest GUI
+a97f85f was Jamie Bird, 23 hours ago, message: Fixes to ipfix-portscan and ipfix-ddos
+3e586ec was Jamie Bird, 2 days ago, message: Added DDoS snort threshold
+d655e49 was Jamie Bird, 3 days ago, message: Variable/selective treatment fields.
+15e3427 was Jamie Bird, 3 days ago, message: Logging and config added to NBI
+cf6608b was Jamie Bird, 3 days ago, message: Add pkt field back to snort alerts
+2d694c7 was Jamie Bird, 7 days ago, message: Merge branch 'master' of d31-git.lancaster.ac.uk:mervyn/mervyn
+2ab9f00 was Jamie Bird, 7 days ago, message: 'Resolved' GUI permisson issue.
+b14aa56 was Lyndon, 8 days ago, message: Added IPFIX threshold for 20 -> 5 which mirrors to snort. Snort treatment will then block the flow if intrusion detected
+e889c6e was Jamie Bird, 8 days ago, message: Added gui src
+989f640 was Jamie Bird, 8 days ago, message: Merge branch 'master' of d31-git.lancaster.ac.uk:mervyn/mervyn
+b340990 was Jamie Bird, 8 days ago, message: Gracefull exiting of apps.
+7d137d6 was Lyndon, 8 days ago, message: Removed old TODO messages
+312f882 was Jamie Bird, 9 days ago, message: Added pip requirements.
+d4544b6 was Jamie Bird, 9 days ago, message: Integration of northbound apps. Including the GUI which is hosted.
+bc99918 was Jamie Bird, 10 days ago, message: Refactored snort thresholds to match on alertmsg not the rule. Coord no longer in charge of installing rules.
+7d442d7 was Jamie Bird, 11 days ago, message: Removed excessive printing
+c5c87a4 was Jamie Bird, 11 days ago, message: Extract pkt info from snort alerts to be used as treatment fields. Optional treatment_fields now specified in thresholds.
+b9f38fd was Jamie Bird, 2 weeks ago, message: Default applications loaded and started with mervyn.
+aad4d56 was Jamie Bird, 2 weeks ago, message: Started work on application management in the NBI
+cb1f32e was Jamie Bird, 2 weeks ago, message: Added Access-Control-Allow-Origin for gui
+8fa3dfc was Jamie Bird, 3 weeks ago, message: Fixed protocolIdentifier for sflow
+7448b57 was Jamie Bird, 3 weeks ago, message: Added snort/query to NBI for RT alerts
+c7e8495 was Jamie Bird, 3 weeks ago, message: Implemented avg_bps and avg_pps ipfix metrics
+9995083 was Jamie Bird, 3 weeks ago, message: yaml syntax fix
+73a3fa2 was Jamie Bird, 3 weeks ago, message: Save working version of thresholds to yaml to be reloaded in on restart
+1e6d658 was Jamie Bird, 3 weeks ago, message: Thresholds allow IP ranges for easier white/black list definitinos.
+e51ef4d was Jamie Bird, 4 weeks ago, message: Alternative fix for uint64 issue
+c51a2eb was Andrew Wright, 4 weeks ago, message: MongoDB 8-Byte issue
+7826062 was Andrew Wright, 4 weeks ago, message: Merge branch 'master' of d31-git.lancaster.ac.uk:mervyn/mervyn
+4927b52 was Andrew Wright, 4 weeks ago, message: Update for sFlow message handling format
+7bb7ca9 was Jamie Bird, 4 weeks ago, message: Removed sflow TODOs
+c659100 was Jamie Bird, 4 weeks ago, message: Merge branch 'master' of d31-git.lancaster.ac.uk:mervyn/mervyn
+a41d337 was Jamie Bird, 4 weeks ago, message: ipfix now a treatment. Fix ipfix to onos url protocolIdentifier translation
+a31be1d was Andrew Wright, 4 weeks ago, message: Updates for sFlow alert handling
+67e5ee9 was Andrew Wright, 4 weeks ago, message: Updates for sFlow alert handling
+ebe5439 was Jamie Bird, 4 weeks ago, message: IPFIX times now in ISO 8601
+a9de72f was Jamie Bird, 5 weeks ago, message: IPFIX thresholds complete
+29f1082 was Jamie Bird, 5 weeks ago, message: IPFIX(interfix) now periodic
+3e5590b was Jamie Bird, 6 weeks ago, message: Get switch dpid from ipfix messages
+2d4ce3d was Jamie Bird, 6 weeks ago, message: Coord NBI version 1 tested
+a76ac42 was Jamie Bird, 6 weeks ago, message: Update thresholds done. Remove thresholds done. Better return codes.
+29c72a8 was Jamie Bird, 6 weeks ago, message: IPFIX and snort thresholds now fully dynamic via NBI with required values check
+3b1b3f4 was Jamie Bird, 6 weeks ago, message: Snort query done. Threshold query done. Started work on threshold modification.
+5b3085b was Jamie Bird, 7 weeks ago, message: IPFIX query now timely on a per app basis
+b9e0159 was Jamie Bird, 7 weeks ago, message: Fixed Mac address serialisation in collectors
+e41c3dd was Jamie Bird, 7 weeks ago, message: NBI basics for IPFIX
+e730a74 was Andrew Wright, 7 weeks ago, message: Indentation fixes
+612bde3 was Jamie Bird, 7 weeks ago, message: Started coord NBI outline
+29af1b8 was Jamie Bird, 7 weeks ago, message: Timeouts and connection refused catches added to REST calls to avoid full coord crash.
+f33a8ce was Jamie Bird, 7 weeks ago, message: Alert path fix (temp)
+d69f3ea was Jamie Bird, 8 weeks ago, message: Merge
+ac9a6e8 was Jamie Bird, 8 weeks ago, message: Coord now report to multiple snort instances. Defined in config.yaml
+8e65cb1 was Jamie Bird, 8 weeks ago, message: Started work on snort management
+f55be78 was Jamie Bird, 8 weeks ago, message: Clear existing snort instances from onos on start up. TODO: Configure multiple snort instances in onos from config file
+6f2d439 was Jamie Bird, 9 weeks ago, message: Further TODOs
+5a9ce1a was Lyndon, 4 months ago, message: Added todo notes for integrating sFlowRT into decision making
+88ced10 was Lyndon, 4 months ago, message: Updated readme with collector changes
+3c9aa3c was Lyndon, 4 months ago, message: sFlowRT messages now pushed to ZeroMQ
+2a10ac3 was Lyndon, 4 months ago, message: Created new collector for sFlowRT alerts.
+f63653f was Lyndon John Fawcett, 9 weeks ago, message: Updated todo list
+1775a82 was Bryce Anderson-Cooper, 8 months ago, message: Updated README
+f7c9241 was Bryce Anderson-Cooper, 9 months ago, message: Fixed truffle HTTP POST message
+d6896b3 was Bryce Anderson-Cooper, 9 months ago, message: Fixed dictionary and string comparison
+2630e2f was Bryce Anderson-Cooper, 9 months ago, message: Fixed pyobj parsing and changed listening IP
+785f201 was Bryce Anderson-Cooper, 9 months ago, message: Fixed Snort Rule parsing
+21c0e37 was Bryce Anderson-Cooper, 9 months ago, message: Updated config and thresholds
+dd730f7 was Bryce Anderson-Cooper, 9 months ago, message: Updated Readme

coordinator/README.md

@@ -0,0 +1,34 @@
+# Coordinator
+
+The TENNISON Coordinator is responsible for receiving messages from potentially multiple IPFIX and Snort exporters. It will collect then parse these messages, and determine appropriate courses of action. The action will be determined by the highest matching threshold in the thresholds.yaml file. This may include upgrading monitoring from lightweight-detection (IPFIX) to heavyweight-detection (Snort). 
+
+## Architecture
+
+When initialised, tennison.py reads all the thresholds and asks rpc.py to send a list of all the Snort rules to add to pigrelay. tennison.py then starts up multiple collectors (collector.py) as daemons, with each collector only collecting one type of event (prefix, ipfix, interfix, snort, sFlowRT). These parse collected packets and place them into a distributed messaging queue (ZeroMQ). These messages are collected by tennison.py, to be passed on to messagehandler.py. At the same time a copy of the packet is sent to MongoDB. The message handler finds the highest matching threshold and sends the packet to treatment.py where the appropriate treatment from the threshold is picked. rpc.py sends a message on to the TENNISONAPI app with a request that matches the treatment. 
+
+## Dependencies
+
+TENNISON Coordinator is built using Python 3.x. It uses a MongoDB database to store messages received from the collectors. This can be located remotely if necessary. Please see the sample configuration file (examples/config.yaml) for details on how to connect to a remote database.
+
+## Installation
+
+To run the coordinator, you will first need to install the necessary Python dependencies. These can be installed with:
+
+```pip3 install -r pip3.3requirements.txt```
+
+```pip install -r pip2.7requirements.txt```
+
+
+TODO add requirements for applications and bower.
+
+Install python3 python python3-pip python-pip npm bower
+
+## Running
+
+Once the dependencies are installed, run the coordinator with:
+
+```sudo python3 tennison.py```
+
+Alternatively, the coordinator can also be run with a different configuration file:
+
+```sudo python3 tennison.py --config=other_configuration_file.yaml```

coordinator/TODO.md

@@ -0,0 +1,5 @@
+– Implement the InterFIX periodic polling.
+
+- Add collector (and everything associated with that in coordinator) for sFlow.
+
+Also see FIXME and TODO comments in-line.

coordinator/app-requirements.txt

@@ -0,0 +1,2 @@
+numpy==1.12.1
+pyflux==0.4.14

coordinator/apps/gui-legacy/README.md

@@ -0,0 +1,40 @@
+
+# IPFix Monitor Interface
+
+### Main files
+###### Javascript polling & chart file : js/pollingbck.js
+###### Configuration file : js/ipfixConfig.js (contains app-name, ports..etc. Can be utilised more)
+###### Home Page : ipfix-flows.html
+
+
+Go to http://127.0.0.1:8080/ to see the GUI
+
+
+
+Use the disable graphs flag to stop graphs from loading (currently this is required as a bug in the graphs halts the system).
+`http://127.0.0.1:8080/?disable_graphs=true`
+
+#### Configuration:
+
+Go to ipfixConfig.json to configure the ip address of the machine running TENNISON before starting the GUI.
+
+
+#### Procedure:
+
+1. The system registers the APP name, and makes the first call for the existing thresholds to render the chart and information DIV elements.
+2. Once the DIV elements scaffolding is in place, the system polls for 
+Threshold information from the NBI
+IPFix information from the NBI
+3. Once the information has arrived the system loops through the IPFix records to check if the record matches ALL fields of any thresholds in the threshold information given
+For each FULL fields match ascertained there is a further check to see if the IPfix record has been plotted already, and to ascertain a consistent source of information (ONLY accept IPFix records from one switch)
+4. The system then takes the most recent stored information from the last plot to calculate the period's throughput (currentOctetCount-previousOctetCount)/(currentRecordTime-previousRecordTime)
+5. Once this has been ascertained the flow information is stored in an array which keeps historical ipfix octetCount and flowEndMilliSecond time (or another timeframe depending on the ipfix subtype).
+6. Depending on what record subtype it is (ipfix,prefix,interfix) the system will put the data needed in the array for plotting data.
+- For a prefix record, the system plots 2 points : the flowStart time with 0 throughput as the first record, and the flowEnd time with the calculated throughput. This is so that the chart is seen staring from zero on the chart at the time of the flow starting.
+- For an interfix record it plots just one point, which has the current flowEnd time with the calculated throughput.
+- For an ipfix record, it plots 2 points : The current flowEnd time with the calculated throughput, and the CURRENT TIME with 0 throughput. This shows as the end of the flow on the chart.
+7. At the same time the current threshold value is collected and put into a array which will be fed to the same chart for the corresponding ipfix records to be plotted.
+8. The chart for the threshold being looped is then plotted and the the process (from point 3. ) is repeated until all records have been checked.
+
+The colours are selected randomly using a random colour generator, but the colour for a flow after generated is kept throughout the lifetime of a flow, even if it's started and stopped, unless the page is refreshed.
+

coordinator/apps/gui-legacy/app-management.html

@@ -0,0 +1,110 @@
+
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+
+    <meta charset="utf-8">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <meta name="description" content="">
+    <meta name="author" content="">
+
+    <title>IPFIX Monitor</title>
+
+    <!-- Bootstrap Core CSS -->
+    <link href="vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
+
+    <!-- MetisMenu CSS -->
+    <link href="vendor/metisMenu/metisMenu.min.css" rel="stylesheet">
+
+    <!-- Custom CSS -->
+    <link href="dist/css/sb-admin-2.css" rel="stylesheet">
+
+    <!-- Morris Charts CSS -->
+    <link href="vendor/morrisjs/morris.css" rel="stylesheet">
+
+    <!-- Custom Fonts -->
+    <link href="vendor/font-awesome/css/font-awesome.min.css" rel="stylesheet" type="text/css">
+
+    <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
+    <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+    <!--[if lt IE 9]>
+        <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
+        <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
+    <![endif]-->
+
+    <style>
+        .legendcolors {
+           float: left;
+           width: 12px;
+           height: 12px;
+           margin: 5px;
+           border: 1px solid rgba(0, 0, 0, .2);
+        }
+    </style>
+
+</head>
+
+<body>
+
+    <div id="wrapper">
+        <div id="page-wrapper">
+            <div class="row">
+                <div class="col-lg-12">
+                    <h1 class="page-header">Application Management</h1>
+                </div>
+                <!-- /.col-lg-12 -->
+            </div>
+            <!-- /.row -->
+
+
+            <div class="row">
+                <div class="col-lg-12">
+                    <div class="">
+
+                        <!-- .panel-heading -->
+                        <div class="panel-body">
+                            <div class="panel-group" id="accordion">
+
+                            </div>
+                        </div>
+                        <!-- .panel-body -->
+                    </div>
+                    <!-- /.panel -->
+                </div>
+                <!-- /.col-lg-12 -->
+            </div>
+            <!-- /.row -->
+
+        </div>
+        <!-- /#page-wrapper -->
+
+    </div>
+    <!-- /#wrapper -->
+
+    <!-- jQuery -->
+    <script src="vendor/jquery/jquery.min.js"></script>
+
+    <!-- Bootstrap Core JavaScript -->
+    <script src="vendor/bootstrap/js/bootstrap.min.js"></script>
+
+    <!-- Metis Menu Plugin JavaScript -->
+    <script src="vendor/metisMenu/metisMenu.min.js"></script>
+
+    <!-- Flot Charts JavaScript -->
+    <script src="vendor/flot/excanvas.min.js"></script>
+    <script src="vendor/flot/jquery.flot.js"></script>
+    <script src="vendor/flot/jquery.flot.pie.js"></script>
+    <script src="vendor/flot/jquery.flot.resize.js"></script>
+    <script src="vendor/flot/jquery.flot.time.js"></script>
+    <script src="vendor/flot-tooltip/jquery.flot.tooltip.min.js"></script>
+    <script src="js/ipfixConfig.js"></script>
+    <script src="js/apps.js"></script>
+
+    <!-- Custom Theme JavaScript -->
+    <script src="dist/js/sb-admin-2.js"></script>
+
+</body>
+
+</html>

coordinator/apps/gui-legacy/config.json

@@ -0,0 +1,3 @@
+{
+   "default_app" : false
+}