initial commit

LICENSE

@@ -0,0 +1,35 @@
+/*
+ *	
+ * Copyright (c) 2019 Cisco Systems, Inc.
+ * All rights reserved.
+ * 
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 
+ *   Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ * 
+ *   Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ * 
+ *   Neither the name of the Cisco Systems, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived
+ *   from this software without specific prior written permission.
+ * 
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */

Makefile

@@ -0,0 +1,37 @@
+# Makefile for mercury
+#
+
+.PHONY: mercury test
+mercury:
+ifneq ($(wildcard src/Makefile), src/Makefile)
+	@echo "error: run ./configure before running make (src/Makefile is missing)"
+else
+	cd src && $(MAKE)
+endif
+
+test:
+	cd src && $(MAKE) test
+
+.PHONY: clean 
+clean:
+	for file in Makefile README.md configure.ac; do if [ -e "$$file~" ]; then rm -f "$$file~" ; fi; done
+ifneq ($(wildcard src/Makefile), src/Makefile)
+	@echo "error: run ./configure before running make (src/Makefile is missing)"
+	@/bin/false
+else
+	cd src && $(MAKE) clean
+	cd test && $(MAKE) clean
+endif
+
+.PHONY: distclean
+distclean: clean
+	rm -rf autom4te.cache config.log config.status
+ifneq ($(wildcard src/Makefile), src/Makefile)
+	@echo "error: run ./configure before running make (src/Makefile is missing)"
+	@/bin/false
+else
+	cd src  && $(MAKE) distclean
+	cd test && $(MAKE) distclean
+endif
+
+# EOF

README.md

@@ -1 +1,104 @@
-# mercury
+# Mercury: network fingerprinting and packet metadata capture
+<img align="right" src="./mercury.png" width="200"> 
+
+This package contains two programs for fingerprinting network traffic and capturing and analyzing packet metadata: **mercury**, a Linux application that leverages the modern Linux kernel's high-performance networking capabilities (AF_PACKET and TPACKETv3), which is described below, and [**pmercury**](python/README.md), a portable python application, which is described [here](python/README.md).
+
+## Building and running mercury
+In the root directory, run 
+```
+./configure 
+make
+```
+to build the package and install the required pip3 modules (dpkt ujson numpy pyasn hpack pypcap).  If you do not have **python3**, **cython**, and **pip3** installed, then you either need to install them (using apt, yum, or whatever your preferred package management tool is), or you need to run 
+```
+./configure --disable-python
+make
+```
+With the **--disable-python** flag, the configure script can build mercury in a way that omits the fingerprint analysis module (which is implemented using cython and python3).  Without the analysis module, mercury can still perform fingerprint and metadata capture.  
+
+#### Compile-time options
+There are compile-time options that can tune mercury for your hardware, or generate debugging output.  Each of these options is set via a C/C++ preprocessor directive, which should be passed as an argument to "make".   For instance, to turn on debugging, first run **make clean** to remove the previous build, then run **make "OPTFLAGS=-DDEBUG"**.   This runs make, telling it to pass the string "-DDEBUG" to the C/C++ compiler.  The available compile time options are:
+   * -DDEBUG, which turns on debugging, and
+   * -FBUFSIZE=16384, which sets the fwrite/fread buffer to 16,384 bytes.
+If multiple compile time options are used, then they must be passed to make together in the OPTFLAGS string, e.g. "OPTFLAGS=-DDEBUG -DFBUFSIZE=16384".
+
+### Running mercury
+```
+mercury: packet metadata capture and analysis
+./src/mercury INPUT [OUTPUT] [OPTIONS]:
+INPUT
+   [-c or --capture] capture_interface   # capture packets from interface
+   [-r or --read] read_file              # read packets from file
+OUTPUT
+   [-f or --fingerprint] json_file_name  # write fingerprints to JSON file
+   [-w or --write] pcap_file_name        # write packets to PCAP/MCAP file
+   no output option                      # write JSON packet summary to stdout
+--capture OPTIONS
+   [-b or --buffer] b                    # set RX_RING size to (b * PHYS_MEM)
+   [-t or --threads] [num_threads | cpu] # set number of threads
+   [-u or --user] u                      # set UID and GID to those of user u
+--read OPTIONS
+   [-m or --multiple] count              # loop over read_file count >= 1 times
+GENERAL OPTIONS
+   [-a or --analysis]                    # analyze fingerprints
+   [-s or --select]                      # select only packets with metadata
+   [-l or --limit] l                     # rotate JSON files after l records
+   [-h or --help]                        # extended help, with examples
+```
+
+### Details
+
+   **[-c or --capture] c** captures packets from interface c with Linux AF_PACKET
+   using a separate ring buffer for each worker thread.  **[-t or --thread] t**
+   sets the number of worker threads to t, if t is a positive integer; if t is
+   "cpu", then the number of threads will be set to the number of available
+   processors.  **[-b or --buffer] b** sets the total size of all ring buffers to
+   (b * PHYS_MEM) where b is a decimal number between 0.0 and 1.0 and PHYS_MEM
+   is the available memory; USE b < 0.1 EXCEPT WHEN THERE ARE GIGABYTES OF SPARE
+   RAM to avoid OS failure due to memory starvation.  When multiple threads are
+   configured, the output is a *file set*: a directory into which each thread
+   writes its own file; all packets in a flow are written to the same file.
+
+   **[-f or --fingerprint] f** writes a JSON record for each fingerprint observed,
+   which incorporates the flow key and the time of observation, into the file or
+   file set f.  With **[-a or --analysis]**, fingerprints and destinations are
+   analyzed and the results are included in the JSON output.  The analysis output
+   is documented [in the pmercury README](python/README.md).
+
+   **[-w or --write] w** writes packets to the file or file set w, in PCAP format.
+   With **[-s or --select]**, packets are filtered so that only ones with
+   fingerprint metadata are written.
+
+   **[r or --read] r** reads packets from the file or file set r, in PCAP format.
+   A single worker thread is used to process each input file; if r is a file set
+   then the output will be a file set as well.  With **[-m or --multiple] m**, the
+   input file or file set is read and processed m times in sequence; this is
+   useful for testing.
+
+   **[-u or --user] u** sets the UID and GID to those of user u; output file(s)
+   are owned by this user.  With **[-l or --limit] l**, each JSON output file has
+   at most l records; output files are rotated, and filenames include a sequence
+   number.
+
+   **[-h or --help]** writes this extended help message to stdout.
+
+### Examples
+```bash
+   mercury -c eth0 -w foo.pcap           # capture from eth0, write to foo.pcap
+   mercury -c eth0 -w foo.pcap -t cpu    # as above, with one thread per CPU
+   mercury -c eth0 -w foo.mcap -t cpu -s # as above, selecting packet metadata
+   mercury -r foo.mcap -f foo.json       # read foo.mcap, write fingerprints
+   mercury -r foo.mcap -f foo.json -a    # as above, with fingerprint analysis
+   mercury -c eth0 -t cpu -f foo.json -a # capture and analyze fingerprints
+```
+
+## Ethics
+Mercury is intended for defensive network monitoring and security research and forensics.  Researchers, administrators, penetration testers, and security operations teams can use these tools to protect networks, detect vulnerabilities, and benefit the broader community through improved awareness and defensive posture. As with any packet monitoring tool, Mercury could potentially be misused. **Do not run it on any network of which you are not the owner or the administrator**.
+
+## Credits
+Mercury was developed by David McGrew, Brandon Enright, Blake Anderson, Shekhar Acharya, and Adam Weller, with input from Brian Long, Bill Hudson, and others.  Pmercury was developed by Blake Anderson, with input from the others.  
+
+## Acknowledgments
+This software includes GeoLite2 data created by MaxMind, available from [https://www.maxmind.com](https://www.maxmind.com).
+
+We make use of Mozilla's [Public Suffix List](https://publicsuffix.org/list/) which is subject to the terms of the [Mozilla Public License, v. 2.0](https://mozilla.org/MPL/2.0/).