T2 release!

Initial release of the Nightmare fuzzing suite for the T2 conference in Finland.

AUTHORS

@@ -0,0 +1,2 @@
+Joxean Koret (admin [AT] joxeankoret DOT com)
+

NEWS.txt

@@ -0,0 +1,21 @@
+Nightmare 0.0.4 - ?????????
+---------------------------
+
+ * Added option in the web interface to download project results (only one crash per different bug + corresponding diff file if any).
+ * Ported Nightmare to SQLite database. You can use either MySQL or SQLite now.
+ * Added mutator ole_file_mutator.py and ole_file_mutator_multiple.py to fuzz OLE2 documents (surprise) using OleFileIO_PL.
+
+Nightmare 0.0.3 - July 2014
+---------------------------
+
+ * Added generic_client_server.py fuzzer.
+ * Added support for minimizing crash files.
+ * Added blind_coverage_fuzzer.py fuzzer.
+ * Added mutator zzuf_mutator.py and zzuf_mutator_multiple.py.
+ * Added support to find the original template file for a crash file.
+ * Added a half intelligent MachO format aware mutator.
+ * The "Charlie Miller" mutators have been enhanced and now they are 
+   fast. Before, they were painfully slow.
+
+---
+Copyright (c) 2013, 2014 Joxean Koret

README.txt

@@ -0,0 +1,23 @@
+Nightmare
+---------
+
+Nightmare is a simple fuzzing suite that was created for an underground
+conference (LaCon 2013). It was later on enhanced for the conference
+SYSCAN 2014 (www.syscan.org), is actively mantained and was released for
+T2 2014 conference.
+
+Important notes
+---------------
+
+PLEASE DO READ ALL OF THE (SMALL) .TXT FILES IN THE "DOC/" DIRECTORY.
+I'LL NOT ANSWER QUESTIONS ALREADY ANSWERED THERE (unless friend).
+
+Install
+-------
+
+Please, refer to doc/install.txt for installation details.
+
+Contact
+-------
+
+joxeankoret AT yahoo DOT es

TODO.txt

@@ -0,0 +1,17 @@
+Nigtmare
+--------
+
+ [ ] Crash verifier.
+ [ ] Specific sample from a set finder.
+ [ ] Valgrind runner.
+ [ ] Exploitability checker.
+ [/] Test case reduction.
+  [ ] Test case reduction for highly changed files.
+  [ ] Build diff files for binary patches.
+  [ ] Improve nfp_min.py to handle such diff files.
+ [ ] Integrate 'bcf' with the queues system.
+  [ ] Providing maximized versions of the files?
+  [ ] Use it for generating maximized files.
+  [ ] Generate nice genetic graphs with this tool.
+ [ ] Mutators
+  [ ] A "chunkenizer" for binary data.

doc/architecture.txt

@@ -0,0 +1,53 @@
+Architecture of nightmare
+-------------------------
+
+Nightmare is a fuzzing suite that was created, initially, to do a talk
+about fuzzing in an underground conference. This fuzzing suite is a 
+distributed fuzzing suite with central management, which means that:
+
+ * Clients can run any where.
+ * Management interface run only in one host.
+
+Components
+----------
+
+There are 5 main components in Nightmare:
+
+ * Beanstalkd: This is the queue server used in Nightmare. Connections
+   to the queue server are established by both client and server
+   processes.
+ * Database: The MySQL (currently) database server will be used to store
+   all the crash information (register values at crashing time, memory 
+   at crashing registers, signal received, etc...). Only the server 
+   processes (nfp_engine.py and nightmare_frontend.py) needs to connect
+   to the database.
+ * nfp_engine.py: This tool is the responsible of creating the mutated
+   samples for the projects enabled at that moment and storing back in
+   the database all the crashes. For this purpose it uses a serie of
+   beanstalk tubes to put samples and retrieve the crashes, if any.
+ * nightmare_fronted.py: This is the web.py application used to check
+   the results of the fuzzing sessions as well as to configure mutators,
+   projects, manage crashes, etc...
+ * Clients: The clients are the fuzzers. The client can be any tool that
+   connects to the appropriate tube to pull samples from and pushes back
+   crash information in the, again, appropriate tube.
+
+There are 2 generic clients with this version of Nightmare:
+
+ * generic_fuzzer.py
+ * generic_client_server.py
+
+The 1st one can be used in either Windows or Linux (as it was tested in
+both operating systems) and the last one is only for fuzzing client and
+server application under Unix/Linux.
+
+The clients, as explained before, can be deployed any where as long as
+they can connect to the port where the beanstalkd is listening for new
+connections.
+
+And this is the basic explaination of the architecture of Nightmare. If
+you have any question, drop an e-mail at joxeankoret AT yah00 DOT es or
+write me a message in twitter (@matalaz).
+
+---
+Copyright (c) 2013, 2014 Joxean Koret

doc/beanstalk/beanstalkd.recommended

@@ -0,0 +1,9 @@
+## Defaults for the beanstalkd init script, /etc/init.d/beanstalkd on
+## Debian systems. Append ``-b /var/lib/beanstalkd'' for persistent
+## storage.
+BEANSTALKD_LISTEN_ADDR=0.0.0.0
+BEANSTALKD_LISTEN_PORT=11300
+DAEMON_OPTS="-l $BEANSTALKD_LISTEN_ADDR -p $BEANSTALKD_LISTEN_PORT -z 55000000"
+
+## Uncomment to enable startup during boot.
+START=yes

doc/blind_code_coverage_fuzzer.txt

@@ -0,0 +1,144 @@
+Blind Code Coverage Fuzzer (BCCF)
+---------------------------------
+
+This is a new addition to the Nightmare fuzzing suite. This tool tries
+to do one of the following 3 tasks:
+
+ * Maximize code coverage of one sample.
+ * Create new generations that can be used as templates for fuzzing
+   covering more features that then original template.
+ * Discover bugs by executing code not typically covered.
+
+It can be considered an evolutionary fuzzer.
+
+How it works
+------------
+
+This fuzzer takes as input some template file and tries to make changes
+here and there, randomly, in order to increase code coverage. The target
+application will run under instrumentation (currently using DynamoRIO)
+in order to be able to determine the total number of different basic 
+blocks that were executed in each run.
+
+Initially, before making mutations, it will meassure after a number of
+executions (10 by default) that maximum, minimum and average number of
+different basic blocks executed per each execution (as it may vary 
+depending on the target application) and then start making many random 
+modifications any where in the sample in order to maximize execution. By
+default, a maximum number of 10 generations (i.e., modifications that
+caused the application to execute more new basic blocks) will be saved
+in memory.
+
+A generic is dropped after a number of "failures". I mean, if a single
+modification makes it execute more basic blocks but after a number of
+added modifications the number of basic blocks is always the same or it
+drops to a bad number, the generation will be eventually dropped. If, on
+the other hand, a new change discover more new basic blocks, the current
+generation and statistics will be stored (in memory) and the last new
+generation will be considered the current generation.
+
+While this approach is not bullet proof, it certainly can:
+
+ * Discover new bugs.
+ * Maximize code coverage of a sample or set of samples in order to use
+   them as new templates for another mutator (like radamsa).
+
+Configuration
+-------------
+
+This is an experimental tool and, as such, there are so many little
+configuration directives and options. However, using the supplied .cfg
+file (bcf.cfg) starting with this fuzzer is easy. 
+
+For example, let's say that you want to fuzz the tool "readelf". We need
+to configure in bcf.cfg:
+
+ * The BCF tool section.
+ * The DynamoRIO section.
+ * The specific target section.
+
+For BCF and DynamoRIO the following options lines are all we need:
+
+#-----------------------------------------------------------------------
+# Configuration for the BCF fuzzer
+#-----------------------------------------------------------------------
+[BCF]
+templates-path=/home/joxean/Documentos/research/nightmare/samples/av
+# Use *ONLY* iterative algorithm instead of all algorithms?
+#iterative=1
+# Use *ONLY* radamsa instead of all the implemented algorithms?
+#radamsa=1
+
+#-----------------------------------------------------------------------
+# Configuration for DynamoRIO (requied for code coverage)
+#-----------------------------------------------------------------------
+[DynamoRIO]
+path=/home/joxean/devel/dynamorio/DynamoRIO-Linux-4.2.0-3/
+
+The options "iterative" and "radamsa" are completely uninteresting yet,
+so let's forget them for now. After those lines we will need to add the
+directive for the target application we want to fuzz, in our example
+case, "readelf". The following lines are what we need:
+
+#-----------------------------------------------------------------------
+# Configuration for "readelf" tool
+#-----------------------------------------------------------------------
+[readelf]
+# Command line to launch it
+command=/usr/bin/readelf -a
+# Base tube name
+basetube=readelf
+# The tube the fuzzer will use to pull of samples 
+tube=%(basetube)s-samples
+# The tube the fuzzer will use to record crashes
+crash-tube=%(basetube)s-crash
+# Extension for the files to be fuzzed
+extension=.fil
+# Timeout for this fuzzer
+timeout=15
+# Environment
+environment=common-environment
+current-state-file=current-state-readelf
+generation-bottom-level=-500
+hide-output=1
+skip-bytes=4
+non-uniques=True
+#iterative=1
+#radamsa=1
+
+[common-environment]
+MALLOC_CHECK_=2
+
+The options are either self-explanatory or already commented so I think
+there is no need to go through all of them with the only exception of
+the following ones:
+
+ * iterative: Use the iterative algorithm? The iterative algorithm will
+   make modifications byte-per-byte starting from the value "skip-bytes"
+   value.
+ * radamsa: Use only radamsa to make new generations?
+
+Both configuration directives can be set at either BCF level (global) or
+specific fuzzer level (the recommended setting).
+
+Once we have the BCF tool, DynamoRIO section and our specific target
+configuration section(s) already configured in our bcf.cfg file we need
+to simply run the following command:
+
+ $ cd $NIGHTMARE_DIR/fuzzers
+ $ bcf.py 32 bcf.cfg readelf inputtemplate output_dir
+
+The first argument given to bcf.py is the architecture: 32 for 32bits
+architectures (Intel x86) and 64 for 64bits (x86_64).
+
+As for the input template(s): this tool is can be considered "smart". We
+can feed to it an (almost) empty file and it will discover the relevant
+modifications required to execute more code and, hopefully, discover new
+bugs. For example, given a 1024 bytes file filled with 0x00 characters
+and only with the typical "\x1FELF" header, the tool will manage to
+create files that will make to crash "readelf" or, at the very least,
+that will execute more code (new different basic blocks) in the target
+application.
+
+---
+Copyright (c) 2013, 2014 Joxean Koret

doc/generic_client_server.txt

@@ -0,0 +1,50 @@
+Generic client/server fuzzer
+----------------------------
+
+This fuzzer was created with the idea of fuzzing client/server Unix
+applications. In order to use this fuzzing it must be created a new
+section in the generic.cfg configuration file (or any other *.cfg file)
+similar to the following (highly commented) one:
+
+#-----------------------------------------------------------------------
+# Configuration for Avast server version
+#-----------------------------------------------------------------------
+[AvastServer]
+# Command line to launch clients
+client-command=/bin/scan -a -p -f
+# UID and GID for the client processes
+client-uid=1000
+client-gid=1000
+# Command line to spawn the server
+command=/bin/avast -n
+# UID and GID for the server process
+server-uid=0
+server-gid=0
+# Command to run before starting up the server process
+pre-command=killall -9 avast
+# Base tube name
+basetube=avast
+# The tube the fuzzer will use to pull of samples 
+tube=%(basetube)s-samples
+# The tube the fuzzer will use to record crashes
+crash-tube=%(basetube)s-crash
+# Extension for the files to be fuzzed
+extension=.fil
+# Timeout for this fuzzer (every month...)
+timeout=25920000
+
+Once a fuzzer section similar to the previous one is added, the fuzzer
+can be executed issuing a command like the following one:
+
+ $ python generic_client_server.py file.cfg SectionName
+
+For example, in order to run the Avast client/server fuzzer:
+
+ $ python generic_client_server.py generic.cfg AvastServer
+
+And that's all! It will take care of spawning the client and server
+processes as well as catching signals and storing in the database all
+the relevant crashing information, diff files, POCs, etc...
+
+---
+Copyright (c) 2013, 2014 Joxean Koret

doc/generic_fuzzer.txt

@@ -0,0 +1,55 @@
+Generic fuzzer
+--------------
+
+The generic_fuzzer.py is, naturally, the generic command line fuzzer
+for Nightmare. It can be used to fuzz any application that:
+
+ * Does not fork and create sub-processes (that we want to inspect).
+ * Does not connect to server applications where the crash should be
+   catched instead of in the client application.
+
+For example, let's say that you want to fuzz the typical Unix antivirus
+"ClamAV". There are 2 tools: clamscan and clamdscan. The former is the
+independent analyser, the last one (clamdscan) is a tool that connects
+to a socket. This fuzzer would only be able to correctly fuzz the 1st
+of the tools.
+
+Once this is explained, we need to add a configuration directive for
+this antivirus we want to fuzz in either generic.cfg or in any .cfg file
+we want. The following is the currently in use configuration settings
+for ClamAV available in generic.cfg:
+
+#-----------------------------------------------------------------------
+# Configuration for ClamAV
+#-----------------------------------------------------------------------
+[ClamAV]
+# Command line to launch it
+command=/usr/bin/clamscan --quiet
+# Base tube name
+basetube=clamav
+# The tube the fuzzer will use to pull of samples 
+tube=%(basetube)s-samples
+# The tube the fuzzer will use to record crashes
+crash-tube=%(basetube)s-crash
+# Extension for the files to be fuzzed
+extension=.fil
+# Timeout for this fuzzer
+timeout=90
+# Environment
+environment=clamav-environment
+
+[clamav-environment]
+MALLOC_CHECK_=3
+
+Once we have configured our fuzzer in generic.cfg we can just run the
+fuzzer issuing a command like the following one:
+
+ $ cd $NIGHTMARE_DIR/fuzzers
+ $ python generic_fuzzer.py generic.cfg ClamAV
+
+And that's all! It will take care of launching the command and killing
+it if a time-out occurs as well as handling all the signals and storing
+in the database all of the crashing information, POCs, diffs, etc...
+
+---
+Copyright (c) 2013, 2014 Joxean Koret