The Autobahn|Testsuite provides a fully automated test suite to verify client and server implementations of The WebSocket Protocol for specification conformance and implementation robustness.
Autobahn|Testsuite also provides a couple of other tools useful for WebSocket (and WAMP) implementors and developers.
For some current reports on the test coverage of Autobahn|Python see
The test suite will check an implementation by doing basic WebSocket conversations, extensive protocol compliance verification and performance and limits testing.
Autobahn|Testsuite is used across the industry and contains over 500 test cases covering
- Framing
- Pings/Pongs
- Reserved Bits
- Opcodes
- Fragmentation
- UTF-8 Handling
- Limits/Performance
- Closing Handshake
- Opening Handshake (under development)
- WebSocket compression (permessage-deflate extension)
Besides the automated testsuite (aka "fuzzing" server/client), wstest also includes a number of other handy modes:
- WebSocket echo server and client
- WebSocket broadcast server (and client driver)
- Testee modes to test AutobahnPython against the test suite
- wsperf controller and master (see below for more)
- WAMP server and client, for developing WAMP implementations
- WebSocket Mass-Connect
The following recipe still works, but the new, recommended way is using a Docker toolchain image we provide. Please checkout this - much easier and repeatable.
The testsuite comes as a single command line tool, wstest. You will need Python 2 or PyPy (recommended).
Right now we only support Python 2 and Python 3 will not work. The testsuite is developed and tested on CPython 2 and PyPy. The latter is a high-performance Python implementation.
The recommended way to install wstest is into it's own, dedicated virtualenv.
On Debian/Ubuntu systems, you can install virtualenv like
sudo apt-get install python-virtualenv.
Create a new virtualenv in your HOME and install Autobahn testsuite:
virtualenv ~/wstest
source ~/wstest/bin/activate
pip install autobahntestsuiteYou will now have the wstest tool:
(wstest)oberstet@thinkpad-t430s:~$ which wstest
/home/oberstet/wstest/bin/wstest
(wstest)oberstet@thinkpad-t430s:~$ wstest -a
Autobahn 0.10.9
AutobahnTestSuite 0.7.4
(wstest)oberstet@thinkpad-t430s:~$ wstest --help
Usage: wstest [options]
Options:
-d, --debug Debug output [default: off].
-a, --autobahnversion Print version information for Autobahn and
AutobahnTestSuite.
-m, --mode= Test mode, one of: echoserver, echoclient,
broadcastclient, broadcastserver, fuzzingserver,
fuzzingclient, testeeserver, testeeclient, massconnect,
serializer [required]
-t, --testset= Run a test set from an import test spec.
-s, --spec= Test specification file [required in some modes].
-o, --outfile= Output filename for modes that generate testdata.
-w, --wsuri= WebSocket URI [required in some modes].
-u, --webport= Web port for running an embedded HTTP Web server;
defaults to 8080; set to 0 to disable. [optionally used
in some modes: fuzzingserver, echoserver,
broadcastserver, wsperfmaster]. [default: 8080]
-i, --ident= Testee client identifier [optional for client testees].
-k, --key= Server private key file for secure WebSocket (WSS)
[required in server modes for WSS].
-c, --cert= Server certificate file for secure WebSocket (WSS)
[required in server modes for WSS].
--version Display Twisted version and exit.
--help Display this help and exit.
(wstest)oberstet@thinkpad-t430s:~$To test a WebSocket server implementation and generate compliance test reports, first start the WebSocket server that you want to test. Here, we are using a example from Autobahn|Python:
(python279_1)oberstet@thinkpad-t430s:~/scm/crossbario/autobahn-python/examples/twisted/websocket/testee$ python testee_server.py
2015-12-21 21:31:54+0100 [-] Log opened.
2015-12-21 21:31:54+0100 [-] TesteeServerFactory starting on 9001
2015-12-21 21:31:54+0100 [-] Starting factory <__main__.TesteeServerFactory object at 0x7faf23551210>Then, run wstest in fuzzing client mode:
cd ~
mkdir test
cd test
wstest -m fuzzingclientThe testsuite will now start a WebSocket fuzzing client connecting on TCP port 9001 to the WebSocket servers to be tested:
(wstest)oberstet@thinkpad-t430s:~/test$ wstest -m fuzzingclient
Auto-generating spec file 'fuzzingclient.json'
Loading spec from /home/oberstet/test/fuzzingclient.json
...
Autobahn Fuzzing WebSocket Client (Autobahn Version 0.7.4 / Autobahn Testsuite Version 0.10.9)
Ok, will run 521 test cases against 1 servers
...To test a WebSocket client implementation and generate compliance test reports, start wstest in fuzzing server mode:
cd ~
mkdir test
cd test
wstest -m fuzzingserverThe testsuite will now start a WebSocket fuzzing server listening on TCP port 9001 for WebSocket clients to be tested:
(wstest)oberstet@thinkpad-t430s:~/test$ wstest -m fuzzingserver
Auto-generating spec file 'fuzzingserver.json'
Loading spec from /home/oberstet/test/fuzzingserver.json
...
Autobahn WebSockets 0.7.4/0.10.9 Fuzzing Server (Port 9001)
Ok, will run 521 test cases for any clients connecting
...Note: The fuzzing server mode will also start a Web server on port 8080 that renders a HTML page for browser WebSocket clients to be tested.
On first run, the tool will auto-generated a test configuration file:
(wstest)oberstet@thinkpad-t430s:~/test$ cat fuzzingserver.json
{
"url": "ws://127.0.0.1:9001",
"outdir": "./reports/clients",
"cases": ["*"],
"exclude-cases": [],
"exclude-agent-cases": {}
}You can tweak that file to run only some tests, e.g. "cases: ["1.*", "2.1.*"]" will run only the tests under section 1.* and subsection 2.1.*.
For more information take a look at the following information
Get in touch on IRC #autobahn on chat.freenode.net or join the mailing list.