End-to-End Tests for Socorro

Continuous Integration

This directory holds Socorro client-based end-to-end tests which is why they're different than the rest of the code in this repository.

To review the specific Python packages the tests utilize, please review requirements.txt.

Set up and run Socorro tests

Review the documentation for pytest-selenium and decide which browser environment you wish to target.

We suggest using a different virtual environment for these tests than the rest of Socorro so you're not mixing requirements:

$ mkvirtualenv socorro-tests
$ # make sure you DON'T run the next command in
$ # your dev virtualenv.
$ pip install -r requirements.txt

An additional constraint for Firefox users, with the release of Firefox 48, Webdriver support is currently broken until GeckoDriver is out of Alpha. We suggest using Firefox 47.0.2 which can be downloaded here.

If you have multiple versions of Firefox installed, you can specifiy a specific one by using the --firefox-path <path to firefox binary> flag.

Running the tests on stage

$ py.test --driver Firefox tests/

Running tests against localhost

$ py.test --driver Firefox --base-url http://localhost:8000 tests/
$ py.test --driver Firefox --firefox-path /path/to/firefox/binary --base-url http://localhost:8000 tests/

Running tests against production

$ py.test --driver Firefox --base-url https://crash-stats.mozilla.com tests/

Running specific tests

You can run tests in a given file::

$ py.test --driver Firefox tests/desktop/test_search.py

You can run tests that match a specific name:

$ py.test --driver Firefox -k test_search_for_unrealistic_data

You can run tests whose names match a specific pattern:

$ py.test --driver Firefox -k test_search

Output

Output of a test run should look something like this:

============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.2.4
collected 73 items

tests/test_crash_reports.py .........................xx..........x.xx....x.
tests/test_layout.py xx
tests/test_search.py .........x.x.
tests/test_smoke_tests.py ...........

============== 63 passed, 10 xfailed in 970.27 seconds ===============

Note "~" will not resolve to the home directory when used in the py.test command line.

The pytest plugin that we use for running tests has a number of advanced command line options available. To see the options available, run py.test --help. The full documentation for the plugin can be found here.

Troubleshooting

If the test run hangs with Firefox open but no URL gets entered in the address box, some combinations of the Firefox version, and the python Selenium bindings version may not be compatible. Upgrading each of them to latest often fixes it.

Tips and tricks

Because Selenium opens real browser windows, it will steal focus and switch workspaces. Firefox doesn't have a headless mode of operation, so we can't simply turn off the UI.

Use a different driver

pytest-selenium provides the ability to run tests against many other browser environments -- consider using a different driver executable or external provider.

It is important to note that tests must pass with Firefox driver otherwise they will not be accepted for merging.

$ # tests must pass with Firefox driver before submitting a pull request
$ py.test --driver PhantomJS --driver-path `which phantomjs` ...

xvfb

Xvfb provides a fairly easily work around on Linux.

On Linux:

Install Xvfb and run the tests with it's xvfb-run binary. For
example, if you run tests like::

    $ py.test ...


You can switch to this to run with Xvfb::

    $ xvfb-run py.test ...


This creates a virtual X session for Firefox to run in, and sets
up all the fiddly environment variables to get this working
well. The tests will run as normal, and no windows will open, if
all is working right.

License

This software is licensed under the MPL 2.0:

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.