Skip to content

Commit

Permalink
Save network interactions to allow tests to run offline by providing …
Browse files Browse the repository at this point in the history
…mocked responses (Qiskit#697)

The current commit introduces several changes to the testing infrastructure.

First, it provides a way of running IBM-Q tests against pre-recorded responses. The software in charge of recording and mocking-up the responses is VCRPy[1] and it is configured and used in `test/common.py`. Cassettes can be updated by using `make test_recording`.

[1] https://github.com/kevin1024/vcrpy

The commit also includes the latest responses (under `tests/casettes`) for a successful test run but a default execution with `make test` will only use them if no IBM-Q credentials are found. To force a run against mocked responses, use `make test_mock`.

Finally, the commit also unifies the several test options in a unique environmental variable called `QISKIT_TESTS` (in `tests/python/_test_options.py`). These are the main equivalences but the set of options is described in the developer introduction [2]:

- `SKIP_SLOW_TESTS=False` is now achieved by `QISKIT_TESTS=run_slow`
- `SKIP_ONLINE_TESTS=1` is now achieved by `QISKIT_TESTS=skip_online`

[2] https://github.com/Qiskit/qiskit-terra/blob/master/doc/dev_introduction.rst#testing-options
  • Loading branch information
1ucian0 authored and delapuente committed Aug 20, 2018
1 parent 1683c8a commit 97d24d1
Show file tree
Hide file tree
Showing 60 changed files with 25,669 additions and 116 deletions.
3 changes: 0 additions & 3 deletions .github/CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -205,9 +205,6 @@ Windows:
C:\..\> set LOG_LEVEL="INFO"
C:\..\> python -m unittest test/python/test_apps.py
Additionally, an environment variable ``SKIP_ONLINE_TESTS`` can be used for
toggling the execution of the tests that require network access to the API.

Style guide
~~~~~~~~~~~

Expand Down
9 changes: 8 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# This source code is licensed under the Apache License, Version 2.0 found in
# the LICENSE.txt file in the root directory of this source tree.

.PHONY: env env-dev lint test run doc
.PHONY: env env-dev lint test run doc test_record test_mock

# Dependencies need to be installed on the Anaconda virtual environment.
env:
Expand All @@ -29,6 +29,13 @@ style:
test:
python3 -m unittest discover -s test -v

test_mock:
env QISKIT_TESTS=mock_online python3 -m unittest discover -s test -v

test_recording:
-rm test/cassettes/*
env QISKIT_TESTS=rec python3 -m unittest discover -s test -v

profile:
python3 -m unittest discover -p "profile*.py" -v

Expand Down
5 changes: 0 additions & 5 deletions doc/de/dev_introduction.rst
Original file line number Diff line number Diff line change
Expand Up @@ -214,8 +214,3 @@ Windows:
C:\..\> set LOG_LEVEL="INFO"
C:\..\> python -m unittest test/python/test_apps.py
Zusätzlich kann die Umgebungsvariable ``SKIP_ONLINE_TESTS`` verwendet werden
zum An- und Ausschalten der Ausführung der Tests, die Netzwerk Zugang für die
API benötigen, und ``SKIP_SLOW_TESTS`` zum An- und Ausschalten von Test, die
besonders langsam sind (standardmäßig auf ``True`` gesetzt).
25 changes: 19 additions & 6 deletions doc/dev_introduction.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Several unroller backends and their outputs are summarized here:


Logging
-------
=======

The SDK uses the `standard Python "logging" library
<https://docs.python.org/3/library/logging.html>`_ for emitting several messages using the
Expand Down Expand Up @@ -125,7 +125,7 @@ the messages. For example, if the module is `qiskit/some/module.py`:
Testing
-------
=======
The SDK uses the `standard Pyton "unittest" framework
<https://docs.python.org/3/library/unittest.html>`_ for the testing of the
Expand Down Expand Up @@ -195,7 +195,20 @@ Windows:
C:\..\> set LOG_LEVEL="INFO"
C:\..\> python -m unittest test/python/test_apps.py
Additionally, an environment variable ``SKIP_ONLINE_TESTS`` can be used for
toggling the execution of the tests that require network access to the API and
``SKIP_SLOW_TESTS`` can be used to toggling execution of tests that are
particularly slow (default is ``True``).
Testing options
===============
By default, and if there is no user credentials available, the tests that require online access are run with recorded (mocked) information. This is, the remote requests are replayed from a `test/cassettes` and not real HTTP requests is generated.
If user credentials are found, in that cases it use them to make the network requests.
How and which tests are executed is controlled by a environment variable `QISKIT_TESTS`. The options are (where `uc_available = True` if the user credentials are available, and `False` otherwise):
| Option | Description | Default | If `True`, forces |
| ------------- |--------------| -----| -----|
| `skip_online` | Skips tests that require remote requests (also, no mocked information is used). Does not require user credentials. | `False`| `rec = False`
| `mock_online` | It runs the online tests using mocked information. Does not require user credentials. | `not uc_available` | `skip_online = False`
| `run_slow` | It runs tests tagged as *slow*. | `False` | |
| `rec` | It records the remote requests. It requires user credentials. | `False` |`skip_online = False` `run_slow = False`
It is possible to provide more than one option separated with commas.
The order of precedence in the options is right to left. For example, `QISKIT_TESTS=skip_online,rec` will set the options as `skip_online == False` and `rec == True`.
1 change: 1 addition & 0 deletions requirements-dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ sphinxcontrib-fulltoc==1.2.0
coverage>=4.4.0
jsonschema>=2.6,<2.7
wheel
vcrpy
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ ply>=3.10
scipy>=0.19
sympy>=1.0
pillow>=4.2.1
vcrpy
129 changes: 129 additions & 0 deletions test/cassettes/test_aliases
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
interactions:
- request:
body: apiToken=apiToken_dummy
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
Content-Length: ['137']
Content-Type: [application/x-www-form-urlencoded]
method: POST
uri: https://quantumexperience.ng.bluemix.net/api/users/loginWithToken
response:
body: {string: '{"id": "dummyapiusersloginWithTokenid01", "ttl": 1209600, "created":
"dummyapiusersloginWithTokencreated01", "userId": "dummyapiusersloginWithTokenuserId01"}'}
headers:
Access-Control-Allow-Credentials: ['true']
Access-Control-Allow-Origin: ['https://quantumexperience.mybluemix.net/']
Cache-Control: ['no-store, no-cache, must-revalidate, proxy-revalidate']
Connection: [Keep-Alive]
Content-Type: [application/json; charset=utf-8]
Expires: ['0']
Pragma: [no-cache]
Set-Cookie: dummy_cookie
Strict-Transport-Security: [max-age=86400]
Surrogate-Control: [no-store]
Transfer-Encoding: [chunked]
Vary: ['Origin, Accept-Encoding']
X-Backside-Transport: [OK OK]
X-Content-Type-Options: [nosniff]
X-Download-Options: [noopen]
X-Frame-Options: [SAMEORIGIN]
X-Xss-Protection: [1; mode=block]
status: {code: 200, message: OK}
- request:
body: null
headers:
Accept: ['*/*']
Accept-Encoding: ['gzip, deflate']
Connection: [keep-alive]
method: GET
uri: https://quantumexperience.ng.bluemix.net/api/Backends?access_token=dummyapiusersloginWithTokenid01
response:
body: {string: '[{"name": "Device Real5Qv1", "status": "off", "serialNumber":
"Real5Qv1", "description": "Device Real5Qv1", "deleted": false, "id": "Device
Real5Qv1", "topologyId": "dummyapiBackendstopologyId01", "internalId": "dummyapiBackendsinternalId01",
"simulator": false, "allowQObject": false, "nQubits": 5, "couplingMap": [[0,
1], [0, 2], [1, 2], [3, 2], [3, 4], [4, 2]]}, {"name": "ibmqx4", "version":
"1.2.0", "status": "on", "serialNumber": "ibmqx4", "description": "5 qubit
transmon bowtie chip 3", "gateSet": "SU2+CNOT", "basisGates": "u1,u2,u3,cx,id",
"onlineDate": "2017-09-18T00:00:00.000Z", "chipName": "Raven", "deleted":
false, "id": "ibmqx4", "topologyId": "dummyapiBackendstopologyId02", "url":
"https://ibm.biz/qiskit-ibmqx4", "internalId": "dummyapiBackendsinternalId02",
"simulator": false, "allowQObject": false, "nQubits": 5, "couplingMap": [[1,
0], [2, 0], [2, 1], [3, 2], [3, 4], [4, 2]]}, {"name": "ibmqx5", "version":
"1.1.0", "status": "on", "serialNumber": "ibmqx5", "description": "16 transmon
2x8 ladder", "gateSet": "SU2+CNOT", "basisGates": "u1,u2,u3,cx,id", "onlineDate":
"2017-09-21T00:00:00.000Z", "chipName": "Albatross", "deleted": false, "id":
"ibmqx5", "topologyId": "dummyapiBackendstopologyId03", "url": "https://ibm.biz/qiskit-ibmqx5",
"internalId": "dummyapiBackendsinternalId03", "simulator": false, "allowQObject":
false, "nQubits": 16, "couplingMap": [[1, 0], [1, 2], [2, 3], [3, 4], [3,
14], [5, 4], [6, 5], [6, 7], [6, 11], [7, 10], [8, 7], [9, 8], [9, 10], [11,
10], [12, 5], [12, 11], [12, 13], [13, 4], [13, 14], [15, 0], [15, 2], [15,
14]]}, {"name": "ibmqx3", "version": "1", "status": "off", "serialNumber":
"ibmqx3", "description": "16 transmon 2x8 ladder", "basisGates": "u1,u2,u3,cx,id",
"onlineDate": "2017-06-06T11:00:00.000Z", "chipName": "Albatross", "deleted":
false, "id": "ibmqx3", "topologyId": "dummyapiBackendstopologyId04", "url":
"https://ibm.biz/qiskit-ibmqx3", "internalId": "dummyapiBackendsinternalId04",
"simulator": false, "allowQObject": false, "nQubits": 16, "couplingMap": [[0,
1], [1, 2], [2, 3], [3, 14], [4, 3], [4, 5], [6, 7], [6, 11], [7, 10], [8,
7], [9, 8], [9, 10], [11, 10], [12, 5], [12, 11], [12, 13], [13, 4], [13,
14], [15, 0], [15, 14]]}, {"name": "QS1_1", "version": "1", "status": "standby",
"serialNumber": "QS1_1", "description": "20 qubit device v1", "basisGates":
"SU2+CNOT", "onlineDate": "2017-10-20T11:00:00.000Z", "chipName": "Qubert",
"deleted": false, "id": "QS1_1", "topologyId": "dummyapiBackendstopologyId05",
"internalId": "dummyapiBackendsinternalId05", "simulator": false, "allowQObject":
false, "nQubits": 20, "couplingMap": [[0, 1], [0, 5], [1, 0], [1, 2], [1,
6], [1, 7], [2, 1], [2, 7], [3, 4], [3, 9], [4, 3], [4, 8], [4, 9], [5, 0],
[5, 6], [5, 10], [5, 11], [6, 1], [6, 5], [6, 7], [6, 10], [6, 11], [7, 1],
[7, 2], [7, 6], [7, 12], [7, 13], [8, 4], [8, 9], [8, 12], [8, 13], [9, 3],
[9, 4], [9, 8], [9, 14], [10, 5], [10, 6], [10, 11], [10, 15], [11, 5], [11,
6], [11, 10], [11, 12], [11, 16], [11, 17], [12, 7], [12, 8], [12, 11], [12,
13], [12, 16], [12, 17], [13, 7], [13, 8], [13, 12], [13, 14], [13, 18], [13,
19], [14, 9], [14, 13], [14, 18], [14, 19], [15, 10], [15, 16], [16, 11],
[16, 12], [16, 15], [16, 17], [17, 11], [17, 12], [17, 16], [17, 18], [18,
13], [18, 14], [18, 17], [18, 19], [19, 13], [19, 14], [19, 18]]}, {"name":
"ibmqx2", "version": "1.1.0", "status": "on", "serialNumber": "Real5Qv2",
"description": "5 qubit transmon bowtie chip 2", "gateSet": "SU2+CNOT", "basisGates":
"u1,u2,u3,cx,id", "onlineDate": "2017-01-10T00:00:00.000Z", "chipName": "Sparrow",
"deleted": false, "id": "ibmqx2", "topologyId": "dummyapiBackendstopologyId01",
"url": "https://ibm.biz/qiskit-ibmqx2", "internalId": "dummyapiBackendsinternalId06",
"simulator": false, "allowQObject": false, "nQubits": 5, "couplingMap": [[0,
1], [0, 2], [1, 2], [3, 2], [3, 4], [4, 2]]}, {"name": "ibmq_20_tokyo", "version":
"1", "status": "standby", "serialNumber": "ibmq_20_tokyo", "description":
"20 qubit device Tokyo", "basisGates": "SU2+CNOT", "onlineDate": "2018-05-10T00:00:00.000Z",
"chipName": "Qubert", "deleted": false, "id": "ibmq_20_tokyo", "topologyId":
"dummyapiBackendstopologyId06", "internalId": "dummyapiBackendsinternalId07",
"simulator": false, "allowQObject": false, "nQubits": 20, "couplingMap": [[0,
1], [0, 5], [1, 0], [1, 2], [1, 6], [1, 7], [2, 1], [2, 3], [2, 6], [3, 2],
[3, 8], [3, 9], [4, 8], [4, 9], [5, 0], [5, 6], [5, 10], [5, 11], [6, 1],
[6, 2], [6, 5], [6, 7], [6, 10], [6, 11], [7, 1], [7, 6], [7, 8], [7, 12],
[7, 13], [8, 3], [8, 4], [8, 7], [8, 9], [8, 12], [8, 13], [9, 3], [9, 4],
[9, 8], [10, 5], [10, 6], [10, 11], [10, 15], [11, 5], [11, 6], [11, 10],
[11, 12], [11, 16], [11, 17], [12, 7], [12, 8], [12, 11], [12, 13], [12, 16],
[13, 7], [13, 8], [13, 12], [13, 14], [13, 18], [13, 19], [14, 13], [15, 10],
[15, 16], [16, 11], [16, 12], [16, 15], [16, 17], [17, 11], [17, 16], [18,
13], [19, 13]]}, {"name": "ibmq_qasm_simulator", "status": "on", "description":
"online qasm simulator", "basisGates": "u1,u2,u3,cx,id", "onlineDate": "2017-12-09T12:00:00.000Z",
"deleted": false, "internalId": "dummyapiBackendsinternalId08", "simulator":
true, "allowQObject": false, "nQubits": 32, "couplingMap": "all-to-all"}]'}
headers:
Access-Control-Allow-Credentials: ['true']
Access-Control-Allow-Origin: ['https://quantumexperience.mybluemix.net/']
Cache-Control: ['no-store, no-cache, must-revalidate, proxy-revalidate']
Connection: [Keep-Alive]
Content-Type: [application/json; charset=utf-8]
Expires: ['0']
Pragma: [no-cache]
Set-Cookie: dummy_cookie
Strict-Transport-Security: [max-age=86400]
Surrogate-Control: [no-store]
Transfer-Encoding: [chunked]
Vary: ['Origin, Accept-Encoding']
X-Backside-Transport: [OK OK]
X-Content-Type-Options: [nosniff]
X-Download-Options: [noopen]
X-Frame-Options: [SAMEORIGIN]
X-Xss-Protection: [1; mode=block]
status: {code: 200, message: OK}
version: 1
Loading

0 comments on commit 97d24d1

Please sign in to comment.