-
Notifications
You must be signed in to change notification settings - Fork 1.2k
/
Copy pathREADME.pytest
139 lines (107 loc) · 4.92 KB
/
README.pytest
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
Apache httpd pytest suite
=========================
Using pytest (<https://docs.pytest.org/en/6.2.x/>) and a Python >= 3.8
for a more flexible testing of Apache httpd.
Install
-------
If not already installed, you will need to install 'pytest' and 'OpenSSL' for
python:
> apt install python3-pip
> pip install -U pytest
> pip install -U pyopenssl
And for 'h2load':
> apt install nghttp2-client
Usage
-----
In your httpd source checkout, do:
> make install
> pytest
and all tests defined run on the installed executable and modules.
> pytest test/modules/core
runs all the core tests. You can run tests also by name selection:
> pytest -k test_core
runs all test cases starting with 'test_core'. Similar:
> pytest -k test_core_001_04
runs the test cases starting with that.
Test output gets more verbose, by adding one or several '-v'. This
also raises the error log level of the tested modules.
> pytest -vvv -k test_h2_004_01
run the specific test with mod_http2 at log level TRACE2.
By default, test cases will configure httpd with mpm_event. You
can change that with the invocation:
> MPM=worker pytest test/modules/http2
Some tests rely on additional tools installed in your environment
and will 'skip' if those are not present. In a non-verbose run,
these will appear as 's' in the output. If you run pytest more
verbose, the skipped test cases will mention a reason for why
they are disabled.
For example, most tests in test/modules/md require an installation
of 'pebble', an ACME test server, and look for it in $PATH.
Workings
--------
All tests start httpd on their own and try hard to shut it down
afterwards. You can abort tests with ^C and they clean up.
httpd is started/stopped repeatedly in testing as configurations
for individual test cases are changed. This is a main difference to
the Perl test framework which starts the server with all possible
combinations configured that are needed by tests.
In test/gen/apache a server root is created with config, logs and htdocs
directories. test/gen/apache/logs/error_log will be the log.
Configs start in test/gen/apache/conf/httpd.conf. modules.conf is
dynamically created for the list of modules that a test suite needs.
Test cases write their specific setup in test.conf and reload/restart
the httpd process. This makes for a small configuration in a test case.
Development
-----------
Adding a test in an existing file is done by adding a method. Its name
must start with 'test_' and the common practice is to have the name
of the test suite there as well. All http2 tests start with 'test_h2_'.
Following this can be any characters. If you make test cases of a
certain feature with a common prefix, it is easier to invoke just
them using the '-k' selector on the command line.
You can also add just a new file to a test suite, if you do a new
range of test cases that do not fit anywhere else. A very simple
one is found in test/modules/http2/test_001_httpd_alive.py.
There is a python class defined with 2 methods. One is the test
method itself and the other one ' is
@pytest.fixture(autouse=True, scope='class')
def _class_scope(self, env):
code
is marked as a pytest 'fixture'. This is some pytest magic.
'autouse=True' means that this fixture is run, even though
no test case uses it directly. scope='class' means that it
is run once for all test cases in this class.
As you see, this fixture gets a parameter named 'env' and
that is the name of another pytest fixture, defined in the
file 'conftest.py' in the same directory.
@pytest.fixture(scope="package")
def env(pytestconfig) -> H2TestEnv:
code
This one runs one time per 'package', meaning for all test
cases in this directory. And it gets the 'pytestconfig'
as parameter which is a standard pytest fixture.
So, when you run 'pytest -k test_h2_004', pytest will look
at _all_ test cases defined and collect those with that
prefix. For each directory with test cases found, it will
process the 'conftest.py', boot-strapping the 'env' fixture,
and the process the files with active test cases.
As a result, if you invoke just a single test case, only
the fixtures needed for that test case are created. This
gives good turn around times when debugging a test case.
If you want to add a new test suite, create a new directory.
Add the files '__init__.py', 'conftest.py' and a first
'test_suitename_something.py'. test/modules/core is the
simplest example. 'test/modules/http2' shows how you load
other modules. 'test/modules/md' checks and starts external
processes (an ACME test server).
Infrastructure
--------------
The test cases rely on the classes provided in 'test/pyhttpd'
for common code in managing a httpd test instance and do
provide some base setups:
- a small set of virtual hosts with some files
- access to paths and definitions (env fixture)
- start/stop httpd and inspect the error log
- run clients like curl and nghttp
- create certificates for your hosts and make curl use
the root certs (so no --insecure calls by curl).