Artifactspublic-v0

curl

master
ply4 git calls28 ms290 KiB3 cachedprotocol v2
206 lines7.9 KiBRaw

The curl HTTP Test Suite

This is an additional test suite using a combination of Apache httpd and nghttpx servers to perform various tests beyond the capabilities of the standard curl test suite.

Usage

The test cases and necessary files are in tests/http. You can invoke pytest from there or from the top level curl checkout and it finds all tests.

sh
curl> pytest tests/http
platform darwin -- Python 3.9.15, pytest-6.2.0, py-1.10.0, pluggy-0.13.1
rootdir: /Users/sei/projects/curl
collected 5 items

tests/http/test_01_basic.py .....

Pytest takes arguments. -v increases its verbosity and can be used several times. -k <expr> can be used to run only matching test cases. The expr can be something resembling a python test or a string that needs to match test cases in their names.

sh
curl/tests/http> pytest -vv -k test_01_02

runs all test cases that have test_01_02 in their name. This does not have to be the start of the name.

Depending on your setup, some test cases may be skipped and appear as s in the output. If you run pytest verbose, it also gives you the reason for skipping.

Prerequisites

You need:

  1. a recent Python, pytest and the other modules listed in tests/http/requirements.txt
  2. Apache httpd and its development files. On Debian/Ubuntu, the packages apache2-bin and apache2-dev have these.
  3. the Apache mod_ssl, mod_http2 and mod_proxy modules. On Debian/Ubuntu, these modules are part of the apache2-bin package, but other distributions may package them separately.
  4. a local curl project build
  5. optionally, nghttpx with HTTP/3 enabled or h3 test cases are skipped

Configuration

Via curl's configure script you may specify:

  • --with-test-nghttpx=<path-of-nghttpx> if you have nghttpx to use somewhere outside your $PATH.

  • --with-test-h2o=<path-of-h2o> if you have h2o to use somewhere outside your $PATH.

  • --with-test-httpd=<httpd-install-path> if you have an Apache httpd installed somewhere else. On Debian/Ubuntu it otherwise looks into /usr/bin and /usr/sbin to find those.

  • --with-test-caddy=<caddy-install-path> if you have a Caddy web server installed somewhere else.

  • --with-test-vsftpd=<vsftpd-install-path> if you have a vsftpd ftp server installed somewhere else.

  • --with-test-danted=<danted-path> if you have dante-server installed

Usage Tips

Several test cases are parameterized, for example with the HTTP version to use. If you want to run a test with a particular protocol only, use a command line like:

sh
curl/tests/http> pytest -k "test_02_06 and h2"

Test cases can be repeated, with the pytest-repeat module (pip install pytest-repeat). Like in:

sh
curl/tests/http> pytest -k "test_02_06 and h2" --count=100

which then runs this test case a hundred times. In case of flaky tests, you can make pytest stop on the first one with:

sh
curl/tests/http> pytest -k "test_02_06 and h2" --count=100 --maxfail=1

which allow you to inspect output and log files for the failed run. Speaking of log files, the verbosity of pytest is also used to collect curl trace output. If you specify -v three times, the curl command is started with --trace:

sh
curl/tests/http> pytest -vvv -k "test_02_06 and h2" --count=100 --maxfail=1

all of curl's output and trace file are found in tests/http/gen/curl.

Writing Tests

There is a lot of pytest documentation with examples. No use in repeating that here. Assuming you are somewhat familiar with it, it is useful how this general test suite is setup. Especially if you want to add test cases.

Servers

In conftest.py 3 "fixtures" are defined that are used by all test cases:

  1. env: the test environment. It is an instance of class testenv/env.py:Env. It holds all information about paths, availability of features (HTTP/3), port numbers to use, domains and SSL certificates for those.
  2. httpd: the Apache httpd instance, configured and started, then stopped at the end of the test suite. It has sites configured for the domains from env. It also loads a local module mod_curltest? and makes it available in certain locations. (more on mod_curltest below).
  3. nghttpx: an instance of nghttpx that provides HTTP/3 support. nghttpx proxies those requests to the httpd server. In a direct mapping, so you may access all the resources under the same path as with HTTP/2. Only the port number used for HTTP/3 requests are different.

pytest manages these fixture so that they are created once and terminated before exit. This means you can Ctrl-C a running pytest and the server then shutdowns. Only when you brutally chop its head off, might there be servers left behind.

Test Cases

Tests making use of these fixtures have them in their parameter list. This tells pytest that a particular test needs them, so it has to create them. Since one can invoke pytest for a single test, it is important that a test references the ones it needs.

All test cases start with test_ in their name. We use a double number scheme to group them. This makes it ease to run only specific tests and also give a short mnemonic to communicate trouble with others in the project. Otherwise you are free to name test cases as you think fitting.

Tests are grouped thematically in a file with a single Python test class. This is convenient if you need a special "fixture" for several tests. "fixtures" can have "class" scope.

There is a curl helper class that knows how to invoke curl and interpret its output. Among other things, it does add the local CA to the command line, so that SSL connections to the test servers are verified. Nothing prevents anyone from running curl directly, for specific uses not covered by the CurlClient class.

mod_curltest

The module source code is found in testenv/mod_curltest. It is compiled using the apxs command, commonly provided via the apache2-dev package. Compilation is quick and done once at the start of a test run.

The module adds 2 "handlers" to the Apache server (right now). Handler are pieces of code that receive HTTP requests and generate the response. Those handlers are:

  • curltest-echo: hooked up on the path /curltest/echo. This one echoes a request and copies all data from the request body to the response body. Useful for simulating upload and checking that the data arrived as intended.

  • curltest-tweak: hooked up on the path /curltest/tweak. This handler is more of a Swiss army knife. It interprets parameters from the URL query string to drive its behavior.

    • status=nnn: generate a response with HTTP status code nnn.
    • chunks=n: generate n chunks of data in the response body, defaults to 3.
    • chunk_size=nnn: each chunk should contain nnn bytes of data. Maximum is 16KB right now.
    • chunkd_delay=duration: wait duration time between writing chunks
    • delay=duration: wait duration time to send the response headers
    • body_error=(timeout|reset): produce an error after the first chunk in the response body
    • id=str: add str in the response header request-id

duration values are integers, optionally followed by a unit. Units are:

  • d: days (probably not useful here)
  • h: hours
  • mi: minutes
  • s: seconds (the default)
  • ms: milliseconds

As you can see, mod_curltest's tweak handler allows Apache to simulate many kinds of responses. An example of its use is test_03_01 where responses are delayed using chunk_delay. This gives the response a defined duration and the test uses that to reload httpd in the middle of the first request. A graceful reload in httpd lets ongoing requests finish, but closes the connection afterwards and tears down the serving process. The following request then needs to open a new connection. This is verified by the test case.

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# The curl HTTP Test Suite
9This is an additional test suite using a combination of Apache httpd and
10nghttpx servers to perform various tests beyond the capabilities of the
11standard curl test suite.
13# Usage
15The test cases and necessary files are in `tests/http`. You can invoke
16`pytest` from there or from the top level curl checkout and it finds all
17tests.
19```sh
20curl> pytest tests/http
21platform darwin -- Python 3.9.15, pytest-6.2.0, py-1.10.0, pluggy-0.13.1
22rootdir: /Users/sei/projects/curl
23collected 5 items
25tests/http/test_01_basic.py .....
26```
28Pytest takes arguments. `-v` increases its verbosity and can be used several
29times. `-k <expr>` can be used to run only matching test cases. The `expr` can
30be something resembling a python test or a string that needs to match test
31cases in their names.
33```sh
34curl/tests/http> pytest -vv -k test_01_02
35```
37runs all test cases that have `test_01_02` in their name. This does not have
38to be the start of the name.
40Depending on your setup, some test cases may be skipped and appear as `s` in
41the output. If you run pytest verbose, it also gives you the reason for
42skipping.
44# Prerequisites
46You need:
481. a recent Python, `pytest` and the other modules listed in
49 `tests/http/requirements.txt`
502. Apache httpd and its development files. On Debian/Ubuntu, the packages
51 `apache2-bin` and `apache2-dev` have these.
523. the Apache `mod_ssl`, `mod_http2` and `mod_proxy` modules. On Debian/Ubuntu, these
53 modules are part of the `apache2-bin` package, but other distributions may
54 package them separately.
554. a local `curl` project build
565. optionally, `nghttpx` with HTTP/3 enabled or h3 test cases are skipped
58### Configuration
60Via curl's `configure` script you may specify:
62 * `--with-test-nghttpx=<path-of-nghttpx>` if you have nghttpx to use
63 somewhere outside your `$PATH`.
65 * `--with-test-h2o=<path-of-h2o>` if you have h2o to use somewhere
66 outside your `$PATH`.
68 * `--with-test-httpd=<httpd-install-path>` if you have an Apache httpd
69 installed somewhere else. On Debian/Ubuntu it otherwise looks into
70 `/usr/bin` and `/usr/sbin` to find those.
72 * `--with-test-caddy=<caddy-install-path>` if you have a Caddy web server
73 installed somewhere else.
75 * `--with-test-vsftpd=<vsftpd-install-path>` if you have a vsftpd ftp
76 server installed somewhere else.
78 * `--with-test-danted=<danted-path>` if you have `dante-server` installed
80## Usage Tips
82Several test cases are parameterized, for example with the HTTP version to
83use. If you want to run a test with a particular protocol only, use a command
84line like:
86```sh
87curl/tests/http> pytest -k "test_02_06 and h2"
88```
90Test cases can be repeated, with the `pytest-repeat` module (`pip install
91pytest-repeat`). Like in:
93```sh
94curl/tests/http> pytest -k "test_02_06 and h2" --count=100
95```
97which then runs this test case a hundred times. In case of flaky tests, you
98can make pytest stop on the first one with:
100```sh
101curl/tests/http> pytest -k "test_02_06 and h2" --count=100 --maxfail=1
102```
104which allow you to inspect output and log files for the failed run. Speaking
105of log files, the verbosity of pytest is also used to collect curl trace
106output. If you specify `-v` three times, the `curl` command is started with
107`--trace`:
109```sh
110curl/tests/http> pytest -vvv -k "test_02_06 and h2" --count=100 --maxfail=1
111```
113all of curl's output and trace file are found in `tests/http/gen/curl`.
115## Writing Tests
117There is a lot of [`pytest` documentation](https://docs.pytest.org/) with
118examples. No use in repeating that here. Assuming you are somewhat familiar
119with it, it is useful how *this* general test suite is setup. Especially if
120you want to add test cases.
122### Servers
124In `conftest.py` 3 "fixtures" are defined that are used by all test cases:
1261. `env`: the test environment. It is an instance of class
127 `testenv/env.py:Env`. It holds all information about paths, availability of
128 features (HTTP/3), port numbers to use, domains and SSL certificates for
129 those.
1302. `httpd`: the Apache httpd instance, configured and started, then stopped at
131 the end of the test suite. It has sites configured for the domains from
132 `env`. It also loads a local module `mod_curltest?` and makes it available
133 in certain locations. (more on mod_curltest below).
1343. `nghttpx`: an instance of nghttpx that provides HTTP/3 support. `nghttpx`
135 proxies those requests to the `httpd` server. In a direct mapping, so you
136 may access all the resources under the same path as with HTTP/2. Only the
137 port number used for HTTP/3 requests are different.
139`pytest` manages these fixture so that they are created once and terminated
140before exit. This means you can `Ctrl-C` a running pytest and the server then
141shutdowns. Only when you brutally chop its head off, might there be servers
142left behind.
144### Test Cases
146Tests making use of these fixtures have them in their parameter list. This
147tells pytest that a particular test needs them, so it has to create them.
148Since one can invoke pytest for a single test, it is important that a test
149references the ones it needs.
151All test cases start with `test_` in their name. We use a double number scheme
152to group them. This makes it ease to run only specific tests and also give a
153short mnemonic to communicate trouble with others in the project. Otherwise
154you are free to name test cases as you think fitting.
156Tests are grouped thematically in a file with a single Python test class. This
157is convenient if you need a special "fixture" for several tests. "fixtures"
158can have "class" scope.
160There is a curl helper class that knows how to invoke curl and interpret its
161output. Among other things, it does add the local CA to the command line, so
162that SSL connections to the test servers are verified. Nothing prevents anyone
163from running curl directly, for specific uses not covered by the `CurlClient`
164class.
166### mod_curltest
168The module source code is found in `testenv/mod_curltest`. It is compiled
169using the `apxs` command, commonly provided via the `apache2-dev` package.
170Compilation is quick and done once at the start of a test run.
172The module adds 2 "handlers" to the Apache server (right now). Handler are
173pieces of code that receive HTTP requests and generate the response. Those
174handlers are:
176* `curltest-echo`: hooked up on the path `/curltest/echo`. This one echoes
177 a request and copies all data from the request body to the response body.
178 Useful for simulating upload and checking that the data arrived as intended.
180* `curltest-tweak`: hooked up on the path `/curltest/tweak`. This handler is
181 more of a Swiss army knife. It interprets parameters from the URL query
182 string to drive its behavior.
184 * `status=nnn`: generate a response with HTTP status code `nnn`.
185 * `chunks=n`: generate `n` chunks of data in the response body, defaults to 3.
186 * `chunk_size=nnn`: each chunk should contain `nnn` bytes of data. Maximum is 16KB right now.
187 * `chunkd_delay=duration`: wait `duration` time between writing chunks
188 * `delay=duration`: wait `duration` time to send the response headers
189 * `body_error=(timeout|reset)`: produce an error after the first chunk in the response body
190 * `id=str`: add `str` in the response header `request-id`
192`duration` values are integers, optionally followed by a unit. Units are:
194 * `d`: days (probably not useful here)
195 * `h`: hours
196 * `mi`: minutes
197 * `s`: seconds (the default)
198 * `ms`: milliseconds
200As you can see, `mod_curltest`'s tweak handler allows Apache to simulate many
201kinds of responses. An example of its use is `test_03_01` where responses are
202delayed using `chunk_delay`. This gives the response a defined duration and the
203test uses that to reload `httpd` in the middle of the first request. A graceful
204reload in httpd lets ongoing requests finish, but closes the connection
205afterwards and tears down the serving process. The following request then needs
206to open a new connection. This is verified by the test case.
Diagnostics4 git calls · 28 ms
CallStatusSizeTime028 ms
discover200152 B3 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit3 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=3
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2003.6 KiB · hit22 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=22
git.response.bytes=3650
git.response.cache=HIT
git.response.duration_ms=22
git.fetch.wants=79f3fac200547e57891b18f8cf3489b8cd95623b
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=3650
git.pkt.payload.max=3629
git.sideband.band1.bytes=3628
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e1ed9a0d6befe9
Request
93546d98…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format