Artifactspublic-v0

curl

master
ply4 git calls152 ms292 KiB2 cachedprotocol v2
307 lines12 KiBRaw

The curl Test Suite

Running

See the "Requires to run" section for prerequisites.

In the root of the curl repository:

./configure && make && make test

To run a specific set of tests (e.g. 303 and 410):

make test TFLAGS="303 410"

To run the tests faster, pass the -j (parallelism) flag:

make test TFLAGS="-j10"

make test builds the test suite support code and invokes the runtests.pl perl script to run all the tests. The value of TFLAGS is passed directly to runtests.pl.

When you run tests via make, the flags -a and -s are passed, meaning to continue running tests even after one fails, and to emit short output.

If you would like to not use those flags, you can run runtests.pl directly. You must chdir into the tests directory, then you can run it like so:

./runtests.pl 303 410

You must have run make test at least once first to build the support code.

To see what flags are available for runtests.pl, and what output it emits, see docs/runtests.md.

After a test fails, examine the tests/log directory for stdout, stderr, and output from the servers used in the test.

Requires to run

  • perl (and a Unix-style shell)
  • python (and a Unix-style shell, for TELNET tests)
  • diff (when a test fails, a diff is shown)
  • stunnel (for HTTPS and FTPS tests)
  • openssl (the command line tool, for generating test server certificates)
  • openssh or SunSSH (for SCP and SFTP tests)
  • nghttpx (for HTTP/2 and HTTP/3 tests)

Event-based

If curl is built with Debug enabled (see below), then the runtests.pl script offers a -e option (or --test-event) that makes it perform event-based. Such tests invokes the curl tool with --test-event, a debug-only option made for this purpose.

Performing event-based means that the curl tool uses the curl_multi_socket_action() API call to drive the transfer(s), instead of the otherwise "normal" functions it would use. This allows us to test drive the socket_action API. Transfers done this way should work exactly the same as with the non-event based API.

To be able to use --test-event together with --parallel, curl requires libuv to be present and enabled in the build: configure --with-libuv

Duplicated handles

If curl is built with Debug enabled (see below), then the runtests.pl script offers a --test-duphandle option. When enabled, curl always duplicates the easy handle and does its transfers using the new one instead of the original. This is done entirely for testing purpose to verify that everything works exactly the same when this is done; confirming that the curl_easy_duphandle() function duplicates everything that it should.

Port numbers used by test servers

All test servers run on "random" port numbers. All tests must be written to use the suitable variables instead of fixed port numbers so that test cases continue to work independently of what port numbers the test servers actually use.

See FILEFORMAT for the port number variables.

Test servers

The test suite runs stand-alone servers on random ports to which it makes requests. For SSL tests, it runs stunnel to handle encryption to the regular servers. For SSH, it runs a standard OpenSSH server.

The listen port numbers for the test servers are picked randomly to allow users to run multiple test cases concurrently and to not collide with other existing services that might listen to ports on the machine.

The HTTP server supports listening on a Unix domain socket, the default location is 'http.sock'.

For HTTP/2 and HTTP/3 testing an installed nghttpx is used. HTTP/3 tests check if nghttpx supports the protocol. To override the nghttpx used, set the environment variable NGHTTPX. The default can also be changed by specifying --with-test-nghttpx=<path> as argument to configure.

DNS server

There is a test DNS server to allow tests to resolve hostnames to verify those code paths. This server is started like all the other servers within the <servers> section.

To make a curl build actually use the test DNS server requires a debug build. When such a test runs, the environment variable CURL_DNS_SERVER is set to identify the IP address and port number of the DNS server to use.

  • curl built to use c-ares for resolving automatically asks that server for host information

  • curl built to use getaddrinfo() for resolving and is built with c-ares 1.26.0 or later, gets a special workaround. In such builds, when the environment variable is set, curl instead invokes a getaddrinfo wrapper that emulates the function and acknowledges the DNS server environment variable. This way, the getaddrinfo-using code paths in curl are verified, and yet the custom responses from the test DNS server are used.

curl that is built to support a custom DNS server in a test gets the override-dns feature set.

When curl ask for HTTPS-RR, c-ares is always used and in debug builds such asks respects the dns server environment variable as well.

The test DNS server only has a few limited responses. When asked for

  • type A response, it returns the address 127.0.0.1 three times
  • type AAAA response, it returns the address ::1 three times
  • other types, it returns a blank response without answers

Shell startup scripts

Tests which use the ssh test server, SCP/SFTP tests, might be badly influenced by the output of system wide or user specific shell startup scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which output text messages or escape sequences on user login. When these shell startup messages or escape sequences are output they might corrupt the expected stream of data which flows to the sftp-server or from the ssh client which can result in bad test behavior or even prevent the test server from running.

If the test suite ssh or sftp server fails to start up and logs the message 'Received message too long' then you are certainly suffering the unwanted output of a shell startup script. Locate, cleanup or adjust the shell script.

Memory test

The test script checks that all allocated memory is freed properly IF curl has been built with the DEBUGBUILD define set. The script automatically detects if that is the case, and it uses the memanalyze.pl script to analyze the memory debugging output.

Also, if you run tests on a machine where valgrind is found, the script uses valgrind to run the test with (unless you use -n) to further verify correctness.

The runtests.pl -t option enables torture testing mode. It runs each test many times and makes each different memory allocation fail on each successive run. This tests the out of memory error handling code to ensure that memory leaks do not occur even in those situations. It can help to compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to ensure that the memory log file is properly written even if curl crashes.

If a specific test takes a long time to run in memory test mode, you can disable it individually by adding !torture to its <features> section.

Debug

If a test case fails, you can conveniently get the script to invoke the debugger (gdb) for you with the server running and the same command line parameters that failed. Invoke runtests.pl <test number> -g and then type 'run' in the debugger to perform the command through the debugger.

Logs

All logs are generated in the log/ subdirectory (it is emptied first in the runtests.pl script). They remain in there after a test run.

Log Verbosity

A curl build with --enable-debug offers more verbose output in the logs. This applies not only for test cases, but also when running it standalone with curl -v. While a curl debug built is *not suitable for production*, it is often helpful in tracking down problems.

Sometimes, one needs detailed logging of operations, but does not want to drown in output. The newly introduced connection filters allows one to dynamically increase log verbosity for a particular filter type. Example:

CURL_DEBUG=ssl curl -v https://curl.se/

makes the ssl connection filter log more details. One may do that for every filter type and also use a combination of names, separated by , or space.

CURL_DEBUG=ssl,http/2 curl -v https://curl.se/

The order of filter type names is not relevant. Names used here are case insensitive. Note that these names are implementation internals and subject to change.

Some, likely stable names are tcp, ssl, http/2. For a current list, one may search the sources for struct Curl_cftype definitions and find the names there. Also, some filters are only available with certain build options, of course.

Test input files

All test cases are put in the data/ subdirectory. Each test is stored in the file named according to the test number.

See FILEFORMAT for a description of the test case file format.

Code coverage

gcc provides a tool that can determine the code coverage figures for the test suite. To use it, configure curl with CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal and torture tests to get more full coverage, i.e. do:

make test
make test-torture

The graphical tool ggcov can be used to browse the source and create coverage reports on *nix hosts:

ggcov -r lib src

The text mode tool gcov may also be used, but it does not handle object files in more than one directory correctly.

Remote testing

The runtests.pl script provides some hooks to allow curl to be tested on a machine where perl can not be run. The test framework in this case runs on a workstation where perl is available, while curl itself is run on a remote system using ssh or some other remote execution method. See the comments at the beginning of runtests.pl for details.

Test case numbering

Test cases used to be numbered by category ranges, but the ranges filled up. Subsets of tests can now be selected by passing keywords to the runtests.pl script via the make TFLAGS variable.

New tests are added by finding a free number in tests/data/Makefile.am. The script scripts/testnum automates this search for you.

Write tests

Here's a quick description on writing test cases. We have three kinds of tests: the ones that test the curl tool, the ones that build small applications and test libcurl directly and the unit tests that test individual (possibly internal) functions.

test data

Each test has a master file that controls all the test data. What to read, what the protocol exchange should look like, what exit code to expect and what command line arguments to use etc.

These files are tests/data/test[num] where [num] is a unique identifier described above, and the XML-like file format of them is described in the separate FILEFORMAT document.

curl tests

A test case that runs the curl tool and verifies that it gets the correct data, it sends the correct data, it uses the correct protocol primitives etc.

libcurl tests

The libcurl tests are identical to the curl ones, except that they use a specific and dedicated custom-built program to run instead of "curl". This tool is built from source code placed in tests/libtest and if you want to make a new libcurl test that is where you add your code.

unit tests

Unit tests are placed in tests/unit. There is a tests/unit/README.md describing the specific set of checks and macros that may be used when writing tests that verify behaviors of specific individual functions.

The unit tests depend on curl being built with debug enabled.

test bundles

Individual tests are bundled into single executables, one for libtests, one for unit tests and one for servers. The executables' first argument is the name of libtest, unit test or server respectively. In these executables, the build process automatically renames the entry point to a unique symbol. test becomes test_<tool>, e.g. test_lib1598 or test_unit1305. For servers main becomes main_sws for the sws server, and so on. Other common symbols may also be suffixed the same way.

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# The curl Test Suite
9# Running
11See the "Requires to run" section for prerequisites.
13In the root of the curl repository:
15 ./configure && make && make test
17To run a specific set of tests (e.g. 303 and 410):
19 make test TFLAGS="303 410"
21To run the tests faster, pass the -j (parallelism) flag:
23 make test TFLAGS="-j10"
25`make test` builds the test suite support code and invokes the `runtests.pl`
26perl script to run all the tests. The value of `TFLAGS` is passed directly
27to `runtests.pl`.
29When you run tests via make, the flags `-a` and `-s` are passed, meaning to
30continue running tests even after one fails, and to emit short output.
32If you would like to not use those flags, you can run `runtests.pl`
33directly. You must `chdir` into the tests directory, then you can run it
34like so:
36 ./runtests.pl 303 410
38You must have run `make test` at least once first to build the support code.
40To see what flags are available for `runtests.pl`, and what output it emits,
41see `docs/runtests.md`.
43After a test fails, examine the tests/log directory for stdout, stderr, and
44output from the servers used in the test.
46## Requires to run
48- `perl` (and a Unix-style shell)
49- `python` (and a Unix-style shell, for TELNET tests)
50- `diff` (when a test fails, a diff is shown)
51- `stunnel` (for HTTPS and FTPS tests)
52- `openssl` (the command line tool, for generating test server certificates)
53- `openssh` or `SunSSH` (for SCP and SFTP tests)
54- `nghttpx` (for HTTP/2 and HTTP/3 tests)
56## Event-based
58If curl is built with `Debug` enabled (see below), then the `runtests.pl`
59script offers a `-e` option (or `--test-event`) that makes it perform
60*event-based*. Such tests invokes the curl tool with `--test-event`, a
61debug-only option made for this purpose.
63Performing event-based means that the curl tool uses the
64`curl_multi_socket_action()` API call to drive the transfer(s), instead of
65the otherwise "normal" functions it would use. This allows us to test drive
66the socket_action API. Transfers done this way should work exactly the same
67as with the non-event based API.
69To be able to use `--test-event` together with `--parallel`, curl requires
70*libuv* to be present and enabled in the build: `configure --with-libuv`
72## Duplicated handles
74If curl is built with `Debug` enabled (see below), then the `runtests.pl`
75script offers a `--test-duphandle` option. When enabled, curl always
76duplicates the easy handle and does its transfers using the new one instead
77of the original. This is done entirely for testing purpose to verify that
78everything works exactly the same when this is done; confirming that the
79`curl_easy_duphandle()` function duplicates everything that it should.
81### Port numbers used by test servers
83All test servers run on "random" port numbers. All tests must be written to
84use the suitable variables instead of fixed port numbers so that test cases
85continue to work independently of what port numbers the test servers
86actually use.
88See [`FILEFORMAT`](FILEFORMAT.md) for the port number variables.
90### Test servers
92The test suite runs stand-alone servers on random ports to which it makes
93requests. For SSL tests, it runs stunnel to handle encryption to the regular
94servers. For SSH, it runs a standard OpenSSH server.
96The listen port numbers for the test servers are picked randomly to allow
97users to run multiple test cases concurrently and to not collide with other
98existing services that might listen to ports on the machine.
100The HTTP server supports listening on a Unix domain socket, the default
101location is 'http.sock'.
103For HTTP/2 and HTTP/3 testing an installed `nghttpx` is used. HTTP/3 tests
104check if nghttpx supports the protocol. To override the nghttpx used, set
105the environment variable `NGHTTPX`. The default can also be changed by
106specifying `--with-test-nghttpx=<path>` as argument to `configure`.
108### DNS server
110There is a test DNS server to allow tests to resolve hostnames to verify
111those code paths. This server is started like all the other servers within
112the `<servers>` section.
114To make a curl build actually use the test DNS server requires a debug
115build. When such a test runs, the environment variable `CURL_DNS_SERVER` is
116set to identify the IP address and port number of the DNS server to use.
118- curl built to use c-ares for resolving automatically asks that server for
119 host information
121- curl built to use `getaddrinfo()` for resolving *and* is built with c-ares
122 1.26.0 or later, gets a special workaround. In such builds, when the
123 environment variable is set, curl instead invokes a getaddrinfo wrapper
124 that emulates the function and acknowledges the DNS server environment
125 variable. This way, the getaddrinfo-using code paths in curl are verified,
126 and yet the custom responses from the test DNS server are used.
128curl that is built to support a custom DNS server in a test gets the
129`override-dns` feature set.
131When curl ask for HTTPS-RR, c-ares is always used and in debug builds such
132asks respects the dns server environment variable as well.
134The test DNS server only has a few limited responses. When asked for
136- type `A` response, it returns the address `127.0.0.1` three times
137- type `AAAA` response, it returns the address `::1` three times
138- other types, it returns a blank response without answers
140### Shell startup scripts
142Tests which use the ssh test server, SCP/SFTP tests, might be badly
143influenced by the output of system wide or user specific shell startup
144scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
145output text messages or escape sequences on user login. When these shell
146startup messages or escape sequences are output they might corrupt the
147expected stream of data which flows to the sftp-server or from the ssh
148client which can result in bad test behavior or even prevent the test server
149from running.
151If the test suite ssh or sftp server fails to start up and logs the message
152'Received message too long' then you are certainly suffering the unwanted
153output of a shell startup script. Locate, cleanup or adjust the shell
154script.
156### Memory test
158The test script checks that all allocated memory is freed properly IF curl
159has been built with the `DEBUGBUILD` define set. The script automatically
160detects if that is the case, and it uses the `memanalyze.pl` script to
161analyze the memory debugging output.
163Also, if you run tests on a machine where valgrind is found, the script uses
164valgrind to run the test with (unless you use `-n`) to further verify
165correctness.
167The `runtests.pl` `-t` option enables torture testing mode. It runs each
168test many times and makes each different memory allocation fail on each
169successive run. This tests the out of memory error handling code to ensure
170that memory leaks do not occur even in those situations. It can help to
171compile curl with `CPPFLAGS=-DMEMDEBUG_LOG_SYNC` when using this option, to
172ensure that the memory log file is properly written even if curl crashes.
174If a specific test takes a long time to run in memory test mode, you can
175disable it individually by adding `!torture` to its `<features>` section.
177### Debug
179If a test case fails, you can conveniently get the script to invoke the
180debugger (gdb) for you with the server running and the same command line
181parameters that failed. Invoke `runtests.pl <test number> -g` and then
182type 'run' in the debugger to perform the command through the debugger.
184### Logs
186All logs are generated in the log/ subdirectory (it is emptied first in the
187runtests.pl script). They remain in there after a test run.
189### Log Verbosity
191A curl build with `--enable-debug` offers more verbose output in the logs.
192This applies not only for test cases, but also when running it standalone
193with `curl -v`. While a curl debug built is
194***not suitable for production***, it is often helpful in tracking down
195problems.
197Sometimes, one needs detailed logging of operations, but does not want
198to drown in output. The newly introduced *connection filters* allows one to
199dynamically increase log verbosity for a particular *filter type*. Example:
201 CURL_DEBUG=ssl curl -v https://curl.se/
203makes the `ssl` connection filter log more details. One may do that for
204every filter type and also use a combination of names, separated by `,` or
205space.
207 CURL_DEBUG=ssl,http/2 curl -v https://curl.se/
209The order of filter type names is not relevant. Names used here are
210case insensitive. Note that these names are implementation internals and
211subject to change.
213Some, likely stable names are `tcp`, `ssl`, `http/2`. For a current list,
214one may search the sources for `struct Curl_cftype` definitions and find
215the names there. Also, some filters are only available with certain build
216options, of course.
218### Test input files
220All test cases are put in the `data/` subdirectory. Each test is stored in
221the file named according to the test number.
223See [`FILEFORMAT`](FILEFORMAT.md) for a description of the test case file
224format.
226### Code coverage
228gcc provides a tool that can determine the code coverage figures for the
229test suite. To use it, configure curl with `CFLAGS='-fprofile-arcs
230-ftest-coverage -g -O0'`. Make sure you run the normal and torture tests to
231get more full coverage, i.e. do:
233 make test
234 make test-torture
236The graphical tool `ggcov` can be used to browse the source and create
237coverage reports on \*nix hosts:
239 ggcov -r lib src
241The text mode tool `gcov` may also be used, but it does not handle object
242files in more than one directory correctly.
244### Remote testing
246The runtests.pl script provides some hooks to allow curl to be tested on a
247machine where perl can not be run. The test framework in this case runs on
248a workstation where perl is available, while curl itself is run on a remote
249system using ssh or some other remote execution method. See the comments at
250the beginning of runtests.pl for details.
252## Test case numbering
254Test cases used to be numbered by category ranges, but the ranges filled
255up. Subsets of tests can now be selected by passing keywords to the
256runtests.pl script via the make `TFLAGS` variable.
258New tests are added by finding a free number in `tests/data/Makefile.am`.
259The script `scripts/testnum` automates this search for you.
261## Write tests
263Here's a quick description on writing test cases. We have three kinds of
264tests: the ones that test the curl tool, the ones that build small
265applications and test libcurl directly and the unit tests that test individual
266(possibly internal) functions.
268### test data
270Each test has a master file that controls all the test data. What to read,
271what the protocol exchange should look like, what exit code to expect and
272what command line arguments to use etc.
274These files are `tests/data/test[num]` where `[num]` is a unique identifier
275described above, and the XML-like file format of them is described in the
276separate [`FILEFORMAT`](FILEFORMAT.md) document.
278### curl tests
280A test case that runs the curl tool and verifies that it gets the correct
281data, it sends the correct data, it uses the correct protocol primitives
282etc.
284### libcurl tests
286The libcurl tests are identical to the curl ones, except that they use a
287specific and dedicated custom-built program to run instead of "curl". This
288tool is built from source code placed in `tests/libtest` and if you want to
289make a new libcurl test that is where you add your code.
291### unit tests
293Unit tests are placed in `tests/unit`. There is a tests/unit/README.md
294describing the specific set of checks and macros that may be used when
295writing tests that verify behaviors of specific individual functions.
297The unit tests depend on curl being built with debug enabled.
299### test bundles
301Individual tests are bundled into single executables, one for libtests, one
302for unit tests and one for servers. The executables' first argument is
303the name of libtest, unit test or server respectively.
304In these executables, the build process automatically renames the entry point
305to a unique symbol. `test` becomes `test_<tool>`, e.g. `test_lib1598` or
306`test_unit1305`. For servers `main` becomes `main_sws` for the `sws` server,
307and so on. Other common symbols may also be suffixed the same way.
Diagnostics4 git calls · 152 ms
CallStatusSizeTime0152 ms
discover200152 B3 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit3 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=3
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2005.1 KiB146 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=146
git.response.bytes=5232
git.response.cache=MISS
git.response.duration_ms=146
git.fetch.wants=56e6b15227ff32cfb345397af9033c77a229f185
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=5232
git.pkt.payload.max=5211
git.sideband.band1.bytes=5210
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e30288ba037929
Request
37f4bed9…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format