Artifactspublic-v0

curl

master
ply4 git calls47 ms292 KiB3 cachedprotocol v2
349 lines11 KiBRaw

NAME

runtests.pl - run one or more test cases

SYNOPSIS

runtests.pl [options] [tests]

DESCRIPTION

runtests.pl runs one, several or all the existing test cases in curl's test suite. It is often called from the root Makefile of the curl package with 'make test'.

TESTS

Specify which test(s) to run by specifying test numbers or keywords.

If no test number or keyword is given, all existing tests that the script can find are considered for running. You can specify single test cases to run by specifying test numbers space-separated, like 1 3 5 7 11, and you can specify a range of tests like 45 to 67.

Specify tests to not run with a leading exclamation point, like !66, which runs all available tests except number 66.

Prefix a test number with a tilde (~) to still run it, but ignore the results.

It is also possible to specify tests based on a keyword describing the test(s) to run, like FTPS. The keywords are strings used in the individual tests.

Features are included as keywords with the feat: prefix (e.g., feat:debug). Specify a feature to run only tests requiring it, or exclude tests using !feat:<feature>, like !feat:proxy, to disable tests which depend on that feature.

With special keywords !with-server and !without-server you can exclude tests using a test server or those that are not, respectively.

You can also specify keywords with a leading exclamation point and the keyword or phrase, like "!HTTP NTLM auth" to run all tests except those using this keyword. Remember that the exclamation marks and spaces need to be quoted somehow when entered at many command shells.

Prefix a keyword with a tilde (~) to still run it, but ignore the results.

OUTPUT

When running without -s (short output), for instance when running runtests.pl directly rather than via make, each test emits a pair of lines like this:

Test 0045...[simple HTTP Location: without protocol in initial URL]
--pd---e-v- OK (45  out of 1427, remaining: 16:08, took 6.188s, duration: 00:31)

the first line contains the test number and a description. On the second line, the characters at the beginning are flags indicating which aspects of curl's behavior were checked by the test:

s stdout
r stderr
p protocol
d data
u upload
P proxy
o output
e exit code
m memory
v valgrind
E the test was run event-based

The remainder of the second line contains the test result, current test sequence, total number of tests to be run and an estimated amount of time to complete the test run.

OPTIONS

-a

Continue running the rest of the test cases even if one test fails. By default, the test script stops as soon as an error is detected.

-ac \<curl\>

Provide a path to a curl binary to talk to APIs (currently only CI test APIs).

-am

Display test results in automake style output (PASS/FAIL: [number] [name]).

--buildinfo

Dump buildinfo.txt.

-c \<curl\>

Provide a path to a custom curl binary to run the tests with. Default is the curl executable in the build tree.

-d

Enable protocol debug: have the servers display protocol output. If used in conjunction with parallel testing, it is difficult to associate the logs with the specific test being run.

-E \<exclude_file\>

Load the exclude_file with additional reasons why certain tests should be skipped. Useful when testing with external HTTP proxies in which case some of the tests are not appropriate.

The file contains colon-delimited lines. The first field contains the type of exclusion, the second field contains a pattern and the final field contains the reason why matching tests should be skipped. The exclusion types are keyword, test, and tool.

-e or --test-event

Run the test event-based (if possible). This makes runtests invoke curl with --test-event option. This option only works if both curl and libcurl were built debug-enabled.

-f

Force the test to run even if mentioned in DISABLED.

-g

Run the given test(s) with gdb. This is best used on a single test case and curl built --disable-shared. This then fires up gdb with command line set to run the specified test case. Set a break-point if needed and type 'run' to start.

-gl

Run the given test(s) with lldb. This is best used on a single test case and curl built --disable-shared. This then fires up lldb with command line set to run the specified test case. Set a break-point if needed and type 'run' to start.

-gw

Run the given test(s) with gdb as a windowed application.

-h, --help

Displays a help text about this program's command line options.

-j[num]

Spawn the given number of processes to run tests in. This defaults to 0 to run tests serially within a single process. Using a number greater than one allows multiple tests to run in parallel, speeding up a test run. The optimum number is dependent on the system and set of tests to run, but 7 times the number of CPU cores is a good figure to start with, or 1.3 times if Valgrind is in use, or 5 times for torture tests. Enabling parallel tests is not recommended in conjunction with the -g option.

-L \<file\>

Load and execute the specified file which should contain perl code. This option allows one to change runtests.pl behavior by overwriting functions and variables and is useful when testing external proxies using curl's regression test suite.

-l

Lists all test case names.

-m=[seconds]

Set timeout for curl commands in tests.

--min=[count]

Set the minimum number of tests to run. This overrides the CURL_TEST_MIN environment variable, if any.

-n

Disable the check for and use of valgrind.

--no-debuginfod

Delete the DEBUGINFOD_URLS variable if that is defined. Makes valgrind, gdb etc not able to use this functionality.

-o \<variablename=value\>

Overwrite the specified internal variable with value. Useful to change variables that did not get a dedicated flag to change them. Check the source to see which variables are available.

-P \<proxy\>

Use the specified HTTP proxy when executing tests, even if the tests themselves do not specify a proxy. This option allows one to test external proxies using curl's regression test suite.

-p

Prints out all files in the log directory to stdout when a test case fails. Practical when used in the automated and distributed tests since then the people checking the failures and the reasons for them might not have physical access to the machine and logs.

-R

Run the tests in a scrambled, or randomized, order instead of sequentially.

The random seed initially set for this is fixed per month and can be set with --seed.

-r

Display run time statistics. (Requires the Perl Time::HiRes module)

-rf

Display full run time statistics. (Requires the Perl Time::HiRes module)

--repeat=[num]

This repeats the given set of test numbers this many times. If no test numbers are given, it repeats ALL tests this many times. It adds the new repeated sequence at the end of the initially given one.

If -R option is also used, the scrambling is done after the repeats have extended the test sequence.

--retry=[num]

Number of attempts for the whole test run to retry failed tests. It applies to tests using servers, or with keywords flaky, threads, timing-dependent, may-retry, or may-retry-on-valgrind (limited to valgrind tests).

-S

Prints out all files in the log directory to stdout when a test case passes. Practical when comparing against a similar test that fails, or the same test failing in a different build.

-s

Shorter output. Speaks less than default.

--seed=[num]

When using --shallow or -R that randomize certain aspects of the behavior, this option can set the initial seed. If not set, the random seed is set based on the currently set local year and month and the first line of the "curl -V" output.

--shallow=[num]

Used together with -t. This limits the number of tests to fail in torture mode to no more than num per test case. If this reduces the amount, the script randomly discards entries to fail until the amount is num.

The random seed initially set for this is fixed per month and can be set with --seed.

--subset=<section>/<parts>

Tell runtests to run a subset of the available tests. The test selection is done first based on existing options. The list of selected tests is then divided into a number of parts. The selected section specifier is the set of tests this invoke runs. Note that both numbers must be provided and section is zero indexed so it must be smaller than parts.

Using this option you can for example split up an identical test run into three separate invokes that combined run all the tests:

 ./runtests.pl --subset=0/3
 ./runtests.pl --subset=1/3
 ./runtests.pl --subset=2/3

This option works fine in combination with -R but consider also using --seed for that.

-t[num]

Selects a torture test for the given tests. This makes runtests.pl first run the tests once and count the number of memory allocations made. It then reruns the test that number of times, each time forcing one of the allocations to fail until all allocations have been tested. By setting num you can force the allocation with that number to be set to fail at once instead of looping through everyone, which is handy when debugging and then often in combination with -g.

--test-duphandle

Passes the --test-duphandle option to curl when invoked. This command line option only exists in debug builds and runs curl normally, but duplicates the easy handle before the transfer and use the duplicate instead of the original handle. This verifies that the duplicate works exactly as good as the original handle.

Because of how the curl tool uses a share object to store and keep some data, not everything is however perfectly copied in the duplicate. In particular HSTS data is not. A specific test case can be set to avoid using --test-duphandle by disabling it on a per test basis.

-u

Error instead of warning on server unexpectedly alive.

-v

Enable verbose output. Speaks more than by default. If used in conjunction with parallel testing, it is difficult to associate the logs with the specific test being run.

-vc \<curl\>

Provide a path to a custom curl binary to run when verifying that the servers running are indeed our test servers. Default is the curl executable in the build tree.

RUNNING TESTS

Many tests have conditions that must be met before the test case can run fine. They could depend on built-in features in libcurl or features present in the operating system or even in third-party libraries that curl may or may not use.

The test script checks most of these by itself to determine when it is safe to attempt to run each test. Those which cannot be run due to failed requirements are skipped and listed at the completion of all test cases. In some unusual configurations, the test script cannot make the correct determination for all tests. In these cases, the problematic tests can be skipped using the "!keyword" skip feature documented earlier.

WRITING TESTS

The simplest way to write test cases is to start with a similar existing test, save it with a new number and then adjust it to fit. There is an attempt to document the test case file format in tests/FILEFORMAT.md.

1---
2c: Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
3SPDX-License-Identifier: curl
4Title: runtests.pl
5Section: 1
6Source: runtests
7See-also:
8 - runtests.pl
9Added-in: 7.5
10---
12# NAME
14runtests.pl - run one or more test cases
16# SYNOPSIS
18**runtests.pl [options] [tests]**
20# DESCRIPTION
22*runtests.pl* runs one, several or all the existing test cases in curl's
23test suite. It is often called from the root Makefile of the curl package with
24'make test'.
26# TESTS
28Specify which test(s) to run by specifying test numbers or keywords.
30If no test number or keyword is given, all existing tests that the script can
31find are considered for running. You can specify single test cases to run by
32specifying test numbers space-separated, like `1 3 5 7 11`, and you can
33specify a range of tests like `45 to 67`.
35Specify tests to not run with a leading exclamation point, like `!66`, which
36runs all available tests except number 66.
38Prefix a test number with a tilde (~) to still run it, but ignore the results.
40It is also possible to specify tests based on a keyword describing the test(s)
41to run, like `FTPS`. The keywords are strings used in the individual tests.
43Features are included as keywords with the `feat:` prefix (e.g., `feat:debug`).
44Specify a feature to run only tests requiring it, or exclude tests using
45`!feat:<feature>`, like `!feat:proxy`, to disable tests which depend on that
46feature.
48With special keywords `!with-server` and `!without-server` you can
49exclude tests using a test server or those that are not, respectively.
51You can also specify keywords with a leading exclamation point and the keyword
52or phrase, like "!HTTP NTLM auth" to run all tests **except** those using this
53keyword. Remember that the exclamation marks and spaces need to be quoted
54somehow when entered at many command shells.
56Prefix a keyword with a tilde (~) to still run it, but ignore the results.
58# OUTPUT
60When running without `-s` (short output), for instance when running
61runtests.pl directly rather than via make, each test emits a pair of lines
62like this:
64 Test 0045...[simple HTTP Location: without protocol in initial URL]
65 --pd---e-v- OK (45 out of 1427, remaining: 16:08, took 6.188s, duration: 00:31)
67the first line contains the test number and a description. On the second line,
68the characters at the beginning are flags indicating which aspects of curl's
69behavior were checked by the test:
71 s stdout
72 r stderr
73 p protocol
74 d data
75 u upload
76 P proxy
77 o output
78 e exit code
79 m memory
80 v valgrind
81 E the test was run event-based
83The remainder of the second line contains the test result, current test sequence,
84total number of tests to be run and an estimated amount of time to complete the
85test run.
87# OPTIONS
89## `-a`
91Continue running the rest of the test cases even if one test fails. By
92default, the test script stops as soon as an error is detected.
94## `-ac \<curl\>`
96Provide a path to a curl binary to talk to APIs (currently only CI test APIs).
98## `-am`
100Display test results in automake style output (`PASS/FAIL: [number] [name]`).
102## `--buildinfo`
104Dump `buildinfo.txt`.
106## `-c \<curl\>`
108Provide a path to a custom curl binary to run the tests with. Default is the
109curl executable in the build tree.
111## `-d`
113Enable protocol debug: have the servers display protocol output. If used in
114conjunction with parallel testing, it is difficult to associate the logs with
115the specific test being run.
117## `-E \<exclude_file\>`
119Load the **exclude_file** with additional reasons why certain tests should be
120skipped. Useful when testing with external HTTP proxies in which case some of
121the tests are not appropriate.
123The file contains colon-delimited lines. The first field contains the type of
124exclusion, the second field contains a pattern and the final field contains
125the reason why matching tests should be skipped. The exclusion types are
126*keyword*, *test*, and *tool*.
128## `-e` or `--test-event`
130Run the test event-based (if possible). This makes runtests invoke curl with
131--test-event option. This option only works if both curl and libcurl were
132built debug-enabled.
134## `-f`
136Force the test to run even if mentioned in DISABLED.
138## `-g`
140Run the given test(s) with gdb. This is best used on a single test case and
141curl built --disable-shared. This then fires up gdb with command line set to
142run the specified test case. Set a break-point if needed and type 'run' to
143start.
145## `-gl`
147Run the given test(s) with lldb. This is best used on a single test case and
148curl built --disable-shared. This then fires up lldb with command line set to
149run the specified test case. Set a break-point if needed and type 'run' to
150start.
152## `-gw`
154Run the given test(s) with gdb as a windowed application.
156## `-h, --help`
158Displays a help text about this program's command line options.
160## `-j[num]`
162Spawn the given number of processes to run tests in. This defaults to 0 to run
163tests serially within a single process. Using a number greater than one allows
164multiple tests to run in parallel, speeding up a test run. The optimum number
165is dependent on the system and set of tests to run, but 7 times the number of
166CPU cores is a good figure to start with, or 1.3 times if Valgrind is in use,
167or 5 times for torture tests. Enabling parallel tests is not recommended in
168conjunction with the -g option.
170## `-L \<file\>`
172Load and execute the specified file which should contain perl code. This
173option allows one to change *runtests.pl* behavior by overwriting functions
174and variables and is useful when testing external proxies using curl's
175regression test suite.
177## `-l`
179Lists all test case names.
181## `-m=[seconds]`
183Set timeout for curl commands in tests.
185## `--min=[count]`
187Set the minimum number of tests to run. This overrides the `CURL_TEST_MIN`
188environment variable, if any.
190## `-n`
192Disable the check for and use of valgrind.
194## `--no-debuginfod`
196Delete the `DEBUGINFOD_URLS` variable if that is defined. Makes valgrind, gdb
197etc not able to use this functionality.
199## `-o \<variablename=value\>`
201Overwrite the specified internal **variable** with **value**. Useful to change
202variables that did not get a dedicated flag to change them. Check the source to
203see which variables are available.
205## `-P \<proxy\>`
207Use the specified HTTP proxy when executing tests, even if the tests
208themselves do not specify a proxy. This option allows one to test external
209proxies using curl's regression test suite.
211## `-p`
213Prints out all files in the log directory to stdout when a test case fails.
214Practical when used in the automated and distributed tests since then the
215people checking the failures and the reasons for them might not have physical
216access to the machine and logs.
218## `-R`
220Run the tests in a scrambled, or randomized, order instead of sequentially.
222The random seed initially set for this is fixed per month and can be set with
223*--seed*.
225## `-r`
227Display run time statistics. (Requires the `Perl Time::HiRes` module)
229## `-rf`
231Display full run time statistics. (Requires the `Perl Time::HiRes` module)
233## `--repeat=[num]`
235This repeats the given set of test numbers this many times. If no test numbers
236are given, it repeats ALL tests this many times. It adds the new repeated
237sequence at the end of the initially given one.
239If **-R** option is also used, the scrambling is done after the repeats have
240extended the test sequence.
242## `--retry=[num]`
244Number of attempts for the whole test run to retry failed tests. It applies to
245tests using servers, or with keywords `flaky`, `threads`, `timing-dependent`,
246`may-retry`, or `may-retry-on-valgrind` (limited to valgrind tests).
248## `-S`
250Prints out all files in the log directory to stdout when a test case passes.
251Practical when comparing against a similar test that fails, or the same test
252failing in a different build.
254## `-s`
256Shorter output. Speaks less than default.
258## `--seed=[num]`
260When using *--shallow* or *-R* that randomize certain aspects of the behavior,
261this option can set the initial seed. If not set, the random seed is set based
262on the currently set local year and month and the first line of the "curl -V"
263output.
265## `--shallow=[num]`
267Used together with **-t**. This limits the number of tests to fail in torture
268mode to no more than **num** per test case. If this reduces the amount, the
269script randomly discards entries to fail until the amount is **num**.
271The random seed initially set for this is fixed per month and can be set with
272*--seed*.
274## `--subset=<section>/<parts>`
276Tell runtests to run a subset of the available tests. The test selection is
277done first based on existing options. The list of selected tests is then
278divided into a number of *parts*. The selected *section* specifier is the
279set of tests this invoke runs. Note that both numbers must be provided and
280*section* is zero indexed so it must be smaller than *parts*.
282Using this option you can for example split up an identical test run into
283three separate invokes that combined run all the tests:
285 ./runtests.pl --subset=0/3
286 ./runtests.pl --subset=1/3
287 ./runtests.pl --subset=2/3
289This option works fine in combination with `-R` but consider also using
290`--seed` for that.
292## `-t[num]`
294Selects a **torture** test for the given tests. This makes runtests.pl first
295run the tests once and count the number of memory allocations made. It then
296reruns the test that number of times, each time forcing one of the allocations
297to fail until all allocations have been tested. By setting *num* you can force
298the allocation with that number to be set to fail at once instead of looping
299through everyone, which is handy when debugging and then often in combination
300with *-g*.
302## `--test-duphandle`
304Passes the `--test-duphandle` option to curl when invoked. This command line
305option only exists in debug builds and runs curl normally, but duplicates the
306easy handle before the transfer and use the duplicate instead of the original
307handle. This verifies that the duplicate works exactly as good as the original
308handle.
310Because of how the curl tool uses a share object to store and keep some data,
311not everything is however perfectly copied in the duplicate. In particular
312HSTS data is not. A specific test case can be set to avoid using
313`--test-duphandle` by disabling it on a per test basis.
315## `-u`
317Error instead of warning on server unexpectedly alive.
319## `-v`
321Enable verbose output. Speaks more than by default. If used in conjunction
322with parallel testing, it is difficult to associate the logs with the specific
323test being run.
325## `-vc \<curl\>`
327Provide a path to a custom curl binary to run when verifying that the servers
328running are indeed our test servers. Default is the curl executable in the
329build tree.
331# RUNNING TESTS
333Many tests have conditions that must be met before the test case can run fine.
334They could depend on built-in features in libcurl or features present in the
335operating system or even in third-party libraries that curl may or may not
336use.
338The test script checks most of these by itself to determine when it is safe to
339attempt to run each test. Those which cannot be run due to failed requirements
340are skipped and listed at the completion of all test cases. In some
341unusual configurations, the test script cannot make the correct determination
342for all tests. In these cases, the problematic tests can be skipped using the
343"!keyword" skip feature documented earlier.
345# WRITING TESTS
347The simplest way to write test cases is to start with a similar existing test,
348save it with a new number and then adjust it to fit. There is an attempt to
349document the test case file format in **tests/FILEFORMAT.md**.
Diagnostics4 git calls · 47 ms
CallStatusSizeTime047 ms
discover200152 B2 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit2 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=2
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=2
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=41
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2004.9 KiB · hit42 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=5025
git.response.cache=HIT
git.response.duration_ms=42
git.fetch.wants=7678bcf286cd8da0f95b41d4bb98dd50ebc9b7ca
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=5025
git.pkt.payload.max=5004
git.sideband.band1.bytes=5003
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e9e6c03d6deefa
Request
b64b5c16…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format