#*************************************************************************** # _ _ ____ _ # Project ___| | | | _ \| | # / __| | | | |_) | | # | (__| |_| | _ <| |___ # \___|\___/|_| \_\_____| # # Copyright (C) Daniel Stenberg, , et al. # # This software is licensed as described in the file COPYING, which # you should have received as part of this distribution. The terms # are also available at https://curl.se/docs/copyright.html. # # You may opt to use, copy, modify, merge, publish, distribute and/or sell # copies of the Software, and permit persons to whom the Software is # furnished to do so, under the terms of the COPYING file. # # This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY # KIND, either express or implied. # # SPDX-License-Identifier: curl # ########################################################################### # import contextlib import logging import os import socket import threading import time from datetime import datetime, timedelta, timezone from typing import Dict, Generator import pytest from testenv import CurlClient, Env from testenv.ports import alloc_ports_and_do log = logging.getLogger(__name__) def pr_process(self): log.info("pr accepting") try: while self._socket and not self._done: conn, _ = self._socket.accept() with conn: conn.sendall(HttpPureResp.RESPONSE) log.info("response sent") except ConnectionAbortedError: return class HttpPureResp: RESPONSE = (b'HTTP/1.1 200 OK\r\n' b'Content-Length: 3\r\n' b'Connection: close\r\n' b'\r\n' b'ok\n') def __init__(self, env): self.name = 'httppureresp' self.env = env self.run_dir = os.path.join(env.gen_dir, self.name) self._thread = None self._socket = None self._done = False self.port = 0 def start(self): log.info("pr start") def startup(ports: Dict[str, int]) -> bool: self.port = ports[self.name] log.info(f"pr port={self.port}") self._socket = socket.create_server(('127.0.0.1', self.port)) log.info("pr socket open") self._thread = threading.Thread(target=pr_process, daemon=True, args=[self]) self._thread.start() self.env.update_ports(ports) return True port_spec = { self.name: socket.SOCK_STREAM } assert alloc_ports_and_do(port_spec, startup, self.env.gen_root, max_tries=3) assert self._socket return self.wait_live(timeout=timedelta(seconds=Env.SERVER_TIMEOUT)) def stop(self): self._done = True if self._socket: self._socket.close() self._socket = None log.info("pr stopped") def wait_live(self, timeout: timedelta): curl = CurlClient(env=self.env, run_dir=self.run_dir, timeout=timeout.total_seconds()) try_until = datetime.now(timezone.utc) + timeout while datetime.now(timezone.utc) < try_until: r = curl.http_get(url=f'http://{self.env.domain1}:{self.port}/') if r.exit_code == 0: return True time.sleep(.1) log.error(f"Server still not responding after {timeout}") return False @pytest.mark.skipif(condition=not Env.httpd_is_at_least('2.4.55'), reason=f"httpd version too old for this: {Env.httpd_version()}") class TestErrors: # download 1 file, check that we get CURLE_PARTIAL_FILE @pytest.mark.parametrize("proto", Env.http_protos()) def test_05_01_partial_1(self, env: Env, httpd, nghttpx, proto): if proto == 'h3' and env.curl_uses_lib('quiche') and \ not env.curl_lib_version_at_least('quiche', '0.29.0'): pytest.skip("quiche issue #2277 not fixed") count = 1 curl = CurlClient(env=env) urln = f'https://{env.authority_for(env.domain1, proto)}' \ f'/curltest/tweak?id=[0-{count - 1}]'\ '&chunks=3&chunk_size=16000&body_error=reset' r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[ '--retry', '0' ]) r.check_exit_code(False) invalid_stats = [] for idx, s in enumerate(r.stats): if 'exitcode' not in s or s['exitcode'] not in [18, 56, 92, 95]: invalid_stats.append(f'request {idx} exit with {s["exitcode"]}') assert len(invalid_stats) == 0, f'failed: {invalid_stats}' # download files, check that we get CURLE_PARTIAL_FILE for all @pytest.mark.parametrize("proto", Env.http_mplx_protos()) def test_05_02_partial_20(self, env: Env, httpd, nghttpx, proto): if proto == 'h3' and env.curl_uses_lib('quiche') and \ not env.curl_lib_version_at_least('quiche', '0.29.0'): pytest.skip("quiche issue #2277 not fixed") count = 20 curl = CurlClient(env=env) urln = f'https://{env.authority_for(env.domain1, proto)}' \ f'/curltest/tweak?id=[0-{count - 1}]'\ '&chunks=5&chunk_size=16000&body_error=reset' r = curl.http_download(urls=[urln], alpn_proto=proto, extra_args=[ '--retry', '0', '--parallel', ]) r.check_exit_code(False) assert len(r.stats) == count, f'did not get all stats: {r}' invalid_stats = [] for idx, s in enumerate(r.stats): if 'exitcode' not in s or s['exitcode'] not in [18, 55, 56, 92, 95]: invalid_stats.append(f'request {idx} exit with {s["exitcode"]}\n{r.dump_logs()}') assert len(invalid_stats) == 0, f'failed: {invalid_stats}\n{r.dump_logs()}' # access a resource that, on h2, RST the stream with HTTP_1_1_REQUIRED @pytest.mark.skipif(condition=not Env.have_h2_curl(), reason="curl without h2") def test_05_03_required(self, env: Env, httpd, nghttpx): curl = CurlClient(env=env) proto = 'http/1.1' urln = f'https://{env.authority_for(env.domain1, proto)}/curltest/1_1' r = curl.http_download(urls=[urln], alpn_proto=proto) r.check_exit_code(0) r.check_response(http_status=200, count=1) proto = 'h2' urln = f'https://{env.authority_for(env.domain1, proto)}/curltest/1_1' r = curl.http_download(urls=[urln], alpn_proto=proto) r.check_exit_code(0) r.check_response(http_status=200, count=1) # check that we did a downgrade assert r.stats[0]['http_version'] == '1.1', r.dump_logs() # On the URL used here, Apache is doing an "unclean" TLS shutdown, # meaning it sends no shutdown notice and closes TCP. # The HTTP response delivers a body without Content-Length. We expect: # - http/1.0 to fail since it relies on a clean connection close to # detect the end of the body # - http/1.1 to work since it will used "chunked" transfer encoding # and stop receiving when that signals the end # - h2 to work since it will signal the end of the response before # and not see the "unclean" close either @pytest.mark.parametrize("proto", ['http/1.0', 'http/1.1', 'h2']) def test_05_04_unclean_tls_shutdown(self, env: Env, httpd, nghttpx, proto): if proto == 'h2' and not env.have_h2_curl(): pytest.skip("h2 not supported") if proto == 'h3' and not env.have_h3(): pytest.skip("h3 not supported") count = 10 if proto == 'h2' else 1 curl = CurlClient(env=env) url = f'https://{env.authority_for(env.domain1, proto)}'\ f'/curltest/shutdown_unclean?id=[0-{count-1}]&chunks=4' r = curl.http_download(urls=[url], alpn_proto=proto, extra_args=[ '--parallel', '--trace-config', 'ssl' ]) if proto == 'http/1.0': # we are inconsistent if we fail or not in missing TLS shutdown # openssl code ignore such errors intentionally in non-debug builds r.check_exit_code(56) else: r.check_exit_code(0) r.check_response(http_status=200, count=count) # Make a connect with a fail for '::1' and wrong certificate for '127.0.0.1' # The error message reported needs to be from the certificate. # verifies issue #20608 @pytest.mark.parametrize("proto", Env.http_h1_h2_protos()) def test_05_05_failed_peer(self, env: Env, proto, httpd, nghttpx): domain = f'invalid.{env.tld}' url = f'https://{domain}:{env.port_for(proto)}/' run_env = os.environ.copy() run_env['CURL_DBG_SOCK_FAIL_IPV6'] = '1' curl = CurlClient(env=env, run_env=run_env) r = curl.http_download(urls=[url], alpn_proto=proto, extra_args=[ '--resolve', f'{domain}:{env.port_for(proto)}:::1,127.0.0.1', ]) assert r.exit_code == 60, f'{r}' assert r.stats[0]['errormsg'] != 'CURL_DBG_SOCK_FAIL_IPV6: failed to open socket' # Get, retry on 502 def test_05_06_retry_502(self, env: Env, httpd, nghttpx): proto = 'http/1.1' curl = CurlClient(env=env) url = f'https://{env.authority_for(env.domain1, proto)}/curltest/tweak?status=502' r = curl.http_download(urls=[url], alpn_proto=proto, extra_args=[ '--retry', '2', '--retry-all-errors', '--retry-delay', '1', ]) r.check_response(http_status=502) assert r.stats[0]['num_retries'] == 2, f'{r}' # curious, since curl does the retries, it finds the previous # connection in the cache and reports that no connects were done assert r.stats[0]['num_connects'] == 0, f'{r}' # Get, retry on 502 in parallel mode def test_05_07_retry_502_parallel(self, env: Env, httpd, nghttpx): proto = 'http/1.1' curl = CurlClient(env=env) url = f'https://{env.authority_for(env.domain1, proto)}/curltest/tweak?status=502' r = curl.http_download(urls=[url], alpn_proto=proto, extra_args=[ '--retry', '2', '--retry-all-errors', '--retry-delay', '1', '--parallel' ]) r.check_response(http_status=502) assert r.stats[0]['num_retries'] == 2, f'{r}' # Get, retry on 401, not happening def test_05_08_retry_401(self, env: Env, httpd, nghttpx): proto = 'http/1.1' curl = CurlClient(env=env) url = f'https://{env.authority_for(env.domain1, proto)}/curltest/tweak?status=401' r = curl.http_download(urls=[url], alpn_proto=proto, extra_args=[ '--retry', '2', '--retry-all-errors', '--retry-delay', '1' ]) r.check_response(http_status=401) # No retries on a 401 assert r.stats[0]['num_retries'] == 0, f'{r}' # Server closes the connection immediately after accept, def test_05_09_handshake_eof(self, env: Env): with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server: server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) server.bind(('127.0.0.1', 0)) server.listen(1) port = server.getsockname()[1] # accept one connection and immediately close it def accept_and_close(): # ignore expected socket error with contextlib.suppress(OSError): conn, _ = server.accept() conn.recv(1) # wait for ClientHello conn.close() t = threading.Thread(target=accept_and_close) t.start() run_env = os.environ.copy() run_env['CURL_DEBUG'] = 'all' curl = CurlClient(env=env, timeout=env.test_timeout, run_env=run_env) url = f'https://127.0.0.1:{port}/' r = curl.run_direct(args=[url, '--insecure', '-v']) t.join(timeout=10) # We expect an error code, not success (0) and not timeout (-1) # Expected error codes are: # - CURLE_SSL_CONNECT_ERROR (35) - common for handshake failures # - CURLE_RECV_ERROR (56) - some TLS backends fail with that assert r.exit_code in [35, 56], \ f'unexpected error {r.exit_code}\n{r.dump_logs()}' # Get a resource many times with limited requests @pytest.mark.skipif(condition=not Env.have_h2_curl(), reason='curl without HTTP/2') def test_05_10_limits(self, env: Env, httpd): proto = 'h2' count = 6 curl = CurlClient(env=env) url = f'https://{env.authority_for(env.domain1, proto)}/curltest/limit?id=[0-{count-1}]' r = curl.http_download(urls=[url], alpn_proto=proto, extra_args=[ '--parallel', '--retry', '5' ]) r.check_stats(count=count, http_status=200, exitcode=0) @pytest.fixture(scope='class') @classmethod def pr_server(cls, env: Env) -> Generator[HttpPureResp, None, None]: server = HttpPureResp(env=env) server.start() yield server server.stop() def test_05_11_pure_resp(self, env: Env, pr_server): proto = 'http/1.1' run_env = os.environ.copy() run_env['CURL_DBG_SOCK_CONNECT_WAIT_MS'] = '1000' curl = CurlClient(env=env, run_env=run_env) url = f'http://{env.domain1}:{pr_server.port}/xxx' r = curl.http_download(urls=[url], alpn_proto=proto, extra_args=[ '--connect-timeout', '5', '--max-time', '5' ]) r.check_exit_code(0) r.check_stats(count=1, http_status=200, exitcode=0)