Artifactspublic-v0

curl

master
ply4 git calls30 ms295 KiB3 cachedprotocol v2
492 lines19 KiBRaw

Building curl with HTTPS-RR and ECH support

We have added support for ECH to curl. It can use HTTPS RRs published in the DNS if curl uses DoH, or else can accept the relevant ECHConfigList values from the command line. This works with AWS-LC, BoringSSL, OpenSSL, Rustls or wolfSSL as the TLS provider.

This feature is EXPERIMENTAL. DO NOT USE IN PRODUCTION.

This should however provide enough of a proof-of-concept to prompt an informed discussion about a good path forward for ECH support in curl.

OpenSSL Build

To build OpenSSL 4.0.0+:

sh
cd $HOME/code
git clone --depth 1 --branch openssl-4.0.0 https://github.com/openssl/openssl
cd openssl
./config --libdir=lib --prefix=$HOME/code/openssl-local-inst
...stuff...
make -j8
...more stuff...
make install_sw
...a little bit of stuff...

To build curl ECH-enabled, making use of the above:

sh
cd $HOME/code
git clone --depth 1 https://github.com/curl/curl
cd curl
autoreconf -fi
LDFLAGS="-Wl,-rpath,$HOME/code/openssl-local-inst/lib/" ./configure --with-ssl=$HOME/code/openssl-local-inst --enable-ech
...lots of output...
WARNING: ECH is enabled but marked EXPERIMENTAL...
make
...lots more output...

If you do not get that WARNING at the end of the configure command, then ECH is not enabled, so go back some steps and re-do whatever needs re-doing:-) If you want to debug curl then you should add --enable-debug to the configure command.

In a recent (2024-05-20) build on one machine, configure failed to find the ECH-enabled SSL library, apparently due to the existence of $HOME/code/openssl-local-inst/lib/pkgconfig as a directory containing various settings. Deleting that directory worked around the problem but may not be the best solution.

Using ECH and DoH

curl supports using DoH for A/AAAA lookups so it was relatively easy to add retrieval of HTTPS RRs in that situation. To use ECH and DoH together:

sh
cd $HOME/code/curl
LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl --ech true --doh-url https://one.one.one.one/dns-query https://defo.ie/ech-check.php
...
SSL_ECH_STATUS: success <img src="greentick-small.png" alt="good" /> <br/>
...

The output snippet above is within the HTML for the webpage, when things work.

The above works for these test sites:

sh
https://defo.ie/ech-check.php
https://crypto.cloudflare.com/cdn-cgi/trace
https://tls-ech.dev/

The list above has 4 different server technologies, implemented by 3 different parties, and includes a case (the port 8414 server) where HelloRetryRequest (HRR) is forced.

We currently support the following new curl command line arguments/options:

  • --ech <config> - the config value can be one of:
    • false says to not attempt ECH
    • true says to attempt ECH, if possible
    • grease if attempting ECH is not possible, then send a GREASE ECH extension
    • hard hard-fail the connection if ECH cannot be attempted
    • ecl:<b64value> a base64 encoded ECHConfigList, rather than one accessed from the DNS
    • pn:<name> override the public_name from an ECHConfigList

Note that in the above "attempt ECH" means the client emitting a TLS ClientHello with a "real" ECH extension, but that does not mean that the relevant server can succeed in decrypting, as things can fail for other reasons.

Supplying an ECHConfigList on the command line

To supply the ECHConfigList on the command line, you might need a bit of cut-and-paste, e.g.:

sh
dig +short https defo.ie
1 . ipv4hint=213.108.108.101 ech=AED+DQA8PAAgACD8WhlS7VwEt5bf3lekhHvXrQBGDrZh03n/LsNtAodbUAAEAAEAAQANY292ZXIuZGVmby5pZQAA
  ipv6hint=2a00:c6c0:0:116:5::10

Then paste the base64 encoded ECHConfigList onto the curl command line:

sh
LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl --ech ecl:AED+DQA8PAAgACD8WhlS7VwEt5bf3lekhHvXrQBGDrZh03n/LsNtAodbUAAEAAEAAQANY292ZXIuZGVmby5pZQAA \
  https://defo.ie/ech-check.php
...
SSL_ECH_STATUS: success <img src="greentick-small.png" alt="good" /> <br/>
...

The output snippet above is within the HTML for the webpage.

If you paste in the wrong ECHConfigList (it changes hourly for defo.ie) you should get an error like this:

sh
LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl -vvv --ech ecl:AED+DQA8yAAgACDRMQo+qYNsNRNj+vfuQfFIkrrUFmM4vogucxKj/4nzYgAEAAEAAQANY292ZXIuZGVmby5pZQAA \
  https://defo.ie/ech-check.php
...
* OpenSSL/3.3.0: error:0A00054B:SSL routines::ech required
...

There is a reason to want this command line option - for use before publishing an ECHConfigList in the DNS as per the Internet-draft A well-known URI for publishing ECHConfigList values.

If you do use a wrong ECHConfigList value, then the server might return a good value, via the retry_configs mechanism. You can see that value in the verbose output, e.g.:

sh
LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl -vvv --ech ecl:AED+DQA8yAAgACDRMQo+qYNsNRNj+vfuQfFIkrrUFmM4vogucxKj/4nzYgAEAAEAAQANY292ZXIuZGVmby5pZQAA \
  https://defo.ie/ech-check.php
...
* ECH: retry_configs AQD+DQA8DAAgACBvYqJy+Hgk33wh/ZLBzKSPgwxeop7gvojQzfASq7zeZQAEAAEAAQANY292ZXIuZGVmby5pZQAA/g0APEMAIAAgXkT5r4cYs8z19q5rdittyIX
  8gfQ3ENW4wj1fVoiJZBoABAABAAEADWNvdmVyLmRlZm8uaWUAAP4NADw2ACAAINXSE9EdXzEQIJZA7vpwCIQsWqsFohZARXChgPsnfI1kAAQAAQABAA1jb3Zlci5kZWZvLmllAAD+DQA8c
  QAgACASeiD5F+UoSnVoHvA2l1EifUVMFtbVZ76xwDqmMPraHQAEAAEAAQANY292ZXIuZGVmby5pZQAA
* ECH: retry_configs for defo.ie from cover.defo.ie, 319
...

At that point, you could copy the base64 encoded value above and try again. For now, this only works for the OpenSSL and AWS-LC/BoringSSL builds.

Default settings

curl has various ways to configure default settings, e.g. in $HOME/.curlrc, so one can set the DoH URL and enable ECH that way:

sh
cat ~/.curlrc
doh-url=https://one.one.one.one/dns-query
silent
ech=true

Note that when you use the system's curl command (rather than our ECH-enabled build), it is liable to warn that ech is an unknown option. If that is an issue (e.g. if some script re-directs stdout and stderr somewhere) then adding the silent line above seems to be a good enough fix. (Though of course, yet another script could depend on non-silent behavior, so you may have to figure out what you prefer yourself.) That seems to have changed with the latest build, previously silent=TRUE was what I used in ~/.curlrc but now that seems to cause a problem, so that the following line(s) are ignored.

If you want to always use our OpenSSL build you can set LD_LIBRARY_PATH in the environment:

sh
export LD_LIBRARY_PATH=$HOME/code/openssl

When you do the above, there can be a mismatch between OpenSSL versions for applications that check that. A git push for example fails so you should unset LD_LIBRARY_PATH before doing that or use a different shell.

sh
git push
OpenSSL version mismatch. Built against 30000080, you have 30200000
...

With all that setup as above the command line gets simpler:

sh
./src/curl https://defo.ie/ech-check.php
...
SSL_ECH_STATUS: success <img src="greentick-small.png" alt="good" /> <br/>
...

The --ech true option is opportunistic, so tries to do ECH but does not fail if the client for example cannot find any ECHConfig values. The --ech hard option hard-fails if there is no ECHConfig found in DNS, so for now, that is not a good option to set as a default. Once ECH has really been attempted by the client, if decryption on the server side fails, then curl fails.

Code changes for ECH support when using DoH

Code changes are #ifdef protected via USE_ECH or USE_HTTPSRR:

  • USE_HTTPSRR is used for HTTPS RR retrieval code that could be generically used should non-ECH uses for HTTPS RRs be identified, e.g. use of ALPN values or IP address hints.

  • USE_ECH protects ECH specific code.

There are various obvious code blocks for handling the new command line arguments which are not described here, but should be fairly clear.

As shown in the configure usage above, there are configure.ac changes that allow separately dis/enabling USE_HTTPSRR and USE_ECH. If USE_ECH is enabled, then USE_HTTPSRR is forced. In both cases CURL_DISABLE_DOH must not be enabled. (There may be some configuration conflicts available for the determined :-)

The main functional change, as you would expect, is in lib/vtls/openssl.c where an ECHConfig, if available from command line or DNS cache, is fed into the OpenSSL library via the new APIs implemented in our OpenSSL fork for that purpose. This code also implements the opportunistic (--ech true) or hard-fail (--ech hard) logic.

Other than that, the main additions are in lib/doh.c where we reuse dohprobe() to retrieve an HTTPS RR value for the target domain. If such a value is found, that is stored using a new doh_store_https() function in a new field in the dohentry structure.

The qname for the DoH query is modified if the port number is not 443, as defined in the SVCB specification.

When the DoH process has worked, Curl_doh_take_result() now also returns the relevant HTTPS RR value data in the Curl_dns_entry structure. That is later accessed when the TLS session is being established, if ECH is enabled (from lib/vtls/openssl.c as described above).

Limitations

Things that need fixing, but that can probably be ignored for the moment:

  • We could easily add code to make use of an alpn= value found in an HTTPS RR, passing that on to OpenSSL for use as the "inner" ALPN value, but have yet to do that.

Current limitations (more interesting than the above):

  • Only the first HTTPS RR value retrieved is actually processed as described above, that could be extended in future, though picking the "right" HTTPS RR could be non-trivial if multiple RRs are published - matching IP address hints versus A/AAAA values might be a good basis for that. Last I checked though, browsers supporting ECH did not handle multiple HTTPS RRs well, though that needs re-checking as it has been a while.

  • It is unclear how one should handle any IP address hints found in an HTTPS RR. It may be that a bit of consideration of how "multi-CDN" deployments might emerge would provide good answers there, but for now, it is not clear how best curl might handle those values when present in the DNS.

  • The SVCB/HTTPS RR specification supports a new "CNAME at apex" indirection ("aliasMode") - the current code takes no account of that at all. One could envisage implementing the equivalent of following CNAMEs in such cases, but it is not clear if that'd be a good plan. (As of now, chrome browsers do not seem to have any support for that "aliasMode" and we have not checked Firefox for that recently.)

  • We have not investigated what related changes or additions might be needed for applications using libcurl, as opposed to use of curl as a command line tool.

  • We have not yet implemented tests as part of the usual curl test harness as doing so would seem to require re-implementing an ECH-enabled server as part of the curl test harness. For now, we have a ./tests/ech_test.sh script that attempts ECH with various test servers and with many combinations of the allowed command line options. While that is a useful test and has find issues, it is not comprehensive and we are not (as yet) sure what would be the right level of coverage. When running that script you should not have a $HOME/.curlrc file that affects ECH or some of the negative tests could produce spurious failures.

Building with cmake

To build with cmake, assuming our ECH-enabled OpenSSL is as before:

sh
cd $HOME/code
git clone --depth 1 https://github.com/curl/curl
cd curl
mkdir build
cd build
cmake -DOPENSSL_ROOT_DIR=$HOME/code/openssl -DUSE_ECH=1 ..
...
make
...
[100%] Built target curl

The binary produced by the cmake build does not need any ECH-specific LD_LIBRARY_PATH setting.

BoringSSL build

BoringSSL is also supported by curl and also supports ECH, so to build with that, instead of our ECH-enabled OpenSSL:

sh
cd $HOME/code
git clone --depth 1 https://boringssl.googlesource.com/boringssl
cd boringssl
cmake -DCMAKE_INSTALL_PREFIX:PATH=$HOME/code/boringssl/inst -DBUILD_SHARED_LIBS=1
make
...
make install

Then:

sh
cd $HOME/code
git clone --depth 1 https://github.com/curl/curl
cd curl
autoreconf -fi
LDFLAGS="-Wl,-rpath,$HOME/code/boringssl/inst/lib" ./configure --with-ssl=$HOME/code/boringssl/inst --enable-ech
...lots of output...
WARNING: ECH HTTPSRR enabled but marked EXPERIMENTAL. Use with caution.
make

The AWS-LC/BoringSSL APIs are fairly similar to those in our ECH-enabled OpenSSL fork, so code changes are also in lib/vtls/openssl.c, protected via #ifdef OPENSSL_IS_BORINGSSL and are mostly obvious API variations.

The AWS-LC/BoringSSL APIs however do not support the --ech pn: command line variant as of now.

wolfSSL build

wolfSSL also supports ECH and can be used by curl, so here's how:

sh
cd $HOME/code
git clone --depth 1 https://github.com/wolfSSL/wolfssl
cd wolfssl
./autogen.sh
./configure --prefix=$HOME/code/wolfssl/inst --enable-ech --enable-debug --enable-opensslextra
make
make install

The install prefix (inst) in the above causes wolfSSL to be installed there and we seem to need that for the curl configure command to work out. The --enable-opensslextra turns out (after much faffing about;-) to be important or else we get build problems with curl below.

sh
cd $HOME/code
git clone --depth 1 https://github.com/curl/curl
cd curl
autoreconf -fi
./configure --with-wolfssl=$HOME/code/wolfssl/inst --enable-ech
make

There are some known issues with the ECH implementation in wolfSSL:

  • The main issue is that the client currently handles HelloRetryRequest incorrectly. HRR issue.) The HRR issue means that the client does not work for this ECH test web site and any other similarly configured sites.
  • There is also an issue related to so-called middlebox compatibility mode. middlebox compatibility issue

Code changes to support wolfSSL

There are what seem like oddball differences:

  • The DoH URL in$HOME/.curlrc can use 1.1.1.1 for OpenSSL but has to be one.one.one.one for wolfSSL. The latter works for both, so OK, we us that.
  • There seems to be some difference in CA databases too - the wolfSSL version does not like defo.ie, whereas the system and OpenSSL ones do. We can ignore that for our purposes via --insecure/-k but would need to fix for a real setup. (Browsers do like those certificates though.)

Then there are some functional code changes:

  • tweak to configure.ac to check if wolfSSL has ECH or not
  • added code to lib/vtls/wolfssl.c mirroring what's done in the OpenSSL equivalent above.
  • wolfSSL does not support --ech false or the --ech pn: command line argument.

The lack of support for --ech false is because wolfSSL has decided to always at least GREASE if built to support ECH. In other words, GREASE is a compile time choice for wolfSSL, but a runtime choice for OpenSSL or AWS-LC/BoringSSL. (Both are reasonable.)

Additional notes

Supporting ECH without DoH

All of the above only applies if DoH is being used. There should be a use-case for ECH when DoH is not used by curl - if a system stub resolver supports DoT or DoH, then, considering only ECH and the network threat model, it would make sense for curl to support ECH without curl itself using DoH. The author for example uses a combination of stubby+unbound as the system resolver listening on localhost:53, so would fit this use-case. That said, it is unclear if this is a niche that is worth trying to address. (The author is happy to let curl use DoH to talk to the same public recursive that stubby might use:-)

Assuming for the moment this is a use-case we would like to support, then if DoH is not being used by curl, it is not clear at this time how to provide support for ECH. One option would seem to be to extend the c-ares library to support HTTPS RRs, but in that case it is not now clear whether such changes would be attractive to the c-ares maintainers, nor whether the "tag=value" extensibility inherent in the HTTPS/SVCB specification is a good match for the c-ares approach of defining structures specific to decoded answers for each supported RRtype. We are also not sure how many downstream curl deployments actually make use of the c-ares library, which would affect the utility of such changes. Another option might be to consider using some other generic DNS library that does support HTTPS RRs, but it is unclear if such a library could or would be used by all or almost all curl builds and downstream releases of curl.

Our current conclusion is that doing the above is likely best left until we have some experience with the "using DoH" approach, so we are going to punt on this for now.

Localhost testing

It can be useful to be able to run against a localhost OpenSSL s_server for testing. We have published instructions for such localhost tests in another repository. Once you have that set up, you can start a server and then run curl against that:

sh
cd $HOME/code/ech-dev-utils
./scripts/echsvr.sh -d
...

The echsvr.sh script supports many ECH-related options. Use echsvr.sh -h for details.

In another window:

sh
cd $HOME/code/curl/
./src/curl -vvv --insecure --connect-to foo.example.com:8443:localhost:8443 \
  --ech ecl:AD7+DQA6uwAgACBix2B78sX+EQhEbxMspDOc8Z3xVS5aQpYP0Cxpc2AWPAAEAAEAAQALZXhhbXBsZS5jb20AAA==

Automated use of retry_configs not supported so far...

As of now we have not added support for using retry_config handling in the application - for a command line tool, one can use dig (or kdig) to get the HTTPS RR and pass the ECHConfigList from that on the command line, if needed, or one can access the value from command line output in verbose more and then reuse that in another invocation.

Both our OpenSSL fork and AWS-LC/BoringSSL have APIs for both controlling GREASE and accessing and logging retry_configs, it seems wolfSSL has neither.

Testing ECH

We have yet to add a robust test setup for ECH as that requires an ECH-enabled test server.

We have added two basic tests though, aiming to ensure that the client sends a GREASE or real ECH extension when requested, and reacts correctly to the failure of ECH in the latter case. (Given that stunnel has no ECH support.)

As with other similar tests, those tests require the stunnel tool be installed. On Ubuntu sudo apt install stunnel4 achieves that.

The test cases are:

  • data/test4000: GREASE ECH, expected result: connection succeeds
  • data/test4001: real ECH, connection fails with error 101 (ECH required)
1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# Building curl with HTTPS-RR and ECH support
9We have added support for ECH to curl. It can use HTTPS RRs published in the
10DNS if curl uses DoH, or else can accept the relevant ECHConfigList values
11from the command line. This works with AWS-LC, BoringSSL, OpenSSL, Rustls or
12wolfSSL as the TLS provider.
14This feature is EXPERIMENTAL. DO NOT USE IN PRODUCTION.
16This should however provide enough of a proof-of-concept to prompt an informed
17discussion about a good path forward for ECH support in curl.
19## OpenSSL Build
21To build OpenSSL 4.0.0+:
23```sh
24cd $HOME/code
25git clone --depth 1 --branch openssl-4.0.0 https://github.com/openssl/openssl
26cd openssl
27./config --libdir=lib --prefix=$HOME/code/openssl-local-inst
28...stuff...
29make -j8
30...more stuff...
31make install_sw
32...a little bit of stuff...
33```
35To build curl ECH-enabled, making use of the above:
37```sh
38cd $HOME/code
39git clone --depth 1 https://github.com/curl/curl
40cd curl
41autoreconf -fi
42LDFLAGS="-Wl,-rpath,$HOME/code/openssl-local-inst/lib/" ./configure --with-ssl=$HOME/code/openssl-local-inst --enable-ech
43...lots of output...
44WARNING: ECH is enabled but marked EXPERIMENTAL...
45make
46...lots more output...
47```
49If you do not get that WARNING at the end of the `configure` command, then
50ECH is not enabled, so go back some steps and re-do whatever needs re-doing:-)
51If you want to debug curl then you should add `--enable-debug` to the
52`configure` command.
54In a recent (2024-05-20) build on one machine, configure failed to find the
55ECH-enabled SSL library, apparently due to the existence of
56`$HOME/code/openssl-local-inst/lib/pkgconfig` as a directory containing
57various settings. Deleting that directory worked around the problem but may
58not be the best solution.
60## Using ECH and DoH
62curl supports using DoH for A/AAAA lookups so it was relatively easy to add
63retrieval of HTTPS RRs in that situation. To use ECH and DoH together:
65```sh
66cd $HOME/code/curl
67LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl --ech true --doh-url https://one.one.one.one/dns-query https://defo.ie/ech-check.php
68...
69SSL_ECH_STATUS: success <img src="greentick-small.png" alt="good" /> <br/>
70...
71```
73The output snippet above is within the HTML for the webpage, when things work.
75The above works for these test sites:
77```sh
78https://defo.ie/ech-check.php
79https://crypto.cloudflare.com/cdn-cgi/trace
80https://tls-ech.dev/
81```
83The list above has 4 different server technologies, implemented by 3 different
84parties, and includes a case (the port 8414 server) where HelloRetryRequest
85(HRR) is forced.
87We currently support the following new curl command line arguments/options:
89- `--ech <config>` - the `config` value can be one of:
90 - `false` says to not attempt ECH
91 - `true` says to attempt ECH, if possible
92 - `grease` if attempting ECH is not possible, then send a GREASE ECH extension
93 - `hard` hard-fail the connection if ECH cannot be attempted
94 - `ecl:<b64value>` a base64 encoded ECHConfigList, rather than one accessed from the DNS
95 - `pn:<name>` override the `public_name` from an ECHConfigList
97Note that in the above "attempt ECH" means the client emitting a TLS
98ClientHello with a "real" ECH extension, but that does not mean that the
99relevant server can succeed in decrypting, as things can fail for other
100reasons.
102## Supplying an ECHConfigList on the command line
104To supply the ECHConfigList on the command line, you might need a bit of
105cut-and-paste, e.g.:
107```sh
108dig +short https defo.ie
1091 . ipv4hint=213.108.108.101 ech=AED+DQA8PAAgACD8WhlS7VwEt5bf3lekhHvXrQBGDrZh03n/LsNtAodbUAAEAAEAAQANY292ZXIuZGVmby5pZQAA
110 ipv6hint=2a00:c6c0:0:116:5::10
111```
113Then paste the base64 encoded ECHConfigList onto the curl command line:
115```sh
116LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl --ech ecl:AED+DQA8PAAgACD8WhlS7VwEt5bf3lekhHvXrQBGDrZh03n/LsNtAodbUAAEAAEAAQANY292ZXIuZGVmby5pZQAA \
117 https://defo.ie/ech-check.php
118...
119SSL_ECH_STATUS: success <img src="greentick-small.png" alt="good" /> <br/>
120...
121```
123The output snippet above is within the HTML for the webpage.
125If you paste in the wrong ECHConfigList (it changes hourly for `defo.ie`) you
126should get an error like this:
128```sh
129LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl -vvv --ech ecl:AED+DQA8yAAgACDRMQo+qYNsNRNj+vfuQfFIkrrUFmM4vogucxKj/4nzYgAEAAEAAQANY292ZXIuZGVmby5pZQAA \
130 https://defo.ie/ech-check.php
131...
132* OpenSSL/3.3.0: error:0A00054B:SSL routines::ech required
133...
134```
136There is a reason to want this command line option - for use before publishing
137an ECHConfigList in the DNS as per the Internet-draft [A well-known URI for
138publishing ECHConfigList values](https://datatracker.ietf.org/doc/html/draft-ietf-tls-wkech/).
140If you do use a wrong ECHConfigList value, then the server might return a
141good value, via the `retry_configs` mechanism. You can see that value in
142the verbose output, e.g.:
144```sh
145LD_LIBRARY_PATH=$HOME/code/openssl ./src/curl -vvv --ech ecl:AED+DQA8yAAgACDRMQo+qYNsNRNj+vfuQfFIkrrUFmM4vogucxKj/4nzYgAEAAEAAQANY292ZXIuZGVmby5pZQAA \
146 https://defo.ie/ech-check.php
147...
148* ECH: retry_configs AQD+DQA8DAAgACBvYqJy+Hgk33wh/ZLBzKSPgwxeop7gvojQzfASq7zeZQAEAAEAAQANY292ZXIuZGVmby5pZQAA/g0APEMAIAAgXkT5r4cYs8z19q5rdittyIX
149 8gfQ3ENW4wj1fVoiJZBoABAABAAEADWNvdmVyLmRlZm8uaWUAAP4NADw2ACAAINXSE9EdXzEQIJZA7vpwCIQsWqsFohZARXChgPsnfI1kAAQAAQABAA1jb3Zlci5kZWZvLmllAAD+DQA8c
150 QAgACASeiD5F+UoSnVoHvA2l1EifUVMFtbVZ76xwDqmMPraHQAEAAEAAQANY292ZXIuZGVmby5pZQAA
151* ECH: retry_configs for defo.ie from cover.defo.ie, 319
152...
153```
155At that point, you could copy the base64 encoded value above and try again.
156For now, this only works for the OpenSSL and AWS-LC/BoringSSL builds.
158## Default settings
160curl has various ways to configure default settings, e.g. in `$HOME/.curlrc`,
161so one can set the DoH URL and enable ECH that way:
163```sh
164cat ~/.curlrc
165doh-url=https://one.one.one.one/dns-query
166silent
167ech=true
168```
170Note that when you use the system's curl command (rather than our ECH-enabled
171build), it is liable to warn that `ech` is an unknown option. If that is an
172issue (e.g. if some script re-directs stdout and stderr somewhere) then adding
173the `silent` line above seems to be a good enough fix. (Though of
174course, yet another script could depend on non-silent behavior, so you may have
175to figure out what you prefer yourself.) That seems to have changed with the
176latest build, previously `silent=TRUE` was what I used in `~/.curlrc` but
177now that seems to cause a problem, so that the following line(s) are ignored.
179If you want to always use our OpenSSL build you can set `LD_LIBRARY_PATH`
180in the environment:
182```sh
183export LD_LIBRARY_PATH=$HOME/code/openssl
184```
186When you do the above, there can be a mismatch between OpenSSL versions
187for applications that check that. A `git push` for example fails so you
188should unset `LD_LIBRARY_PATH` before doing that or use a different shell.
190```sh
191git push
192OpenSSL version mismatch. Built against 30000080, you have 30200000
193...
194```
196With all that setup as above the command line gets simpler:
198```sh
199./src/curl https://defo.ie/ech-check.php
200...
201SSL_ECH_STATUS: success <img src="greentick-small.png" alt="good" /> <br/>
202...
203```
205The `--ech true` option is opportunistic, so tries to do ECH but does not fail if
206the client for example cannot find any ECHConfig values. The `--ech hard`
207option hard-fails if there is no ECHConfig found in DNS, so for now, that is not
208a good option to set as a default. Once ECH has really been attempted by
209the client, if decryption on the server side fails, then curl fails.
211## Code changes for ECH support when using DoH
213Code changes are `#ifdef` protected via `USE_ECH` or `USE_HTTPSRR`:
215- `USE_HTTPSRR` is used for HTTPS RR retrieval code that could be generically
216 used should non-ECH uses for HTTPS RRs be identified, e.g. use of ALPN values
217 or IP address hints.
219- `USE_ECH` protects ECH specific code.
221There are various obvious code blocks for handling the new command line
222arguments which are not described here, but should be fairly clear.
224As shown in the `configure` usage above, there are `configure.ac` changes
225that allow separately dis/enabling `USE_HTTPSRR` and `USE_ECH`. If `USE_ECH`
226is enabled, then `USE_HTTPSRR` is forced. In both cases `CURL_DISABLE_DOH`
227must not be enabled. (There may be some configuration conflicts available for the
228determined :-)
230The main functional change, as you would expect, is in `lib/vtls/openssl.c`
231where an ECHConfig, if available from command line or DNS cache, is fed into
232the OpenSSL library via the new APIs implemented in our OpenSSL fork for that
233purpose. This code also implements the opportunistic (`--ech true`) or hard-fail
234(`--ech hard`) logic.
236Other than that, the main additions are in `lib/doh.c`
237where we reuse `dohprobe()` to retrieve an HTTPS RR value for the target
238domain. If such a value is found, that is stored using a new `doh_store_https()`
239function in a new field in the `dohentry` structure.
241The qname for the DoH query is modified if the port number is not 443, as
242defined in the SVCB specification.
244When the DoH process has worked, `Curl_doh_take_result()` now also returns
245the relevant HTTPS RR value data in the `Curl_dns_entry` structure.
246That is later accessed when the TLS session is being established, if ECH is
247enabled (from `lib/vtls/openssl.c` as described above).
249## Limitations
251Things that need fixing, but that can probably be ignored for the
252moment:
254- We could easily add code to make use of an `alpn=` value found in an HTTPS
255 RR, passing that on to OpenSSL for use as the "inner" ALPN value, but have
256 yet to do that.
258Current limitations (more interesting than the above):
260- Only the first HTTPS RR value retrieved is actually processed as described
261 above, that could be extended in future, though picking the "right" HTTPS RR
262 could be non-trivial if multiple RRs are published - matching IP address
263 hints versus A/AAAA values might be a good basis for that. Last I checked
264 though, browsers supporting ECH did not handle multiple HTTPS RRs well, though
265 that needs re-checking as it has been a while.
267- It is unclear how one should handle any IP address hints found in an HTTPS RR.
268 It may be that a bit of consideration of how "multi-CDN" deployments might
269 emerge would provide good answers there, but for now, it is not clear how best
270 curl might handle those values when present in the DNS.
272- The SVCB/HTTPS RR specification supports a new "CNAME at apex" indirection
273 ("aliasMode") - the current code takes no account of that at all. One could
274 envisage implementing the equivalent of following CNAMEs in such cases, but
275 it is not clear if that'd be a good plan. (As of now, chrome browsers do not
276 seem to have any support for that "aliasMode" and we have not checked Firefox
277 for that recently.)
279- We have not investigated what related changes or additions might be needed
280 for applications using libcurl, as opposed to use of curl as a command line
281 tool.
283- We have not yet implemented tests as part of the usual curl test harness as
284 doing so would seem to require re-implementing an ECH-enabled server as part
285 of the curl test harness. For now, we have a `./tests/ech_test.sh` script
286 that attempts ECH with various test servers and with many combinations of the
287 allowed command line options. While that is a useful test and has find
288 issues, it is not comprehensive and we are not (as yet) sure what would be
289 the right level of coverage. When running that script you should not have a
290 `$HOME/.curlrc` file that affects ECH or some of the negative tests could
291 produce spurious failures.
293## Building with cmake
295To build with cmake, assuming our ECH-enabled OpenSSL is as before:
297```sh
298cd $HOME/code
299git clone --depth 1 https://github.com/curl/curl
300cd curl
301mkdir build
302cd build
303cmake -DOPENSSL_ROOT_DIR=$HOME/code/openssl -DUSE_ECH=1 ..
304...
305make
306...
307[100%] Built target curl
308```
310The binary produced by the cmake build does not need any ECH-specific
311`LD_LIBRARY_PATH` setting.
313## BoringSSL build
315BoringSSL is also supported by curl and also supports ECH, so to build
316with that, instead of our ECH-enabled OpenSSL:
318```sh
319cd $HOME/code
320git clone --depth 1 https://boringssl.googlesource.com/boringssl
321cd boringssl
322cmake -DCMAKE_INSTALL_PREFIX:PATH=$HOME/code/boringssl/inst -DBUILD_SHARED_LIBS=1
323make
324...
325make install
326```
328Then:
330```sh
331cd $HOME/code
332git clone --depth 1 https://github.com/curl/curl
333cd curl
334autoreconf -fi
335LDFLAGS="-Wl,-rpath,$HOME/code/boringssl/inst/lib" ./configure --with-ssl=$HOME/code/boringssl/inst --enable-ech
336...lots of output...
337WARNING: ECH HTTPSRR enabled but marked EXPERIMENTAL. Use with caution.
338make
339```
341The AWS-LC/BoringSSL APIs are fairly similar to those in our ECH-enabled
342OpenSSL fork, so code changes are also in `lib/vtls/openssl.c`, protected
343via `#ifdef OPENSSL_IS_BORINGSSL` and are mostly obvious API variations.
345The AWS-LC/BoringSSL APIs however do not support the `--ech pn:` command
346line variant as of now.
348## wolfSSL build
350wolfSSL also supports ECH and can be used by curl, so here's how:
352```sh
353cd $HOME/code
354git clone --depth 1 https://github.com/wolfSSL/wolfssl
355cd wolfssl
356./autogen.sh
357./configure --prefix=$HOME/code/wolfssl/inst --enable-ech --enable-debug --enable-opensslextra
358make
359make install
360```
362The install prefix (`inst`) in the above causes wolfSSL to be installed there
363and we seem to need that for the curl configure command to work out. The
364`--enable-opensslextra` turns out (after much faffing about;-) to be
365important or else we get build problems with curl below.
367```sh
368cd $HOME/code
369git clone --depth 1 https://github.com/curl/curl
370cd curl
371autoreconf -fi
372./configure --with-wolfssl=$HOME/code/wolfssl/inst --enable-ech
373make
374```
376There are some known issues with the ECH implementation in wolfSSL:
378- The main issue is that the client currently handles HelloRetryRequest
379 incorrectly. [HRR issue](https://github.com/wolfSSL/wolfssl/issues/6802).)
380 The HRR issue means that the client does not work for
381 [this ECH test web site](https://tls-ech.dev/) and any other similarly
382 configured sites.
383- There is also an issue related to so-called middlebox compatibility mode.
384 [middlebox compatibility issue](https://github.com/wolfSSL/wolfssl/issues/6774)
386### Code changes to support wolfSSL
388There are what seem like oddball differences:
390- The DoH URL in`$HOME/.curlrc` can use `1.1.1.1` for OpenSSL but has to be
391 `one.one.one.one` for wolfSSL. The latter works for both, so OK, we us that.
392- There seems to be some difference in CA databases too - the wolfSSL version
393 does not like `defo.ie`, whereas the system and OpenSSL ones do. We can
394 ignore that for our purposes via `--insecure`/`-k` but would need to fix
395 for a real setup. (Browsers do like those certificates though.)
397Then there are some functional code changes:
399- tweak to `configure.ac` to check if wolfSSL has ECH or not
400- added code to `lib/vtls/wolfssl.c` mirroring what's done in the
401 OpenSSL equivalent above.
402- wolfSSL does not support `--ech false` or the `--ech pn:` command line
403 argument.
405The lack of support for `--ech false` is because wolfSSL has decided to
406always at least GREASE if built to support ECH. In other words, GREASE is
407a compile time choice for wolfSSL, but a runtime choice for OpenSSL or
408AWS-LC/BoringSSL. (Both are reasonable.)
410## Additional notes
412### Supporting ECH without DoH
414All of the above only applies if DoH is being used. There should be a use-case
415for ECH when DoH is not used by curl - if a system stub resolver supports DoT
416or DoH, then, considering only ECH and the network threat model, it would make
417sense for curl to support ECH without curl itself using DoH. The author for
418example uses a combination of stubby+unbound as the system resolver listening
419on localhost:53, so would fit this use-case. That said, it is unclear if this
420is a niche that is worth trying to address. (The author is happy to let curl
421use DoH to talk to the same public recursive that stubby might use:-)
423Assuming for the moment this is a use-case we would like to support, then if
424DoH is not being used by curl, it is not clear at this time how to provide
425support for ECH. One option would seem to be to extend the `c-ares` library
426to support HTTPS RRs, but in that case it is not now clear whether such
427changes would be attractive to the `c-ares` maintainers, nor whether the
428"tag=value" extensibility inherent in the HTTPS/SVCB specification is a good
429match for the `c-ares` approach of defining structures specific to decoded
430answers for each supported RRtype. We are also not sure how many downstream
431curl deployments actually make use of the `c-ares` library, which would
432affect the utility of such changes. Another option might be to consider using
433some other generic DNS library that does support HTTPS RRs, but it is unclear
434if such a library could or would be used by all or almost all curl builds and
435downstream releases of curl.
437Our current conclusion is that doing the above is likely best left until we
438have some experience with the "using DoH" approach, so we are going to punt on
439this for now.
441### Localhost testing
443It can be useful to be able to run against a localhost OpenSSL `s_server`
444for testing. We have published instructions for such
445[localhost tests](https://github.com/defo-project/ech-dev-utils/blob/main/howtos/localhost-tests.md)
446in another repository. Once you have that set up, you can start a server
447and then run curl against that:
449```sh
450cd $HOME/code/ech-dev-utils
451./scripts/echsvr.sh -d
452...
453```
455The `echsvr.sh` script supports many ECH-related options. Use `echsvr.sh -h`
456for details.
458In another window:
460```sh
461cd $HOME/code/curl/
462./src/curl -vvv --insecure --connect-to foo.example.com:8443:localhost:8443 \
463 --ech ecl:AD7+DQA6uwAgACBix2B78sX+EQhEbxMspDOc8Z3xVS5aQpYP0Cxpc2AWPAAEAAEAAQALZXhhbXBsZS5jb20AAA==
464```
466### Automated use of `retry_configs` not supported so far...
468As of now we have not added support for using `retry_config` handling in the
469application - for a command line tool, one can use `dig` (or `kdig`) to
470get the HTTPS RR and pass the ECHConfigList from that on the command line, if
471needed, or one can access the value from command line output in verbose more
472and then reuse that in another invocation.
474Both our OpenSSL fork and AWS-LC/BoringSSL have APIs for both controlling GREASE
475and accessing and logging `retry_configs`, it seems wolfSSL has neither.
477### Testing ECH
479We have yet to add a robust test setup for ECH as that requires an ECH-enabled
480test server.
482We have added two basic tests though, aiming to ensure that the client sends a
483GREASE or real ECH extension when requested, and reacts correctly to the
484failure of ECH in the latter case. (Given that `stunnel` has no ECH support.)
486As with other similar tests, those tests require the `stunnel` tool be
487installed. On Ubuntu `sudo apt install stunnel4` achieves that.
489The test cases are:
491- data/test4000: GREASE ECH, expected result: connection succeeds
492- data/test4001: real ECH, connection fails with error 101 (ECH required)
Diagnostics4 git calls · 30 ms
CallStatusSizeTime030 ms
discover200152 B4 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit4 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=4
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=4
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=23
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2008.0 KiB · hit23 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=8206
git.response.cache=HIT
git.response.duration_ms=23
git.fetch.wants=8a0153209d8c147dcddc21139e8aedfd8a42931e
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=8206
git.pkt.payload.max=8185
git.sideband.band1.bytes=8184
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e24f5838390ecd
Request
4bc6b0f9…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format