Artifactspublic-v0

curl

master
ply4 git calls26 ms290 KiB3 cachedprotocol v2
168 lines7.1 KiBRaw

curl client writers

Client writers is a design in the internals of libcurl, not visible in its public API. They were started in curl 8.5.0. This document describes the concepts, its high level implementation and the motivations.

Naming

libcurl operates between clients and servers. A client is the application using libcurl, like the command line tool curl itself. Data to be uploaded to a server is read from the client and send to the server, the servers response is received by libcurl and then written to the client.

With this naming established, client writers are concerned with writing responses from the server to the application. Applications register callbacks via CURLOPT_WRITEFUNCTION and CURLOPT_HEADERFUNCTION to be invoked by libcurl when the response is received.

Invoking

All code in libcurl that handles response data is ultimately expected to forward this data via Curl_client_write() to the application. The exact prototype of this function is:

c
CURLcode Curl_client_write(struct Curl_easy *data, int type, const char *buf, size_t blen);

The type argument specifies what the bytes in buf actually are. The following bits are defined:

c
#define CLIENTWRITE_BODY    (1 << 0) /* non-meta information, BODY */
#define CLIENTWRITE_INFO    (1 << 1) /* meta information, not a HEADER */
#define CLIENTWRITE_HEADER  (1 << 2) /* meta information, HEADER */
#define CLIENTWRITE_STATUS  (1 << 3) /* a special status HEADER */
#define CLIENTWRITE_CONNECT (1 << 4) /* a CONNECT related HEADER */
#define CLIENTWRITE_1XX     (1 << 5) /* a 1xx response related HEADER */
#define CLIENTWRITE_TRAILER (1 << 6) /* a trailer HEADER */

The main types here are CLIENTWRITE_BODY and CLIENTWRITE_HEADER. They are mutually exclusive. The other bits are enhancements to CLIENTWRITE_HEADER to specify what the header is about. They are only used in HTTP and related protocols (RTSP and WebSocket).

The implementation of Curl_client_write() uses a chain of client writer instances to process the call and make sure that the bytes reach the proper application callbacks. This is similar to the design of connection filters: client writers can be chained to process the bytes written through them. The definition is:

c
struct Curl_cwtype {
  const char *name;
  CURLcode (*do_init)(struct Curl_easy *data,
                      struct Curl_cwriter *writer);
  CURLcode (*do_write)(struct Curl_easy *data,
                       struct Curl_cwriter *writer, int type,
                       const char *buf, size_t nbytes);
  void (*do_close)(struct Curl_easy *data,
                   struct Curl_cwriter *writer);
};

struct Curl_cwriter {
  const struct Curl_cwtype *cwt;  /* type implementation */
  struct Curl_cwriter *next;  /* Downstream writer. */
  Curl_cwriter_phase phase; /* phase at which it operates */
};

Curl_cwriter is a writer instance with a next pointer to form the chain. It has a type cwt which provides the implementation. The main callback is do_write() that processes the data and calls then the next writer. The others are for setup and tear down.

Phases and Ordering

Since client writers may transform the bytes written through them, the order in which the are called is relevant for the outcome. When a writer is created, one property it gets is the phase in which it operates. Writer phases are defined like:

c
typedef enum {
  CURL_CW_RAW,  /* raw data written, before any decoding */
  CURL_CW_TRANSFER_DECODE, /* remove transfer-encodings */
  CURL_CW_PROTOCOL, /* after transfer, but before content decoding */
  CURL_CW_CONTENT_DECODE, /* remove content-encodings */
  CURL_CW_CLIENT  /* data written to client */
} Curl_cwriter_phase;

If a writer for phase PROTOCOL is added to the chain, it is always added after any RAW or TRANSFER_DECODE and before any CONTENT_DECODE and CLIENT phase writer. If there is already a writer for the same phase present, the new writer is inserted before that one.

All transfers have a chain of 3 writers by default. A specific protocol handler may alter that by adding additional writers. The 3 standard writers are (name, phase):

  1. "raw", CURL_CW_RAW : if the transfer is verbose, it forwards the body data to the debug function.
  2. "download", CURL_CW_PROTOCOL: checks that protocol limits are kept and updates progress counters. When a download has a known length, it checks that it is not exceeded and errors otherwise.
  3. "client", CURL_CW_CLIENT: the main work horse. It invokes the application callbacks or writes to the configured file handles. It chops large writes into smaller parts, as documented for CURLOPT_WRITEFUNCTION. If also handles pausing of transfers when the application callback returns CURL_WRITEFUNC_PAUSE.

With these writers always in place, libcurl's protocol handlers automatically have these implemented.

Enhanced Use

HTTP is the protocol in curl that makes use of the client writer chain by adding writers to it. When the libcurl application set CURLOPT_ACCEPT_ENCODING (as curl does with --compressed), the server is offered an Accept-Encoding header with the algorithms supported. The server then may choose to send the response body compressed. For example using gzip or brotli or even both.

In the server's response, if there is a Content-Encoding header listing the encoding applied. If supported by libcurl it then decompresses the content before writing it out to the client. How does it do that?

The HTTP protocol adds client writers in phase CURL_CW_CONTENT_DECODE on seeing such a header. For each encoding listed, it adds the corresponding writer. The response from the server is then passed through Curl_client_write() to the writers that decode it. If several encodings had been applied the writer chain decodes them in the proper order.

When the server provides a Content-Length header, that value applies to the compressed content. Length checks on the response bytes must happen before it gets decoded. That is why this check happens in phase CURL_CW_PROTOCOL which always is ordered before writers in phase CURL_CW_CONTENT_DECODE.

What else?

Well, HTTP servers may also apply a Transfer-Encoding to the body of a response. The most well-known one is chunked, but algorithms like gzip and friends could also be applied. The difference to content encodings is that decoding needs to happen before protocol checks, for example on length, are done.

That is why transfer decoding writers are added for phase CURL_CW_TRANSFER_DECODE. Which makes their operation happen before phase CURL_CW_PROTOCOL where length may be checked.

Summary

By adding the common behavior of all protocols into Curl_client_write() we make sure that they do apply everywhere. Protocol handler have less to worry about. Changes to default behavior can be done without affecting handler implementations.

Having a writer chain as implementation allows protocol handlers with extra needs, like HTTP, to add to this for special behavior. The common way of writing the actual response data stays the same.

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# curl client writers
9Client writers is a design in the internals of libcurl, not visible in its
10public API. They were started in curl 8.5.0. This document describes the
11concepts, its high level implementation and the motivations.
13## Naming
15`libcurl` operates between clients and servers. A *client* is the application
16using libcurl, like the command line tool `curl` itself. Data to be uploaded
17to a server is **read** from the client and **send** to the server, the
18servers response is **received** by `libcurl` and then **written** to the
19client.
21With this naming established, client writers are concerned with writing
22responses from the server to the application. Applications register callbacks
23via `CURLOPT_WRITEFUNCTION` and `CURLOPT_HEADERFUNCTION` to be invoked by
24`libcurl` when the response is received.
26## Invoking
28All code in `libcurl` that handles response data is ultimately expected to
29forward this data via `Curl_client_write()` to the application. The exact
30prototype of this function is:
32```c
33CURLcode Curl_client_write(struct Curl_easy *data, int type, const char *buf, size_t blen);
34```
36The `type` argument specifies what the bytes in `buf` actually are.
37The following bits are defined:
38```c
39#define CLIENTWRITE_BODY (1 << 0) /* non-meta information, BODY */
40#define CLIENTWRITE_INFO (1 << 1) /* meta information, not a HEADER */
41#define CLIENTWRITE_HEADER (1 << 2) /* meta information, HEADER */
42#define CLIENTWRITE_STATUS (1 << 3) /* a special status HEADER */
43#define CLIENTWRITE_CONNECT (1 << 4) /* a CONNECT related HEADER */
44#define CLIENTWRITE_1XX (1 << 5) /* a 1xx response related HEADER */
45#define CLIENTWRITE_TRAILER (1 << 6) /* a trailer HEADER */
46```
48The main types here are `CLIENTWRITE_BODY` and `CLIENTWRITE_HEADER`. They are
49mutually exclusive. The other bits are enhancements to `CLIENTWRITE_HEADER`
50to specify what the header is about. They are only used in HTTP and related
51protocols (RTSP and WebSocket).
53The implementation of `Curl_client_write()` uses a chain of *client writer*
54instances to process the call and make sure that the bytes reach the proper
55application callbacks. This is similar to the design of connection filters:
56client writers can be chained to process the bytes written through them. The
57definition is:
59```c
60struct Curl_cwtype {
61 const char *name;
62 CURLcode (*do_init)(struct Curl_easy *data,
63 struct Curl_cwriter *writer);
64 CURLcode (*do_write)(struct Curl_easy *data,
65 struct Curl_cwriter *writer, int type,
66 const char *buf, size_t nbytes);
67 void (*do_close)(struct Curl_easy *data,
68 struct Curl_cwriter *writer);
69};
71struct Curl_cwriter {
72 const struct Curl_cwtype *cwt; /* type implementation */
73 struct Curl_cwriter *next; /* Downstream writer. */
74 Curl_cwriter_phase phase; /* phase at which it operates */
75};
76```
78`Curl_cwriter` is a writer instance with a `next` pointer to form the chain.
79It has a type `cwt` which provides the implementation. The main callback is
80`do_write()` that processes the data and calls then the `next` writer. The
81others are for setup and tear down.
83## Phases and Ordering
85Since client writers may transform the bytes written through them, the order
86in which the are called is relevant for the outcome. When a writer is created,
87one property it gets is the `phase` in which it operates. Writer phases are
88defined like:
90```c
91typedef enum {
92 CURL_CW_RAW, /* raw data written, before any decoding */
93 CURL_CW_TRANSFER_DECODE, /* remove transfer-encodings */
94 CURL_CW_PROTOCOL, /* after transfer, but before content decoding */
95 CURL_CW_CONTENT_DECODE, /* remove content-encodings */
96 CURL_CW_CLIENT /* data written to client */
97} Curl_cwriter_phase;
98```
100If a writer for phase `PROTOCOL` is added to the chain, it is always added
101*after* any `RAW` or `TRANSFER_DECODE` and *before* any `CONTENT_DECODE` and
102`CLIENT` phase writer. If there is already a writer for the same phase
103present, the new writer is inserted before that one.
105All transfers have a chain of 3 writers by default. A specific protocol
106handler may alter that by adding additional writers. The 3 standard writers
107are (name, phase):
1091. `"raw", CURL_CW_RAW `: if the transfer is verbose, it forwards the body data
110 to the debug function.
1111. `"download", CURL_CW_PROTOCOL`: checks that protocol limits are kept and
112 updates progress counters. When a download has a known length, it checks
113 that it is not exceeded and errors otherwise.
1141. `"client", CURL_CW_CLIENT`: the main work horse. It invokes the application
115 callbacks or writes to the configured file handles. It chops large writes
116 into smaller parts, as documented for `CURLOPT_WRITEFUNCTION`. If also
117 handles *pausing* of transfers when the application callback returns
118 `CURL_WRITEFUNC_PAUSE`.
120With these writers always in place, libcurl's protocol handlers automatically
121have these implemented.
123## Enhanced Use
125HTTP is the protocol in curl that makes use of the client writer chain by
126adding writers to it. When the `libcurl` application set
127`CURLOPT_ACCEPT_ENCODING` (as `curl` does with `--compressed`), the server is
128offered an `Accept-Encoding` header with the algorithms supported. The server
129then may choose to send the response body compressed. For example using `gzip`
130or `brotli` or even both.
132In the server's response, if there is a `Content-Encoding` header listing the
133encoding applied. If supported by `libcurl` it then decompresses the content
134before writing it out to the client. How does it do that?
136The HTTP protocol adds client writers in phase `CURL_CW_CONTENT_DECODE` on
137seeing such a header. For each encoding listed, it adds the corresponding
138writer. The response from the server is then passed through
139`Curl_client_write()` to the writers that decode it. If several encodings had
140been applied the writer chain decodes them in the proper order.
142When the server provides a `Content-Length` header, that value applies to the
143*compressed* content. Length checks on the response bytes must happen *before*
144it gets decoded. That is why this check happens in phase `CURL_CW_PROTOCOL`
145which always is ordered before writers in phase `CURL_CW_CONTENT_DECODE`.
147What else?
149Well, HTTP servers may also apply a `Transfer-Encoding` to the body of a
150response. The most well-known one is `chunked`, but algorithms like `gzip` and
151friends could also be applied. The difference to content encodings is that
152decoding needs to happen *before* protocol checks, for example on length, are
153done.
155That is why transfer decoding writers are added for phase
156`CURL_CW_TRANSFER_DECODE`. Which makes their operation happen *before* phase
157`CURL_CW_PROTOCOL` where length may be checked.
159## Summary
161By adding the common behavior of all protocols into `Curl_client_write()` we
162make sure that they do apply everywhere. Protocol handler have less to worry
163about. Changes to default behavior can be done without affecting handler
164implementations.
166Having a writer chain as implementation allows protocol handlers with extra
167needs, like HTTP, to add to this for special behavior. The common way of
168writing the actual response data stays the same.
Diagnostics4 git calls · 26 ms
CallStatusSizeTime026 ms
discover200152 B2 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit2 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=2
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=2
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=21
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2003.0 KiB · hit21 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=3088
git.response.cache=HIT
git.response.duration_ms=21
git.fetch.wants=4e28d188ab0e26f3cf8f378028c6e34cfe8ee32a
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=3088
git.pkt.payload.max=3067
git.sideband.band1.bytes=3066
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e3441d4dc86e2a
Request
e89d7a40…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format