Artifactspublic-v0

curl

master
ply4 git calls28 ms291 KiB3 cachedprotocol v2
206 lines9.1 KiBRaw

curl client readers

Client readers is a design in the internals of libcurl, not visible in its public API. They were started in curl 8.7.0. This document describes the concepts, its high level implementation and the motivations.

Naming

libcurl operates between clients and servers. A client is the application using libcurl, like the command line tool curl itself. Data to be uploaded to a server is read from the client and sent to the server, the servers response is received by libcurl and then written to the client.

With this naming established, client readers are concerned with providing data from the application to the server. Applications register callbacks via CURLOPT_READFUNCTION, data via CURLOPT_POSTFIELDS and other options to be used by libcurl when the request is send.

Invoking

The transfer loop that sends and receives, is using Curl_client_read() to get more data to send for a transfer. If no specific reader has been installed yet, the default one that uses CURLOPT_READFUNCTION is added. The prototype is

c
CURLcode Curl_client_read(struct Curl_easy *data, char *buf, size_t blen,
                          size_t *nread, bool *eos);

The arguments are the transfer to read for, a buffer to hold the read data, its length, the actual number of bytes placed into the buffer and the eos (end of stream) flag indicating that no more data is available. The eos flag may be set for a read amount, if that amount was the last. That way curl can avoid to read an additional time.

The implementation of Curl_client_read() uses a chain of client reader instances to get the data. This is similar to the design of client writers. The chain of readers allows processing of the data to send.

The definition of a reader is:

c
struct Curl_crtype {
  const char *name;        /* writer name. */
  CURLcode (*do_init)(struct Curl_easy *data, struct Curl_creader *writer);
  CURLcode (*do_read)(struct Curl_easy *data, struct Curl_creader *reader,
                      char *buf, size_t blen, size_t *nread, bool *eos);
  void (*do_close)(struct Curl_easy *data, struct Curl_creader *reader);
  bool (*needs_rewind)(struct Curl_easy *data, struct Curl_creader *reader);
  curl_off_t (*total_length)(struct Curl_easy *data,
                             struct Curl_creader *reader);
  CURLcode (*resume_from)(struct Curl_easy *data,
                          struct Curl_creader *reader, curl_off_t offset);
  CURLcode (*rewind)(struct Curl_easy *data, struct Curl_creader *reader);
};

struct Curl_creader {
  const struct Curl_crtype *crt;  /* type implementation */
  struct Curl_creader *next;  /* Downstream reader. */
  Curl_creader_phase phase; /* phase at which it operates */
};

Curl_creader is a reader instance with a next pointer to form the chain. It as a type crt which provides the implementation. The main callback is do_read() which provides the data to the caller. The others are for setup and tear down. needs_rewind() is explained further below.

Phases and Ordering

Since client readers may transform the data being read through the chain, the order in which they are called is relevant for the outcome. When a reader is created, it gets the phase property in which it operates. Reader phases are defined like:

c
typedef enum {
  CURL_CR_NET,  /* data send to the network (connection filters) */
  CURL_CR_TRANSFER_ENCODE, /* add transfer-encodings */
  CURL_CR_PROTOCOL, /* before transfer, but after content decoding */
  CURL_CR_CONTENT_ENCODE, /* add content-encodings */
  CURL_CR_CLIENT  /* data read from client */
} Curl_creader_phase;

If a reader for phase PROTOCOL is added to the chain, it is always added after any NET or TRANSFER_ENCODE readers and before and CONTENT_ENCODE and CLIENT readers. If there is already a reader for the same phase, the new reader is added before the existing one(s).

Example: chunked reader

In http_chunks.c a client reader for chunked uploads is implemented. This one operates at phase CURL_CR_TRANSFER_ENCODE. Any data coming from the reader "below" has the HTTP/1.1 chunk handling applied and returned to the caller.

When this reader sees an eos from below, it generates the terminal chunk, adding trailers if provided by the application. When that last chunk is fully returned, it also sets eos to the caller.

Example: lineconv reader

In sendf.c a client reader that does line-end conversions is implemented. It operates at CURL_CR_CONTENT_ENCODE and converts any "\n" to "\r\n". This is used for FTP ASCII uploads or when the general crlf options has been set.

Example: null reader

Implemented in sendf.c for phase CURL_CR_CLIENT, this reader has the simple job of providing transfer bytes of length 0 to the caller, immediately indicating an eos. This reader is installed by HTTP for all GET/HEAD requests and when authentication is being negotiated.

Example: buf reader

Implemented in sendf.c for phase CURL_CR_CLIENT, this reader get a buffer pointer and a length and provides exactly these bytes. This one is used in HTTP for sending postfields provided by the application.

Request retries

Sometimes it is necessary to send a request with client data again. Transfer handling can inquire via Curl_client_read_needs_rewind() if a rewind (e.g. a reset of the client data) is necessary. This asks all installed readers if they need it and give FALSE of none does.

Upload Size

Many protocols need to know the amount of bytes delivered by the client readers in advance. They may invoke Curl_creader_total_length(data) to retrieve that. Not all reader chains know the exact value beforehand. In that case, the call returns -1 for "unknown".

Even if the length of the "raw" data is known, the length that is send may not. Example: with option --crlf the uploaded content undergoes line-end conversion. The line converting reader does not know in advance how many newlines it may encounter. Therefore it must return -1 for any positive raw content length.

In HTTP, once the correct client readers are installed, the protocol asks the readers for the total length. If that is known, it can set Content-Length: accordingly. If not, it may choose to add an HTTP "chunked" reader.

In addition, there is Curl_creader_client_length(data) which gives the total length as reported by the reader in phase CURL_CR_CLIENT without asking other readers that may transform the raw data. This is useful in estimating the size of an upload. The HTTP protocol uses this to determine if Expect: 100-continue shall be done.

Resuming

Uploads can start at a specific offset, if so requested. The "resume from" that offset. This applies to the reader in phase CURL_CR_CLIENT that delivers the "raw" content. Resumption can fail if the installed reader does not support it or if the offset is too large.

The total length reported by the reader changes when resuming. Example: resuming an upload of 100 bytes by 25 reports a total length of 75 afterwards.

If resume_from() is invoked twice, it is additive. There is currently no way to undo a resume.

Rewinding

When a request is retried, installed client readers are discarded and replaced by new ones. This works only if the new readers upload the same data. For many readers, this is not an issue. The "null" reader always does the same. Also the buf reader, initialized with the same buffer, does this.

Readers operating on callbacks to the application need to "rewind" the underlying content. For example, when reading from a FILE*, the reader needs to fseek() to the beginning. The following methods are used:

  1. Curl_creader_needs_rewind(data): tells if a rewind is necessary, given the current state of the reader chain. If nothing really has been read so far, this returns FALSE.
  2. Curl_creader_will_rewind(data): tells if the reader chain rewinds at the start of the next request.
  3. Curl_creader_set_rewind(data, TRUE): marks the reader chain for rewinding at the start of the next request.
  4. Curl_client_start(data): tells the readers that a new request starts and they need to rewind if requested.

Summary and Outlook

By adding the client reader interface, any protocol can control how/if it wants the curl transfer to send bytes for a request. The transfer loop becomes then blissfully ignorant of the specifics.

The protocols on the other hand no longer have to care to package data most efficiently. At any time, should more data be needed, it can be read from the client. This is used when sending HTTP requests headers to add as much request body data to the initial sending as there is room for.

Future enhancements based on the client readers:

  • expect-100 handling: place that into an HTTP specific reader at CURL_CR_PROTOCOL and eliminate the checks in the generic transfer parts.
  • eos forwarding: transfer should forward an eos flag to the connection filters. Filters like HTTP/2 and HTTP/3 can make use of that, terminating streams early. This would also eliminate length checks in stream handling.
1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# curl client readers
9Client readers is a design in the internals of libcurl, not visible in its
10public API. They were started in curl 8.7.0. This document describes
11the concepts, its high level implementation and the motivations.
13## Naming
15`libcurl` operates between clients and servers. A *client* is the application
16using libcurl, like the command line tool `curl` itself. Data to be uploaded
17to a server is **read** from the client and **sent** to the server, the servers
18response is **received** by `libcurl` and then **written** to the client.
20With this naming established, client readers are concerned with providing data
21from the application to the server. Applications register callbacks via
22`CURLOPT_READFUNCTION`, data via `CURLOPT_POSTFIELDS` and other options to be
23used by `libcurl` when the request is send.
25## Invoking
27The transfer loop that sends and receives, is using `Curl_client_read()` to get
28more data to send for a transfer. If no specific reader has been installed yet,
29the default one that uses `CURLOPT_READFUNCTION` is added. The prototype is
31```c
32CURLcode Curl_client_read(struct Curl_easy *data, char *buf, size_t blen,
33 size_t *nread, bool *eos);
34```
35The arguments are the transfer to read for, a buffer to hold the read data, its
36length, the actual number of bytes placed into the buffer and the `eos` (*end of
37stream*) flag indicating that no more data is available. The `eos` flag may be
38set for a read amount, if that amount was the last. That way curl can avoid to
39read an additional time.
41The implementation of `Curl_client_read()` uses a chain of *client reader*
42instances to get the data. This is similar to the design of *client writers*.
43The chain of readers allows processing of the data to send.
45The definition of a reader is:
47```c
48struct Curl_crtype {
49 const char *name; /* writer name. */
50 CURLcode (*do_init)(struct Curl_easy *data, struct Curl_creader *writer);
51 CURLcode (*do_read)(struct Curl_easy *data, struct Curl_creader *reader,
52 char *buf, size_t blen, size_t *nread, bool *eos);
53 void (*do_close)(struct Curl_easy *data, struct Curl_creader *reader);
54 bool (*needs_rewind)(struct Curl_easy *data, struct Curl_creader *reader);
55 curl_off_t (*total_length)(struct Curl_easy *data,
56 struct Curl_creader *reader);
57 CURLcode (*resume_from)(struct Curl_easy *data,
58 struct Curl_creader *reader, curl_off_t offset);
59 CURLcode (*rewind)(struct Curl_easy *data, struct Curl_creader *reader);
60};
62struct Curl_creader {
63 const struct Curl_crtype *crt; /* type implementation */
64 struct Curl_creader *next; /* Downstream reader. */
65 Curl_creader_phase phase; /* phase at which it operates */
66};
67```
69`Curl_creader` is a reader instance with a `next` pointer to form the chain.
70It as a type `crt` which provides the implementation. The main callback is
71`do_read()` which provides the data to the caller. The others are for setup
72and tear down. `needs_rewind()` is explained further below.
74## Phases and Ordering
76Since client readers may transform the data being read through the chain,
77the order in which they are called is relevant for the outcome. When a reader
78is created, it gets the `phase` property in which it operates. Reader phases
79are defined like:
81```c
82typedef enum {
83 CURL_CR_NET, /* data send to the network (connection filters) */
84 CURL_CR_TRANSFER_ENCODE, /* add transfer-encodings */
85 CURL_CR_PROTOCOL, /* before transfer, but after content decoding */
86 CURL_CR_CONTENT_ENCODE, /* add content-encodings */
87 CURL_CR_CLIENT /* data read from client */
88} Curl_creader_phase;
89```
91If a reader for phase `PROTOCOL` is added to the chain, it is always added
92*after* any `NET` or `TRANSFER_ENCODE` readers and *before* and
93`CONTENT_ENCODE` and `CLIENT` readers. If there is already a reader for the
94same phase, the new reader is added before the existing one(s).
96### Example: `chunked` reader
98In `http_chunks.c` a client reader for chunked uploads is implemented. This
99one operates at phase `CURL_CR_TRANSFER_ENCODE`. Any data coming from the
100reader "below" has the HTTP/1.1 chunk handling applied and returned to the
101caller.
103When this reader sees an `eos` from below, it generates the terminal chunk,
104adding trailers if provided by the application. When that last chunk is fully
105returned, it also sets `eos` to the caller.
107### Example: `lineconv` reader
109In `sendf.c` a client reader that does line-end conversions is implemented. It
110operates at `CURL_CR_CONTENT_ENCODE` and converts any "\n" to "\r\n". This is
111used for FTP ASCII uploads or when the general `crlf` options has been set.
113### Example: `null` reader
115Implemented in `sendf.c` for phase `CURL_CR_CLIENT`, this reader has the
116simple job of providing transfer bytes of length 0 to the caller, immediately
117indicating an `eos`. This reader is installed by HTTP for all GET/HEAD
118requests and when authentication is being negotiated.
120### Example: `buf` reader
122Implemented in `sendf.c` for phase `CURL_CR_CLIENT`, this reader get a buffer
123pointer and a length and provides exactly these bytes. This one is used in
124HTTP for sending `postfields` provided by the application.
126## Request retries
128Sometimes it is necessary to send a request with client data again. Transfer
129handling can inquire via `Curl_client_read_needs_rewind()` if a rewind (e.g. a
130reset of the client data) is necessary. This asks all installed readers if
131they need it and give `FALSE` of none does.
133## Upload Size
135Many protocols need to know the amount of bytes delivered by the client
136readers in advance. They may invoke `Curl_creader_total_length(data)` to
137retrieve that. Not all reader chains know the exact value beforehand. In that
138case, the call returns `-1` for "unknown".
140Even if the length of the "raw" data is known, the length that is send may
141not. Example: with option `--crlf` the uploaded content undergoes line-end
142conversion. The line converting reader does not know in advance how many
143newlines it may encounter. Therefore it must return `-1` for any positive raw
144content length.
146In HTTP, once the correct client readers are installed, the protocol asks the
147readers for the total length. If that is known, it can set `Content-Length:`
148accordingly. If not, it may choose to add an HTTP "chunked" reader.
150In addition, there is `Curl_creader_client_length(data)` which gives the total
151length as reported by the reader in phase `CURL_CR_CLIENT` without asking
152other readers that may transform the raw data. This is useful in estimating
153the size of an upload. The HTTP protocol uses this to determine if `Expect:
154100-continue` shall be done.
156## Resuming
158Uploads can start at a specific offset, if so requested. The "resume from"
159that offset. This applies to the reader in phase `CURL_CR_CLIENT` that
160delivers the "raw" content. Resumption can fail if the installed reader does
161not support it or if the offset is too large.
163The total length reported by the reader changes when resuming. Example:
164resuming an upload of 100 bytes by 25 reports a total length of 75 afterwards.
166If `resume_from()` is invoked twice, it is additive. There is currently no way
167to undo a resume.
169## Rewinding
171When a request is retried, installed client readers are discarded and replaced
172by new ones. This works only if the new readers upload the same data. For many
173readers, this is not an issue. The "null" reader always does the same. Also
174the `buf` reader, initialized with the same buffer, does this.
176Readers operating on callbacks to the application need to "rewind" the
177underlying content. For example, when reading from a `FILE*`, the reader needs
178to `fseek()` to the beginning. The following methods are used:
1801. `Curl_creader_needs_rewind(data)`: tells if a rewind is necessary, given
181 the current state of the reader chain. If nothing really has been read so
182 far, this returns `FALSE`.
1832. `Curl_creader_will_rewind(data)`: tells if the reader chain rewinds at
184 the start of the next request.
1853. `Curl_creader_set_rewind(data, TRUE)`: marks the reader chain for rewinding
186 at the start of the next request.
1874. `Curl_client_start(data)`: tells the readers that a new request starts and
188 they need to rewind if requested.
190## Summary and Outlook
192By adding the client reader interface, any protocol can control how/if it wants
193the curl transfer to send bytes for a request. The transfer loop becomes then
194blissfully ignorant of the specifics.
196The protocols on the other hand no longer have to care to package data most
197efficiently. At any time, should more data be needed, it can be read from the
198client. This is used when sending HTTP requests headers to add as much request
199body data to the initial sending as there is room for.
201Future enhancements based on the client readers:
202* `expect-100` handling: place that into an HTTP specific reader at
203 `CURL_CR_PROTOCOL` and eliminate the checks in the generic transfer parts.
204* `eos forwarding`: transfer should forward an `eos` flag to the connection
205 filters. Filters like HTTP/2 and HTTP/3 can make use of that, terminating
206 streams early. This would also eliminate length checks in stream handling.
Diagnostics4 git calls · 28 ms
CallStatusSizeTime028 ms
discover200152 B3 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit3 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=3
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit2 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=22
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=2
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2003.8 KiB · hit23 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=3896
git.response.cache=HIT
git.response.duration_ms=23
git.fetch.wants=8b1040698822d57e211728ca8f759d3f7664263f
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=3896
git.pkt.payload.max=3875
git.sideband.band1.bytes=3874
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e32dab0ccd8821
Request
1740c640…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format