Artifactspublic-v0

curl

master
ply4 git calls29 ms290 KiB3 cachedprotocol v2
184 lines6.5 KiBRaw

bufq

This is an internal module for managing I/O buffers. A bufq can be written to and read from. It manages read and write positions and has a maximum size.

read/write

Its basic read/write functions have a similar signature and return code handling as many internal curl read and write ones.

c
CURLcode Curl_bufq_write(struct bufq *q,
                         const uint8_t *buf, size_t len,
                         size_t *pnwritten);
  • sets pnwritten to the length written into q or -1 on error.
  • writing to a full q sets pnwritten to -1 and returns CURLE_AGAIN
CURLcode Curl_bufq_read(struct bufq *q, uint8_t *buf, size_t len,
                        size_t *pnread);
  • sets pnread to the length read from q or -1 on error.
  • reading from an empty q sets pnread to -1 and returns CURLE_AGAIN

To pass data into a bufq without an extra copy, read callbacks can be used.

c
typedef CURLcode Curl_bufq_reader(void *reader_ctx,
                                  uint8_t *buf, size_t len,
                                  size_t *pnread);

CURLcode Curl_bufq_slurp(struct bufq *q, Curl_bufq_reader *reader,
                         void *reader_ctx, size_t *pnread);

Curl_bufq_slurp() invokes the given reader callback, passing it its own internal buffer memory to write to. It may invoke the reader several times, as long as it has space and while the reader always returns the length that was requested. There are variations of slurp that call the reader at most once or only read in a maximum amount of bytes.

The analog mechanism for write out buffer data is:

c
typedef CURLcode Curl_bufq_writer(void *writer_ctx,
                                  const uint8_t *buf, size_t len,
                                  size_t *pwritten);

CURLcode Curl_bufq_pass(struct bufq *q, Curl_bufq_writer *writer,
                        void *writer_ctx, size_t *pwritten);

Curl_bufq_pass() invokes the writer, passing its internal memory and remove the amount that writer reports.

peek and skip

It is possible to get access to the memory of data stored in a bufq with:

c
bool Curl_bufq_peek(struct bufq *q,
                    const uint8_t **pbuf, size_t *plen);

On returning TRUE, pbuf points to internal memory with plen bytes that one may read. This is only valid until another operation on bufq is performed.

Instead of reading bufq data, one may skip it:

c
void Curl_bufq_skip(struct bufq *q, size_t amount);

This removes amount number of bytes from the bufq.

lifetime

bufq is initialized and freed similar to the dynbuf module. Code using bufq holds a struct bufq somewhere. Before it uses it, it invokes:

c
void Curl_bufq_init(struct bufq *q, size_t chunk_size, size_t max_chunks);

The bufq is told how many "chunks" of data it shall hold at maximum and how large those "chunks" should be. There are some variants of this, allowing for more options. How "chunks" are handled in a bufq is presented in the section about memory management.

The user of the bufq has the responsibility to call:

c
void Curl_bufq_free(struct bufq *q);

to free all resources held by q. It is possible to reset a bufq to empty via:

c
void Curl_bufq_reset(struct bufq *q);

memory management

Internally, a bufq uses allocation of fixed size, e.g. the "chunk_size", up to a maximum number, e.g. "max_chunks". These chunks are allocated on demand, therefore writing to a bufq may return CURLE_OUT_OF_MEMORY. Once the max number of chunks are used, the bufq reports that it is "full".

Each chunks has a read and write index. A bufq keeps its chunks in a list. Reading happens always at the head chunk, writing always goes to the tail chunk. When the head chunk becomes empty, it is removed. When the tail chunk becomes full, another chunk is added to the end of the list, becoming the new tail.

Chunks that are no longer used are returned to a spare list by default. If the bufq is created with option BUFQ_OPT_NO_SPARES those chunks are freed right away.

If a bufq is created with a bufc_pool, the no longer used chunks are returned to the pool. Also bufq asks the pool for a chunk when it needs one. More in section "pools".

empty, full and overflow

One can ask about the state of a bufq with methods such as Curl_bufq_is_empty(q), Curl_bufq_is_full(q), etc. The amount of data held by a bufq is the sum of the data in all its chunks. This is what is reported by Curl_bufq_len(q).

Note that a bufq length and it being "full" are only loosely related. A simple example:

  • create a bufq with chunk_size=1000 and max_chunks=4.
  • write 4000 bytes to it, it reports "full"
  • read 1 bytes from it, it still reports "full"
  • read 999 more bytes from it, and it is no longer "full"

The reason for this is that full really means: bufq uses max_chunks and the last one cannot be written to.

When you read 1 byte from the head chunk in the example above, the head still hold 999 unread bytes. Only when those are also read, can the head chunk be removed and a new tail be added.

There is another variation to this. If you initialized a bufq with option BUFQ_OPT_SOFT_LIMIT, it allows writes beyond the max_chunks. It reports full, but one can still write. This option is necessary, if partial writes need to be avoided. It means that you need other checks to keep the bufq from growing ever larger and larger.

pools

A struct bufc_pool may be used to create chunks for a bufq and keep spare ones around. It is initialized and used via:

c
void Curl_bufcp_init(struct bufc_pool *pool,
                     size_t chunk_size, size_t spare_max);

void Curl_bufq_initp(struct bufq *q, struct bufc_pool *pool,
                     size_t max_chunks, int opts);

The pool gets the size and the mount of spares to keep. The bufq gets the pool and the max_chunks. It no longer needs to know the chunk sizes, as those are managed by the pool.

A pool can be shared between many bufqs, as long as all of them operate in the same thread. In curl that would be true for all transfers using the same multi handle. The advantages of a pool are:

  • when all bufqs are empty, only memory for max_spare chunks in the pool is used. Empty bufqs holds no memory.
  • the latest spare chunk is the first to be handed out again, no matter which bufq needs it. This keeps the footprint of "recently used" memory smaller.
1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# bufq
9This is an internal module for managing I/O buffers. A `bufq` can be written
10to and read from. It manages read and write positions and has a maximum size.
12## read/write
14Its basic read/write functions have a similar signature and return code
15handling as many internal curl read and write ones.
17```c
18CURLcode Curl_bufq_write(struct bufq *q,
19 const uint8_t *buf, size_t len,
20 size_t *pnwritten);
21```
23- sets `pnwritten` to the length written into `q` or -1 on error.
24- writing to a full `q` sets `pnwritten` to -1 and returns CURLE_AGAIN
26```
27CURLcode Curl_bufq_read(struct bufq *q, uint8_t *buf, size_t len,
28 size_t *pnread);
29```
31- sets `pnread` to the length read from `q` or -1 on error.
32- reading from an empty `q` sets `pnread` to -1 and returns CURLE_AGAIN
34To pass data into a `bufq` without an extra copy, read callbacks can be used.
36```c
37typedef CURLcode Curl_bufq_reader(void *reader_ctx,
38 uint8_t *buf, size_t len,
39 size_t *pnread);
41CURLcode Curl_bufq_slurp(struct bufq *q, Curl_bufq_reader *reader,
42 void *reader_ctx, size_t *pnread);
43```
45`Curl_bufq_slurp()` invokes the given `reader` callback, passing it its own
46internal buffer memory to write to. It may invoke the `reader` several times,
47as long as it has space and while the `reader` always returns the length that
48was requested. There are variations of `slurp` that call the `reader` at most
49once or only read in a maximum amount of bytes.
51The analog mechanism for write out buffer data is:
53```c
54typedef CURLcode Curl_bufq_writer(void *writer_ctx,
55 const uint8_t *buf, size_t len,
56 size_t *pwritten);
58CURLcode Curl_bufq_pass(struct bufq *q, Curl_bufq_writer *writer,
59 void *writer_ctx, size_t *pwritten);
60```
62`Curl_bufq_pass()` invokes the `writer`, passing its internal memory and
63remove the amount that `writer` reports.
65## peek and skip
67It is possible to get access to the memory of data stored in a `bufq` with:
69```c
70bool Curl_bufq_peek(struct bufq *q,
71 const uint8_t **pbuf, size_t *plen);
72```
74On returning TRUE, `pbuf` points to internal memory with `plen` bytes that one
75may read. This is only valid until another operation on `bufq` is performed.
77Instead of reading `bufq` data, one may skip it:
79```c
80void Curl_bufq_skip(struct bufq *q, size_t amount);
81```
83This removes `amount` number of bytes from the `bufq`.
85## lifetime
87`bufq` is initialized and freed similar to the `dynbuf` module. Code using
88`bufq` holds a `struct bufq` somewhere. Before it uses it, it invokes:
90```c
91void Curl_bufq_init(struct bufq *q, size_t chunk_size, size_t max_chunks);
92```
94The `bufq` is told how many "chunks" of data it shall hold at maximum and how
95large those "chunks" should be. There are some variants of this, allowing for
96more options. How "chunks" are handled in a `bufq` is presented in the section
97about memory management.
99The user of the `bufq` has the responsibility to call:
101```c
102void Curl_bufq_free(struct bufq *q);
103```
105to free all resources held by `q`. It is possible to reset a `bufq` to empty via:
107```c
108void Curl_bufq_reset(struct bufq *q);
109```
111## memory management
113Internally, a `bufq` uses allocation of fixed size, e.g. the "chunk_size", up
114to a maximum number, e.g. "max_chunks". These chunks are allocated on demand,
115therefore writing to a `bufq` may return `CURLE_OUT_OF_MEMORY`. Once the max
116number of chunks are used, the `bufq` reports that it is "full".
118Each chunks has a `read` and `write` index. A `bufq` keeps its chunks in a
119list. Reading happens always at the head chunk, writing always goes to the
120tail chunk. When the head chunk becomes empty, it is removed. When the tail
121chunk becomes full, another chunk is added to the end of the list, becoming
122the new tail.
124Chunks that are no longer used are returned to a `spare` list by default. If
125the `bufq` is created with option `BUFQ_OPT_NO_SPARES` those chunks are freed
126right away.
128If a `bufq` is created with a `bufc_pool`, the no longer used chunks are
129returned to the pool. Also `bufq` asks the pool for a chunk when it needs one.
130More in section "pools".
132## empty, full and overflow
134One can ask about the state of a `bufq` with methods such as
135`Curl_bufq_is_empty(q)`, `Curl_bufq_is_full(q)`, etc. The amount of data held
136by a `bufq` is the sum of the data in all its chunks. This is what is reported
137by `Curl_bufq_len(q)`.
139Note that a `bufq` length and it being "full" are only loosely related. A
140simple example:
142* create a `bufq` with chunk_size=1000 and max_chunks=4.
143* write 4000 bytes to it, it reports "full"
144* read 1 bytes from it, it still reports "full"
145* read 999 more bytes from it, and it is no longer "full"
147The reason for this is that full really means: *bufq uses max_chunks and the
148last one cannot be written to*.
150When you read 1 byte from the head chunk in the example above, the head still
151hold 999 unread bytes. Only when those are also read, can the head chunk be
152removed and a new tail be added.
154There is another variation to this. If you initialized a `bufq` with option
155`BUFQ_OPT_SOFT_LIMIT`, it allows writes **beyond** the `max_chunks`. It
156reports **full**, but one can **still** write. This option is necessary, if
157partial writes need to be avoided. It means that you need other checks to keep
158the `bufq` from growing ever larger and larger.
160## pools
162A `struct bufc_pool` may be used to create chunks for a `bufq` and keep spare
163ones around. It is initialized and used via:
165```c
166void Curl_bufcp_init(struct bufc_pool *pool,
167 size_t chunk_size, size_t spare_max);
169void Curl_bufq_initp(struct bufq *q, struct bufc_pool *pool,
170 size_t max_chunks, int opts);
171```
173The pool gets the size and the mount of spares to keep. The `bufq` gets the
174pool and the `max_chunks`. It no longer needs to know the chunk sizes, as
175those are managed by the pool.
177A pool can be shared between many `bufq`s, as long as all of them operate in
178the same thread. In curl that would be true for all transfers using the same
179multi handle. The advantages of a pool are:
181* when all `bufq`s are empty, only memory for `max_spare` chunks in the pool
182 is used. Empty `bufq`s holds no memory.
183* the latest spare chunk is the first to be handed out again, no matter which
184 `bufq` needs it. This keeps the footprint of "recently used" memory smaller.
Diagnostics4 git calls · 29 ms
CallStatusSizeTime029 ms
discover200152 B2 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit2 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=2
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=2
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit2 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=25
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=2
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2002.6 KiB · hit25 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=2700
git.response.cache=HIT
git.response.duration_ms=25
git.fetch.wants=a977fcc8bdc1d2031ac3d60440fc48ca46a40ad2
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=2700
git.pkt.payload.max=2679
git.sideband.band1.bytes=2678
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e2f18c5a24fda9
Request
19b1f0b4…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format