Artifactspublic-v0

curl

master
ply4 git calls25 ms289 KiB2 cachedprotocol v2
137 lines5.4 KiBRaw

uint32_t Sets

The multi handle tracks added easy handles via an uint32_t it calls an mid. There are four data structures for uint32_t optimized for the multi use case.

uint32_tbl

uint32_table, implemented in uint-table.[ch] manages an array of void *. The uint32_t is the index into this array. It is created with a capacity which can be resized. The table assigns the index when a void * is added. It keeps track of the last assigned index and uses the next available larger index for a subsequent add. Reaching capacity it wraps around.

The table can not store NULL values. The largest possible index is UINT32_MAX - 1.

The table is iterated over by asking for the first existing index, meaning the smallest number that has an entry, if the table is not empty. To get the next entry, one passes the index of the previous iteration step. It does not matter if the previous index is still in the table. Sample code for a table iteration would look like this:

c
uint32_t int mid;
void *entry;

if(Curl_uint32_tbl_first(tbl, &mid, &entry)) {
  do {
     /* operate on entry with index mid */
  }
  while(Curl_uint_tbl_next(tbl, mid, &mid, &entry));
}

This iteration has the following properties:

  • entries in the table can be added/removed safely.
  • all entries that are not removed during the iteration are visited.
  • the table may be resized to a larger capacity without affecting visited entries.
  • entries added with a larger index than the current are visited.

Memory

For storing 1000 entries, the table would allocate one block of 8KB on a 64-bit system, plus the pointer and 3 uint32_t in its base struct uint32_tbl. A resize allocates a completely new pointer array, copy the existing entries and free the previous one.

Performance

Lookups of entries are only an index into the array, O(1) with a tiny 1. Adding entries and iterations are more work:

  1. adding an entry means "find the first free index larger than the previous assigned one". Worst case for this is a table with only a single free index where capacity - 1 checks on NULL values would be performed, O(N). If the single free index is randomly distributed, this would be O(N/2).
  2. iterating a table scans for the first not NULL entry after the start index. This makes a complete iteration O(N) work.

In the multi use case, point 1 is remedied by growing the table so that a good chunk of free entries always exists.

Point 2 is less of an issue for a multi, since it does not really matter when the number of transfer is relatively small. A multi managing a larger set needs to operate event based anyway and table iterations rarely are needed.

For these reasons, the simple implementation was preferred. Should this become a concern, there are options like "free index lists" or, alternatively, an internal bitset that scans better.

uint32_bset

A bitset for uint32_t values, allowing fast add/remove operations. It is initialized with a capacity, meaning it can store only the numbers in the range [0, capacity-1]. It can be resized and safely iterated. uint32_bset is designed to operate in combination with uint_tbl.

The bitset keeps an array of uint64_t. The first array entry keeps the numbers 0 to 63, the second 64 to 127 and so on. A bitset with capacity 1024 would therefore allocate an array of 16 64-bit values (128 bytes). Operations for an unsigned int divide it by 64 for the array index and then check/set/clear the bit of the remainder.

Iterator works the same as with uint32_tbl: ask the bitset for the first number present and then use that to get the next higher number present. Like the table, this safe for adds/removes and growing the set while iterating.

Memory

The set only needs 1 bit for each possible number. A bitset for 40000 transfers occupies 5KB of memory.

Performance

Operations for add/remove/check are O(1). Iteration needs to scan for the next bit set. The number of scans is small (see memory footprint) and, for checking bits, many compilers offer primitives for special CPU instructions.

uint32_spbset

While the memory footprint of uint32_bset is good, it still needs 5KB to store the single number 40000. This is not optimal when many are needed. For example, in event based processing, each socket needs to keep track of the transfers involved. There are many sockets potentially, but each one mostly tracks a single transfer or few (on HTTP/2 connection borderline up to 100).

For such uses cases, the uint32_spbset is intended: track a small number of unsigned int, potentially rather "close" together. It keeps "chunks" with an offset and has no capacity limit.

Example: adding the number 40000 to an empty sparse bitset would have one chunk with offset 39936, keeping track of the numbers 39936 to 40192 (a chunk has 4 64-bit values). The numbers in that range can be handled without further allocations.

The worst case is then storing 100 numbers that lie in separate intervals. Then 100 chunks would need to be allocated and linked, resulting in overall 4 KB of memory used.

Iterating a sparse bitset works the same as for bitset and table.

uint32_hash

At last, there are places in libcurl such as the HTTP/2 and HTTP/3 protocol implementations that need to store their own data related to a transfer. uint32_hash allows then to associate an unsigned int, e.g. the transfer's mid, to their own data.

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# `uint32_t` Sets
9The multi handle tracks added easy handles via an `uint32_t` it calls an
10`mid`. There are four data structures for `uint32_t` optimized for the multi
11use case.
13## `uint32_tbl`
15`uint32_table`, implemented in `uint-table.[ch]` manages an array of `void *`.
16The `uint32_t` is the index into this array. It is created with a *capacity*
17which can be *resized*. The table assigns the index when a `void *` is
18*added*. It keeps track of the last assigned index and uses the next available
19larger index for a subsequent add. Reaching *capacity* it wraps around.
21The table *can not* store `NULL` values. The largest possible index
22is `UINT32_MAX - 1`.
24The table is iterated over by asking for the *first* existing index, meaning
25the smallest number that has an entry, if the table is not empty. To get
26the *next* entry, one passes the index of the previous iteration step. It does
27not matter if the previous index is still in the table. Sample code for a table
28iteration would look like this:
30```c
31uint32_t int mid;
32void *entry;
34if(Curl_uint32_tbl_first(tbl, &mid, &entry)) {
35 do {
36 /* operate on entry with index mid */
37 }
38 while(Curl_uint_tbl_next(tbl, mid, &mid, &entry));
41```
43This iteration has the following properties:
45* entries in the table can be added/removed safely.
46* all entries that are not removed during the iteration are visited.
47* the table may be resized to a larger capacity without affecting visited
48 entries.
49* entries added with a larger index than the current are visited.
51### Memory
53For storing 1000 entries, the table would allocate one block of 8KB on
54a 64-bit system, plus the pointer and 3 `uint32_t` in its base `struct
55uint32_tbl`. A resize allocates a completely new pointer array, copy
56the existing entries and free the previous one.
58### Performance
60Lookups of entries are only an index into the array, O(1) with a tiny 1.
61Adding entries and iterations are more work:
631. adding an entry means "find the first free index larger than the previous
64 assigned one". Worst case for this is a table with only a single free index
65 where `capacity - 1` checks on `NULL` values would be performed, O(N). If
66 the single free index is randomly distributed, this would be O(N/2).
672. iterating a table scans for the first not `NULL` entry after the start
68 index. This makes a complete iteration O(N) work.
70In the multi use case, point 1 is remedied by growing the table so that a good
71chunk of free entries always exists.
73Point 2 is less of an issue for a multi, since it does not really matter when
74the number of transfer is relatively small. A multi managing a larger set
75needs to operate event based anyway and table iterations rarely are needed.
77For these reasons, the simple implementation was preferred. Should this become
78a concern, there are options like "free index lists" or, alternatively, an
79internal bitset that scans better.
81## `uint32_bset`
83A bitset for `uint32_t` values, allowing fast add/remove operations. It is
84initialized with a *capacity*, meaning it can store only the numbers in the
85range `[0, capacity-1]`. It can be *resized* and safely *iterated*.
86`uint32_bset` is designed to operate in combination with `uint_tbl`.
88The bitset keeps an array of `uint64_t`. The first array entry keeps the
89numbers 0 to 63, the second 64 to 127 and so on. A bitset with capacity 1024
90would therefore allocate an array of 16 64-bit values (128 bytes). Operations
91for an unsigned int divide it by 64 for the array index and then
92check/set/clear the bit of the remainder.
94Iterator works the same as with `uint32_tbl`: ask the bitset for the *first*
95number present and then use that to get the *next* higher number present. Like
96the table, this safe for adds/removes and growing the set while iterating.
98### Memory
100The set only needs 1 bit for each possible number.
101A bitset for 40000 transfers occupies 5KB of memory.
103## Performance
105Operations for add/remove/check are O(1). Iteration needs to scan for the next
106bit set. The number of scans is small (see memory footprint) and, for checking
107bits, many compilers offer primitives for special CPU instructions.
109## `uint32_spbset`
111While the memory footprint of `uint32_bset` is good, it still needs 5KB to
112store the single number 40000. This is not optimal when many are needed. For
113example, in event based processing, each socket needs to keep track of the
114transfers involved. There are many sockets potentially, but each one mostly
115tracks a single transfer or few (on HTTP/2 connection borderline up to 100).
117For such uses cases, the `uint32_spbset` is intended: track a small number of
118unsigned int, potentially rather "close" together. It keeps "chunks" with an
119offset and has no capacity limit.
121Example: adding the number 40000 to an empty sparse bitset would have one
122chunk with offset 39936, keeping track of the numbers 39936 to 40192 (a chunk
123has 4 64-bit values). The numbers in that range can be handled without further
124allocations.
126The worst case is then storing 100 numbers that lie in separate intervals.
127Then 100 chunks would need to be allocated and linked, resulting in overall 4
128KB of memory used.
130Iterating a sparse bitset works the same as for bitset and table.
132## `uint32_hash`
134At last, there are places in libcurl such as the HTTP/2 and HTTP/3 protocol
135implementations that need to store their own data related to a transfer.
136`uint32_hash` allows then to associate an unsigned int, e.g. the transfer's
137`mid`, to their own data.
Diagnostics4 git calls · 25 ms
CallStatusSizeTime025 ms
discover200152 B2 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B2 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=2
git.response.bytes=149
git.response.cache=UPDATING
git.response.duration_ms=2
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=20
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2002.5 KiB · hit20 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=2599
git.response.cache=HIT
git.response.duration_ms=20
git.fetch.wants=22ba858c61050c3034d73e4c9567692e6f7ccb0c
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=2599
git.pkt.payload.max=2578
git.sideband.band1.bytes=2577
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e2eaf13fcc4bc2
Request
8be91c93…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format