Artifactspublic-v0

curl

master
ply4 git calls26 ms288 KiB3 cachedprotocol v2
73 lines3.3 KiBRaw

Multi Identifiers (mid)

All transfers (easy handles) added to a multi handle are assigned a unique identifier until they are removed again. The multi handle keeps a table multi->xfers that allow O(1) access to the easy handle by its mid.

References to other easy handles should keep their mids instead of a pointer (not all code has been converted as of now). This solves problems in easy and multi handle life cycle management as well as iterating over handles where operations may add/remove other handles.

Values and Lifetime

An mid is an unsigned int. There are two reserved values:

  • 0: is the mid of an internal "admin" handle. Multi and share handles each have their own admin handle for maintenance operations, like shutting down connections.
  • UINT_MAX: the "invalid" mid. Easy handles are initialized with this value. They get it assigned again when removed from a multi handle.

This makes potential range of mids from 1 to UINT_MAX - 1 inside the same multi handle at the same time. The multi->xfers table reuses mid values from previous transfers that have been removed.

multi->xfers is created with an initial capacity. At the time of this writing that is 16 for "multi_easy" handles (used in curl_easy_perform() and 512 for multi handles created with curl_multi_init().

The first added easy handle gets mid == 1 assigned. The second one receives 2, even when the first one has been removed already. Every added handle gets an mid one larger than the previously assigned one. Until the capacity of the table is reached and it starts looking for a free id at 1 again (0 is always in the table).

When adding a new handle, the multi checks the amount of free entries in the multi->xfers table. If that drops below a threshold (currently 25%), the table is resized. This serves two purposes: one, a previous mid is not reused immediately and second, table resizes are not needed that often.

The table is implemented in uint-table.[ch]. More details in UINT_SETS.

Tracking mids

There are several places where transfers need to be tracked:

  • the multi tracks process, pending and msgsent transfers. A transfer is in at most one of these at a time. A dirty set, which may overlap with process, tracks transfers that need to run immediately.
  • connections track the transfers that are attached to them.
  • multi event handling tracks transfers interested in a specific socket.
  • DoH handles track the handle they perform lookups for (and vice versa).

There are two bitset implemented for storing mids: uint_bset and uint_spbset. The first is a bitset optimal for storing a large number of unsigned int values. The second one is a "sparse" variant good for storing a small set of numbers. More details about these in UINT_SETS.

A multi uses uint_bsets for process, dirty, pending and msgsent. Connections and sockets use the sparse variant as both often track only a single transfer and at most 100 on an HTTP/2 or HTTP/3 connection/socket.

These sets allow safe iteration while being modified. This allows a multi to iterate over its "process" set while existing transfers are removed or new ones added.

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# Multi Identifiers (mid)
9All transfers (easy handles) added to a multi handle are assigned
10a unique identifier until they are removed again. The multi handle
11keeps a table `multi->xfers` that allow O(1) access to the easy
12handle by its `mid`.
14References to other easy handles *should* keep their `mid`s instead
15of a pointer (not all code has been converted as of now). This solves
16problems in easy and multi handle life cycle management as well as
17iterating over handles where operations may add/remove other handles.
19### Values and Lifetime
21An `mid` is an `unsigned int`. There are two reserved values:
23* `0`: is the `mid` of an internal "admin" handle. Multi and share handles
24 each have their own admin handle for maintenance operations, like
25 shutting down connections.
26* `UINT_MAX`: the "invalid" `mid`. Easy handles are initialized with
27 this value. They get it assigned again when removed from
28 a multi handle.
30This makes potential range of `mid`s from `1` to `UINT_MAX - 1` *inside the
31same multi handle at the same time*. The `multi->xfers` table reuses `mid`
32values from previous transfers that have been removed.
34`multi->xfers` is created with an initial capacity. At the time of this
35writing that is `16` for "multi_easy" handles (used in `curl_easy_perform()`
36and `512` for multi handles created with `curl_multi_init()`.
38The first added easy handle gets `mid == 1` assigned. The second one receives `2`,
39even when the first one has been removed already. Every added handle gets an
40`mid` one larger than the previously assigned one. Until the capacity of
41the table is reached and it starts looking for a free id at `1` again (`0`
42is always in the table).
44When adding a new handle, the multi checks the amount of free entries
45in the `multi->xfers` table. If that drops below a threshold (currently 25%),
46the table is resized. This serves two purposes: one, a previous `mid` is not
47reused immediately and second, table resizes are not needed that often.
49The table is implemented in `uint-table.[ch]`. More details in [`UINT_SETS`](UINT_SETS.md).
51### Tracking `mid`s
53There are several places where transfers need to be tracked:
55* the multi tracks `process`, `pending` and `msgsent` transfers. A transfer is
56 in at most one of these at a time. A `dirty` set, which may overlap with
57 `process`, tracks transfers that need to run immediately.
58* connections track the transfers that are *attached* to them.
59* multi event handling tracks transfers interested in a specific socket.
60* DoH handles track the handle they perform lookups for (and vice versa).
62There are two bitset implemented for storing `mid`s: `uint_bset` and `uint_spbset`.
63The first is a bitset optimal for storing a large number of unsigned int values.
64The second one is a "sparse" variant good for storing a small set of numbers.
65More details about these in [`UINT_SETS`](UINT_SETS.md).
67A multi uses `uint_bset`s for `process`, `dirty`, `pending` and `msgsent`.
68Connections and sockets use the sparse variant as both often track only a
69single transfer and at most 100 on an HTTP/2 or HTTP/3 connection/socket.
71These sets allow safe iteration while being modified. This allows a multi
72to iterate over its "process" set while existing transfers are removed
73or new ones added.
Diagnostics4 git calls · 26 ms
CallStatusSizeTime026 ms
discover200152 B4 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit4 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=4
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=4
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=19
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2001.6 KiB · hit19 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=1647
git.response.cache=HIT
git.response.duration_ms=19
git.fetch.wants=fddc3d3a2fabd4fc26c84793c2a92b59edb13288
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=1647
git.pkt.payload.max=1626
git.sideband.band1.bytes=1625
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e2e0cc78554af4
Request
32b624dd…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format