Artifactspublic-v0

curl

master
ply4 git calls347 ms289 KiB2 cachedprotocol v2
116 lines4.5 KiBRaw

Adding a new protocol?

Every once in a while, someone comes up with the idea of adding support for yet another protocol to curl. After all, curl already supports 25 something protocols and it is the Internet transfer machine for the world.

In the curl project we love protocols and we love supporting many protocols and doing it well.

How do you proceed to add a new protocol and what are the requirements?

No fixed set of requirements

This document is an attempt to describe things to consider. There is no checklist of the twenty-seven things you need to cross off. We view the entire effort as a whole and then judge if it seems to be the right thing - for now. The more things that look right, fit our patterns and are done in ways that align with our thinking, the better are the chances that we agree that supporting this protocol is a grand idea.

Mutual benefit is preferred

curl is not here for your protocol. Your protocol is not here for curl. The best cooperation and end result occur when all involved parties mutually see and agree that supporting this protocol in curl would be good for everyone. Heck, for the world.

Consider "selling us" the idea that we need an implementation merged in curl, to be fairly important. Why do we want curl to support this new protocol?

Protocol requirements

Client-side

The protocol implementation is for a client's side of a "communication session".

Transfer oriented

The protocol itself should be focused on transfers. Be it uploads or downloads or both. It should at least be possible to view the transfers as such, like we can view reading emails over POP3 as a download and sending emails over SMTP as an upload.

If you cannot even shoehorn the protocol into a transfer focused view, then you are up for a tough argument.

URL

There should be a documented URL format. If there is an RFC for it there is no question about it but the syntax does not have to be a published RFC. It could be enough if it is already in use by other implementations.

If you make up the syntax in order to be able to propose it to curl, then you are in a bad place. URLs are designed and defined for interoperability. There should at least be a good chance that other clients and servers can be implemented supporting the same URL syntax and work the same or similar way.

URLs work on registered 'schemes'. There is a register of all officially recognized schemes. If your protocol is not in there, is it really a protocol we want?

Wide and public use

The protocol shall already be used or have an expectation of getting used widely. Experimental protocols are better off worked on in experiments first, to prove themselves before they are adopted by curl.

Code

Of course the code needs to be written, provided, licensed agreeably and it should follow our code guidelines and review comments have to be dealt with. If the implementation needs third party code, that third party code should not have noticeably lesser standards than the curl project itself.

Tests

As much of the protocol implementation as possible needs to be verified by curl test cases. We must have the implementation get tested by CI jobs, torture tests and more.

We have experienced many times in the past how new implementations were brought to curl and immediately once the code had been merged, the originator vanished from the face of the earth. That is fine, but we need to take the necessary precautions so when it happens we are still fine.

Our test infrastructure is powerful enough to test almost every protocol - but it might require a bit of an effort to make it happen.

Documentation

We cannot assume that users are particularly familiar with details and peculiarities of the protocol. It needs documentation.

Maybe it even needs some internal documentation so that the developers who try to debug something five years from now can figure out functionality a little easier.

The protocol specification itself should be freely available without requiring a non-disclosure agreement or similar.

Do not compare

We are constantly raising the bar and we are constantly improving the project. A lot of things we did in the past would not be acceptable if done today. Therefore, you might be tempted to use shortcuts or "hacks" you can spot other - existing - protocol implementations have used, but there is nothing to gain from that. The bar has been raised. Former "cheats" may not tolerated anymore.

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# Adding a new protocol?
9Every once in a while, someone comes up with the idea of adding support for yet
10another protocol to curl. After all, curl already supports 25 something
11protocols and it is the Internet transfer machine for the world.
13In the curl project we love protocols and we love supporting many protocols
14and doing it well.
16How do you proceed to add a new protocol and what are the requirements?
18## No fixed set of requirements
20This document is an attempt to describe things to consider. There is no
21checklist of the twenty-seven things you need to cross off. We view the entire
22effort as a whole and then judge if it seems to be the right thing - for now.
23The more things that look right, fit our patterns and are done in ways that
24align with our thinking, the better are the chances that we agree that
25supporting this protocol is a grand idea.
27## Mutual benefit is preferred
29curl is not here for your protocol. Your protocol is not here for curl. The
30best cooperation and end result occur when all involved parties mutually see
31and agree that supporting this protocol in curl would be good for everyone.
32Heck, for the world.
34Consider "selling us" the idea that we need an implementation merged in curl,
35to be fairly important. *Why* do we want curl to support this new protocol?
37## Protocol requirements
39### Client-side
41The protocol implementation is for a client's side of a "communication
42session".
44### Transfer oriented
46The protocol itself should be focused on *transfers*. Be it uploads or
47downloads or both. It should at least be possible to view the transfers as
48such, like we can view reading emails over POP3 as a download and sending
49emails over SMTP as an upload.
51If you cannot even shoehorn the protocol into a transfer focused view, then
52you are up for a tough argument.
54### URL
56There should be a documented URL format. If there is an RFC for it there is no
57question about it but the syntax does not have to be a published RFC. It could
58be enough if it is already in use by other implementations.
60If you make up the syntax in order to be able to propose it to curl, then you
61are in a bad place. URLs are designed and defined for interoperability. There
62should at least be a good chance that other clients and servers can be
63implemented supporting the same URL syntax and work the same or similar way.
65URLs work on registered 'schemes'. There is a register of [all officially
66recognized
67schemes](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml). If
68your protocol is not in there, is it really a protocol we want?
70### Wide and public use
72The protocol shall already be used or have an expectation of getting used
73widely. Experimental protocols are better off worked on in experiments first,
74to prove themselves before they are adopted by curl.
76## Code
78Of course the code needs to be written, provided, licensed agreeably and it
79should follow our code guidelines and review comments have to be dealt with.
80If the implementation needs third party code, that third party code should not
81have noticeably lesser standards than the curl project itself.
83## Tests
85As much of the protocol implementation as possible needs to be verified by
86curl test cases. We must have the implementation get tested by CI jobs,
87torture tests and more.
89We have experienced many times in the past how new implementations were brought
90to curl and immediately once the code had been merged, the originator vanished
91from the face of the earth. That is fine, but we need to take the necessary
92precautions so when it happens we are still fine.
94Our test infrastructure is powerful enough to test almost every protocol - but
95it might require a bit of an effort to make it happen.
97## Documentation
99We cannot assume that users are particularly familiar with details and
100peculiarities of the protocol. It needs documentation.
102Maybe it even needs some internal documentation so that the developers who try
103to debug something five years from now can figure out functionality a little
104easier.
106The protocol specification itself should be freely available without requiring
107a non-disclosure agreement or similar.
109## Do not compare
111We are constantly raising the bar and we are constantly improving the project.
112A lot of things we did in the past would not be acceptable if done today.
113Therefore, you might be tempted to use shortcuts or "hacks" you can spot
114other - existing - protocol implementations have used, but there is nothing to
115gain from that. The bar has been raised. Former "cheats" may not tolerated
116anymore.
Diagnostics4 git calls · 347 ms
CallStatusSizeTime0347 ms
discover200152 B3 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit3 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=3
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit4 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=340
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=4
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2002.2 KiB340 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=2266
git.response.cache=MISS
git.response.duration_ms=340
git.fetch.wants=832f8e843dbd70ef30e39c7044ba6133101a61b0
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=2266
git.pkt.payload.max=2245
git.sideband.band1.bytes=2244
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e2e4091d802947
Request
8f218c84…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format