Artifactspublic-v0

curl

master
ply4 git calls32 ms290 KiB3 cachedprotocol v2
171 lines6.7 KiBRaw

HTTP Cookies

Cookies are name=contents pairs that an HTTP server tells the client to hold and then the client sends back those to the server on subsequent requests to the same domains and paths for which the cookies were set.

Cookies are either "session cookies" which typically are forgotten when the session is over which is often translated to equal when browser quits, or the cookies are not session cookies they have expiration dates after which the client throws them away.

Cookies are set to the client with the Set-Cookie: header and are sent to servers with the Cookie: header.

For a long time, the only spec explaining how to use cookies was the original Netscape spec from 1994.

In 2011, RFC 6265 was finally published and details how cookies work within HTTP. In 2016, an update which added support for prefixes was proposed, and in 2017, another update was drafted to deprecate modification of 'secure' cookies from non-secure origins. Both of these drafts have been incorporated into a proposal to replace RFC 6265. Cookie prefixes and secure cookie modification protection has been implemented by curl.

curl considers http://localhost to be a secure context, meaning that it allows and uses cookies marked with the secure keyword even when done over plain HTTP for this host. curl does this to match how popular browsers work with secure cookies.

Super cookies

A single cookie can be set for a domain that matches multiple hosts. Like if set for example.com it gets sent to both aa.example.com as well as bb.example.com.

A challenge with this concept is that there are certain domains for which cookies should not be allowed at all, because they are Public Suffixes. Similarly, a client never accepts cookies set directly for the top-level domain like for example .com. Cookies set for too broad domains are generally referred to as super cookies.

If curl is built with PSL (Public Suffix List) support, it detects and discards cookies that are specified for such suffix domains that should not be allowed to have cookies.

if curl is not built with PSL support, it has no ability to stop super cookies.

Cookies saved to disk

Netscape once created a file format for storing cookies on disk so that they would survive browser restarts. curl adopted that file format to allow sharing the cookies with browsers, only to see browsers move away from that format. Modern browsers no longer use it, while curl still does.

The Netscape cookie file format stores one cookie per physical line in the file with a bunch of associated meta data, each field separated with TAB. That file is called the cookie jar in curl terminology.

When libcurl saves a cookie jar, it creates a file header of its own in which there is a URL mention that links to the web version of this document.

The cookie file format is text based and stores one cookie per line. Lines that start with # are treated as comments. An exception is lines that start with #HttpOnly_, which is a prefix for cookies that have the HttpOnly attribute set.

Each line that specifies a single cookie consists of seven text fields separated with TAB characters. A valid line must end with a newline character.

Fields in the file

Field number, what type and example data and the meaning of it:

  1. string example.com - the domain name
  2. boolean FALSE - include subdomains
  3. string /foobar/ - path
  4. boolean TRUE - send/receive over HTTPS only
  5. number 1462299217 - expires at - seconds since Jan 1st 1970, or 0
  6. string person - name of the cookie
  7. string daniel - value of the cookie

Cookies with curl the command line tool

curl has a full cookie "engine" built in. If you activate it, you can have curl receive and send cookies exactly as mandated in the specs.

Command line options:

-b, --cookie

tell curl a file to read cookies from and start the cookie engine, or if it is not a file it passes on the given string. -b name=var works and so does -b cookiefile.

-j, --junk-session-cookies

when used in combination with -b, it skips all "session cookies" on load so as to appear to start a new cookie session.

-c, --cookie-jar

tell curl to start the cookie engine and write cookies to the given file after the request(s)

Cookies with libcurl

libcurl offers several ways to enable and interface the cookie engine. These options are the ones provided by the native API. libcurl bindings may offer access to them using other means.

CURLOPT_COOKIE

Is used when you want to specify the exact contents of a cookie header to send to the server.

CURLOPT_COOKIEFILE

Tell libcurl to activate the cookie engine, and to read the initial set of cookies from the given file. Read-only.

CURLOPT_COOKIEJAR

Tell libcurl to activate the cookie engine, and when the easy handle is closed save all known cookies to the given cookie jar file. Write-only.

CURLOPT_COOKIELIST

Provide detailed information about a single cookie to add to the internal storage of cookies. Pass in the cookie as an HTTP header with all the details set, or pass in a line from a Netscape cookie file. This option can also be used to flush the cookies etc.

CURLOPT_COOKIESESSION

Tell libcurl to ignore all cookies it is about to load that are session cookies.

CURLINFO_COOKIELIST

Extract cookie information from the internal cookie storage as a linked list.

Cookies with JavaScript

These days a lot of the web is built up by JavaScript. The web browser loads complete programs that render the page you see. These JavaScript programs can also set and access cookies.

Since curl and libcurl are plain HTTP clients without any knowledge of or capability to handle JavaScript, such cookies are not detected or used.

Often, if you want to mimic what a browser does on such websites, you can record web browser HTTP traffic when using such a site and then repeat the cookie operations using curl or libcurl.

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# HTTP Cookies
9## Cookie overview
11Cookies are `name=contents` pairs that an HTTP server tells the client to
12hold and then the client sends back those to the server on subsequent
13requests to the same domains and paths for which the cookies were set.
15Cookies are either "session cookies" which typically are forgotten when the
16session is over which is often translated to equal when browser quits, or
17the cookies are not session cookies they have expiration dates after which
18the client throws them away.
20Cookies are set to the client with the Set-Cookie: header and are sent to
21servers with the Cookie: header.
23For a long time, the only spec explaining how to use cookies was the
24original [Netscape spec from 1994](https://curl.se/rfc/cookie_spec.html).
26In 2011, [RFC 6265](https://datatracker.ietf.org/doc/html/rfc6265) was finally
27published and details how cookies work within HTTP. In 2016, an update which
28added support for prefixes was
29[proposed](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-cookie-prefixes-00),
30and in 2017, another update was
31[drafted](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-cookie-alone-01)
32to deprecate modification of 'secure' cookies from non-secure origins. Both
33of these drafts have been incorporated into a proposal to
34[replace](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-rfc6265bis-11)
35RFC 6265. Cookie prefixes and secure cookie modification protection has been
36implemented by curl.
38curl considers `http://localhost` to be a *secure context*, meaning that it
39allows and uses cookies marked with the `secure` keyword even when done over
40plain HTTP for this host. curl does this to match how popular browsers work
41with secure cookies.
43## Super cookies
45A single cookie can be set for a domain that matches multiple hosts. Like if
46set for `example.com` it gets sent to both `aa.example.com` as well as
47`bb.example.com`.
49A challenge with this concept is that there are certain domains for which
50cookies should not be allowed at all, because they are *Public
51Suffixes*. Similarly, a client never accepts cookies set directly for the
52top-level domain like for example `.com`. Cookies set for *too broad*
53domains are generally referred to as *super cookies*.
55If curl is built with PSL (**Public Suffix List**) support, it detects and
56discards cookies that are specified for such suffix domains that should not
57be allowed to have cookies.
59if curl is *not* built with PSL support, it has no ability to stop super
60cookies.
62## Cookies saved to disk
64Netscape once created a file format for storing cookies on disk so that they
65would survive browser restarts. curl adopted that file format to allow
66sharing the cookies with browsers, only to see browsers move away from that
67format. Modern browsers no longer use it, while curl still does.
69The Netscape cookie file format stores one cookie per physical line in the
70file with a bunch of associated meta data, each field separated with
71TAB. That file is called the cookie jar in curl terminology.
73When libcurl saves a cookie jar, it creates a file header of its own in
74which there is a URL mention that links to the web version of this document.
76## Cookie file format
78The cookie file format is text based and stores one cookie per line. Lines
79that start with `#` are treated as comments. An exception is lines that
80start with `#HttpOnly_`, which is a prefix for cookies that have the
81`HttpOnly` attribute set.
83Each line that specifies a single cookie consists of seven text fields
84separated with TAB characters. A valid line must end with a newline
85character.
87### Fields in the file
89Field number, what type and example data and the meaning of it:
910. string `example.com` - the domain name
921. boolean `FALSE` - include subdomains
932. string `/foobar/` - path
943. boolean `TRUE` - send/receive over HTTPS only
954. number `1462299217` - expires at - seconds since Jan 1st 1970, or 0
965. string `person` - name of the cookie
976. string `daniel` - value of the cookie
99## Cookies with curl the command line tool
101curl has a full cookie "engine" built in. If you activate it, you can have
102curl receive and send cookies exactly as mandated in the specs.
104Command line options:
106[`-b, --cookie`](https://curl.se/docs/manpage.html#-b)
108tell curl a file to read cookies from and start the cookie engine, or if it
109is not a file it passes on the given string. `-b name=var` works and so does
110`-b cookiefile`.
112[`-j, --junk-session-cookies`](https://curl.se/docs/manpage.html#-j)
114when used in combination with -b, it skips all "session cookies" on load so
115as to appear to start a new cookie session.
117[`-c, --cookie-jar`](https://curl.se/docs/manpage.html#-c)
119tell curl to start the cookie engine and write cookies to the given file
120after the request(s)
122## Cookies with libcurl
124libcurl offers several ways to enable and interface the cookie engine. These
125options are the ones provided by the native API. libcurl bindings may offer
126access to them using other means.
128[`CURLOPT_COOKIE`](https://curl.se/libcurl/c/CURLOPT_COOKIE.html)
130Is used when you want to specify the exact contents of a cookie header to
131send to the server.
133[`CURLOPT_COOKIEFILE`](https://curl.se/libcurl/c/CURLOPT_COOKIEFILE.html)
135Tell libcurl to activate the cookie engine, and to read the initial set of
136cookies from the given file. Read-only.
138[`CURLOPT_COOKIEJAR`](https://curl.se/libcurl/c/CURLOPT_COOKIEJAR.html)
140Tell libcurl to activate the cookie engine, and when the easy handle is
141closed save all known cookies to the given cookie jar file. Write-only.
143[`CURLOPT_COOKIELIST`](https://curl.se/libcurl/c/CURLOPT_COOKIELIST.html)
145Provide detailed information about a single cookie to add to the internal
146storage of cookies. Pass in the cookie as an HTTP header with all the
147details set, or pass in a line from a Netscape cookie file. This option can
148also be used to flush the cookies etc.
150[`CURLOPT_COOKIESESSION`](https://curl.se/libcurl/c/CURLOPT_COOKIESESSION.html)
152Tell libcurl to ignore all cookies it is about to load that are session
153cookies.
155[`CURLINFO_COOKIELIST`](https://curl.se/libcurl/c/CURLINFO_COOKIELIST.html)
157Extract cookie information from the internal cookie storage as a linked
158list.
160## Cookies with JavaScript
162These days a lot of the web is built up by JavaScript. The web browser loads
163complete programs that render the page you see. These JavaScript programs
164can also set and access cookies.
166Since curl and libcurl are plain HTTP clients without any knowledge of or
167capability to handle JavaScript, such cookies are not detected or used.
169Often, if you want to mimic what a browser does on such websites, you can
170record web browser HTTP traffic when using such a site and then repeat the
171cookie operations using curl or libcurl.
Diagnostics4 git calls · 32 ms
CallStatusSizeTime032 ms
discover200152 B2 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=2
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit3 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=3
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit4 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=4
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=4
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch2002.9 KiB · hit25 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=25
git.response.bytes=3003
git.response.cache=HIT
git.response.duration_ms=25
git.fetch.wants=2a32aae6053caad7ed96635fe359ec8ac9005441
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=3003
git.pkt.payload.max=2982
git.sideband.band1.bytes=2981
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e1efad1c89efe9
Request
c8dce27a…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format