Artifactspublic-v0

curl

master
ply4 git calls40 ms298 KiB3 cachedprotocol v2
705 lines27 KiBRaw

The Art Of Scripting HTTP Requests Using curl

Background

This document assumes that you are familiar with HTML and general networking.

The increasing amount of applications moving to the web has made "HTTP Scripting" more frequently requested and wanted. To be able to automatically extract information from the web, to fake users, to post or upload data to web servers are all important tasks today.

curl is a command line tool for doing all sorts of URL manipulations and transfers, but this particular document focuses on how to use it when doing HTTP requests for fun and profit. This documents assumes that you know how to invoke curl --help or curl --manual to get basic information about it.

curl is not written to do everything for you. It makes the requests, it gets the data, it sends data and it retrieves the information. You probably need to glue everything together using some kind of script language or repeated manual invokes.

The HTTP Protocol

HTTP is the protocol used to fetch data from web servers. It is a simple protocol that is built upon TCP/IP. The protocol also allows information to get sent to the server from the client using a few different methods, as is shown here.

HTTP is plain ASCII text lines being sent by the client to a server to request a particular action, and then the server replies a few text lines before the actual requested content is sent to the client.

The client, curl, sends an HTTP request. The request contains a method (like GET, POST, HEAD etc), a number of request headers and sometimes a request body. The HTTP server responds with a status line (indicating if things went well), response headers and most often also a response body. The "body" part is the plain data you requested, like the actual HTML or the image etc.

See the Protocol

Using curl's option --verbose (-v as a short option) displays what kind of commands curl sends to the server, as well as a few other informational texts.

--verbose is the single most useful option when it comes to debug or even understand the curl<->server interaction.

Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer even more details as they show everything curl sends and receives. Use it like this:

curl --trace-ascii debugdump.txt https://www.example.com/

See the Timing

Many times you may wonder what exactly is taking all the time, or you want to know the amount of milliseconds between two points in a transfer. For those, and other similar situations, the --trace-time option is what you need. It prepends the time to each trace output line:

curl --trace-ascii d.txt --trace-time https://example.com/

See which Transfer

When doing parallel transfers, it is relevant to see which transfer is doing what. When response headers are received (and logged) you need to know which transfer these are for. --trace-ids option is what you need. It prepends the transfer and connection identifier to each trace output line:

curl --trace-ascii d.txt --trace-ids https://example.com/

See the Response

By default curl sends the response to stdout. You need to redirect it somewhere to avoid that, most often that is done with -o or -O.

URL

Spec

The Uniform Resource Locator format is how you specify the address of a particular resource on the Internet. You know these, you have seen URLs like https://curl.se/ or https://example.com/ a million times. RFC 3986 is the canonical spec. The formal name is not URL, it is URI.

Host

The hostname is usually resolved using DNS or your /etc/hosts file to an IP address and that is what curl communicates with. Alternatively you specify the IP address directly in the URL instead of a name.

For development and other trying out situations, you can point to a different IP address for a hostname than what would otherwise be used, by using curl's --resolve option:

curl --resolve www.example.org:80:127.0.0.1 https://www.example.org/

Port number

Each protocol curl supports operates on a default port number, be it over TCP or in some cases UDP. Normally you do not have to take that into consideration, but at times you run test servers on other ports or similar. Then you can specify the port number in the URL with a colon and a number immediately following the hostname. Like when doing HTTP to port 1234:

curl https://www.example.org:1234/

The port number you specify in the URL is the number that the server uses to offer its services. Sometimes you may use a proxy, and then you may need to specify that proxy's port number separately from what curl needs to connect to the server. Like when using an HTTP proxy on port 4321:

curl --proxy http://proxy.example.org:4321 https://remote.example.org/

Username and password

Some services are setup to require HTTP authentication and then you need to provide name and password which is then transferred to the remote site in various ways depending on the exact authentication protocol used.

You can opt to either insert the user and password in the URL or you can provide them separately:

curl https://user:password@example.org/

or

curl -u user:password https://example.org/

You need to pay attention that this kind of HTTP authentication is not what is usually done and requested by user-oriented websites these days. They tend to use forms and cookies instead.

Path part

The path part is sent off to the server to request that it sends back the associated response. The path is what is to the right side of the slash that follows the hostname and possibly port number.

Fetch a page

GET

The simplest and most common request/operation made using HTTP is to GET a URL. The URL could itself refer to a webpage, an image or a file. The client issues a GET request to the server and receives the document it asked for. If you issue the command line

curl https://curl.se/

you get a webpage returned in your terminal window. The entire HTML document this URL identifies.

All HTTP replies contain a set of response headers that are normally hidden, use curl's --include (-i) option to display them as well as the rest of the document.

You can ask the remote server for ONLY the headers by using the --head (-I) option which makes curl issue a HEAD request. In some special cases servers deny the HEAD method while others still work, which is a particular kind of annoyance.

The HEAD method is defined and made so that the server returns the headers exactly the way it would do for a GET, but without a body. It means that you may see a Content-Length: in the response headers, but there must not be an actual body in the HEAD response.

Multiple URLs in a single command line

A single curl command line may involve one or many URLs. The most common case is probably to use one, but you can specify any amount of URLs. Yes any. No limits. You then get requests repeated over and over for all the given URLs.

Example, send two GET requests:

curl https://url1.example.com https://url2.example.com

If you use --data to POST to the URL, using multiple URLs means that you send that same POST to all the given URLs.

Example, send two POSTs:

curl --data name=curl https://url1.example.com https://url2.example.com

Multiple HTTP methods in a single command line

Sometimes you need to operate on several URLs in a single command line and do different HTTP methods on each. For this, you might enjoy the --next option. It is a separator that separates a bunch of options from the next. All the URLs before --next get the same method and get all the POST data merged into one.

When curl reaches the --next on the command line, it resets the method and the POST data and allow a new set.

Perhaps this is best shown with a few examples. To send first a HEAD and then a GET:

curl -I https://example.com --next https://example.com

To first send a POST and then a GET:

curl -d score=10 https://example.com/post.cgi --next https://example.com/results.html

HTML forms

Forms explained

Forms are the general way a website can present an HTML page with fields for the user to enter data in, and then press some kind of 'OK' or 'Submit' button to get that data sent to the server. The server then typically uses the posted data to decide how to act. Like using the entered words to search in a database, or to add the info in a bug tracking system, display the entered address on a map or using the info as a login-prompt verifying that the user is allowed to see what it is about to see.

Of course there has to be some kind of program on the server end to receive the data you send. You cannot invent something out of the air.

GET

A GET-form uses the method GET, as specified in HTML like:

html
<form method="GET" action="junk.cgi">
  <input type=text name="birthyear">
  <input type=submit name=press value="OK">
</form>

In your favorite browser, this form appears with a text box to fill in and a press-button labeled "OK". If you fill in '1905' and press the OK button, your browser then creates a new URL to get for you. The URL gets junk.cgi?birthyear=1905&press=OK appended to the path part of the previous URL.

If the original form was seen on the page www.example.com/when/birth.html, the second page you get becomes www.example.com/when/junk.cgi?birthyear=1905&press=OK.

Most search engines work this way.

To make curl do the GET form post for you, enter the expected created URL:

curl "https://www.example.com/when/junk.cgi?birthyear=1905&press=OK"

POST

The GET method makes all input field names get displayed in the URL field of your browser. That is generally a good thing when you want to be able to bookmark that page with your given data, but it is an obvious disadvantage if you entered secret information in one of the fields or if there are a large amount of fields creating a long and unreadable URL.

The HTTP protocol then offers the POST method. This way the client sends the data separated from the URL and thus you do not see any of it in the URL address field.

The form would look similar to the previous one:

html
<form method="POST" action="junk.cgi">
  <input type=text name="birthyear">
  <input type=submit name=press value=" OK ">
</form>

To use curl to post this form with the same data filled in as before, we could do it like:

curl --data "birthyear=1905&press=%20OK%20" https://www.example.com/when/junk.cgi

This kind of POST uses the Content-Type application/x-www-form-urlencoded and is the most widely used POST kind.

The data you send to the server MUST already be properly encoded, curl does not do that for you. For example, if you want the data to contain a space, you need to replace that space with %20, etc. Failing to comply with this most likely causes your data to be received wrongly and messed up.

Recent curl versions can in fact URL encode POST data for you, like this:

curl --data-urlencode "name=I am Daniel" https://www.example.com

If you repeat --data several times on the command line, curl concatenates all the given data pieces - and put a & symbol between each data segment.

File Upload POST

Back in late 1995 they defined an additional way to post data over HTTP. It is documented in the RFC 1867, why this method sometimes is referred to as RFC 1867-posting.

This method is mainly designed to better support file uploads. A form that allows a user to upload a file could be written like this in HTML:

<form method="POST" enctype='multipart/form-data' action="upload.cgi">
  <input name=upload type=file>
  <input type=submit name=press value="OK">
</form>

This clearly shows that the Content-Type about to be sent is multipart/form-data.

To post to a form like this with curl, you enter a command line like:

curl --form upload=@localfilename --form press=OK [URL]

Hidden Fields

A common way for HTML based applications to pass state information between pages is to add hidden fields to the forms. Hidden fields are already filled in, they are not displayed to the user and they get passed along as all the other fields.

A similar example form with one visible field, one hidden field and one submit button could look like:

html
<form method="POST" action="foobar.cgi">
  <input type=text name="birthyear">
  <input type=hidden name="person" value="daniel">
  <input type=submit name="press" value="OK">
</form>

To POST this with curl, you do not have to think about if the fields are hidden or not. To curl they are all the same:

curl --data "birthyear=1905&press=OK&person=daniel" [URL]

Figure Out What A POST Looks Like

When you are about to fill in a form and send it to a server by using curl instead of a browser, you are of course interested in sending a POST exactly the way your browser does.

An easy way to get to see this, is to save the HTML page with the form on your local disk, modify the 'method' to a GET, and press the submit button (you could also change the action URL if you want to).

You then clearly see the data get appended to the URL, separated with a ?-letter as GET forms are supposed to.

HTTP upload

PUT

Perhaps the best way to upload data to an HTTP server is to use PUT. Then again, this of course requires that someone put a program or script on the server end that knows how to receive an HTTP PUT stream.

Put a file to an HTTP server with curl:

curl --upload-file uploadfile https://www.example.com/receive.cgi

HTTP Authentication

Basic Authentication

HTTP Authentication is the ability to tell the server your username and password so that it can verify that you are allowed to do the request you are doing. The Basic authentication used in HTTP (which is the type curl uses by default) is plain text based, which means it sends username and password only slightly obfuscated, but still fully readable by anyone that sniffs on the network between you and the remote server.

To tell curl to use a user and password for authentication:

curl --user myname:password https://www.example.com

Other Authentication

The site might require a different authentication method (check the headers returned by the server), and then --digest, --negotiate or even --anyauth might be options that suit you.

Proxy Authentication

Sometimes your HTTP access is only available through the use of an HTTP proxy. This seems to be especially common at various companies. An HTTP proxy may require its own user and password to allow the client to get through to the Internet. To specify those with curl, run something like:

curl --proxy-user proxyuser:proxypassword curl.se

If your proxy requires the authentication to be done using Digest use --proxy-digest.

If you use any one of these user+password options but leave out the password part, curl prompts for the password interactively.

Hiding credentials

Do note that when a program is run, its parameters might be possible to see when listing the running processes of the system. Thus, other users may be able to watch your passwords if you pass them as plain command line options. There are ways to circumvent this.

It is worth noting that while this is how HTTP Authentication works, many websites do not use this concept when they provide logins etc. See the Web Login chapter further below for more details on that.

More HTTP Headers

Referer

An HTTP request may include a 'referer' field (yes it is misspelled), which can be used to tell from which URL the client got to this particular resource. Some programs/scripts check the referer field of requests to verify that this was not arriving from an external site or an unknown page. While this is a stupid way to check something so easily forged, many scripts still do it. Using curl, you can put anything you want in the referer-field and thus more easily be able to fool the server into serving your request.

Use curl to set the referer field with:

curl --referer https://www.example.come https://www.example.com

User Agent

Similar to the referer field, all HTTP requests may set the User-Agent field. It names what user agent (client) that is being used. Many applications use this information to decide how to display pages. Silly web programmers try to make different pages for users of different browsers to make them look the best possible for their particular browsers. They usually also do different kinds of JavaScript etc.

At times, you may learn that getting a page with curl does not return the same page that you see when getting the page with your browser. Then you know it is time to set the User Agent field to fool the server into thinking you are one of those browsers.

By default, curl uses curl/VERSION, such as User-Agent: curl/8.11.0.

To make curl look like Internet Explorer 5 on a Windows 2000 box:

curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]

Or why not look like you are using Netscape 4.73 on an old Linux box:

curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]

Redirects

Location header

When a resource is requested from a server, the reply from the server may include a hint about where the browser should go next to find this page, or a new page keeping newly generated output. The header that tells the browser to redirect is Location:.

curl does not follow Location: headers by default, but displays such pages in the same manner it displays all HTTP replies. It does however feature an option that makes it attempt to follow the Location: pointers.

To tell curl to follow a Location:

curl --location https://www.example.com

If you use curl to POST to a site that immediately redirects you to another page, you can safely use --location (-L) and --data/--form together. curl only uses POST in the first request, and then revert to GET in the following operations.

Other redirects

Browsers typically support at least two other ways of redirects that curl does not: first the html may contain a meta refresh tag that asks the browser to load a specific URL after a set number of seconds, or it may use JavaScript to do it.

Cookies

The way the web browsers do "client side state control" is by using cookies. Cookies are names with associated contents. The cookies are sent to the client by the server. The server tells the client for what path and hostname it wants the cookie sent back, and it also sends an expiration date and a few more properties.

When a client communicates with a server with a name and path as previously specified in a received cookie, the client sends back the cookies and their contents to the server, unless of course they are expired.

Many applications and servers use this method to connect a series of requests into a single logical session. To be able to use curl in such occasions, we must be able to record and send back cookies the way the web application expects them. The same way browsers deal with them.

The simplest way to send a few cookies to the server when getting a page with curl is to add them on the command line like:

curl --cookie "name=Daniel" https://www.example.com

Cookies are sent as common HTTP headers. This is practical as it allows curl to record cookies by recording headers. Record cookies with curl by using the --dump-header (-D) option like:

curl --dump-header headers_and_cookies https://www.example.com

(Take note that the --cookie-jar option described below is a better way to store cookies.)

curl has a full blown cookie parsing engine built-in that comes in use if you want to reconnect to a server and use cookies that were stored from a previous connection (or hand-crafted manually to fool the server into believing you had a previous connection). To use previously stored cookies, you run curl like:

curl --cookie stored_cookies_in_file https://www.example.com

curl's "cookie engine" gets enabled when you use the --cookie option. If you only want curl to understand received cookies, use --cookie with a file that does not exist. Example, if you want to let curl understand cookies from a page and follow a location (and thus possibly send back cookies it received), you can invoke it like:

curl --cookie nada --location https://www.example.com

curl has the ability to read and write cookie files that use the same file format that Netscape and Mozilla once used. It is a convenient way to share cookies between scripts or invokes. The --cookie (-b) switch automatically detects if a given file is such a cookie file and parses it, and by using the --cookie-jar (-c) option you make curl write a new cookie file at the end of an operation:

curl --cookie cookies.txt --cookie-jar newcookies.txt \
  https://www.example.com

HTTPS

HTTPS is HTTP secure

There are a few ways to do secure HTTP transfers. By far the most common protocol for doing this is what is generally known as HTTPS, HTTP over SSL. SSL encrypts all the data that is sent and received over the network and thus makes it harder for attackers to spy on sensitive information.

SSL (or TLS as the current version of the standard is called) offers a set of advanced features to do secure transfers over HTTP.

curl supports encrypted fetches when built to use a TLS library and it can be built to use one out of a fairly large set of libraries - curl -V shows which one your curl was built to use (if any). To get a page from an HTTPS server, run curl like:

curl https://secure.example.com

Certificates

In the HTTPS world, you use certificates to validate that you are the one you claim to be, as an addition to normal passwords. curl supports client- side certificates. All certificates are locked with a passphrase, which you need to enter before the certificate can be used by curl. The passphrase can be specified on the command line or if not, entered interactively when curl queries for it. Use a certificate with curl on an HTTPS server like:

curl --cert mycert.pem https://secure.example.com

curl also tries to verify that the server is who it claims to be, by verifying the server's certificate against a locally stored CA cert bundle. Failing the verification causes curl to deny the connection. You must then use --insecure (-k) in case you want to tell curl to ignore that the server cannot be verified.

More about server certificate verification and CA cert bundles can be read in the SSLCERTS document.

At times you may end up with your own CA cert store and then you can tell curl to use that to verify the server's certificate:

curl --cacert ca-bundle.pem https://example.com/

Custom Request Elements

Modify method and headers

Doing fancy stuff, you may need to add or change elements of a single curl request.

For example, you can change the POST method to PROPFIND and send the data as Content-Type: text/xml (instead of the default Content-Type) like this:

curl --data "<xml>" --header "Content-Type: text/xml" \
  --request PROPFIND example.com

You can delete a default header by providing one without content. Like you can ruin the request by chopping off the Host: header:

curl --header "Host:" https://www.example.com

You can add headers the same way. Your server may want a Destination: header, and you can add it:

curl --header "Destination: nowhere" https://example.com

More on changed methods

It should be noted that curl selects which methods to use on its own depending on what action to ask for. -d makes a POST, -I makes a HEAD and so on. If you use the --request / -X option you can change the method keyword curl selects, but you do not modify curl's behavior. This means that if you for example use -d "data" to do a POST, you can modify the method to a PROPFIND with -X and curl still thinks it sends a POST. You can change the normal GET to a POST method by adding -X POST in a command line like:

curl -X POST https://example.org/

curl however still acts as if it sent a GET so it does not send any request body etc.

Web Login

Some login tricks

While not strictly HTTP related, it still causes a lot of people problems so here's the executive run-down of how the vast majority of all login forms work and how to login to them using curl.

It can also be noted that to do this properly in an automated fashion, you most certainly need to script things and do multiple curl invokes etc.

First, servers mostly use cookies to track the logged-in status of the client, so you need to capture the cookies you receive in the responses. Then, many sites also set a special cookie on the login page (to make sure you got there through their login page) so you should make a habit of first getting the login-form page to capture the cookies set there.

Some web-based login systems feature various amounts of JavaScript, and sometimes they use such code to set or modify cookie contents. Possibly they do that to prevent programmed logins, like this manual describes how to... Anyway, if reading the code is not enough to let you repeat the behavior manually, capturing the HTTP requests done by your browsers and analyzing the sent cookies is usually a working method to work out how to shortcut the JavaScript need.

In the actual <form> tag for the login, lots of sites fill-in random/session or otherwise secretly generated hidden tags and you may need to first capture the HTML code for the login form and extract all the hidden fields to be able to do a proper login POST. Remember that the contents need to be URL encoded when sent in a normal POST.

Debug

Some debug tricks

Many times when you run curl on a site, you notice that the site does not seem to respond the same way to your curl requests as it does to your browser's.

Then you need to start making your curl requests more similar to your browser's requests:

  • Use the --trace-ascii option to store fully detailed logs of the requests for easier analyzing and better understanding

  • Make sure you check for and use cookies when needed (both reading with --cookie and writing with --cookie-jar)

  • Set user-agent (with -A) to one like a recent popular browser does

  • Set referer (with -E) like it is set by the browser

  • If you use POST, make sure you send all the fields and in the same order as the browser does it.

Check what the browsers do

A good helper to make sure you do this right, is the web browsers' developers tools that let you view all headers you send and receive (even when using HTTPS).

A more raw approach is to capture the HTTP traffic on the network with tools such as Wireshark or tcpdump and check what headers that were sent and received by the browser. (HTTPS forces you to use SSLKEYLOGFILE to do that.)

1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
4SPDX-License-Identifier: curl
5-->
7# The Art Of Scripting HTTP Requests Using curl
9## Background
11This document assumes that you are familiar with HTML and general networking.
13The increasing amount of applications moving to the web has made "HTTP
14Scripting" more frequently requested and wanted. To be able to automatically
15extract information from the web, to fake users, to post or upload data to
16web servers are all important tasks today.
18curl is a command line tool for doing all sorts of URL manipulations and
19transfers, but this particular document focuses on how to use it when doing
20HTTP requests for fun and profit. This documents assumes that you know how to
21invoke `curl --help` or `curl --manual` to get basic information about it.
23curl is not written to do everything for you. It makes the requests, it gets
24the data, it sends data and it retrieves the information. You probably need
25to glue everything together using some kind of script language or repeated
26manual invokes.
28## The HTTP Protocol
30HTTP is the protocol used to fetch data from web servers. It is a simple
31protocol that is built upon TCP/IP. The protocol also allows information to
32get sent to the server from the client using a few different methods, as is
33shown here.
35HTTP is plain ASCII text lines being sent by the client to a server to
36request a particular action, and then the server replies a few text lines
37before the actual requested content is sent to the client.
39The client, curl, sends an HTTP request. The request contains a method (like
40GET, POST, HEAD etc), a number of request headers and sometimes a request
41body. The HTTP server responds with a status line (indicating if things went
42well), response headers and most often also a response body. The "body" part
43is the plain data you requested, like the actual HTML or the image etc.
45## See the Protocol
47Using curl's option [`--verbose`](https://curl.se/docs/manpage.html#-v) (`-v`
48as a short option) displays what kind of commands curl sends to the server,
49as well as a few other informational texts.
51`--verbose` is the single most useful option when it comes to debug or even
52understand the curl<->server interaction.
54Sometimes even `--verbose` is not enough. Then
55[`--trace`](https://curl.se/docs/manpage.html#-trace) and
56[`--trace-ascii`](https://curl.se/docs/manpage.html#--trace-ascii)
57offer even more details as they show **everything** curl sends and
58receives. Use it like this:
60 curl --trace-ascii debugdump.txt https://www.example.com/
62## See the Timing
64Many times you may wonder what exactly is taking all the time, or you want to
65know the amount of milliseconds between two points in a transfer. For those,
66and other similar situations, the
67[`--trace-time`](https://curl.se/docs/manpage.html#--trace-time) option is
68what you need. It prepends the time to each trace output line:
70 curl --trace-ascii d.txt --trace-time https://example.com/
72## See which Transfer
74When doing parallel transfers, it is relevant to see which transfer is doing
75what. When response headers are received (and logged) you need to know which
76transfer these are for.
77[`--trace-ids`](https://curl.se/docs/manpage.html#--trace-ids) option is what
78you need. It prepends the transfer and connection identifier to each trace
79output line:
81 curl --trace-ascii d.txt --trace-ids https://example.com/
83## See the Response
85By default curl sends the response to stdout. You need to redirect it
86somewhere to avoid that, most often that is done with `-o` or `-O`.
88# URL
90## Spec
92The Uniform Resource Locator format is how you specify the address of a
93particular resource on the Internet. You know these, you have seen URLs like
94https://curl.se/ or https://example.com/ a million times. RFC 3986 is the
95canonical spec. The formal name is not URL, it is **URI**.
97## Host
99The hostname is usually resolved using DNS or your /etc/hosts file to an IP
100address and that is what curl communicates with. Alternatively you specify
101the IP address directly in the URL instead of a name.
103For development and other trying out situations, you can point to a different
104IP address for a hostname than what would otherwise be used, by using curl's
105[`--resolve`](https://curl.se/docs/manpage.html#--resolve) option:
107 curl --resolve www.example.org:80:127.0.0.1 https://www.example.org/
109## Port number
111Each protocol curl supports operates on a default port number, be it over TCP
112or in some cases UDP. Normally you do not have to take that into
113consideration, but at times you run test servers on other ports or
114similar. Then you can specify the port number in the URL with a colon and a
115number immediately following the hostname. Like when doing HTTP to port
1161234:
118 curl https://www.example.org:1234/
120The port number you specify in the URL is the number that the server uses to
121offer its services. Sometimes you may use a proxy, and then you may
122need to specify that proxy's port number separately from what curl needs to
123connect to the server. Like when using an HTTP proxy on port 4321:
125 curl --proxy http://proxy.example.org:4321 https://remote.example.org/
127## Username and password
129Some services are setup to require HTTP authentication and then you need to
130provide name and password which is then transferred to the remote site in
131various ways depending on the exact authentication protocol used.
133You can opt to either insert the user and password in the URL or you can
134provide them separately:
136 curl https://user:password@example.org/
138or
140 curl -u user:password https://example.org/
142You need to pay attention that this kind of HTTP authentication is not what
143is usually done and requested by user-oriented websites these days. They tend
144to use forms and cookies instead.
146## Path part
148The path part is sent off to the server to request that it sends back the
149associated response. The path is what is to the right side of the slash that
150follows the hostname and possibly port number.
152# Fetch a page
154## GET
156The simplest and most common request/operation made using HTTP is to GET a
157URL. The URL could itself refer to a webpage, an image or a file. The client
158issues a GET request to the server and receives the document it asked for.
159If you issue the command line
161 curl https://curl.se/
163you get a webpage returned in your terminal window. The entire HTML document
164this URL identifies.
166All HTTP replies contain a set of response headers that are normally hidden,
167use curl's [`--include`](https://curl.se/docs/manpage.html#-i) (`-i`)
168option to display them as well as the rest of the document.
170## HEAD
172You can ask the remote server for ONLY the headers by using the
173[`--head`](https://curl.se/docs/manpage.html#-I) (`-I`) option which makes
174curl issue a HEAD request. In some special cases servers deny the HEAD method
175while others still work, which is a particular kind of annoyance.
177The HEAD method is defined and made so that the server returns the headers
178exactly the way it would do for a GET, but without a body. It means that you
179may see a `Content-Length:` in the response headers, but there must not be an
180actual body in the HEAD response.
182## Multiple URLs in a single command line
184A single curl command line may involve one or many URLs. The most common case
185is probably to use one, but you can specify any amount of URLs. Yes any. No
186limits. You then get requests repeated over and over for all the given URLs.
188Example, send two GET requests:
190 curl https://url1.example.com https://url2.example.com
192If you use [`--data`](https://curl.se/docs/manpage.html#-d) to POST to
193the URL, using multiple URLs means that you send that same POST to all the
194given URLs.
196Example, send two POSTs:
198 curl --data name=curl https://url1.example.com https://url2.example.com
200## Multiple HTTP methods in a single command line
202Sometimes you need to operate on several URLs in a single command line and do
203different HTTP methods on each. For this, you might enjoy the
204[`--next`](https://curl.se/docs/manpage.html#-:) option. It is a separator
205that separates a bunch of options from the next. All the URLs before `--next`
206get the same method and get all the POST data merged into one.
208When curl reaches the `--next` on the command line, it resets the method and
209the POST data and allow a new set.
211Perhaps this is best shown with a few examples. To send first a HEAD and then
212a GET:
214 curl -I https://example.com --next https://example.com
216To first send a POST and then a GET:
218 curl -d score=10 https://example.com/post.cgi --next https://example.com/results.html
220# HTML forms
222## Forms explained
224Forms are the general way a website can present an HTML page with fields for
225the user to enter data in, and then press some kind of 'OK' or 'Submit'
226button to get that data sent to the server. The server then typically uses
227the posted data to decide how to act. Like using the entered words to search
228in a database, or to add the info in a bug tracking system, display the
229entered address on a map or using the info as a login-prompt verifying that
230the user is allowed to see what it is about to see.
232Of course there has to be some kind of program on the server end to receive
233the data you send. You cannot invent something out of the air.
235## GET
237A GET-form uses the method GET, as specified in HTML like:
239```html
240<form method="GET" action="junk.cgi">
241 <input type=text name="birthyear">
242 <input type=submit name=press value="OK">
243</form>
244```
246In your favorite browser, this form appears with a text box to fill in and a
247press-button labeled "OK". If you fill in '1905' and press the OK button,
248your browser then creates a new URL to get for you. The URL gets
249`junk.cgi?birthyear=1905&press=OK` appended to the path part of the previous
250URL.
252If the original form was seen on the page `www.example.com/when/birth.html`,
253the second page you get becomes
254`www.example.com/when/junk.cgi?birthyear=1905&press=OK`.
256Most search engines work this way.
258To make curl do the GET form post for you, enter the expected created URL:
260 curl "https://www.example.com/when/junk.cgi?birthyear=1905&press=OK"
262## POST
264The GET method makes all input field names get displayed in the URL field of
265your browser. That is generally a good thing when you want to be able to
266bookmark that page with your given data, but it is an obvious disadvantage if
267you entered secret information in one of the fields or if there are a large
268amount of fields creating a long and unreadable URL.
270The HTTP protocol then offers the POST method. This way the client sends the
271data separated from the URL and thus you do not see any of it in the URL
272address field.
274The form would look similar to the previous one:
276```html
277<form method="POST" action="junk.cgi">
278 <input type=text name="birthyear">
279 <input type=submit name=press value=" OK ">
280</form>
281```
283To use curl to post this form with the same data filled in as before, we
284could do it like:
286 curl --data "birthyear=1905&press=%20OK%20" https://www.example.com/when/junk.cgi
288This kind of POST uses the Content-Type `application/x-www-form-urlencoded`
289and is the most widely used POST kind.
291The data you send to the server MUST already be properly encoded, curl does
292not do that for you. For example, if you want the data to contain a space,
293you need to replace that space with `%20`, etc. Failing to comply with this
294most likely causes your data to be received wrongly and messed up.
296Recent curl versions can in fact URL encode POST data for you, like this:
298 curl --data-urlencode "name=I am Daniel" https://www.example.com
300If you repeat `--data` several times on the command line, curl concatenates
301all the given data pieces - and put a `&` symbol between each data segment.
303## File Upload POST
305Back in late 1995 they defined an additional way to post data over HTTP. It
306is documented in the RFC 1867, why this method sometimes is referred to as
307RFC 1867-posting.
309This method is mainly designed to better support file uploads. A form that
310allows a user to upload a file could be written like this in HTML:
312 <form method="POST" enctype='multipart/form-data' action="upload.cgi">
313 <input name=upload type=file>
314 <input type=submit name=press value="OK">
315 </form>
317This clearly shows that the Content-Type about to be sent is
318`multipart/form-data`.
320To post to a form like this with curl, you enter a command line like:
322 curl --form upload=@localfilename --form press=OK [URL]
324## Hidden Fields
326A common way for HTML based applications to pass state information between
327pages is to add hidden fields to the forms. Hidden fields are already filled
328in, they are not displayed to the user and they get passed along as all the
329other fields.
331A similar example form with one visible field, one hidden field and one
332submit button could look like:
334```html
335<form method="POST" action="foobar.cgi">
336 <input type=text name="birthyear">
337 <input type=hidden name="person" value="daniel">
338 <input type=submit name="press" value="OK">
339</form>
340```
342To POST this with curl, you do not have to think about if the fields are
343hidden or not. To curl they are all the same:
345 curl --data "birthyear=1905&press=OK&person=daniel" [URL]
347## Figure Out What A POST Looks Like
349When you are about to fill in a form and send it to a server by using curl
350instead of a browser, you are of course interested in sending a POST exactly
351the way your browser does.
353An easy way to get to see this, is to save the HTML page with the form on
354your local disk, modify the 'method' to a GET, and press the submit button
355(you could also change the action URL if you want to).
357You then clearly see the data get appended to the URL, separated with a
358`?`-letter as GET forms are supposed to.
360# HTTP upload
362## PUT
364Perhaps the best way to upload data to an HTTP server is to use PUT. Then
365again, this of course requires that someone put a program or script on the
366server end that knows how to receive an HTTP PUT stream.
368Put a file to an HTTP server with curl:
370 curl --upload-file uploadfile https://www.example.com/receive.cgi
372# HTTP Authentication
374## Basic Authentication
376HTTP Authentication is the ability to tell the server your username and
377password so that it can verify that you are allowed to do the request you are
378doing. The Basic authentication used in HTTP (which is the type curl uses by
379default) is **plain text** based, which means it sends username and password
380only slightly obfuscated, but still fully readable by anyone that sniffs on
381the network between you and the remote server.
383To tell curl to use a user and password for authentication:
385 curl --user myname:password https://www.example.com
387## Other Authentication
389The site might require a different authentication method (check the headers
390returned by the server), and then
391[`--digest`](https://curl.se/docs/manpage.html#--digest),
392[`--negotiate`](https://curl.se/docs/manpage.html#--negotiate) or even
393[`--anyauth`](https://curl.se/docs/manpage.html#--anyauth) might be options
394that suit you.
396## Proxy Authentication
398Sometimes your HTTP access is only available through the use of an HTTP
399proxy. This seems to be especially common at various companies. An HTTP proxy
400may require its own user and password to allow the client to get through to
401the Internet. To specify those with curl, run something like:
403 curl --proxy-user proxyuser:proxypassword curl.se
405If your proxy requires the authentication to be done using Digest use
406[`--proxy-digest`](https://curl.se/docs/manpage.html#--proxy-digest).
408If you use any one of these user+password options but leave out the password
409part, curl prompts for the password interactively.
411## Hiding credentials
413Do note that when a program is run, its parameters might be possible to see
414when listing the running processes of the system. Thus, other users may be
415able to watch your passwords if you pass them as plain command line
416options. There are ways to circumvent this.
418It is worth noting that while this is how HTTP Authentication works, many
419websites do not use this concept when they provide logins etc. See the Web
420Login chapter further below for more details on that.
422# More HTTP Headers
424## Referer
426An HTTP request may include a 'referer' field (yes it is misspelled), which
427can be used to tell from which URL the client got to this particular
428resource. Some programs/scripts check the referer field of requests to verify
429that this was not arriving from an external site or an unknown page. While
430this is a stupid way to check something so easily forged, many scripts still
431do it. Using curl, you can put anything you want in the referer-field and
432thus more easily be able to fool the server into serving your request.
434Use curl to set the referer field with:
436 curl --referer https://www.example.come https://www.example.com
438## User Agent
440Similar to the referer field, all HTTP requests may set the User-Agent
441field. It names what user agent (client) that is being used. Many
442applications use this information to decide how to display pages. Silly web
443programmers try to make different pages for users of different browsers to
444make them look the best possible for their particular browsers. They usually
445also do different kinds of JavaScript etc.
447At times, you may learn that getting a page with curl does not return the
448same page that you see when getting the page with your browser. Then you know
449it is time to set the User Agent field to fool the server into thinking you
450are one of those browsers.
452By default, curl uses curl/VERSION, such as User-Agent: curl/8.11.0.
454To make curl look like Internet Explorer 5 on a Windows 2000 box:
456 curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
458Or why not look like you are using Netscape 4.73 on an old Linux box:
460 curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
462## Redirects
464## Location header
466When a resource is requested from a server, the reply from the server may
467include a hint about where the browser should go next to find this page, or a
468new page keeping newly generated output. The header that tells the browser to
469redirect is `Location:`.
471curl does not follow `Location:` headers by default, but displays such
472pages in the same manner it displays all HTTP replies. It does however
473feature an option that makes it attempt to follow the `Location:` pointers.
475To tell curl to follow a Location:
477 curl --location https://www.example.com
479If you use curl to POST to a site that immediately redirects you to another
480page, you can safely use [`--location`](https://curl.se/docs/manpage.html#-L)
481(`-L`) and `--data`/`--form` together. curl only uses POST in the first
482request, and then revert to GET in the following operations.
484## Other redirects
486Browsers typically support at least two other ways of redirects that curl
487does not: first the html may contain a meta refresh tag that asks the browser
488to load a specific URL after a set number of seconds, or it may use
489JavaScript to do it.
491# Cookies
493## Cookie Basics
495The way the web browsers do "client side state control" is by using cookies.
496Cookies are names with associated contents. The cookies are sent to the client
497by the server. The server tells the client for what path and hostname it wants
498the cookie sent back, and it also sends an expiration date and a few more
499properties.
501When a client communicates with a server with a name and path as previously
502specified in a received cookie, the client sends back the cookies and their
503contents to the server, unless of course they are expired.
505Many applications and servers use this method to connect a series of requests
506into a single logical session. To be able to use curl in such occasions, we
507must be able to record and send back cookies the way the web application
508expects them. The same way browsers deal with them.
510## Cookie options
512The simplest way to send a few cookies to the server when getting a page with
513curl is to add them on the command line like:
515 curl --cookie "name=Daniel" https://www.example.com
517Cookies are sent as common HTTP headers. This is practical as it allows curl
518to record cookies by recording headers. Record cookies with curl by
519using the [`--dump-header`](https://curl.se/docs/manpage.html#-D) (`-D`)
520option like:
522 curl --dump-header headers_and_cookies https://www.example.com
524(Take note that the
525[`--cookie-jar`](https://curl.se/docs/manpage.html#-c) option described
526below is a better way to store cookies.)
528curl has a full blown cookie parsing engine built-in that comes in use if you
529want to reconnect to a server and use cookies that were stored from a
530previous connection (or hand-crafted manually to fool the server into
531believing you had a previous connection). To use previously stored cookies,
532you run curl like:
534 curl --cookie stored_cookies_in_file https://www.example.com
536curl's "cookie engine" gets enabled when you use the
537[`--cookie`](https://curl.se/docs/manpage.html#-b) option. If you only
538want curl to understand received cookies, use `--cookie` with a file that
539does not exist. Example, if you want to let curl understand cookies from a
540page and follow a location (and thus possibly send back cookies it received),
541you can invoke it like:
543 curl --cookie nada --location https://www.example.com
545curl has the ability to read and write cookie files that use the same file
546format that Netscape and Mozilla once used. It is a convenient way to share
547cookies between scripts or invokes. The `--cookie` (`-b`) switch
548automatically detects if a given file is such a cookie file and parses it,
549and by using the `--cookie-jar` (`-c`) option you make curl write a new
550cookie file at the end of an operation:
552 curl --cookie cookies.txt --cookie-jar newcookies.txt \
553 https://www.example.com
555# HTTPS
557## HTTPS is HTTP secure
559There are a few ways to do secure HTTP transfers. By far the most common
560protocol for doing this is what is generally known as HTTPS, HTTP over
561SSL. SSL encrypts all the data that is sent and received over the network and
562thus makes it harder for attackers to spy on sensitive information.
564SSL (or TLS as the current version of the standard is called) offers a set of
565advanced features to do secure transfers over HTTP.
567curl supports encrypted fetches when built to use a TLS library and it can be
568built to use one out of a fairly large set of libraries - `curl -V` shows
569which one your curl was built to use (if any). To get a page from an HTTPS
570server, run curl like:
572 curl https://secure.example.com
574## Certificates
576In the HTTPS world, you use certificates to validate that you are the one you
577claim to be, as an addition to normal passwords. curl supports client- side
578certificates. All certificates are locked with a passphrase, which you need
579to enter before the certificate can be used by curl. The passphrase can be
580specified on the command line or if not, entered interactively when curl
581queries for it. Use a certificate with curl on an HTTPS server like:
583 curl --cert mycert.pem https://secure.example.com
585curl also tries to verify that the server is who it claims to be, by
586verifying the server's certificate against a locally stored CA cert bundle.
587Failing the verification causes curl to deny the connection. You must then
588use [`--insecure`](https://curl.se/docs/manpage.html#-k) (`-k`) in case you
589want to tell curl to ignore that the server cannot be verified.
591More about server certificate verification and CA cert bundles can be read in
592the [`SSLCERTS` document](https://curl.se/docs/sslcerts.html).
594At times you may end up with your own CA cert store and then you can tell
595curl to use that to verify the server's certificate:
597 curl --cacert ca-bundle.pem https://example.com/
599# Custom Request Elements
601## Modify method and headers
603Doing fancy stuff, you may need to add or change elements of a single curl
604request.
606For example, you can change the POST method to `PROPFIND` and send the data
607as `Content-Type: text/xml` (instead of the default `Content-Type`) like
608this:
610 curl --data "<xml>" --header "Content-Type: text/xml" \
611 --request PROPFIND example.com
613You can delete a default header by providing one without content. Like you
614can ruin the request by chopping off the `Host:` header:
616 curl --header "Host:" https://www.example.com
618You can add headers the same way. Your server may want a `Destination:`
619header, and you can add it:
621 curl --header "Destination: nowhere" https://example.com
623## More on changed methods
625It should be noted that curl selects which methods to use on its own
626depending on what action to ask for. `-d` makes a POST, `-I` makes a HEAD and
627so on. If you use the [`--request`](https://curl.se/docs/manpage.html#-X) /
628`-X` option you can change the method keyword curl selects, but you do not
629modify curl's behavior. This means that if you for example use -d "data" to
630do a POST, you can modify the method to a `PROPFIND` with `-X` and curl still
631thinks it sends a POST. You can change the normal GET to a POST method by
632adding `-X POST` in a command line like:
634 curl -X POST https://example.org/
636curl however still acts as if it sent a GET so it does not send any request
637body etc.
639# Web Login
641## Some login tricks
643While not strictly HTTP related, it still causes a lot of people problems so
644here's the executive run-down of how the vast majority of all login forms work
645and how to login to them using curl.
647It can also be noted that to do this properly in an automated fashion, you
648most certainly need to script things and do multiple curl invokes etc.
650First, servers mostly use cookies to track the logged-in status of the
651client, so you need to capture the cookies you receive in the responses.
652Then, many sites also set a special cookie on the login page (to make sure
653you got there through their login page) so you should make a habit of first
654getting the login-form page to capture the cookies set there.
656Some web-based login systems feature various amounts of JavaScript, and
657sometimes they use such code to set or modify cookie contents. Possibly they
658do that to prevent programmed logins, like this manual describes how to...
659Anyway, if reading the code is not enough to let you repeat the behavior
660manually, capturing the HTTP requests done by your browsers and analyzing the
661sent cookies is usually a working method to work out how to shortcut the
662JavaScript need.
664In the actual `<form>` tag for the login, lots of sites fill-in
665random/session or otherwise secretly generated hidden tags and you may need
666to first capture the HTML code for the login form and extract all the hidden
667fields to be able to do a proper login POST. Remember that the contents need
668to be URL encoded when sent in a normal POST.
670# Debug
672## Some debug tricks
674Many times when you run curl on a site, you notice that the site does not
675seem to respond the same way to your curl requests as it does to your
676browser's.
678Then you need to start making your curl requests more similar to your
679browser's requests:
681- Use the `--trace-ascii` option to store fully detailed logs of the requests
682 for easier analyzing and better understanding
684- Make sure you check for and use cookies when needed (both reading with
685 `--cookie` and writing with `--cookie-jar`)
687- Set user-agent (with [`-A`](https://curl.se/docs/manpage.html#-A)) to
688 one like a recent popular browser does
690- Set referer (with [`-E`](https://curl.se/docs/manpage.html#-E)) like
691 it is set by the browser
693- If you use POST, make sure you send all the fields and in the same order as
694 the browser does it.
696## Check what the browsers do
698A good helper to make sure you do this right, is the web browsers' developers
699tools that let you view all headers you send and receive (even when using
700HTTPS).
702A more raw approach is to capture the HTTP traffic on the network with tools
703such as Wireshark or tcpdump and check what headers that were sent and
704received by the browser. (HTTPS forces you to use `SSLKEYLOGFILE` to do
705that.)
Diagnostics4 git calls · 40 ms
CallStatusSizeTime040 ms
discover200152 B2 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200149 B · hit2 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=2
git.response.bytes=149
git.response.cache=HIT
git.response.duration_ms=2
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=149
git.pkt.payload.max=78
git.ls_refs.count=2
git.ls_refs.head=refs/heads/master
fetchblob:none · deepen 10200287 KiB · hit4 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=33
git.response.bytes=293470
git.response.cache=HIT
git.response.duration_ms=4
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=31c911efe5652725e930f4d2e60c9060e5f6646d
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=10
git.pkt.data=8
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=293470
git.pkt.payload.max=65516
git.sideband.band1.bytes=293354
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=87
git.pack.bytes=293354
git.pack.ofs_delta=0
git.pack.ref_delta=1
git.pack.chain.max=1
git.pack.trailer=6a646bcda6740bab42e321b51998b52ea15146c0
git.pack.full=86
Open pack map
fetch20011 KiB · hit34 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=11093
git.response.cache=HIT
git.response.duration_ms=34
git.fetch.wants=7e5f2d45f62fbf42cf7f4d57bc0f0628dcf018ec
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=11093
git.pkt.payload.max=11072
git.sideband.band1.bytes=11071
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/curl.git
Ref
refs/heads/master
Commit
31c911efe5652725e930f4d2e60c9060e5f6646d
cf-ray
a5e2182e3e85ecd4
Request
42ff181a…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format