others-How to use 'curl' to do some common jobs?
1. Purpose
In this post, I would introduce some tricks of using curl, which is a very useful tool to do some http/https testing.
2. The tricks of curl
2.1 What’s the difference of -v and -vvv when using curl?
There is no difference between the two options -v and -vvv when using curl, but they do have some difference in memchaced:
-v verbose (print errors/warnings while in event loop)
-vv very verbose (also print client commands/reponses)
-vvv extremely verbose (also print internal state transitions)
According to the code of curl, there is no difference because they are all translated to the option TRACE_PLAIN:
case 'v':
if(toggle) {
/* the '%' thing here will cause the trace get sent to stderr */
Curl_safefree(global->trace_dump);
global->trace_dump = strdup("%");
if(!global->trace_dump)
return PARAM_NO_MEM;
if(global->tracetype && (global->tracetype != TRACE_PLAIN))
warnf(config,
"-v, --verbose overrides an earlier trace/verbose option\n");
global->tracetype = TRACE_PLAIN;
}
else
/* verbose is disabled here */
global->tracetype = TRACE_NONE;
break;
2.2 How to follow the redirects of the URLs when using curl?
You can do the auto-redirects with curl by using the option -L:
(HTTP) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place. If used together with -i, --include or -I, --head, headers from all requested pages will be shown. When authentication is used, curl only sends its credentials to the initial host. If a redirect takes curl to a different host, it will not be able to intercept the user+password. See also --location-trusted on how to change this. You can limit the amount of redirects to follow by using the --max-redirs option.
When curl follows a redirect and if the request is a POST, it will send the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.
You can tell curl to not change POST requests to GET after a 30x response by using the dedicated options for that: --post301, --post302 and --post303.
The method set with -X, --request overrides the method curl would otherwise select to use.
Example:
curl -L https://www.bswen.com
2.3 How to show HTTP request/response headers when using curl?
We can use the -v option to view the http request header:
curl -v https://httpbin.org/get
We can use the -i parameter to see the response header of the request URL.
curl -i https://www.bswen.com
We can test the curl -i option as follows:
➜ ~ curl -i https://www.bswen.com
HTTP/2 200
date: Sat, 04 Dec 2021 09:51:13 GMT
content-type: text/html; charset=utf-8
last-modified: Fri, 03 Dec 2021 12:59:34 GMT
cache-control: private, max-age=0, proxy-revalidate, no-store, no-cache, must-revalidate
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cf-cache-status: DYNAMIC
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
<!DOCTYPE html>
<html lang="en">
...
If you just want to view the response header instead of the whole response content, you can use -I option as follows:
➜ ~ curl -I https://www.bswen.com
If you only want to view the headers of http request and response, you can do as follows:
curl -v -I -H "Test_header: Test header so you see this works" THE_URL
For example:
➜ ~ curl -v -I -H "Test_header: Test header so you see this works" https://httpbin.org/get
* Trying 3.216.167.140...
* TCP_NODELAY set
* Connected to httpbin.org (3.216.167.140) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=httpbin.org
* start date: Nov 21 00:00:00 2021 GMT
* expire date: Dec 19 23:59:59 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8713809600)
> HEAD /get HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.64.1
> Accept: */*
> Test_header: Test header so you see this works
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
HTTP/2 200
< date: Sat, 04 Dec 2021 12:48:08 GMT
date: Sat, 04 Dec 2021 12:48:08 GMT
< content-type: application/json
content-type: application/json
< content-length: 310
content-length: 310
< server: gunicorn/19.9.0
server: gunicorn/19.9.0
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-allow-credentials: true
access-control-allow-credentials: true
<
* Connection #0 to host httpbin.org left intact
* Closing connection 0
➜ ~
You can see that our custom header test_header is incorporated in the request.
2.4 How to add or append header to http requests when using curl?
Curl can add HTTP request headers through -H key:value. To set multiple request headers, you can add multiple -H parameters.
curl -H 'Accept-Language: en-US' https://www.bswen.com
Or
curl -v -I -H 'User-Agent: my-user-agent' https://www.bswen.com
You can check as follows:
➜ ~ curl -v -I -H "Test_header: Test header so you see this works" https://httpbin.org/get
* Trying 3.216.167.140...
* TCP_NODELAY set
* Connected to httpbin.org (3.216.167.140) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=httpbin.org
* start date: Nov 21 00:00:00 2021 GMT
* expire date: Dec 19 23:59:59 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8713809600)
> HEAD /get HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.64.1
> Accept: */*
> Test_header: Test header so you see this works
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
HTTP/2 200
< date: Sat, 04 Dec 2021 12:48:08 GMT
date: Sat, 04 Dec 2021 12:48:08 GMT
< content-type: application/json
content-type: application/json
< content-length: 310
content-length: 310
< server: gunicorn/19.9.0
server: gunicorn/19.9.0
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-allow-credentials: true
access-control-allow-credentials: true
<
* Connection #0 to host httpbin.org left intact
* Closing connection 0
➜ ~
2.5 How to send JSON when using curl?
Now JSON is a very popular data format. When making a request using curl, you may want to send data in JSON format. In this case, you need to use the -H parameter to set the Content-Type request header.
curl -d '{"key1": "value1", "key2": "value2"}' -H "Content-Type: application/json" -X POST https://httpbin.org/post
We got this response:
➜ ~ curl -d '{"key1": "value1", "key2": "value2"}' -H "Content-Type: application/json" -X POST https://httpbin.org/post
{
"args": {},
"data": "{\"key1\": \"value1\", \"key2\": \"value2\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "36",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=1-61ab3bed-5e08355c7010494d49a1dff7"
},
"json": {
"key1": "value1",
"key2": "value2"
},
"origin": "121.225.152.1",
"url": "https://httpbin.org/post"
}
2.6 How to send JSON from file when using curl?
You can send JSON data from file using curlas follows:
curl -X POST -H "Content-Type: application/json" -d @FILENAME DESTINATION
For example, Let’s say there is file named package.json which has the following content:
➜ ~ cat package.json
{
"dependencies": {
"create-react-app": "^4.0.3",
"express": "^4.17.1",
"socket.io": "^4.1.3"
}
}
➜ ~
We can then send its content as JSON http request to remote API like this:
curl -X POST -H "Content-Type: application/json" -d @./package.json https://httpbin.org/post
We got this result:
➜ ~ curl -X POST -H "Content-Type: application/json" -d @./package.json https://httpbin.org/post
{
"args": {},
"data": "{ \"dependencies\": { \"create-react-app\": \"^4.0.3\", \"express\": \"^4.17.1\", \"socket.io\": \"^4.1.3\" }}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "107",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=1-61ab3d51-2cd981227e648e1a3266c5f9"
},
"json": {
"dependencies": {
"create-react-app": "^4.0.3",
"express": "^4.17.1",
"socket.io": "^4.1.3"
}
},
"origin": "121.225.152.219",
"url": "https://httpbin.org/post"
}
➜ ~
You can see that our json file is incorporated into the http request.
2.7 How to do basic authentication when using curl?
If the target URL requires HTTP Basic Authentication, you can pass username:password through the -u parameter to authenticate:
curl -u user:pass http://httpbin.org/basic-auth/user/pass
Here is the example:
➜ ~ curl -u user:pass http://httpbin.org/basic-auth/user/pass
{
"authenticated": true,
"user": "user"
}
➜ ~
2.8 How to set referer of the http request when using curl?
Use the -e parameter to set the Referer of the HTTP request header, indicating the source of the request.
curl -e 'https://www.bswen.com' https://httpbin.org/get
We can check as follows:
➜ ~ curl -v -I -e 'https://bswen.com' https://httpbin.org/get
* Trying 3.216.167.140...
* TCP_NODELAY set
* Connected to httpbin.org (3.216.167.140) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=httpbin.org
* start date: Nov 21 00:00:00 2021 GMT
* expire date: Dec 19 23:59:59 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe928809600)
> HEAD /get HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.64.1
> Accept: */*
> Referer: https://bswen.com
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
HTTP/2 200
< date: Sat, 04 Dec 2021 12:58:13 GMT
date: Sat, 04 Dec 2021 12:58:13 GMT
< content-type: application/json
content-type: application/json
< content-length: 294
content-length: 294
< server: gunicorn/19.9.0
server: gunicorn/19.9.0
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-allow-credentials: true
access-control-allow-credentials: true
<
* Connection #0 to host httpbin.org left intact
* Closing connection 0
➜ ~
You can see that our newly added referer is set in the request.
2.9 How to set bandwidth limit or rate limit of http requests when using curl?
By default, curl uses the maximum available bandwidth, but usually we need to slow down for testing. You can use --limit-rate to limit the bandwidth of curl’s request and response to slow down the request and response.
curl --limit-rate 300k https://httpbin.org/get
The above command limits curl to 300K bytes per second.
2.10 How to do url encode when using curl?
When initiating a GET request, we may need to follow the URL with query parameters. If the data requires URL encoding, you can use the –data–urlencode parameter in combination. Just as follows:
curl -G --data-urlencode 'q=CURL 博思闻bswen' https://www.google.com/search
For test:
➜ ~ curl -G --data-urlencode 'q=CURL 博思闻bswen' -v -I https://httpbin.org/get
* Uses proxy env variable https_proxy == 'http://127.0.0.1:7890'
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 7890 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.64.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
HTTP/1.1 200 Connection established
< Content-Length: 0
Content-Length: 0
* Ignoring Content-Length in CONNECT 200 response
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=httpbin.org
* start date: Nov 21 00:00:00 2021 GMT
* expire date: Dec 19 23:59:59 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc9ba008200)
> HEAD /get?q=CURL%20%E5%8D%9A%E6%80%9D%E9%97%BBbswen HTTP/2
> Host: httpbin.org
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
HTTP/2 200
< date: Sat, 04 Dec 2021 13:10:02 GMT
date: Sat, 04 Dec 2021 13:10:02 GMT
< content-type: application/json
content-type: application/json
< content-length: 330
content-length: 330
< server: gunicorn/19.9.0
server: gunicorn/19.9.0
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-allow-credentials: true
access-control-allow-credentials: true
<
* Connection #0 to host 127.0.0.1 left intact
* Closing connection 0
➜ ~
You can see that our query parameters are encoded as follows:
> HEAD /get?q=CURL%20%E5%8D%9A%E6%80%9D%E9%97%BBbswen HTTP/2
3. Summary
In this post, I demonstrated several common usecases when using curl. That’s it, thanks for your reading.