curl is most probably the highest used software in the world. I generally use it daily (directly) in the various scripts at the SecureDrop, starting from inside of Dockerfiles, to Ansible roles or in CI. I never read much about various options available other than a few very basic ones. So I decided to look more into the available options. Here are a few interesting points from that reading:

Protocols supported

This is a very long list, I had no clue!!!!

• dict
• file
• ftp
• ftps
• gopher
• gophers
• http
• https
• imap
• imaps
• ldap
• ldaps
• mqtt
• pop3
• pop3s
• rtsp
• scp
• sftp
• smb
• smbs
• smtp
• smtps
• telnet
• tftp

The --version flag will tell you this details and many things more.

$ curl --version
curl 7.76.1 (x86_64-redhat-linux-gnu) libcurl/7.76.1 OpenSSL/1.1.1k-fips zlib/1.2.11 brotli/1.0.9 libidn2/2.3.1 libpsl/0.21.1 (+libidn2/2.3.0) libssh/0.9.5/openssl/zlib nghttp2/1.43.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

User agent string

We can use -A to provide any given string as User Agent.

$ curl -A "browser/kd 1.0.0" http://httpbin.org/get
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "browser/kd 1.0.0", 
    "X-Amzn-Trace-Id": "Root=1-60ec8985-05acbf60769f18855db925f1"
  }, 
  "origin": "xxx.xxx.xxx.xxx", 
  "url": "http://httpbin.org/get"
}

Download via sequence

We can specify the range in our command. The first example downloads different index files from my blog.

curl -O "https://kushaldas.in/index-[1-10].html"

A few more examples:

curl -O "http://example.com/[1-100].png"

curl -O "http://example.com/[001-100].png"

curl -O "http://example.com/[0-100:2].png"

curl -O "http://example.com/section[a-z].html"

curl -O "http://example.com/{one,two,three,alpha,beta}.html"

Parallel downloads

You can pass -Z to the above command to download in parallel. Try it out yourself.

Configuration file

We can create a configuration file to hold all the command line parameters and pass it to the curl command via -K flag.

curl -K configs.txt https://kushaldas.in

One flag each line, comments via #.

# Change user agent
--user-agent "ACAB/1.0"

Curl checks for a default configuration file at ~/.curlrc unless called with -q.

To view only the headers

-I to check the headers returned by the server.

$ curl -I https://kushaldas.in
HTTP/2 200 
server: nginx/1.14.2
date: Sun, 18 Jul 2021 06:40:37 GMT
content-type: text/html; charset=utf-8
content-length: 25977
last-modified: Mon, 05 Jul 2021 15:24:32 GMT
etag: "60e32430-6579"
strict-transport-security: max-age=31536000
onion-location: https://kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion
permissions-policy: interest-cohort=()
x-frame-options: DENY
x-content-type-options: nosniff
referrer-policy: strict-origin
accept-ranges: bytes

DNS over HTTPS

We can force curl to use a DoH server for the DNS query, pass --doh-url flag to the command along with the server address.

curl -I --doh-url https://adblock.doh.mullvad.net/dns-query https://securedrop.org