howtos/topics/curl.md

curl

Transfer data to/from servers — the Swiss army knife of HTTP.

Basics

# Simple GET
curl https://example.com

# Follow redirects
curl -L https://example.com/old-path

# Silent (no progress bar)
curl -s https://api.example.com/data

# Silent but show errors
curl -sS https://api.example.com/data

# Save to file
curl -o output.html https://example.com
curl -O https://example.com/file.tar.gz      # keep remote filename

# Show response headers + body
curl -i https://example.com

# Show headers only
curl -I https://example.com                   # HEAD request
curl -sI https://example.com                  # silent + headers only

# Verbose (debug connection, TLS, headers)
curl -v https://example.com
curl -vvv https://example.com                 # extra verbose

Common Flags

Flag	Effect
-s	Silent (no progress meter)
-S	Show errors even when silent
-L	Follow redirects
-i	Include response headers in output
-I	Headers only (HEAD request)
-v	Verbose (show request/response details)
-o FILE	Write output to file
-O	Save with remote filename
-X METHOD	HTTP method (GET, POST, PUT, DELETE, PATCH)
-H "K: V"	Add request header
-d DATA	POST data (sets method to POST)
-F "k=v"	Multipart form data (file upload)
-u user:pass	Basic auth
-b FILE	Send cookies from file
-c FILE	Save cookies to file
-k	Ignore TLS certificate errors
-w FORMAT	Write-out format string after transfer
--connect-timeout N	Connection timeout in seconds
-m N	Max total time in seconds
--retry N	Retry N times on transient errors
--compressed	Request and decompress gzip/deflate/br

HTTP Methods

# GET (default)
curl https://api.example.com/users

# POST — form-encoded (default content-type with -d)
curl -X POST -d "name=alice&role=admin" https://api.example.com/users

# POST — JSON
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"name": "alice", "role": "admin"}' \
  https://api.example.com/users

# POST — JSON from file
curl -X POST \
  -H "Content-Type: application/json" \
  -d @payload.json \
  https://api.example.com/users

# POST — read body from stdin
echo '{"key":"value"}' | curl -X POST -H "Content-Type: application/json" -d @- https://api.example.com

# PUT
curl -X PUT \
  -H "Content-Type: application/json" \
  -d '{"role": "superadmin"}' \
  https://api.example.com/users/42

# PATCH
curl -X PATCH \
  -H "Content-Type: application/json" \
  -d '{"role": "viewer"}' \
  https://api.example.com/users/42

# DELETE
curl -X DELETE https://api.example.com/users/42

Headers

# Set custom headers
curl -H "Authorization: Bearer $TOKEN" \
     -H "Accept: application/json" \
     -H "X-Request-ID: abc-123" \
     https://api.example.com/data

# Content-Type shortcuts
curl -H "Content-Type: application/json" -d '...'
curl -H "Content-Type: text/xml" -d @request.xml

# User-Agent
curl -A "my-script/1.0" https://example.com

# Referer
curl -e "https://example.com/origin" https://example.com/target

Authentication

# Basic auth
curl -u alice:s3cret https://api.example.com/private

# Basic auth (prompt for password)
curl -u alice https://api.example.com/private

# Bearer token
curl -H "Authorization: Bearer $TOKEN" https://api.example.com

# API key in header
curl -H "X-API-Key: $API_KEY" https://api.example.com

# Netrc file (~/.netrc)
curl -n https://api.example.com
# ~/.netrc format:
#   machine api.example.com login alice password s3cret

# Client certificate (mTLS)
curl --cert client.pem --key client-key.pem https://secure.example.com

File Upload

# Multipart form upload
curl -F "file=@document.pdf" https://example.com/upload
curl -F "file=@photo.jpg;type=image/jpeg" https://example.com/upload

# Multiple files
curl -F "file1=@doc.pdf" -F "file2=@img.png" https://example.com/upload

# File + form fields
curl -F "file=@doc.pdf" -F "title=Report" -F "public=true" https://example.com/upload

# PUT upload (raw file body)
curl -T file.tar.gz https://example.com/upload/file.tar.gz

Download

# Save with remote filename
curl -O https://example.com/archive.tar.gz

# Save to specific path
curl -o /tmp/data.json https://api.example.com/export

# Resume interrupted download
curl -C - -O https://example.com/large-file.iso

# Download multiple files
curl -O https://example.com/file1.txt -O https://example.com/file2.txt

# Rate limit
curl --limit-rate 1M -O https://example.com/large.iso

# Show progress bar (instead of meter)
curl -# -O https://example.com/file.tar.gz

Cookies

# Send cookie
curl -b "session=abc123" https://example.com

# Save cookies from response
curl -c cookies.txt https://example.com/login

# Load and send saved cookies
curl -b cookies.txt https://example.com/dashboard

# Save + send (session flow)
curl -c cookies.txt -b cookies.txt \
  -d "user=alice&pass=secret" \
  https://example.com/login

Response Inspection with -w

# HTTP status code only
curl -s -o /dev/null -w "%{http_code}" https://example.com

# Timing breakdown
curl -s -o /dev/null -w "\
  dns:     %{time_namelookup}s\n\
  connect: %{time_connect}s\n\
  tls:     %{time_appconnect}s\n\
  start:   %{time_starttransfer}s\n\
  total:   %{time_total}s\n\
  size:    %{size_download} bytes\n\
  code:    %{http_code}\n" \
  https://example.com

# Response content-type
curl -s -o /dev/null -w "%{content_type}" https://example.com

# Effective URL after redirects
curl -sL -o /dev/null -w "%{url_effective}" https://short.url/abc

# Write-out to file
curl -s -o /dev/null -w "%{http_code}" -o response.json https://api.example.com

Useful -w Variables

Variable	Value
%{http_code}	HTTP status code
%{time_total}	Total time in seconds
%{time_namelookup}	DNS resolution time
%{time_connect}	TCP connection time
%{time_appconnect}	TLS handshake time
%{time_starttransfer}	Time to first byte (TTFB)
%{size_download}	Downloaded bytes
%{size_upload}	Uploaded bytes
%{url_effective}	Final URL after redirects
%{content_type}	Content-Type header
%{num_redirects}	Number of redirects followed
%{remote_ip}	Server IP address

Proxies

# HTTP proxy
curl -x http://proxy:8080 https://example.com

# SOCKS5 proxy
curl --socks5-hostname localhost:1080 https://example.com

# No proxy for specific hosts
curl --noproxy "localhost,*.internal" https://example.com

# Env variable (respected by curl)
export https_proxy=http://proxy:8080
curl https://example.com

Practical Recipes

# JSON API call with jq
curl -s https://api.example.com/users | jq '.[] | {name, email}'

# Check if endpoint is reachable
curl -sf -o /dev/null https://example.com && echo "up" || echo "down"

# POST JSON, extract field from response
ID=$(curl -s -X POST \
  -H "Content-Type: application/json" \
  -d '{"name":"test"}' \
  https://api.example.com/items | jq -r '.id')

# Retry with backoff
curl --retry 5 --retry-delay 2 --retry-all-errors https://api.example.com

# Send from config file (avoid long command lines)
curl -K request.conf
# request.conf:
#   url = "https://api.example.com"
#   header = "Authorization: Bearer token123"
#   header = "Accept: application/json"
#   silent

# Loop over paginated API
PAGE=1; while :; do
  DATA=$(curl -s "https://api.example.com/items?page=$PAGE&limit=100")
  echo "$DATA" | jq '.items[]'
  [ "$(echo "$DATA" | jq '.items | length')" -lt 100 ] && break
  ((PAGE++))
done

# Parallel downloads with xargs
cat urls.txt | xargs -P 4 -I{} curl -sO {}

Gotchas

• -d implies POST and sets Content-Type: application/x-www-form-urlencoded — add -H for JSON
• -X GET -d '...' sends a body with GET — legal but unusual; some servers reject it
• Without -L, 3xx redirects return the redirect response, not the final destination
• -k disables all TLS verification — never use in production scripts
• -o /dev/null is needed with -w to suppress body when you only want status/timing
• Spaces in URLs must be encoded (%20) or the URL must be quoted
• --data-urlencode handles encoding for form values; plain -d does not
• Shell variables in single-quoted -d '...' won't expand — use double quotes or --data-raw
• -F and -d cannot be mixed in the same request

See Also

• jq — pipe curl JSON output through jq for parsing
• Everything curl — comprehensive book
• curl --help all — full flag reference