User Tools

Site Tools


linux:sysadmin:cmd:wget

wget

GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

manually setting the host header

Setting the host header manually is handy on web servers with many virtual host distinguished by host header or to verify load balanced web servers.

Example:

wget --header="Host: example.com" http://localhost/images/products/prod-1.jpg

Use wget over a proxy

First you have to register the web proxy address in your environment:

export http_proxy="http://webproxy.example.net:8084"

Make a request a specify proxy-user and password:

wget http://example.com --proxy-user=wolfgran --proxy-password=my_secret

use the following command line options for wget to activate/deactivate a proxy:

  • proxy=on
  • -proxy=off

See server response (header)

Handy for debugging, i.e. caching.

Example:

> wget -S http://www.google.com
--13:12:10--  http://www.google.com/
Resolving www.google.com... 74.125.77.103, 74.125.77.147, 74.125.77.99, ...
Connecting to www.google.com|74.125.77.103|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.0 302 Found
  Location: http://www.google.co.uk/
  Cache-Control: private
  Content-Type: text/html; charset=UTF-8
  Set-Cookie: PREF=ID=d2f4442e70e161fa:TM=1238505130:LM=1238505130:S=BrTYAtqvvpIgualX; expires=Thu, 31-Mar-2011 13:12:10 GMT; path=/; domain=.google.com
  Date: Tue, 31 Mar 2009 13:12:10 GMT
  Server: gws
  Content-Length: 221
  Connection: Keep-Alive
Location: http://www.google.co.uk/ [following]
--13:12:10--  http://www.google.co.uk/
Resolving www.google.co.uk... 74.125.77.103, 74.125.77.147, 74.125.77.99, ...
Reusing existing connection to www.google.com:80.
HTTP request sent, awaiting response...
  HTTP/1.0 200 OK
  Cache-Control: private, max-age=0
  Date: Tue, 31 Mar 2009 13:12:10 GMT
  Expires: -1
  Content-Type: text/html; charset=ISO-8859-1
  Set-Cookie: PREF=ID=d7882aa33babbe94:TM=1238505130:LM=1238505130:S=E1VXZTa5lCyVnKzP; expires=Thu, 31-Mar-2011 13:12:10 GMT; path=/; domain=.google.co.uk
  Server: gws
Length: unspecified [text/html]
Saving to: `index.html'
    [ <=>                                                                         ] 5,840       --.-K/s   in 0.001s
13:12:10 (8.46 MB/s) - `index.html' saved [5840]
/srv/wiki.niwos.com/data/pages/linux/sysadmin/cmd/wget.txt · Last modified: 2009/08/15 12:14 (external edit)