wget [option]... [URL]...

Non-interactive download of files from the Web, supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

Works in the background, allows running from cron

Wget can follow links in HTML pages and create local versions of remote web sites, fully recreating the directory structure of the original site. This is sometimes referred to as recursive downloading.

designed for robustness over slow or unstable network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved. If the server supports regetting, it will instruct the server to continue the download from where it left off.

examples

options

Logging and Input File Options
-o logfile --output-file=logfile	Log messages to logfile. Default STDERR
-a logfile --append-output=logfile
-i file --input-file=file	Read URLs from file URLs on the command line are retrieved first. The file need not be an HTML document .
-F --force-html	When input is read from a file, force it to be treated as an HTML file. This enables retrieval of relative links from existing HTML files on local system, by adding <base href="url"> to HTML, or using --base
-B URL --base=URL	When used in conjunction with -F, prepends URL to relative links in the file specified by -i.
Download Options
-O file --output-document=file	documents will be concatenated to file or STDOUT if - is specified. Sets tries to 1.
-nc --no-clobber	Suppresses creating versions of duplicate files named ffff.n Newer copies of file are not retrieved. -nc may not be specified with --timestamping. Handling a file downloaded more than once in the same directory: Without --timestamping, --noclobber, or --recursive, downloading the same file in the same directory will result in the original copy of file being preserved and the second copy being named ffff.n. With --recursive , but without --timestamping or --noclobber, the last file download is retained. With --timestamping download only newer version of a file When -nc is specified, files with the suffixes .html or .htm will be loaded from the local disk and parsed as if they had been retrieved from the Web.
-N --timestamping	Turn on time-stamping.
-c --continue	Continue a partially-downloaded file. Don't specify this option to retry downloading a file should the connection be lost midway through. This is the default. Only effects resumption of downloads started prior to this invocation of Wget, and whose local files are present. A local file that's smaller than the server one will be considered part of an incomplete download and only "(length(remote) - length(local))" bytes will be downloaded and appended. The server must support continued downloading via the Range header. Using -c with -r, will interpret every file as an "incomplete download" . A garbled file will result HTTP the a proxy that inserts a transfer interrupted string into the local file.
--progress= dot[:style] | bar	default bar . draws an ASCII progress bar (aka thermometer display) indicating the status of retrieval. If the output is not a TTY, the dot bar will be used by default. style dot dots per cluster dots in a line. line Default 1K 10 50 binary 8K 16 48 384K mega 64K 8 48 3M progress in .wgetrc is overridden from the command line, unless the output is not a TTY, the dot progress will be favored over bar. To force the bar output, use --progress=bar:force.
--spider	Pages only checked not downloaded. Useful for checking bookmarks. wget --spider --force-html -i bookmarks.html
--limit-rate=Bps[k|m]	Expressed in bytes (default), kilobytes or megabytes. Example: --limit-rate=20k Implemented by sleeping an appropriate amount of time after network reads. Does not to work with very small files. Specifing bandwidth less than KBps may be ineffecive.
-w seconds --wait=seconds	between retrievals. minutes m suffix, hours use h , or days d. Specifying a large value is useful if the network or the destination host is down. Wget can wait long enough to reasonably expect the network error to be fixed before the retry.
--random-wait	causes the time between requests to vary between 0 and 2 * wait seconds specified using the --wait to mask Wget's presence from analysis.
--proxy=on|off, -Y onoff	On by default if the appropriate environmental variable is defined.
-Q quota --quota=quota	specified in bytes , kilobytes k suffix, or megabytes ( m). Will not affect downloading a single file. quota is respected when retrieving either recursively, or from an input file. Example: wget -Q2m -i sites. Download will be aborted when the quota is reached. Setting quota to 0 or inf specifies NO limit.
--bind-address=ADDRESS	When making client TCP/IP connections, "bind()" to ADDRESS on the local machine. ADDRESS may be specified as a hostname or IP address. Use when wget's host has multiple IPs.
-t number --tries=number	Specify 0 or inf for infinite retrying.
--waitretry=attempts	only wait between retries of failed downloads. Wget will use linear backoff, waiting 1 second after the first failure on a given file, then waiting 2 seconds after the second failure on that file, up to the maximum number of attempts . Therefore, a value of 10 will have Wget wait up to (1 + 2 + ... + 10) = 55 seconds per file. On by default in the global wgetrc file.
-T seconds --timeout=seconds	read timeout. When a network read is issued, the file descriptor is checked for a timeout, which could otherwise leave a pending connection (uninterrupted read). default 900 seconds (fifteen minutes). Setting timeout to 0 will disable checking for timeouts.CAUTION!
Directory Options
-nd --no-directories	Do not create a hierarchy of directories when retrieving recursively, files will be saved to the current directory, without clobbering (if a name shows up more than once, the filenames will get extensions .n).
-x --force-directories	The opposite of -nd---create a hierarchy of directories, even if one would not have been created otherwise. E.g. wget -x http://fly.srk.fer.hr/robots.txt will save the downloaded file to fly.srk.fer.hr/robots.txt.
-nH --no-host-directories	Disable generation of host-prefixed directories. By default, invoking Wget with -r http://fly.srk.fer.hr/ will create a structure of directories beginning with fly.srk.fer.hr/ . This option disables such behavior.
--cut-dirs=number	Ignore number directory components. This is useful for getting a fine-grained control over the directory where recursive retrieval will be saved. For example, the directory at ftp://ftp.xemacs.org/pub/xemacs/. If you retrieve it with -r, it will be saved locally under ftp.xemacs.org/pub/xemacs/. While the -nH option can remove the ftp.xemacs.org/ part, you get pub/xemacs. Using --cut-dirs makes Wget not see number remote directory components. No options -> ftp.xemacs.org/pub/xemacs/ -nH -> pub/xemacs/ -nH --cut-dirs=1 -> xemacs/ -nH --cut-dirs=2 -> . --cut-dirs=1 -> ftp.xemacs.org/xemacs/ ... To supress the directory structure, this option is similar to a combination of -nd and -P. However, unlike -nd, --cut-dirs does not lose with subdirectories---for instance, with -nH --cut-dirs=1, a beta/ subdirectory will be placed to xemacs/beta, as one would expect.
-P prefix --directory-prefix=prefix	The directory prefix is the directory where all other files and subdirectories will be saved to, i.e. the top of the retrieval tree. The default is . (the current directory).
HTTP Options
-E --html-extension	If a file of type text/html is downloaded and the URL does not end with the regexp \.[Hh][Tt][Mm][Ll]?, .html will be appended to the local filename. Use when mirroring a remote site that uses .asp or CGIs. URL http://site.com/article.cgi?25 will be saved as article.cgi?25.html. WARNING: filenames changed in this way will be re-downloaded every time you re-mirror a site. To prevent this use --convert-links and --backup-converted so that the original version of the file will be saved as X.orig.
-s --save-headers	before the actual contents, with an empty line as the separator.
-C on|off --cache=on|off	If off Wget sends Pragma:nocache to disable server-side cache. Used to retrieving out-of-date documents on proxy servers. default:on.
--cookies=on|off
--load-cookies file	Load cookies from file before the first HTTP retrieval. file in the format originally used by Netscape's cookies.txt file. Use this option when mirroring sites that require that you be logged in to access their content. The login process typically works by the web server issuing an HTTP cookie upon receiving and verifying your credentials. The cookie is then resent by the browser when accessing that part of the site, and so sets your identity. Mirroring such a site requires Wget to send the same cookies your browser sends when communicating with the site. This is achieved by --load-cookies---simply point Wget to the location of the cookies.txt file, and it will send the same cookies your browser would send in the same situation. Different browsers keep textual cookie files in different locations: Netscape 4.x. The cookies are in ~/.netscape/cookies.txt. Mozilla and Netscape 6.x. Mozilla's cookie file is also named cookies.txt, located some where under ~/.mozilla, in the directory of your profile. The full path usually ends up looking somewhat like ~/.mozilla/default/some-weird-string/cookies.txt. Internet Explorer. You can produce a cookie file Wget can use by using the File menu, Import and Export, Export Cookies. Other browsers. If you are using a different browser to create your cookies, --load-cookies will only work if you can locate or produce a cookie file in the Netscape format that Wget expects. If you cannot use --load-cookies, there might still be an alternative. If your browser supports a cookie manager, you can use it to view the cookies used when accessing the site you're mirroring. Write down the name and value of the cookie, and manually instruct Wget to send those cookies, bypassing the official cookie support: wget --cookies=off --header "Cookie: I<name>=I<value>"
--save-cookies file	Cookies whose expiry time is not specified, or those that have already expired, are not saved.
--ignore-length	CGI programs send out incorrent "Content-Length" headers.
--header=additional-header	passed to the HTTP servers. Headers must contain a : preceded by one or more non-blank characters, and must not contain newlines. Define more than one additional header by specifying --header more than once. wget --header='Accept-Charset: iso-8859-2' \ --header='Accept-Language: hr' \ http://fly.srk.fer.hr/ Specification of an empty string as the header value will clear all previous user-defined headers.
--proxy-user=user --proxy-passwd=password	for authentication on a proxy server. Encode using the "basic" authentication scheme.
--http-user=user
--http-passwd=password	According to the type of the challenge, Wget will encode them using either the "basic" (insecure) or the "digest" authentication scheme. Another way to specify username and password is in the URL itself. Either method reveals your password to anyone who bothers to run "ps". To prevent the passwords from being seen, store them in .wgetrc or .netrc, and make sure to protect those files from other users with "chmod". If the passwords are really important, do not leave them lying in those files either---edit the files and delete them after Wget has started the download.
--referer=url	Include 'Referer: url' header in HTTP request. Useful for retrieving documents with server-side processing that assume they are always being retrieved by interactive web browsers and only come out properly when Referer is set to one of the pages that point to them.
-U agent-string --user-agent=agent-string	Identify as agent-string to the HTTP server. The HTTP protocol allows the clients to identify themselves using a "User-Agent" header field. This enables distinguishing the WWW software, usually for statistical purposes or for tracing of protocol violations. Wget normally identifies as Wget/version. Some sites tailor the output according to the "User-Agent". This has been abused by servers denying information to clients other than "Mozilla" or Microsoft "Internet Explorer". FTP Options
-nr --dont-remove-listing	Don't remove temporary .listing files generated by FTP retrievals containing the raw directory listings. Useful for debugging , or when you want to be able to easily check on the contents of remote server directories (e.g. to verify that a mirror you're running is complete). This is not a security hole in the scenario of a user making .listing a symbolic link to /etc/passwd or something and asking "root" to run Wget in his directory. Depending on the options used, either Wget will refuse to write to .listing, making the globbing/recursion/time-stamping operation fail, or the symbolic link will be deleted and replaced with the actual .listing file, or the listing will be written to a .listing.number file. root should never run Wget in a non-trusted user's directory. A user could do something as simple as linking index.html to /etc/passwd and asking "root" to run Wget with --timestamping or -r so the file will be overwritten.
-g on|off --glob=on|off	Use the shelllike special characters (wildcards), like *, ?, [ and ] to retrieve more than one file wget ftp://gnjilux.srk.fer.hr/*.msg on by default. Quote the URL to protect it from being expanded by the shell. Globbing makes Wget look for a directory listing, which is system-specific. This is why it currently works only with Unix FTP servers (and the ones emulating Unix "ls" output).
--passive-ftp	the client initiates the data connection.
--retr-symlinks	When retrieving FTP directories recursively and a symbolic link is encountered, the linked-to file is not downloaded. Instead, a matching symbolic link is created on the local filesystem. The pointed-to file will not be downloaded unless this recursive retrieval would have encountered it separately and downloaded. When --retr-symlinks is specified, symbolic links are traversed and the pointed-to files are retrieved. This option does not cause Wget to traverse symlinks to directories and recurse through them. Retrieving a file (not a directory) specified on the commandline, rather than because it was recursed to, this option has no effect. Symbolic links are always traversed in this case. Recursive Retrieval Options
-r --recursive	Turn on recursive retrieving.
-l depth --level=depth	default is 5.
--delete-after	useful for pre-fetching popular pages through a proxy, e.g.: wget -r -nd --delete-after http://whatever.com/~popular/page/ The -r option is to retrieve recursively, and -nd to not create directories. --delete-after deletes files on the local machine. It	does not issue the DELE command to remote FTP sites. when --delete-after is specified, --convert-links is ignored, so .orig files are not .
-k --convert-links	to make them suitable for local viewing. This affects not only the visible hyperlinks, but any part of the document that links to external content, such as embedded images, links to style sheets, hyperlinks to non-HTML content, etc. Each link will be changed in one of the two ways: The links to files that have been downloaded by Wget will be changed to refer to the file they point to as a relative link. Example: if the downloaded file /foo/doc.html links to /bar/img.gif, also downloaded, then the link in doc.html will be modified to point to ../bar/img.gif. This kind of transformation works reliably for arbitrary combinations of directories. The links to files that have not been downloaded by Wget will be changed to include host name and absolute path of the location they point to. Example: if the downloaded file /foo/doc.html links to /bar/img.gif (or to ../bar/img.gif), then the link in doc.html will be modified to point to http://hostname/bar/img.gif. This makes local browsing work reliably: if a linked file was downloaded, the link will refer to its local name; if it was not downloaded, the link will refer to its full Internet address rather than presenting a broken link. The fact that the former links are converted to relative links ensures that you can move the downloaded hierarchy to another directory. only at the end of the download can Wget know which links have been downloaded. Because of that, the work done by -k will be performed at the end of all the downloads.
-K --backup-converted	When converting a file, back up the original version with a .orig suffix. Affects the behavior of --timestamping.
-m --mirror	Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. equivalent to -r --timestamping --level inf -nr.
-p --page-requisites	Download files necessary to display the HTML page including inlined images, sounds, and referenced stylesheets. When downloading a single HTML page, requisite documents that may be needed not downloaded. Using --r together with --level can help. Wget does not distinguish between external and inlined documents, one is left with leaf documents that are missing their requisites. For example, document 1.html contains an <IMG> referencing 1.gif and an <A> pointing to external document 2.html. 2.html has image 2.gif and it links to 3.html with image 3.gif. When executing: wget -recursive --level 2 http://I<site>/1.html 1.html, 1.gif, 2.html, 2.gif, and 3.html will be downloaded. but, 3.gif is not downloaded because the level (up to 2) away from 1.html in order to determine where to stop the recursion. Instead: wget -r --level 2 --page-requisites http://I<site>/1.html all the above files and 3.html's requisite 3.gif will be downloaded. Similarly, wget -r --level 1 --page-requisites http://I<site>/1.html will cause 1.html, 1.gif, 2.html, and 2.gif to be downloaded. wget -r --level 0 --page-requisites http://I<site>/1.html Does not download just 1.html and 1.gif, because -l 0 is equivalent to -l inf, that is, infinite recursion. To download a single HTML page, specified on the commandline or in a -i URL input file and its requisites, omit -r and -l: wget -p http://I<site>/1.html Wget will behave as if -r had been specified, but only that single page and its requisites will be downloaded. Links from that page to external documents will not be followed. To download a single page and all its requisites (even if they exist on separate websites),the following is suggested: wget --html-extension --span-hosts --convert-links --backup-converted --page-requisites http://I<site>/I<document> An external document link is any URL in an <A>, <AREA> or a <LINK> tag other than <LINK REL="stylesheet">.
Recursive Accept/Reject Options
-A acclist --accept acclist -R rejlist --reject rejlist	Comma-separated lists of file name suffixes or patterns to accept or reject.
--follow-tags=list
--ignore-tags=list -G list	Wget has an internal table of HTML tag / attribute pairs that it considers when looking for linked documents during a recursive retrieval. To speciy a subset of tags to be considered, specify them in a comma separated list . To specify tags to be ignored use -G.
-H --span-hosts	Enable spanning across hosts when doing recursive retrieving.
-D list --domains=list	Set domains to be followed. list is a comma-separated list of domains. This does not turn on -H.
--exclude-domains list	Specify the domains not to be followed..
-I list --include-directories=list
-X list --exclude-directories=list	Specify a comma-separated list of directories to follow/exclude when downloading. Elements of list may contain wildcards./exclude
-np --no-parent	Do not ascend to the parent directory when retrieving recursively.
-L --relative	Follow relative links only. Useful for retrieving a specific home page without any distractions, not even those from the same hosts.
--follow-ftp	Follow FTP links from HTML documents. Default: ignore all the FTP links.
-b --background	Go to background immediately after startup. If no output file is specified via the -o, output is redirected to wget-log.
-e command --execute command	Execute command after the commands in .wgetrc.
-q --quiet	Turn off Wget's output.
-v --verbose	Turn on verbose output, with all the available data. default: verbose.
-nv --non-verbose	Turn off verbose without being completely quiet , error messages and basic information get printed.
-S --server-response	Print the headers sent by HTTP servers and responses sent by FTP servers.
-V --version	Display the version of Wget.
-h --help	Print a help message describing all of Wget's command-line options.
-d --debug	if compiled with debug

Respects the Robot Exclusion file (/robots.txt). can convert the links in downloaded HTML files to the local files for offline viewing.

Examples

Simple Usage

1. to download a URL:

   wget http://fly.srk.fer.hr/

   if the connection is slow, and the file is lengthy, The connection may fail before the whole file is retrieved, more than once. Wget will try getting the file until it either gets the whole of it, or exceeds the default number of retries .

2. to work in the background, and write its progress to log file log

   wget --tries 45 -o log http://fly.srk.fer.hr/jpg/flyweb.jpg &

3. For infinite(?) number of retries, use -t inf Caution!.

   wget ftp://gnjilux.srk.fer.hr/welcome.msg

4. If you specify a directory, Wget will retrieve the directory listing, parse it and convert it to HTML. :

   wget ftp://prep.ai.mit.edu/pub/gnu/ links index.html

Advanced Usage

5. a file that contains the URLs to download Use -i :

   wget --input-file=urlfile

   specify - as file name, the URLs will be read from standard input.

6. Create a 5 default --level=n levels deep mirror, with the same directory structure the original has, with only 1 try per document, saving the log of the activities to gnulog:

   wget --recursive http://www.gnu.org/ -o gnulog

7. convertng the links in the HTML files to point to local files, so you can view the documents off-line:

   wget --convert-links -r http://www.gnu.org/ -o gnulog

8. Retrieve only one HTML page, but make sure that all the elements needed for the page to be displayed, such as inline images and external style sheets, are also downloaded. Also make sure the downloaded page references the downloaded links.

   wget -p --convert-links http://www.server.com/dir/page.html

   The HTML page will be saved to www.server.com/dir/page.html, and the images, stylesheets, etc., somewhere under www.server.com/, depending on where they were on the remote server.

9. as the above, without the www.server.com/ directory. save all those files under a download/ subdirectory of the current directory.

   wget -p --convert-links -nH -nd -Pdownload \
   http://www.server.com/dir/page.html

10. Retrieve the index.html of www.lycos.com, showing the original server headers:

    wget -S http://www.lycos.com/

11. Save the server headers with the file, for post-processing.

    wget -s http://www.lycos.com/ more index.html

12. Retrieve the first two levels of wuarchive.wustl.edu, saving them to /tmp.

    wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/

13. download the GIFs from a directory on an HTTP server. wget http://www.server.com/dir/*.gif, didn't work because HTTP retrieval does not support globbing. :

    wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

    -r -l1 means to retrieve recursively, with maximum depth of 1. --no-parent means that references to the parent directory are ignored, and -A.gif means to download only the GIF files. -A "*.gif" would have worked too.

14. interrupted. do not clobber files already present.

    wget -nc -r http://www.gnu.org/

15. encode username and password to HTTP or FTP, use the appropriate URL syntax.

    wget ftp://hniksic:mypassword@unix.server.com/.emacs

    This usage is not advisable on multi-user systems because it reveals your password to anyone who looks at the output of "ps".

16. output documents to standard output instead of to files?

    wget -O - http://jagor.srce.hr/ http://www.srce.hr/

    combine the two options and make pipelines to retrieve the documents from remote hotlists:

    wget -O - http://cool.list.com/ ' wget --force-html -i

Very Advanced Usage

17. keep a mirror of a page (or FTP subdirectories), use --mirror (-m), which is the shorthand for -r -l inf --timestamping. put Wget in the crontab file asking it to recheck a site each Sunday:

    crontab 0 0 * * 0 wget --mirror http://www.gnu.org/ -o /home/me/weeklog

18. links to be converted for local viewing. link conversion doesn't play well with timestamping, also have Wget back up original HTML files before the conversion. :

    wget --mirror --convert-links --backup-converted \ http://www.gnu.org/ -o /home/me/weeklog

19. local viewing doesn't work when HTML files are saved under extensions other than .html, rename all the files served with content-type text/html to name.html.

    wget --mirror --convert-links --backup-converted \ --html-extension -o /home/me/weeklog \ http://www.gnu.org/

    Or, with less typing:

    wget -m -k -K -E http://www.gnu.org/ -o /home/me/weeklog

FILES

BUGS

Before actually submitting a bug report, please try to follow a few simple guidelines.

1. Please try to ascertain that the behaviour you see really is a bug. If Wget crashes, it's a bug. If Wget does not behave as documented, it's a bug. If things work strange, but you are not sure about the way they are supposed to work, it might well be a bug.
2. Try to repeat the bug in as simple circumstances as possible. E.g. if Wget crashes while downloading wget -rl0 -kKE -t5 -Y0 http://yoyodyne.com -o /tmp/log, you should try to see if the crash is repeatable, and if will occur with a simpler set of options. You might even try to start the download at the page where the crash occurred to see if that page somehow triggered the crash. Also, while I will probably be interested to know the contents of your .wgetrc file, just dumping it into the debug message is probably a bad idea. Instead, you should first try to see if the bug repeats with .wgetrc moved out of the way. Only if it turns out that .wgetrc settings affect the bug, mail me the relevant parts of the file.
3. Please start Wget with -d option and send the log (or the relevant parts of it). If Wget was compiled without debug support, recompile it. It is much easier to trace bugs with debug support on.

4. If Wget has crashed, try to run it in a debugger, e.g. "gdb 'which wget' core" and type "where" to get the backtrace.