Solaris OpenCSW
GNU Wget2 is the successor of GNU Wget, a file and recursive website downloader.
Designed and written from scratch it wraps around libwget, that provides the basic functions needed by a web client.
Wget2 works multi-threaded and uses many features to allow fast operation.
In many cases Wget2 downloads much faster than Wget1.x due to HTTP zlib compression, parallel connections and use of If-Modified-Since HTTP header.
GNU Wget2 is licensed under GPLv3+.
Libwget is licensed under LGPLv3+.
A non-exhaustive list of features
- Support for HTTP/1.1 and HTTP/2.0 protocol
- brotli decompression support (Accept-Encoding: br)
- HPKP - HTTP Public Key Pinning (RFC7469) with persistent database
- TCP Fast Open for plain text and for HTTPS
- TLS Session Resumption including persistent session data cache
- TLS False Start (with GnuTLS >= 3.5.0)
- HTTP2 support via nghttp2 and GnuTLS ALPN including streaming/pipelining
- OCSP stapling + OCSP server querying as a fallback (experimental, needs GnuTLS >= 3.3.11)
- Use libpsl for cookie domain checking (using Public Suffix List)
- Support link conversion (-k/--convert-links and -K/--backup-converted)
- Support for RFC 6266 compliant Content-Disposition
- RFC 6797 HSTS (HTTP Strict Transport Security)
- Support for bzip2 Content-Encoding / Accept-Encoding compression type
- New Year 2014 gimmick: added support for XZ Content-Encoding / Accept-Encoding compression type
- Character encoding of input files may be specified despite from local and remote encoding (--input-encoding)
- Support scanning RSS 2.0 feeds from local files (
--force-rss -i <filename>) - Support scanning RSS 2.0 feeds.
- Support scanning Atom 1.0 feeds from local files (
--force-atom -i <filename>) - Support scanning Atom 1.0 feeds.
- Support scanning URLs from local Sitemap XML file (
--force-sitemap -i <filename>) - Support scanning sitemap files given in robots.txt (Sitemap XML, gzipped Sitemap XML, plain text) including sitemap index files.
- Support arbitrary number of proxies for parallel downloads
- Multithreaded download of single files (option --chunk-size)
- Internationalized Domain Names in Applications (compile-selectable IDNA2008 or IDNA2003)
- ICEcast / SHOUTcast support via library (see examples/getstream.c)
- respect /robots.txt "Robot Exclusion Standard" and
<META name="robots" ...> - new option --secure-protocol=PFS to have TLS only plus forcing Perfect Forward Secrecy (PFS)
- IDN support for international domains
- autotools support
- proxy support
- cookies (session/non-session), detection of supercookies via Mozilla Public Suffix List
(use the new option
--cookie-suffixes <filename>, better: put it into ~/.wgetrc) - recursive download of websites with or without spanning hosts
- download of single web pages / resources
- zlib/gzip compressed HTTP/HTTPS downloads (gzip, deflate)
- number of parallel download threads is adjustable
- include directive for config files (wildcards allowed)
- support for keep-alive connections
- included CSS, HTML, XML parser needed for recursive downloads
- gettext support
- HTTPS via libgnutls
- support for Metalink RFC 6249 (Metalink/HTTP: Mirrors and Hashes)
- support for Metalink RFC 5854 (Metalink Download Description Format / .meta4 files)
- support for Metalink 3
- Metalink checksumming via libgnutls
- DNS lookup cache
- IPv4 and IPv6 support
- tested on Debian SID amd64 and OpenBSD 5.0
- compiled and tested with gcc (4.7.1 upto 4.8.2) and clang (3.1.1 upto 3.4)
- tested regularly with static analysis tools
- compiled and linked with hardening options proposed by the Debian project
The following packages are needed to build the software
- autotools (autoconf, autogen, automake, autopoint, libtool)
- python (recommended for faster bootstrap)
- rsync
- tar
- makeinfo (part of texinfo)
- pkg-config >= 0.28 (recommended)
- doxygen (for creating the documentation)
- pandoc (for creating the wget2 man page)
- gettext >= 0.18.2
- libiconv (needed for IRI and IDN support)
- libz >= 1.2.3 (the distribution may call the package zlib*, eg. zlib1g on Debian)
- liblzma >= 5.1.1alpha (optional, if you want HTTP lzma decompression)
- libbz2 >= 1.0.6 (optional, if you want HTTP bzip2 decompression)
- libbrotlidec >= 1.0.0 (optional, if you want HTTP brotli decompression)
- libgnutls (3.3, 3.5 or 3.6)
- libidn2 >= 0.9 + libunistring >= 0.9.3 (libidn >= 1.25 if you don't have libidn2)
- flex >= 2.5.35
- libpsl >= 0.5.0
- libnghttp2 >= 1.3.0 (optional, if you want HTTP/2 support)
- libmicrohttpd >= 0.9.51 (optional, if you want to run the test suite)
- lzip (optional, if you want to build distribution tarballs)
- lcov (optional, for coverage reports)
- libgpgme >= 0.4.2 (optional, for automatic signature verification)
The versions are recommended, but older versions may also work.
Download project and prepare sources with
git clone https://gitlab.com/gnuwget/wget2.git
cd wget2
./bootstrap
# on shell failure try 'bash ./bootstrap'
Build Wget2 with
./configure
make
Test the functionality
make check
Install Wget2 and libwget
sudo make install (or su -c "make install")
To run the test suite with valgrind memcheck
make check-valgrind
or if you want valgrind memcheck by default
./configure --enable-valgrind-tests
make check
To run single tests with valgrind (e.g. test-k)
cd tests
VALGRIND_TESTS=1 ./test-k
Why not directly using valgrind like 'valgrind --leak-check=full ./test-k' ? Well, you want to valgrind 'wget2' and not the test program itself, right ?
To generate and view the test code coverage (works with gcc, not with clang)
make check-coverage
<browser> lcov/index.html
To instrument clang's CFI:
CC="clang-5.0" CFLAGS="-g -fsanitize=cfi -fno-sanitize-trap=all -fno-sanitize=cfi-icall -flto -fvisibility=hidden" NM=/usr/bin/llvm-nm-5.0 RANLIB=/usr/bin/llvm-ranlib-5.0 AR=/usr/bin/llvm-ar-5.0 LD=/usr/bin/gold ./configure
make clean
make check
With clang-5.0 -fsanitize=cfi-icall does not work as expected.
Our logger callback functions are typed correctly, but falsely cause a hiccup.
