Robofinder is a Python script that allows you to search for and retrieve historical robots.txt
files for any given website using Archive.org. This tool is particularly useful for security researchers and web archivists to discover previously accessible paths or directories that were once listed in a site's robots.txt
.
- Fetch historical
robots.txt
files from Archive.org. - Extract and display old paths or directories that were once disallowed or listed.
- Option to save the output to a file.
- Silent mode for unobtrusive execution.
- Multi-threading support to speed up the search process.
Clone the repository and install the required dependencies:
git clone https://github.com/Spix0r/robofinder.git
cd robofinder
pip install -r requirements.txt
Run the program by providing a URL with the -u
flag:
python3 robofinder.py -u https://example.com
- Save output to file:
python3 robofinder.py -u https://example.com -o results.txt
- Concatenate paths with site URL:
python3 robofinder.py -u https://example.com -c
- Run in silent mode (no console output):
python3 robofinder.py -u https://example.com --silent
- Multi-threading (default recommended: 10 threads):
python3 robofinder.py -u https://example.com -t 10 -c -o results.txt
Running Robofinder on example.com
and saving the result to results.txt
with 10 threads:
python3 robofinder.py -u https://example.com -t 10 -o results.txt