Skip to content

A robots.txt to block all those pesky bandwidth wasting bots

License

Notifications You must be signed in to change notification settings

quellhorst/robots

Repository files navigation

Robots.txt Examples

Fend off those pesky robots!

This project resulted from having to specifically block useless bots that requests thousands of pages
per day and send no traffic. Several example bots files are included that could be renamed and copied to
robots.txt in the root of your web application. Your robots file should be accessible at
http://www.yourdomain.com/robots.txt

I have seen sites that had multiple servers just to handle excessive bot load.

Files included

robots.txt Standard bot file should be usable for most sites.
Only disallows know bad bots.

robots.major.txt Has a white-list for major search engine and blocks
everything else

robots.noarchive.txt same as above but disallows archive.org bot which uses
lots of traffic and doesn’t send much traffic

robots.wordpress.txt Entries for Wordpress blogs

robots.none.txt Block all bots

Build by monitoring robots visiting http://abtain.com/

Licensed under the MIT License

About

A robots.txt to block all those pesky bandwidth wasting bots

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published