Add robots.txt to exclude crawlers from walking a 'staging' test publ…

…ishing dir

See https://support.google.com/webmasters/answer/6062608?hl=en&ref_topic=6061961

Testing Done:
```
$ ./build-support/bin/publish_docs.sh
```
robots.txt is copied to root of docs

```
./build-support/bin/publish_docs.sh -d zundel -p
```

Now publishes the site under  the directory to staging/zundel where it will be excluded from crawlers that obey the robots.txt settings.

Bugs closed: 1416

Reviewed at https://rbcommons.com/s/twitter/r/2072/

build-support/bin/publish_docs.sh

@@ -15,7 +15,7 @@ function usage() {
   echo " -h           print out this help message"
   echo " -o           open the doc site locally"
   echo " -p           publish the doc site remotely"
-  echo " -d           publish the site to a subdir at this path (useful for public previews)"
+  echo " -d  <dir>    publish the site to a subdir staging/<dir> (useful for public previews)"
 
   if (( $# > 0 )); then
     die "$@"
@@ -31,7 +31,7 @@ while getopts "hopd:" opt; do
     h) usage ;;
     o) preview="true" ;;
     p) publish="true" ;;
-    d) publish_path="${OPTARG}" ;;
+    d) publish_path="staging/${OPTARG}" ;;
     *) usage "Invalid option: -${OPTARG}" ;;
   esac
 done

src/docs/robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /staging/

src/python/pants/docs/docsite.json

@@ -49,6 +49,7 @@
     "images/intellij-run.png": "src/python/pants/docs/images/intellij-run.png",
     "images/intellij-select-venv.png": "src/python/pants/docs/images/intellij-select-venv.png",
     "images/report-server-page.png": "src/docs/images/report-server-page.png",
+    "robots.txt": "src/docs/robots.txt",
     "logo.ico": "src/python/pants/docs/pants-logo.ico",
     "s.js": "src/python/pants/docs/docsite.js",
     "s.css": "src/python/pants/docs/docsite.css"