Block search engines via robots.txt (zedeus#631)

Prevents instances from being rate limited due to being senselessly crawled by search engines. Since there is no reason to index Nitter instances, simply block all robots. Notably, this does *not* affect link previews (e.g. in various chat software).
alnmy · Jun 4, 2022 · c543a1d · c543a1d
1 parent 778c6c6
commit c543a1d
Showing 1 changed file with 2 additions and 0 deletions.
diff --git a/public/robots.txt b/public/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /