tree f90bd2a65aa757ba01997570421672fcc367d8e8
parent 0d5782d46e32eb9064d51e4955a02950bc1d66aa
author Han-Wen Nienhuys <hanwen@google.com> 1624612381 +0200
committer Han-Wen Nienhuys <hanwen@google.com> 1625737907 +0000

web: serve robots.txt, disallowing /search

The example queries on the entry page are not limited by repository,
so they may touch a large amount of memory. For public servers,
avoiding robots may increase availability. Anecdotally, cs.bazel.build
is having fits of crashing, ostensibly caused by crawlers overloading
the search engine

Change-Id: I4417c9389ce57f4d5ebee7695397e20540d15779