Configurable external robots.txt file

Currently the robots.txt file used by the application is inside the .war
file and thus difficult to modify for users. This patch adds a new
optional configuration parameter that allows the user to specify an
external robots file, so that it is easy to modify and easy to preserve
during upgrades of the application. If no configuration change is made
the original robots file is used, only if the following is added to the
gerrit.config file will the external file be used:

  [httpd]
    robotsFile = etc/myrobots.txt

If the file indicated by this parameter is relative then it will be
resolved as sub directory of the site directory, if it is absolute it
will be used as is.

If the file doesn't exist or can't be read a message will be written to
the log and the default file will be used.

Bug: issue 1968
Change-Id: Iad02dbd97633e9c45dbce15d1f227f3931255e0a
Signed-off-by: Juan Hernandez <juanantonio.hernandez@gmail.com>
3 files changed