Clone this repo:

Branches

  1. e2a5902 Merge "Show more lines in log queries in Grafana" by Thomas Dräbing · 3 days ago master
  2. 6d3c31e Merge "Update Grafana to 6.7.1" by Thomas Dräbing · 11 days ago
  3. d7d1703 Merge changes I2cd9c872,I26cfd395 by Matthias Sohn · 12 days ago
  4. 202a316 Show more lines in log queries in Grafana by Thomas Draebing · 4 weeks ago
  5. 10a0a54 Update Grafana to 6.7.1 by Thomas Draebing · 2 weeks ago

Monitoring setup for Gerrit

This project provides a setup for monitoring Gerrit instances. The setup is based on Prometheus and Grafana running in Kubernetes. In addition, logging will be provided by Grafana Loki.

The setup is provided as a helm chart. It can be installed using Helm (This README expects Helm version 3.0 or higher).

The charts used in this setup are the chart provided in the open source and can be found on GitHub:

This project just provides values.yaml-files that are already configured to work with the metrics-reporter-prometheus-plugin of Gerrit to make the setup easier.

Dependencies

  • Gerrit
    Gerrit requires the following plugin to be installed:

  • Promtail
    Promtail has to be installed with access to the logs-directory in the Gerrit- site. A configuration-file for Promtail will be provided in this setup. Find the documentation for Promtail here

  • Helm
    To install and configure Helm, follow the official guide.

  • ytt
    ytt is a templating tool for yaml-files. It is required for some last moment configuration. Installation instructions can be found here.

  • yq
    yq is a commandline processor for yaml-files. Installation instructions can be found here.

Add dashboards

To have dashboards deployed automatically during installation, export the dashboards to a JSON-file or create JSON-files describing the dashboards in another way. Put these dashboards into the ./dashboards-directory of this repository. During the installation the dashboards will be added to a configmap and with this automatically installed to Grafana.

Configuration

While this project is supposed to provide a specialized and opinionated monitoring setup, some configuration is highly dependent on the specific installation. These options have to be configured in the ./config.yaml before installing and are listed here:

optiondescription
gerritServers.[0].hostHostname (incl. port, if required) of the Gerrit server to monitor
gerritServers.[0].usernameUsername of Gerrit user with ‘View Metrics’ capabilities
gerritServers.[0].passwordPassword of Gerrit user with ‘View Metrics’ capabilities
namespaceThe namespace the charts are installed to
tls.skipVerifyWhether to skip TLS certificate verification
tls.caCertCA certificate used for TLS certificate verification
promtail.storagePathPath to directory, where Promtail is allowed to save files (e.g. positions.yaml)
promtail.logPathPath to directory containing the Gerrit logs (e.g. /var/gerrit/logs)
prometheus.server.hostPrometheus server ingress hostname
prometheus.server.usernameUsername for Prometheus
prometheus.server.passwordPassword for Prometheus
prometheus.server.tls.certTLS certificate
prometheus.server.tls.keyTLS key
prometheus.alertmanager.slack.apiUrlAPI URL of the Slack Webhook
prometheus.alertmanager.slack.channelChannel to which the alerts should be posted
loki.hostLoki ingress hostname
loki.usernameUsername for Loki
loki.passwordPassword for Loki
loki.tls.certTLS certificate
loki.tls.keyTLS key
grafana.hostGrafana ingress hostname
grafana.tls.certTLS certificate
grafana.tls.keyTLS key
grafana.admin.usernameUsername for the admin user
grafana.admin.passwordPassword for the admin user
grafana.ldap.enabledWhether to enable LDAP
grafana.ldap.hostHostname of LDAP server
grafana.ldap.portPort of LDAP server (Has to be quoted!)
grafana.ldap.passwordPassword of LDAP server
grafana.ldap.bind_dnBind DN (username) of the LDAP server
grafana.ldap.accountBasesList of base DNs to discover accounts (Has to have the format "['a', 'b']")
grafana.ldap.groupBasesList of base DNs to discover groups (Has to have the format "['a', 'b']")
grafana.dashboards.editableWhether dashboards can be edited manually in the UI

Encryption

The configuration file contains secrets. Thus, to be able to share the configuration, e.g. with the CI-system, it is meant to be encrypted. The encryption is explained here.

The ./install.sh-script will decrypt the file before templating, if it was encrypted with sops.

Installation

Before beginning with the installation, ensure that the local helm repository is up-to-date:

helm repo add loki https://grafana.github.io/loki/charts
helm repo update

This project provides a script to quickly install the monitoring setup. To use it, run:

./install.sh \
  [--output ./dist] \
  [--dryrun] \
  config.yaml

The command will use the given configuration to create the final files in the directory given by --output (default ./dist) and install/update the Kubernetes resources and charts, if the --dryrun flag is not set.

Configure Promtail

Promtail has to be installed with access to the directory containing the Gerrit logs, e.g. on the same host. The installation as described above will create a configuration file for Promtail, which can be found in ./dist/promtail.yaml. Use it to configure Promtail by using the -config.file=./dist/promtail.yaml- parameter, when starting Promtail. Using the Promtail binary directly this would result in the following command:

$PATH_TO_PROMTAIL/promtail \
  -config.file=./dist/promtail.yaml \
  -client.external-labels=host=$(hostname)

The -client.external-labels=host=$(hostname) option will add a label to each job that contains the hostname. This is useful, if multiple host are scraped for logs and only one Grafana is used to view the logs.

If TLS-verification is activated, the CA-certificate used for verification (usually the one configured for tls.caCert) has to be present in the directory configured for promtail.storagePath in the config.yaml and has to be called promtail.ca.crt.

The Promtail configuration provided here expects the logs to be available in JSON-format. This can be configured by setting log.jsonLogging = true in the gerrit.config.

Uninstallation

To remove the Prometheus chart from the cluster, run

helm uninstall prometheus --namespace $NAMESPACE
helm uninstall loki --namespace $NAMESPACE
helm uninstall grafana --namespace $NAMESPACE
kubectl delete -f ./dist/configuration

To also release the volumes, run

kubectl delete -f ./dist/storage

NOTE: Doing so, all data, which was not backed up will be lost!

Remove the namespace:

kubectl delete -f ./dist/namespace.yaml

The ./uninstall.sh-script will automatically remove the charts installed in by the ./install.sh-script from the configured namespace and delete the namespace as well:

./uninstall.sh config.yaml