|author||Thomas Draebing <email@example.com>||Mon Mar 23 11:53:13 2020 +0100|
|committer||Thomas Draebing <firstname.lastname@example.org>||Mon Mar 23 11:53:13 2020 +0100|
Remove custom labels added to logs during parsing Promtail was configured to create labels for nearly every key in the logs. This was done to support easier label-based querying. Loki however is not optimized to work with labels having a high cardinality. This led to failures in Loki, if it had to handle a high number of logs. In addition, the high number of labels led to a huge number of chunks being created, mostly just containing a single log entry, making querying and storage very inefficient. This change removes all custom made labels, except for the gerrit_version label. Logs should rather be queried using the grep- like syntax of LogQL for which Loki is optimized. Change-Id: I70e2a3ff4f640bc6f5d08d50212958a7bca2eae1
This project provides a setup for monitoring Gerrit instances. The setup is based on Prometheus and Grafana running in Kubernetes. In addition, logging will be provided by Grafana Loki.
The setup is provided as a helm chart. It can be installed using Helm (This README expects Helm version 3.0 or higher).
The charts used in this setup are the chart provided in the open source and can be found on GitHub:
This project just provides
values.yaml-files that are already configured to work with the
metrics-reporter-prometheus-plugin of Gerrit to make the setup easier.
Gerrit requires the following plugin to be installed:
Promtail has to be installed with access to the
logs-directory in the Gerrit- site. A configuration-file for Promtail will be provided in this setup. Find the documentation for Promtail here
To install and configure Helm, follow the official guide.
ytt is a templating tool for yaml-files. It is required for some last moment configuration. Installation instructions can be found here.
yq is a commandline processor for yaml-files. Installation instructions can be found here.
To have dashboards deployed automatically during installation, export the dashboards to a JSON-file or create JSON-files describing the dashboards in another way. Put these dashboards into the
./dashboards-directory of this repository. During the installation the dashboards will be added to a configmap and with this automatically installed to Grafana.
While this project is supposed to provide a specialized and opinionated monitoring setup, some configuration is highly dependent on the specific installation. These options have to be configured in the
./config.yaml before installing and are listed here:
|Hostname (incl. port, if required) of the Gerrit server to monitor|
|Username of Gerrit user with ‘View Metrics’ capabilities|
|Password of Gerrit user with ‘View Metrics’ capabilities|
|The namespace the charts are installed to|
|Whether to skip TLS certificate verification|
|CA certificate used for TLS certificate verification|
|Path to directory, where Promtail is allowed to save files (e.g. |
|Path to directory containing the Gerrit logs (e.g. |
|Prometheus server ingress hostname|
|Username for Prometheus|
|Password for Prometheus|
|API URL of the Slack Webhook|
|Channel to which the alerts should be posted|
|Loki ingress hostname|
|Username for Loki|
|Password for Loki|
|Grafana ingress hostname|
|Username for the admin user|
|Password for the admin user|
|Whether to enable LDAP|
|Hostname of LDAP server|
|Port of LDAP server (Has to be |
|Password of LDAP server|
|Bind DN (username) of the LDAP server|
|List of base DNs to discover accounts (Has to have the format |
|List of base DNs to discover groups (Has to have the format |
|Whether dashboards can be edited manually in the UI|
The configuration file contains secrets. Thus, to be able to share the configuration, e.g. with the CI-system, it is meant to be encrypted. The encryption is explained here.
./install.sh-script will decrypt the file before templating, if it was encrypted with
Before beginning with the installation, ensure that the local helm repository is up-to-date:
helm repo add loki https://grafana.github.io/loki/charts helm repo update
This project provides a script to quickly install the monitoring setup. To use it, run:
./install.sh \ [--output ./dist] \ [--dryrun] \ config.yaml
The command will use the given configuration to create the final files in the directory given by
./dist) and install/update the Kubernetes resources and charts, if the
--dryrun flag is not set.
Promtail has to be installed with access to the directory containing the Gerrit logs, e.g. on the same host. The installation as described above will create a configuration file for Promtail, which can be found in
./dist/promtail.yaml. Use it to configure Promtail by using the
-config.file=./dist/promtail.yaml- parameter, when starting Promtail. Using the Promtail binary directly this would result in the following command:
$PATH_TO_PROMTAIL/promtail \ -config.file=./dist/promtail.yaml \ -client.external-labels=host=$(hostname)
-client.external-labels=host=$(hostname) option will add a label to each job that contains the hostname. This is useful, if multiple host are scraped for logs and only one Grafana is used to view the logs.
If TLS-verification is activated, the CA-certificate used for verification (usually the one configured for
tls.caCert) has to be present in the directory configured for
promtail.storagePath in the
config.yaml and has to be called
The Promtail configuration provided here expects the logs to be available in JSON-format. This can be configured by setting
log.jsonLogging = true in the
To remove the Prometheus chart from the cluster, run
helm uninstall prometheus --namespace $NAMESPACE helm uninstall loki --namespace $NAMESPACE helm uninstall grafana --namespace $NAMESPACE kubectl delete -f ./dist/configuration
To also release the volumes, run
kubectl delete -f ./dist/storage
NOTE: Doing so, all data, which was not backed up will be lost!
Remove the namespace:
kubectl delete -f ./dist/namespace.yaml
./uninstall.sh-script will automatically remove the charts installed in by the
./install.sh-script from the configured namespace and delete the namespace as well: