| commit | 672cc245707f4e1b5480eacfe1df8d3c5879a639 | [log] [tgz] |
|---|---|---|
| author | Thomas Draebing <thomas.draebing@sap.com> | Tue Jun 23 14:46:13 2020 +0200 |
| committer | Thomas Dräbing <thomas.draebing@sap.com> | Mon Aug 26 08:41:08 2024 +0000 |
| tree | e00763fd98928102cc80afb0d6405bcfece98232 | |
| parent | c990d0e8536efbd959d3da01944ad65c73c91373 [diff] |
Enable the use of istio for the monitoring setup The monitoring setup was only usable with an ingress controller like Nginx. However, if being used in a cluster running one or multiple Gerrit instances, it might be worth considering a service mesh to have better control of the network traffic. In this case, it would be worth considering also integrating the monitoring setup with the service mesh to avoid multiple load balancers. This change adds the resources to configure the setup to run in an istio service mesh. Istio is already expected to be installed. The ports 80 and 443 have to be enabled in the istio-ingressgateway. If configured the setup will then set up a gateway, which allows access to Grafana. The access to Prometheus and Loki is only possible with a JWT-token. This was done, since istio does not provide basic authentication like the NGINX ingress controller and neither do Prometheus or Loki. The installer will automatically put a bearer token into the Promtail configurations. If a key is required, the gerrit-monitoring.py -c <config> jwt command can be used to get one. At the moment JWT tokens do not expire, since there is currently no mechanism supported to automatically cycle the tokens. This may be added in a later step. Change-Id: Ic6542a3bdfdbcaa63552d0e049d62e12d1f8acaf
This project provides a setup for monitoring Gerrit instances. The setup is based on Prometheus and Grafana running in Kubernetes. In addition, logging will be provided by Grafana Loki.
The setup is provided as a helm chart. It can be installed using Helm (This README expects Helm version 3.0 or higher).
The charts used in this setup are the chart provided in the open source and can be found on GitHub:
This project just provides values.yaml-files that are already configured to work with the metrics-reporter-prometheus-plugin of Gerrit to make the setup easier.
Gerrit
Gerrit requires the following plugin to be installed:
Promtail
Promtail has to be installed with access to the logs-directory in the Gerrit- site. A configuration-file for Promtail will be provided in this setup. Find the documentation for Promtail here
Helm
To install and configure Helm, follow the official guide.
ytt
ytt is a templating tool for yaml-files. It is required for some last moment configuration. Installation instructions can be found here.
Pipenv
Pipenv sets up a virtual python environment and installs required python packages based on a lock-file, ensuring a deterministic Python environment. Instruction on how Pipenv can be installed, can be found here
Jsonnet
Jsonnet is used to create the JSON-files describing the Grafana dashboards. Instruction on how Jsonnet can be installed, can be found here
Grafonnet
Grafonnet should be installed using jsonnet-bundler and the jsonnetfile.json provided by this project. Install jsonnet-bundler as described here. Then run jb install from this project's root directory.
Kubernetes Cluster
A cluster with at least 3 free CPUs and 4 GB of free memory are required. In addition persistent storage of about 30 GB will be used.
Ingress Controller
The charts currently expect a Nginx ingress controller to be installed in the cluster.
Object store
Loki will store the data chunks in an object store. This store has to be callable via the S3 API.
There are two ways to have dashboards deployed automatically during installation:
One way is to export the dashboards to a JSON-file in the UI or create JSON-files describing the dashboards in another way. Put these dashboards into the ./dashboards-directory of this repository.
The other way is to use Jsonnet/Grafonnet to programmatically create dashboards. Install Grafonnet into the project as described above and put your dashboard jsonnet files into the dashboards-directory or one of its subdirectories. The jsonnet-based dashboards can be transcribed into json manually using the following command:
jsonnet -J grafonnet-lib --ext-code publish=false dashboards/<dashboard>.jsonnet
The external variable publish should be set to false, if the dashboard is imported via API and to true, if it is published to the Grafana homepage or imported via the UI.
While this project is supposed to provide a specialized and opinionated monitoring setup, some configuration is highly dependent on the specific installation. These options have to be configured in the ./config.yaml before installing and are listed here:
| option | description |
|---|---|
gerritServers | List of Gerrit servers to scrape. For details refer to section below |
namespace | The namespace the charts are installed to |
tls.skipVerify | Whether to skip TLS certificate verification |
tls.caCert | CA certificate used for TLS certificate verification |
istio.enabled | Whether to use istio |
istio.crt | TLS cert for Ingress gateway (should have alternative names for URLs of all components) |
istio.key | TLS key for Ingress gateway |
istio.jwt.cert | RSA certificate to be used to create JWT tokens |
istio.jwt.key | RSA key to be used to create JWT tokens |
istio.jwt.issuer | Issuer to be used for tokens (e.g. an email address) |
monitoring.prometheus.server.host | Prometheus server ingress hostname |
monitoring.prometheus.server.username | Username for Prometheus (only required if not using istio) |
monitoring.prometheus.server.password | Password for Prometheus (only required if not using istio) |
monitoring.prometheus.server.tls.cert | TLS certificate |
monitoring.prometheus.server.tls.key | TLS key |
monitoring.prometheus.alertmanager.slack.apiUrl | API URL of the Slack Webhook |
monitoring.prometheus.alertmanager.slack.channel | Channel to which the alerts should be posted |
monitoring.grafana.host | Grafana ingress hostname |
monitoring.grafana.tls.cert | TLS certificate (only required if not using istio) |
monitoring.grafana.tls.key | TLS key (only required if not using istio) |
monitoring.grafana.admin.username | Username for the admin user |
monitoring.grafana.admin.password | Password for the admin user |
monitoring.grafana.ldap.enabled | Whether to enable LDAP |
monitoring.grafana.ldap.host | Hostname of LDAP server |
monitoring.grafana.ldap.port | Port of LDAP server (Has to be quoted!) |
monitoring.grafana.ldap.password | Password of LDAP server |
monitoring.grafana.ldap.bind_dn | Bind DN (username) of the LDAP server |
monitoring.grafana.ldap.accountBases | List of base DNs to discover accounts (Has to have the format "['a', 'b']") |
monitoring.grafana.ldap.groupBases | List of base DNs to discover groups (Has to have the format "['a', 'b']") |
monitoring.grafana.dashboards.editable | Whether dashboards can be edited manually in the UI |
logging.loki.host | Loki ingress hostname |
logging.loki.username | Username for Loki (only required if not using istio) |
logging.loki.password | Password for Loki (only required if not using istio) |
logging.loki.s3.protocol | Protocol used for communicating with S3 |
logging.loki.s3.host | Hostname of the S3 object store |
logging.loki.s3.accessToken | The EC2 accessToken used for authentication with S3 |
logging.loki.s3.secret | The secret associated with the accessToken |
logging.loki.s3.bucket | The name of the S3 bucket |
logging.loki.s3.region | The region in which the S3 bucket is hosted |
logging.loki.tls.cert | TLS certificate (only required if not using istio) |
logging.loki.tls.key | TLS key (only required if not using istio) |
gerritServersTwo types of Gerrit servers are currently supported, which require different configuration parameters:
| option | description |
|---|---|
gerritServers.kubernetes.[*].namespace | Namespace into which Gerrit was deployed |
gerritServers.kubernetes.[*].label.name | Label name used to select deployments |
gerritServers.kubernetes.[*].label.value | Label value to select deployments |
gerritServers.kubernetes.[*].containerName | Name of container in the pod that runs Gerrit |
gerritServers.kubernetes.[*].port | Container port to be used when scraping |
gerritServers.kubernetes.[*].username | Username of Gerrit user with ‘View Metrics’ capabilities |
gerritServers.kubernetes.[*].password | Password of Gerrit user with ‘View Metrics’ capabilities |
| option | description |
|---|---|
gerritServers.federatedPrometheus.[*].host | Host running Gerrit and the Prometheus instance being scraped |
gerritServers.federatedPrometheus.[*].port | Port used by Prometheus |
gerritServers.federatedPrometheus.[*].username | Username for authenticating with Prometheus |
gerritServers.federatedPrometheus.[*].password | Password for authenticating with Prometheus |
| option | description |
|---|---|
gerritServers.other.[*].host | Hostname (incl. port, if required) of the Gerrit server to monitor |
gerritServers.other.[*].username | Username of Gerrit user with ‘View Metrics’ capabilities |
gerritServers.other.[*].password | Password of Gerrit user with ‘View Metrics’ capabilities |
gerritServers.other.[*].healthcheck | Whether to deploy a container that regularly pings the healthcheck plugin endpoint in Gerrit |
gerritServers.other.[*].promtail.storagePath | Path to directory, where Promtail is allowed to save files (e.g. positions.yaml) |
gerritServers.other.[*].promtail.logPath | Path to directory containing the Gerrit logs (e.g. /var/gerrit/logs) |
The configuration file contains secrets. Thus, to be able to share the configuration, e.g. with the CI-system, it is meant to be encrypted. The encryption is explained here.
The gerrit_monitoring.py install-command will decrypt the file before templating, if it was encrypted with sops.
The easiest way of using the monitoring setup, is to use an Ingress Controller, but it is also possible to use the setup within an Istio service mesh. To do this, Istio has to be already installed in the cluster and the istio-ingressgateway has to open the ports 80 and 443. Authentication and authorization for Prometheus and Loki for users outside of the cluster will be done by JWT-tokens. Promtail configurations created by the installer will automatically get a token configured during the installation. Should another token be needed, the follwoing command can be used to create a token:
pipenv run python ./gerrit-monitoring.py \ --config config.yaml \ jwt
Before using the script, set up a python environment using pipenv install.
The installation will use the environment of the current shell. Thus, make sure that the path for ytt, kubectland helm are set. Also the KUBECONFIG-variable has to be set to point to the kubeconfig of the target Kubernetes cluster.
This project provides a script to quickly install the monitoring setup. To use it, run:
pipenv run ./gerrit_monitoring.py \ --config config.yaml \ install \ [--output ./dist] \ [--dryrun] \ [--update-repo]
The command will use the given configuration (--config/-c) to create the final files in the directory given by --output/-o (default ./dist) and install/update the Kubernetes resources and charts, if the --dryrun/-d flag is not set. If the --update-repo-flag is used, the helm repository will be updated before installing the helm charts. This is for example required, if a chart version was updated.
Promtail has to be installed with access to the directory containing the Gerrit logs, e.g. on the same host. The installation as described above will create a configuration file for Promtail, which can be found in ./dist/promtail.yaml. Use it to configure Promtail by using the -config.file=./dist/promtail.yaml- parameter, when starting Promtail. Using the Promtail binary directly this would result in the following command:
$PATH_TO_PROMTAIL/promtail \ -config.file=./dist/promtail.yaml
If TLS-verification is activated, the CA-certificate used for verification (usually the one configured for tls.caCert) has to be present in the directory configured for promtail.storagePath in the config.yaml and has to be called promtail.ca.crt.
The Promtail configuration provided here expects the logs to be available in JSON-format. This can be configured by setting log.jsonLogging = true in the gerrit.config.
To remove the Prometheus chart from the cluster, run
helm uninstall prometheus --namespace $NAMESPACE helm uninstall loki --namespace $NAMESPACE helm uninstall grafana --namespace $NAMESPACE kubectl delete -f ./dist/configuration
To also release the volumes, run
kubectl delete -f ./dist/storage
NOTE: Doing so, all data, which was not backed up will be lost!
Remove the namespace:
kubectl delete -f ./dist/namespace.yaml
The ./gerrit_monitoring.py uninstall-script will automatically remove the charts installed in the configured namespace and delete the namespace as well:
pipenv run ./gerrit_monitoring.py \ --config config.yaml \ uninstall