Allow to monitor Gerrit on Kubernetes
So far it was only possible to monitor single instance Gerrit servers.
This was due to to the fact that a URL had to be used that pointed to
a dedicated instance, since if multiple replicas would be behind the
instance, the metrics of a random replica would be scraped and not of
all.
Prometheus has a service discovery functionality for deployments running
in Kubernetes. This is now used, when monitoring a Gerrit instance in
Kubernetes. This allows to have a variable number of replicas running,
which will be automatically discovered by Prometheus.
The dashboards were adapted accordingly and allow now to select the
replica to be observed. For now, no summary of all replicas can be
displayed in the dashboards, but that feature is planned to be added
in the future.
Change-Id: I96efc63a192cd90f5e3e91a53dace8e1ae83132e
diff --git a/README.md b/README.md
index 0fee843..ff7f45e 100644
--- a/README.md
+++ b/README.md
@@ -76,47 +76,75 @@
These options have to be configured in the `./config.yaml` before installing and
are listed here:
-| option | description |
-|------------------------------------------|------------------------------------------------------------------------------------|
-| `gerritServers.[0].host` | Hostname (incl. port, if required) of the Gerrit server to monitor |
-| `gerritServers.[0].username` | Username of Gerrit user with 'View Metrics' capabilities |
-| `gerritServers.[0].password` | Password of Gerrit user with 'View Metrics' capabilities |
-| `gerritServers.[0].promtail.storagePath` | Path to directory, where Promtail is allowed to save files (e.g. `positions.yaml`) |
-| `gerritServers.[0].promtail.logPath` | Path to directory containing the Gerrit logs (e.g. `/var/gerrit/logs`) |
-| `namespace` | The namespace the charts are installed to |
-| `tls.skipVerify` | Whether to skip TLS certificate verification |
-| `tls.caCert` | CA certificate used for TLS certificate verification |
-| `prometheus.server.host` | Prometheus server ingress hostname |
-| `prometheus.server.username` | Username for Prometheus |
-| `prometheus.server.password` | Password for Prometheus |
-| `prometheus.server.tls.cert` | TLS certificate |
-| `prometheus.server.tls.key` | TLS key |
-| `prometheus.alertmanager.slack.apiUrl` | API URL of the Slack Webhook |
-| `prometheus.alertmanager.slack.channel` | Channel to which the alerts should be posted |
-| `loki.host` | Loki ingress hostname |
-| `loki.username` | Username for Loki |
-| `loki.password` | Password for Loki |
-| `loki.s3.protocol` | Protocol used for communicating with S3 |
-| `loki.s3.host` | Hostname of the S3 object store |
-| `loki.s3.accessToken` | The EC2 accessToken used for authentication with S3 |
-| `loki.s3.secret` | The secret associated with the accessToken |
-| `loki.s3.bucket` | The name of the S3 bucket |
-| `loki.s3.region` | The region in which the S3 bucket is hosted |
-| `loki.tls.cert` | TLS certificate |
-| `loki.tls.key` | TLS key |
-| `grafana.host` | Grafana ingress hostname |
-| `grafana.tls.cert` | TLS certificate |
-| `grafana.tls.key` | TLS key |
-| `grafana.admin.username` | Username for the admin user |
-| `grafana.admin.password` | Password for the admin user |
-| `grafana.ldap.enabled` | Whether to enable LDAP |
-| `grafana.ldap.host` | Hostname of LDAP server |
-| `grafana.ldap.port` | Port of LDAP server (Has to be `quoted`!) |
-| `grafana.ldap.password` | Password of LDAP server |
-| `grafana.ldap.bind_dn` | Bind DN (username) of the LDAP server |
-| `grafana.ldap.accountBases` | List of base DNs to discover accounts (Has to have the format `"['a', 'b']"`) |
-| `grafana.ldap.groupBases` | List of base DNs to discover groups (Has to have the format `"['a', 'b']"`) |
-| `grafana.dashboards.editable` | Whether dashboards can be edited manually in the UI |
+| option | description |
+|-----------------------------------------|----------------------------------------------------------------------------------------|
+| `gerritServers` | List of Gerrit servers to scrape. For details refer to section [below](#gerritServers) |
+| `namespace` | The namespace the charts are installed to |
+| `tls.skipVerify` | Whether to skip TLS certificate verification |
+| `tls.caCert` | CA certificate used for TLS certificate verification |
+| `prometheus.server.host` | Prometheus server ingress hostname |
+| `prometheus.server.username` | Username for Prometheus |
+| `prometheus.server.password` | Password for Prometheus |
+| `prometheus.server.tls.cert` | TLS certificate |
+| `prometheus.server.tls.key` | TLS key |
+| `prometheus.alertmanager.slack.apiUrl` | API URL of the Slack Webhook |
+| `prometheus.alertmanager.slack.channel` | Channel to which the alerts should be posted |
+| `loki.host` | Loki ingress hostname |
+| `loki.username` | Username for Loki |
+| `loki.password` | Password for Loki |
+| `loki.s3.protocol` | Protocol used for communicating with S3 |
+| `loki.s3.host` | Hostname of the S3 object store |
+| `loki.s3.accessToken` | The EC2 accessToken used for authentication with S3 |
+| `loki.s3.secret` | The secret associated with the accessToken |
+| `loki.s3.bucket` | The name of the S3 bucket |
+| `loki.s3.region` | The region in which the S3 bucket is hosted |
+| `loki.tls.cert` | TLS certificate |
+| `loki.tls.key` | TLS key |
+| `grafana.host` | Grafana ingress hostname |
+| `grafana.tls.cert` | TLS certificate |
+| `grafana.tls.key` | TLS key |
+| `grafana.admin.username` | Username for the admin user |
+| `grafana.admin.password` | Password for the admin user |
+| `grafana.ldap.enabled` | Whether to enable LDAP |
+| `grafana.ldap.host` | Hostname of LDAP server |
+| `grafana.ldap.port` | Port of LDAP server (Has to be `quoted`!) |
+| `grafana.ldap.password` | Password of LDAP server |
+| `grafana.ldap.bind_dn` | Bind DN (username) of the LDAP server |
+| `grafana.ldap.accountBases` | List of base DNs to discover accounts (Has to have the format `"['a', 'b']"`) |
+| `grafana.ldap.groupBases` | List of base DNs to discover groups (Has to have the format `"['a', 'b']"`) |
+| `grafana.dashboards.editable` | Whether dashboards can be edited manually in the UI |
+
+### `gerritServers`
+
+Two types of Gerrit servers are currently supported, which require different
+configuration parameters:
+
+* Kubernetes \
+ Gerrit installations running in the same Kubernetes cluster as the monitoring
+ setup. Multiple replicas are supported and automatically discovered.
+
+| option | description |
+|----------------------------------------------|----------------------------------------------------------|
+| `gerritServers.kubernetes.[*].namespace` | Namespace into which Gerrit was deployed |
+| `gerritServers.kubernetes.[*].label.name` | Label name used to select deployments |
+| `gerritServers.kubernetes.[*].label.value` | Label value to select deployments |
+| `gerritServers.kubernetes.[*].containerName` | Name of container in the pod that runs Gerrit |
+| `gerritServers.kubernetes.[*].port` | Container port to be used when scraping |
+| `gerritServers.kubernetes.[*].username` | Username of Gerrit user with 'View Metrics' capabilities |
+| `gerritServers.kubernetes.[*].password` | Password of Gerrit user with 'View Metrics' capabilities |
+
+* Other \
+ Gerrit installations with just one replica that can run anywhere, where they
+ are reachable via HTTP.
+
+| option | description |
+|------------------------------------------------|------------------------------------------------------------------------------------|
+| `gerritServers.other.[*].host` | Hostname (incl. port, if required) of the Gerrit server to monitor |
+| `gerritServers.other.[*].username` | Username of Gerrit user with 'View Metrics' capabilities |
+| `gerritServers.other.[*].password` | Password of Gerrit user with 'View Metrics' capabilities |
+| `gerritServers.other.[*].promtail.storagePath` | Path to directory, where Promtail is allowed to save files (e.g. `positions.yaml`) |
+| `gerritServers.other.[*].promtail.logPath` | Path to directory containing the Gerrit logs (e.g. `/var/gerrit/logs`) |
+
### Encryption
diff --git a/charts/prometheus/configuration/prometheus.secret.yaml b/charts/prometheus/configuration/prometheus.secret.yaml
index 01de104..07757e9 100644
--- a/charts/prometheus/configuration/prometheus.secret.yaml
+++ b/charts/prometheus/configuration/prometheus.secret.yaml
@@ -7,7 +7,11 @@
namespace: #@ data.values.namespace
data:
#@yaml/text-templated-strings
- #@ for gerrit in data.values.gerritServers:
+ #@ for gerrit in data.values.gerritServers.kubernetes:
+ .pwd_(@= gerrit.namespace @): #@ base64.encode(gerrit.password)
+ #@ end
+ #@yaml/text-templated-strings
+ #@ for gerrit in data.values.gerritServers.other:
.pwd_(@= gerrit.host @): #@ base64.encode(gerrit.password)
#@ end
diff --git a/charts/prometheus/prometheus.yaml b/charts/prometheus/prometheus.yaml
index decdc6c..aaeed5d 100644
--- a/charts/prometheus/prometheus.yaml
+++ b/charts/prometheus/prometheus.yaml
@@ -1127,7 +1127,43 @@
- /etc/config/alerts
scrape_configs:
- #@ for gerrit in data.values.gerritServers:
+ #@ for gerrit in data.values.gerritServers.kubernetes:
+ - job_name: #@ "gerrit-{}".format(gerrit.namespace)
+ metrics_path: /a/plugins/metrics-reporter-prometheus/metrics
+ scheme: http
+ basic_auth:
+ username: #@ gerrit.username
+ password_file: #@ "/etc/secrets/.pwd_{}".format(gerrit.namespace)
+
+ kubernetes_sd_configs:
+ - role: pod
+ namespaces:
+ names:
+ - #@ gerrit.namespace
+
+ relabel_configs:
+ - source_labels:
+ - #@ "__meta_kubernetes_pod_label_{}".format(gerrit.label.name)
+ regex: #@ gerrit.label.value
+ action: keep
+ - source_labels: [__meta_kubernetes_pod_container_name]
+ regex: #@ gerrit.containerName
+ action: keep
+ - source_labels: [__meta_kubernetes_pod_container_port_number]
+ regex: #@ gerrit.port
+ action: keep
+ - source_labels: [__meta_kubernetes_namespace]
+ regex: (.*)
+ target_label: instance
+ replacement: kubernetes_$1
+ action: replace
+ - source_labels: [__meta_kubernetes_pod_name]
+ regex: (.*)
+ target_label: replica
+ replacement: $1
+ action: replace
+ #@ end
+ #@ for gerrit in data.values.gerritServers.other:
- job_name: #@ "gerrit-{}".format(gerrit.host)
metrics_path: /a/plugins/metrics-reporter-prometheus/metrics
scheme: https
@@ -1141,6 +1177,8 @@
static_configs:
- targets:
- #@ gerrit.host
+ labels:
+ replica: main
basic_auth:
username: #@ gerrit.username
password_file: #@ "/etc/secrets/.pwd_{}".format(gerrit.host)
diff --git a/config.yaml b/config.yaml
index fffed5a..52e98bd 100644
--- a/config.yaml
+++ b/config.yaml
@@ -1,4 +1,14 @@
gerritServers:
+ kubernetes:
+ - namespace: default
+ label:
+ name: app
+ value: gerrit-slave
+ containerName: gerrit-slave
+ port: 8080
+ username: admin
+ password: secret
+ other:
- host: gerrit.example.com
username: admin
password: secret
diff --git a/dashboards/gerrit-caches.json b/dashboards/gerrit-caches.json
index 0761902..811fd09 100644
--- a/dashboards/gerrit-caches.json
+++ b/dashboards/gerrit-caches.json
@@ -62,22 +62,22 @@
"steppedLine": false,
"targets": [
{
- "expr": "100-caches_memory_hit_ratio_accounts{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_accounts{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "accounts",
"refId": "A"
},
{
- "expr": "100-caches_memory_hit_ratio_groups{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_groups{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "groups",
"refId": "B"
},
{
- "expr": "100-caches_memory_hit_ratio_groups_byuuid{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_groups_byuuid{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "groups_byuuid",
"refId": "C"
},
{
- "expr": "100-caches_memory_hit_ratio_ldap_groups_byinclude{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_ldap_groups_byinclude{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "ldap_groups_byinclude",
"refId": "D"
}
@@ -169,12 +169,12 @@
"steppedLine": false,
"targets": [
{
- "expr": "100-caches_memory_hit_ratio_conflicts{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_conflicts{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "conflicts",
"refId": "A"
},
{
- "expr": "100-caches_memory_hit_ratio_mergeability{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_mergeability{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "mergeability",
"refId": "B"
}
@@ -265,17 +265,17 @@
"steppedLine": false,
"targets": [
{
- "expr": "100-caches_memory_hit_ratio_change_kind{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_change_kind{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "change_kind",
"refId": "B"
},
{
- "expr": "100-caches_memory_hit_ratio_change_notes{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_change_notes{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "change_notes",
"refId": "C"
},
{
- "expr": "100-caches_memory_hit_ratio_changeid_project{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_changeid_project{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "changeid_project",
"refId": "D"
}
@@ -365,12 +365,12 @@
"steppedLine": false,
"targets": [
{
- "expr": "100-caches_memory_hit_ratio_project_list{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_project_list{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "project list",
"refId": "A"
},
{
- "expr": "100-caches_memory_hit_ratio_projects{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_projects{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "projects",
"refId": "B"
}
@@ -461,17 +461,17 @@
"steppedLine": false,
"targets": [
{
- "expr": "100-caches_memory_hit_ratio_diff{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_diff{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "diff",
"refId": "A"
},
{
- "expr": "100-caches_memory_hit_ratio_diff_intraline{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_diff_intraline{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "diff intraline",
"refId": "B"
},
{
- "expr": "100-caches_memory_hit_ratio_diff_summary{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_diff_summary{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "diff summary",
"refId": "C"
}
@@ -563,22 +563,22 @@
"steppedLine": false,
"targets": [
{
- "expr": "100-caches_memory_hit_ratio_web_sessions{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_web_sessions{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "web sessions",
"refId": "A"
},
{
- "expr": "100-caches_memory_hit_ratio_sshkeys{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_sshkeys{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "sshkeys",
"refId": "B"
},
{
- "expr": "100-caches_memory_hit_ratio_git_tags{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_git_tags{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "git tags",
"refId": "D"
},
{
- "expr": "100-caches_memory_hit_ratio_permission_sort{instance=\"$instance\"}",
+ "expr": "100-caches_memory_hit_ratio_permission_sort{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "permission_sort",
"refId": "C"
}
@@ -655,6 +655,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "text": "",
+ "value": ""
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "hide": 0,
+ "includeAll": false,
+ "label": "Replica",
+ "multi": false,
+ "name": "replica",
+ "options": [],
+ "query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "refresh": 2,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
}
]
},
diff --git a/dashboards/gerrit-fetch-clone.json b/dashboards/gerrit-fetch-clone.json
index 99201e5..c1990fe 100644
--- a/dashboards/gerrit-fetch-clone.json
+++ b/dashboards/gerrit-fetch-clone.json
@@ -60,7 +60,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\"}[2m])/2",
+ "expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\",replica=\"$replica\"}[2m])/2",
"legendFormat": "clone",
"refId": "A"
}
@@ -149,7 +149,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\"}[2m])/2",
+ "expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\",replica=\"$replica\"}[2m])/2",
"legendFormat": "fetch",
"refId": "A"
}
@@ -253,7 +253,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "git_upload_pack_pack_bytes_CLONE{instance=\"$instance\"}",
+ "expr": "git_upload_pack_pack_bytes_CLONE{instance=\"$instance\",replica=\"$replica\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "quantile: {{quantile}}",
@@ -360,7 +360,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "git_upload_pack_pack_bytes_FETCH{instance=\"$instance\"}",
+ "expr": "git_upload_pack_pack_bytes_FETCH{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
}
@@ -438,6 +438,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "text": "",
+ "value": ""
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "hide": 0,
+ "includeAll": false,
+ "label": "Replica",
+ "multi": false,
+ "name": "replica",
+ "options": [],
+ "query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "refresh": 2,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
}
]
},
diff --git a/dashboards/gerrit-latency.json b/dashboards/gerrit-latency.json
index 6d77d31..5e8c657 100644
--- a/dashboards/gerrit-latency.json
+++ b/dashboards/gerrit-latency.json
@@ -73,7 +73,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "receivecommits_latency_total{instance=\"$instance\"}",
+ "expr": "receivecommits_latency_total{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -174,7 +174,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_total{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_total{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -275,7 +275,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "query_query_latency_total{instance=\"$instance\"}",
+ "expr": "query_query_latency_total{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -376,7 +376,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_GetDetail{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_GetDetail{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -477,7 +477,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "query_query_latency_changes{instance=\"$instance\"}",
+ "expr": "query_query_latency_changes{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -578,7 +578,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_GetDiff{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_GetDiff{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -679,7 +679,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_ListChangeComments{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_ListChangeComments{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -780,7 +780,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_GetChange{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_GetChange{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -881,7 +881,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_ListChangeRobotComments{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_ListChangeRobotComments{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -983,7 +983,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_GetCommit{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_GetCommit{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -1084,7 +1084,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_PostReview{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_PostReview{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -1185,7 +1185,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_rest_api_server_latency_restapi_change_GetRevisionActions{instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_restapi_change_GetRevisionActions{instance=\"$instance\",replica=\"$replica\"}",
"interval": "",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
@@ -1263,6 +1263,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "text": "",
+ "value": ""
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "hide": 0,
+ "includeAll": false,
+ "label": "Replica",
+ "multi": false,
+ "name": "replica",
+ "options": [],
+ "query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "refresh": 2,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
}
]
},
@@ -1291,4 +1316,4 @@
"list": []
},
"version": 1
-}
\ No newline at end of file
+}
diff --git a/dashboards/gerrit-overview.json b/dashboards/gerrit-overview.json
index 82f819f..8508ec8 100644
--- a/dashboards/gerrit-overview.json
+++ b/dashboards/gerrit-overview.json
@@ -143,7 +143,7 @@
],
"targets": [
{
- "expr": "max(gerrit_build_info{instance=\"$instance\"}) by (instance, version, revision, javaversion)",
+ "expr": "max(gerrit_build_info{instance=\"$instance\",replica=\"$replica\"}) by (instance, version, revision, javaversion)",
"format": "table",
"instant": true,
"interval": "",
@@ -232,7 +232,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
- "expr": "(rate(proc_cpu_usage{instance=\"$instance\"}[5m])/proc_cpu_num_cores{instance=\"$instance\"})*100",
+ "expr": "(rate(proc_cpu_usage{instance=\"$instance\",replica=\"$replica\"}[5m])/proc_cpu_num_cores{instance=\"$instance\",replica=\"$replica\"})*100",
"refId": "A"
}
],
@@ -302,7 +302,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
- "expr": "(proc_jvm_memory_heap_used{instance=\"$instance\"}/proc_jvm_memory_heap_committed{instance=\"$instance\"})*100",
+ "expr": "(proc_jvm_memory_heap_used{instance=\"$instance\",replica=\"$replica\"}/proc_jvm_memory_heap_committed{instance=\"$instance\",replica=\"$replica\"})*100",
"refId": "A"
}
],
@@ -352,17 +352,17 @@
"pluginVersion": "6.7.1",
"targets": [
{
- "expr": "proc_jvm_thread_num_daemon_live{instance=\"$instance\"}",
+ "expr": "proc_jvm_thread_num_daemon_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "current live threads (daemon)",
"refId": "A"
},
{
- "expr": "proc_jvm_thread_num_live{instance=\"$instance\"}",
+ "expr": "proc_jvm_thread_num_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "current live threads",
"refId": "B"
},
{
- "expr": "proc_jvm_thread_num_peak_live{instance=\"$instance\"}",
+ "expr": "proc_jvm_thread_num_peak_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "peak of live threads",
"refId": "C"
}
@@ -420,7 +420,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
- "expr": "http_server_rest_api_server_latency_total{quantile=\"0.99\", instance=\"$instance\"}",
+ "expr": "http_server_rest_api_server_latency_total{quantile=\"0.99\", instance=\"$instance\",replica=\"$replica\"}",
"intervalFactor": 4,
"legendFormat": "quantile {{quantile}}",
"refId": "A"
@@ -493,7 +493,7 @@
"pluginVersion": "6.7.1",
"targets": [
{
- "expr": "(increase(http_server_error_count_total_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\"}[5m])) / (increase(http_server_success_count_total_total{instance=\"$instance\"}[5m]) + increase(http_server_error_count_total_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\"}[5m]))*100",
+ "expr": "(increase(http_server_error_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\",replica=\"$replica\"}[5m])) / (increase(http_server_success_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m]) + increase(http_server_error_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_404_total{instance=\"$instance\",replica=\"$replica\"}[5m]) - increase(http_server_error_count_401_total{instance=\"$instance\",replica=\"$replica\"}[5m]))*100",
"refId": "A"
}
],
@@ -557,7 +557,7 @@
"steppedLine": true,
"targets": [
{
- "expr": "up{instance=\"$instance\"}",
+ "expr": "up{instance=\"$instance\",replica=\"$replica\"}",
"format": "time_series",
"legendFormat": "{{instance}}",
"refId": "A"
@@ -669,7 +669,7 @@
"tableColumn": "",
"targets": [
{
- "expr": "avg_over_time(up{instance=\"$instance\"}[1d])*100",
+ "expr": "avg_over_time(up{instance=\"$instance\",replica=\"$replica\"}[1d])*100",
"format": "time_series",
"legendFormat": "{{instance}}",
"refId": "A"
@@ -798,98 +798,98 @@
"steppedLine": false,
"targets": [
{
- "expr": "(increase(http_server_success_count_200_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_success_count_200_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "200",
"refId": "A"
},
{
- "expr": "(increase(http_server_success_count_201_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_success_count_201_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "201",
"refId": "C"
},
{
- "expr": "(increase(http_server_success_count_204_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_success_count_204_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "204",
"refId": "D"
},
{
- "expr": "(increase(http_server_success_count_301_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_success_count_301_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "301",
"refId": "E"
},
{
- "expr": "(increase(http_server_success_count_304_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_success_count_304_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "304",
"refId": "F"
},
{
- "expr": "(increase(http_server_error_count_400_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_400_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "400",
"refId": "B"
},
{
- "expr": "(increase(http_server_error_count_401_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_401_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "401",
"refId": "G"
},
{
- "expr": "(increase(http_server_error_count_403_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_403_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "403",
"refId": "H"
},
{
- "expr": "(increase(http_server_error_count_404_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_404_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"instant": false,
"intervalFactor": 1,
"legendFormat": "404",
"refId": "I"
},
{
- "expr": "(increase(http_server_error_count_405_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_405_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "405",
"refId": "J"
},
{
- "expr": "(increase(http_server_error_count_409_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_409_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "409",
"refId": "K"
},
{
- "expr": "(increase(http_server_error_count_412_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_412_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "412",
"refId": "L"
},
{
- "expr": "(increase(http_server_error_count_422_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_422_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "422",
"refId": "M"
},
{
- "expr": "(increase(http_server_error_count_500_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_500_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "500",
"refId": "N"
},
{
- "expr": "(increase(http_server_error_count_501_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_501_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "501",
"refId": "O"
},
{
- "expr": "(increase(http_server_error_count_503_total{instance=\"$instance\"}[5m]))/5",
+ "expr": "(increase(http_server_error_count_503_total{instance=\"$instance\",replica=\"$replica\"}[5m]))/5",
"intervalFactor": 1,
"legendFormat": "503",
"refId": "P"
@@ -981,73 +981,73 @@
"steppedLine": false,
"targets": [
{
- "expr": "increase(events_assignee_changed_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_assignee_changed_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "assignee changed",
"refId": "A"
},
{
- "expr": "increase(events_change_abandoned_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_change_abandoned_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "change abandoned",
"refId": "B"
},
{
- "expr": "increase(events_change_merged_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_change_merged_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "change merged",
"refId": "C"
},
{
- "expr": "increase(events_comment_added_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_comment_added_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "comment added",
"refId": "D"
},
{
- "expr": "increase(events_patchset_created_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_patchset_created_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "patchset created",
"refId": "E"
},
{
- "expr": "increase(events_ref_replicated_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_ref_replicated_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "ref replicated",
"refId": "F"
},
{
- "expr": "increase(events_ref_updated_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_ref_updated_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "ref updated",
"refId": "G"
},
{
- "expr": "increase(events_reviewer_added_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_reviewer_added_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "reviewer added",
"refId": "H"
},
{
- "expr": "increase(events_reviewer_deleted_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_reviewer_deleted_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "reviewer deleted",
"refId": "I"
},
{
- "expr": "increase(events_topic_changed_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_topic_changed_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "topic changed",
"refId": "J"
},
{
- "expr": "increase(events_vote_deleted_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_vote_deleted_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "vote deleted",
"refId": "K"
},
{
- "expr": "increase(events_wip_state_changed_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(events_wip_state_changed_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 1,
"legendFormat": "wip state changed",
"refId": "L"
@@ -1140,13 +1140,13 @@
"steppedLine": false,
"targets": [
{
- "expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(git_upload_pack_request_count_FETCH_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 4,
"legendFormat": "Fetch",
"refId": "B"
},
{
- "expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\"}[5m])",
+ "expr": "increase(git_upload_pack_request_count_CLONE_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"intervalFactor": 4,
"legendFormat": "Clone",
"refId": "A"
@@ -1227,6 +1227,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "text": "",
+ "value": ""
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "hide": 0,
+ "includeAll": false,
+ "label": "Replica",
+ "multi": false,
+ "name": "replica",
+ "options": [],
+ "query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "refresh": 2,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
}
]
},
diff --git a/dashboards/gerrit-process.json b/dashboards/gerrit-process.json
index 335dd15..fd76368 100644
--- a/dashboards/gerrit-process.json
+++ b/dashboards/gerrit-process.json
@@ -59,7 +59,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "proc_cpu_system_load{instance=\"$instance\"}",
+ "expr": "proc_cpu_system_load{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "system load",
"refId": "A"
}
@@ -148,23 +148,23 @@
"steppedLine": false,
"targets": [
{
- "expr": "proc_jvm_memory_heap_committed{instance=\"$instance\"}",
+ "expr": "proc_jvm_memory_heap_committed{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "committed heap",
"refId": "C"
},
{
- "expr": "proc_jvm_memory_heap_used{instance=\"$instance\"}",
+ "expr": "proc_jvm_memory_heap_used{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "used heap",
"refId": "B"
},
{
- "expr": "jgit_block_cache_cache_used{instance=\"$instance\"}",
+ "expr": "jgit_block_cache_cache_used{instance=\"$instance\",replica=\"$replica\"}",
"instant": false,
"legendFormat": "JGit block cache",
"refId": "A"
},
{
- "expr": "proc_jvm_memory_non_heap_used{instance=\"$instance\"}",
+ "expr": "proc_jvm_memory_non_heap_used{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "used non-heap",
"refId": "D"
}
@@ -261,12 +261,12 @@
"steppedLine": false,
"targets": [
{
- "expr": "rate(proc_cpu_usage{instance=\"$instance\"}[5m])",
+ "expr": "rate(proc_cpu_usage{instance=\"$instance\",replica=\"$replica\"}[5m])",
"legendFormat": "used CPUs",
"refId": "A"
},
{
- "expr": "proc_cpu_num_cores{instance=\"$instance\"}",
+ "expr": "proc_cpu_num_cores{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "available CPUs",
"refId": "B"
}
@@ -365,12 +365,12 @@
"steppedLine": false,
"targets": [
{
- "expr": "increase(proc_jvm_gc_time_G1_Young_Generation{instance=\"$instance\"}[2m])/increase(proc_uptime{instance=\"$instance\"}[2m])",
+ "expr": "increase(proc_jvm_gc_time_G1_Young_Generation{instance=\"$instance\",replica=\"$replica\"}[2m])/increase(proc_uptime{instance=\"$instance\",replica=\"$replica\"}[2m])",
"legendFormat": "gc time G1 young gen",
"refId": "B"
},
{
- "expr": "increase(proc_jvm_gc_time_G1_Old_Generation{instance=\"$instance\"}[2m])/increase(proc_uptime{instance=\"$instance\"}[2m])",
+ "expr": "increase(proc_jvm_gc_time_G1_Old_Generation{instance=\"$instance\",replica=\"$replica\"}[2m])/increase(proc_uptime{instance=\"$instance\",replica=\"$replica\"}[2m])",
"legendFormat": "gc time G1 old gen",
"refId": "A"
}
@@ -459,7 +459,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "proc_jvm_thread_num_live{instance=\"$instance\"}",
+ "expr": "proc_jvm_thread_num_live{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "Java live threads",
"refId": "A"
}
@@ -548,12 +548,12 @@
"steppedLine": false,
"targets": [
{
- "expr": "jgit_block_cache_open_files{instance=\"$instance\"}",
+ "expr": "jgit_block_cache_open_files{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "jgit block cache",
"refId": "B"
},
{
- "expr": "proc_num_open_fds{instance=\"$instance\"}-jgit_block_cache_open_files{instance=\"$instance\"}",
+ "expr": "proc_num_open_fds{instance=\"$instance\",replica=\"$replica\"}-jgit_block_cache_open_files{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "other",
"refId": "C"
}
@@ -651,14 +651,14 @@
"steppedLine": false,
"targets": [
{
- "expr": "increase(jgit_block_cache_miss_count{instance=\"$instance\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\"}[2m]))",
+ "expr": "increase(jgit_block_cache_miss_count{instance=\"$instance\",replica=\"$replica\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\",replica=\"$replica\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\",replica=\"$replica\"}[2m]))",
"format": "time_series",
"instant": false,
"legendFormat": "miss ratio",
"refId": "A"
},
{
- "expr": "increase(jgit_block_cache_eviction_count{instance=\"$instance\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\"}[2m]))",
+ "expr": "increase(jgit_block_cache_eviction_count{instance=\"$instance\",replica=\"$replica\"}[2m])/(increase(jgit_block_cache_hit_count{instance=\"$instance\",replica=\"$replica\"}[2m])+increase(jgit_block_cache_miss_count{instance=\"$instance\",replica=\"$replica\"}[2m]))",
"format": "time_series",
"instant": false,
"legendFormat": "eviction ratio",
@@ -739,6 +739,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "text": "",
+ "value": ""
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "hide": 0,
+ "includeAll": false,
+ "label": "Replica",
+ "multi": false,
+ "name": "replica",
+ "options": [],
+ "query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "refresh": 2,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
}
]
},
diff --git a/dashboards/gerrit-queues.json b/dashboards/gerrit-queues.json
index fce3587..bf34c48 100644
--- a/dashboards/gerrit-queues.json
+++ b/dashboards/gerrit-queues.json
@@ -83,22 +83,22 @@
"steppedLine": false,
"targets": [
{
- "expr": "queue_ssh_batch_worker_active_threads{instance=\"$instance\"}",
+ "expr": "queue_ssh_batch_worker_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch threads",
"refId": "C"
},
{
- "expr": "queue_ssh_batch_worker_pool_size{instance=\"$instance\"}",
+ "expr": "queue_ssh_batch_worker_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch pool size",
"refId": "D"
},
{
- "expr": "queue_ssh_interactive_worker_active_threads{instance=\"$instance\"}",
+ "expr": "queue_ssh_interactive_worker_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive threads",
"refId": "A"
},
{
- "expr": "queue_ssh_interactive_worker_pool_size{instance=\"$instance\"}",
+ "expr": "queue_ssh_interactive_worker_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive pool size",
"refId": "B"
}
@@ -208,22 +208,22 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_jetty_threadpool_active_threads{instance=\"$instance\"}",
+ "expr": "http_server_jetty_threadpool_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "active threads",
"refId": "A"
},
{
- "expr": "http_server_jetty_threadpool_reserved_threads{instance=\"$instance\"}",
+ "expr": "http_server_jetty_threadpool_reserved_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "reserved threads",
"refId": "C"
},
{
- "expr": "http_server_jetty_threadpool_max_pool_size{instance=\"$instance\"}",
+ "expr": "http_server_jetty_threadpool_max_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "max pool size",
"refId": "B"
},
{
- "expr": "http_server_jetty_threadpool_pool_size{instance=\"$instance\"}",
+ "expr": "http_server_jetty_threadpool_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "pool size",
"refId": "D"
}
@@ -333,22 +333,22 @@
"steppedLine": false,
"targets": [
{
- "expr": "queue_index_batch_active_threads{instance=\"$instance\"}",
+ "expr": "queue_index_batch_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch threads",
"refId": "A"
},
{
- "expr": "queue_index_batch_pool_size{instance=\"$instance\"}",
+ "expr": "queue_index_batch_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch pool size",
"refId": "B"
},
{
- "expr": "queue_index_interactive_active_threads{instance=\"$instance\"}",
+ "expr": "queue_index_interactive_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive threads",
"refId": "C"
},
{
- "expr": "queue_index_interactive_pool_size{instance=\"$instance\"}",
+ "expr": "queue_index_interactive_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive pool size",
"refId": "D"
}
@@ -448,12 +448,12 @@
"steppedLine": false,
"targets": [
{
- "expr": "queue_ssh_batch_worker_scheduled_tasks{instance=\"$instance\"}",
+ "expr": "queue_ssh_batch_worker_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch",
"refId": "C"
},
{
- "expr": "queue_ssh_interactive_worker_scheduled_tasks{instance=\"$instance\"}",
+ "expr": "queue_ssh_interactive_worker_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive",
"refId": "A"
}
@@ -547,7 +547,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "http_server_jetty_threadpool_queue_size{instance=\"$instance\"}",
+ "expr": "http_server_jetty_threadpool_queue_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "http",
"refId": "A"
}
@@ -648,12 +648,12 @@
"steppedLine": false,
"targets": [
{
- "expr": "queue_index_batch_scheduled_tasks{instance=\"$instance\"}",
+ "expr": "queue_index_batch_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "batch",
"refId": "C"
},
{
- "expr": "queue_index_interactive_scheduled_tasks{instance=\"$instance\"}",
+ "expr": "queue_index_interactive_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "interactive",
"refId": "A"
}
@@ -771,32 +771,32 @@
"steppedLine": false,
"targets": [
{
- "expr": "queue_ssh_stream_worker_active_threads{instance=\"$instance\"}",
+ "expr": "queue_ssh_stream_worker_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "stream",
"refId": "A"
},
{
- "expr": "queue_ssh_stream_worker_pool_size{instance=\"$instance\"}",
+ "expr": "queue_ssh_stream_worker_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "stream pool",
"refId": "B"
},
{
- "expr": "queue_send_email_active_threads{instance=\"$instance\"}",
+ "expr": "queue_send_email_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "email",
"refId": "C"
},
{
- "expr": "queue_send_email_pool_size{instance=\"$instance\"}",
+ "expr": "queue_send_email_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "email pool",
"refId": "D"
},
{
- "expr": "queue_receive_commits_active_threads{instance=\"$instance\"}",
+ "expr": "queue_receive_commits_active_threads{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "receive-commit",
"refId": "E"
},
{
- "expr": "queue_receive_commits_pool_size{instance=\"$instance\"}",
+ "expr": "queue_receive_commits_pool_size{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "receive-commit pool",
"refId": "F"
}
@@ -887,7 +887,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "rate(http_server_rest_api_count_total_total{instance=\"$instance\"}[5m])",
+ "expr": "rate(http_server_rest_api_count_total_total{instance=\"$instance\",replica=\"$replica\"}[5m])",
"legendFormat": "REST API request rate",
"refId": "A"
}
@@ -992,17 +992,17 @@
"steppedLine": false,
"targets": [
{
- "expr": "queue_ssh_stream_worker_scheduled_tasks{instance=\"$instance\"}",
+ "expr": "queue_ssh_stream_worker_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "stream",
"refId": "E"
},
{
- "expr": "queue_send_email_scheduled_tasks{instance=\"$instance\"}",
+ "expr": "queue_send_email_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "email",
"refId": "C"
},
{
- "expr": "queue_receive_commits_scheduled_tasks{instance=\"$instance\"}",
+ "expr": "queue_receive_commits_scheduled_tasks{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "receive-commit",
"refId": "D"
}
@@ -1079,6 +1079,31 @@
"tagsQuery": "",
"type": "query",
"useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "text": "",
+ "value": ""
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "hide": 0,
+ "includeAll": false,
+ "label": "Replica",
+ "multi": false,
+ "name": "replica",
+ "options": [],
+ "query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "refresh": 2,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
}
]
},
diff --git a/dashboards/gerrit-replication.json b/dashboards/gerrit-replication.json
index 871be51..e5cdea7 100644
--- a/dashboards/gerrit-replication.json
+++ b/dashboards/gerrit-replication.json
@@ -60,7 +60,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "plugins_replication_replication_delay_$target{instance=\"$instance\"}",
+ "expr": "plugins_replication_replication_delay_$target{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "quantile: {{quantile}}",
"refId": "B"
}
@@ -148,7 +148,7 @@
"steppedLine": false,
"targets": [
{
- "expr": "plugins_replication_replication_latency_$target{instance=\"$instance\"}",
+ "expr": "plugins_replication_replication_latency_$target{instance=\"$instance\",replica=\"$replica\"}",
"legendFormat": "quantile: {{quantile}}",
"refId": "A"
}
@@ -228,6 +228,31 @@
{
"allValue": null,
"current": {
+ "text": "",
+ "value": ""
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "hide": 0,
+ "includeAll": false,
+ "label": "Replica",
+ "multi": false,
+ "name": "replica",
+ "options": [],
+ "query": "label_values(git_upload_pack_phase_writing_total{instance=\"$instance\"}, replica)",
+ "refresh": 2,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
"tags": [],
"text": "",
"value": ""
diff --git a/promtail/promtail.yaml b/promtail/promtail.yaml
index d03e026..a56e625 100644
--- a/promtail/promtail.yaml
+++ b/promtail/promtail.yaml
@@ -1,20 +1,20 @@
#@ load("@ytt:data", "data")
-#@ for i in range(len(data.values.gerritServers)):
+#@ for i in range(len(data.values.gerritServers.other)):
---
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
- filename: #@ "{}/positions.yaml".format(data.values.gerritServers[i].promtail.storagePath)
+ filename: #@ "{}/positions.yaml".format(data.values.gerritServers.other[i].promtail.storagePath)
clients:
- url: #@ "https://{}/loki/api/v1/push".format(data.values.loki.host)
tls_config:
insecure_skip_verify: #@ data.values.tls.skipVerify
#@ if not data.values.tls.skipVerify:
- ca_file: #@ "{}/promtail.ca.crt".format(data.values.gerritServers[i].promtail.storagePath)
+ ca_file: #@ "{}/promtail.ca.crt".format(data.values.gerritServers.other[i].promtail.storagePath)
#@ end
basic_auth:
username: #@ data.values.loki.username
@@ -26,8 +26,8 @@
- localhost
labels:
job: gerrit_error
- __path__: #@ "{}/error_log.json".format(data.values.gerritServers[i].promtail.logPath)
- host: #@ data.values.gerritServers[i].host
+ __path__: #@ "{}/error_log.json".format(data.values.gerritServers.other[i].promtail.logPath)
+ host: #@ data.values.gerritServers.other[i].host
entry_parser: raw
pipeline_stages:
- json:
@@ -57,8 +57,8 @@
- localhost
labels:
job: gerrit_httpd
- __path__: #@ "{}/httpd_log.json".format(data.values.gerritServers[i].promtail.logPath)
- host: #@ data.values.gerritServers[i].host
+ __path__: #@ "{}/httpd_log.json".format(data.values.gerritServers.other[i].promtail.logPath)
+ host: #@ data.values.gerritServers.other[i].host
entry_parser: raw
pipeline_stages:
- json:
@@ -76,8 +76,8 @@
- localhost
labels:
job: gerrit_sshd
- __path__: #@ "{}/sshd_log.json".format(data.values.gerritServers[i].promtail.logPath)
- host: #@ data.values.gerritServers[i].host
+ __path__: #@ "{}/sshd_log.json".format(data.values.gerritServers.other[i].promtail.logPath)
+ host: #@ data.values.gerritServers.other[i].host
entry_parser: raw
pipeline_stages:
- json:
diff --git a/subcommands/install.py b/subcommands/install.py
index 9803bce..a51461a 100644
--- a/subcommands/install.py
+++ b/subcommands/install.py
@@ -231,10 +231,12 @@
namespace = config_manager.get_config()["namespace"]
_create_dashboard_configmaps(output_dir, namespace)
- _create_promtail_configs(config, output_dir)
+ if os.path.exists(os.path.join(output_dir, "promtail.yaml")):
+ _create_promtail_configs(config, output_dir)
+ if not dryrun:
+ _download_promtail(output_dir)
if not dryrun:
- _download_promtail(output_dir)
if update_repo:
_update_helm_repos()
_deploy_loose_resources(output_dir)
