| commit | b141ef4c41ced4b4a631b6c3dede5aace33638f9 | [log] [tgz] |
|---|---|---|
| author | Jacek Centkowski <geminica.programs@gmail.com> | Wed Apr 28 22:19:58 2021 +0200 |
| committer | Jacek Centkowski <geminica.programs@gmail.com> | Fri May 28 16:20:39 2021 +0200 |
| tree | 3a981899f47b93d2d1baecdd69d09d3faa5a49f1 | |
| parent | 52211e251f2a04056a12f8bfa6ffc4d294a70ad3 [diff] |
Add blocked threads check
Introduce check that detects the blocked threads (enabled by default).
In case when it reaches above the configured percent of all threads (by
default 50%) check fails and instance is considered unhealthy.
Check can:
* be disabled with 'healthcheck.blockedthreads.enabled = false'
* have threshold reconfigured with 'healthcheck.blocked.threshold=XX'
Note that one can fine tune the check to certain group of threads by
configuring threshold for threads with name starting wiht prefix.
For instance in case when Gerrit instance should be considered
unhealthy when 33% of SSH-Interactive-Worker threads are 'BLOCKED'
one can configure it with
[healthcheck "blockedthreads"]
threshold = SSH-Interactive-Threads=33
Note that multiple checks for multiple thread group could be
configured (for details with examples see the config.md file).
Bug: Issue 14401
Change-Id: I7d0c4b70decf49465a7318abe4a555cde7318833
Allow having a single entry point to check the availability of the services that Gerrit exposes.
Clone or link this plugin to the plugins directory of Gerrit‘s source tree, and then run bazel build on the plugin’s directory.
Example:
git clone --recursive https://gerrit.googlesource.com/gerrit git clone https://gerrit.googlesource.com/plugins/healthcheck pushd gerrit/plugins && ln -s ../../healthcheck . && popd cd gerrit && bazel build plugins/healthcheck
The output plugin jar is created in:
bazel-genfiles/plugins/healthcheck/healthcheck.jar
Copy the healthcheck.jar into the Gerrit's /plugins directory and wait for the plugin to be automatically loaded. The healthcheck plugin is compatible with both primary Gerrit setups and Gerrit replicas. The only difference to bear in mind is that some checks will be automatically disabled on replicas (e.g. query changes) because the associated subsystem is switched off.
The healthcheck plugin exposes a single endpoint under its root URL and provides a JSON output of the Gerrit health status.
The HTTP status code returned indicates whether Gerrit is healthy (HTTP status 200) or has some issues (HTTP status 500).
The HTTP response payload is a JSON output that contains the details of the checks performed.
Each check returns a JSON payload with the following information:
ts: epoch timestamp in millis of the individual check
elapsed: elapsed time in millis to complete the check
result: result of the health check
Example of a healthy Gerrit response:
GET /config/server/healthcheck~status
200 OK
Content-Type: application/json
)]}'
{
"ts": 139402910202,
"elapsed": 100,
"querychanges": {
"ts": 139402910202,
"elapsed": 20,
"result": "passed"
},
"reviewdb": {
"ts": 139402910202,
"elapsed": 50,
"result": "passed"
},
"projectslist": {
"ts": 139402910202,
"elapsed": 100,
"result": "passed"
},
"jgit": {
"ts": 139402910202,
"elapsed": 80,
"result": "passed"
}
}
Example of a Gerrit instance with the projects list timing out:
GET /config/server/healthcheck~status
500 ERROR
Content-Type: application/json
)]}'
{
"ts": 139402910202,
"elapsed": 100,
"querychanges": {
"ts": 139402910202,
"elapsed": 20,
"result": "passed"
},
"reviewdb": {
"ts": 139402910202,
"elapsed": 50,
"result": "passed"
},
"projectslist": {
"ts": 139402910202,
"elapsed": 100,
"result": "timeout"
},
"jgit": {
"ts": 139402910202,
"elapsed": 80,
"result": "passed"
}
}
As for all other endpoints in Gerrit, some metrics are automatically emitted when the /config/server/healthcheck~status endpoint is hit (thanks to the Dropwizard library).
Some additional metrics are also produced to give extra insights on their result about results and latency of healthcheck sub component, such as jgit, reviewdb, etc.
More information can be found in the config.md file.