| :linkattrs: | |
| = Request Tracing | |
| [[on-demand]] | |
| == On-demand Request Tracing | |
| Gerrit supports on-demand tracing of single requests that results in | |
| additional logs with debug information that are written to the | |
| `error_log`. The logs that correspond to a traced request are | |
| associated with a unique trace ID. This trace ID is returned with the | |
| response and can be used by an administrator to find the matching log | |
| entries. | |
| How tracing is enabled and how the trace ID is returned depends on the | |
| request type: | |
| * REST API: For REST calls tracing can be enabled by setting the | |
| `trace` request parameter or the `X-Gerrit-Trace` header, the trace | |
| ID is returned as `X-Gerrit-Trace` header. More information about | |
| this can be found in the link:rest-api.html#tracing[Request Tracing] | |
| section of the link:rest-api.html[REST API documentation]. | |
| * SSH API: For SSH calls tracing can be enabled by setting the | |
| `--trace` option. More information about this can be found in | |
| the link:cmd-index.html#trace[Trace] section of the | |
| link:cmd-index.html[SSH command documentation]. | |
| * Git Push (requires usage of git protocol v2): For Git pushes tracing | |
| can be enabled by setting the `trace` push option, the trace ID is | |
| returned in the command output. More information about this can be | |
| found in the link:user-upload.html#trace[Trace] section of the | |
| link:user-upload.html[upload documentation]. | |
| * Git Clone/Fetch/Ls-Remote (requires usage of git protocol v2): For | |
| Git clone/fetch/ls-remote tracing can be enabled by setting the | |
| `trace` server option. Use '-o trace=<TRACE-ID>' for `git fetch` and | |
| `git ls-remote`, and '--server-option trace=<TRACE-ID>' for | |
| `git clone`. If the `trace` server option is set without a value | |
| (without trace ID) a trace ID is generated but the generated trace ID | |
| is not returned to the client (hence a trace ID should always be | |
| set). | |
| When request tracing is enabled it is possible to provide an ID that | |
| should be used as trace ID. If a trace ID is not provided a trace ID is | |
| automatically generated. The trace ID must be provided to the support | |
| team (administrator of the server) so that they can find the trace. | |
| When doing traces it is recommended to specify the ID of the issue | |
| that is being investigated as trace ID so that the traces of the issue | |
| can be found more easily. When the issue ID is used as trace ID there | |
| is no need to find the generated trace ID and report it in the issue. | |
| Since tracing consumes additional server resources tracing should only | |
| be enabled for single requests if there is a concrete need for | |
| debugging. In particular bots should never enable tracing for all their | |
| requests by default. | |
| [[auto-retry]] | |
| == Automatic Request Tracing | |
| Gerrit can be link:config-gerrit.html#retry.retryWithTraceOnFailure[ | |
| configured] to automatically retry requests on non-recoverable failures | |
| with tracing enabled. This allows to automatically captures traces of | |
| these failures for further analysis by the Gerrit administrators. | |
| The auto-retry on failure behaves the same way as if the calling user | |
| would retry the failed operation with tracing enabled. | |
| It is expected that the auto-retry fails with the same exception that | |
| triggered the auto-retry, however this is not guaranteed: | |
| * Not all Gerrit operations are fully atomic and it can happen that | |
| some parts of the operation have been successfully performed before | |
| the failure happened. In this case the auto-retry may fail with a | |
| different exception. | |
| * Some exceptions may mistakenly be considered as non-recoverable and | |
| the auto-retry actually succeeds. | |
| [[auto-retry-succeeded]] | |
| If an auto-retry succeeds you may consider filing this as | |
| link:https://bugs.chromium.org/p/gerrit/issues/entry?template=GoogleSource+Issue[ | |
| Gerrit issue,role=external,window=_blank] so that the Gerrit developers can fix this and treat this | |
| exception as recoverable. | |
| The trace IDs for auto-retries are generated and start with | |
| `retry-on-failure-`. For REST requests they are returned to the client | |
| as `X-Gerrit-Trace` header. | |
| The best way to search for auto-retries in logs is to do a grep by | |
| `AutoRetry`. For each auto-retry that happened this should match 1 or 2 | |
| log entries: | |
| * one `FINE` log entry with the exception that triggered the auto-retry | |
| * one `FINE` log entry with the exception that happened on auto-retry | |
| (if this log entry is not present the operation succeeded on | |
| auto-retry) | |
| To inspect single auto-retry occurrences in detail you can do a | |
| link:#find-trace[grep by the trace ID]. The trace ID is part of the log | |
| entries which have been found by the previous grep (watch out for | |
| something like: `retry-on-failure-1534166888910-3985dfba`). | |
| [TIP] | |
| Auto-retrying on failures is only supported by some of the REST | |
| endpoints (change REST endpoints that perform updates). | |
| [[auto-retry-metrics]] | |
| === Metrics | |
| If auto-retry is link:config-gerrit.html#retry.retryWithTraceOnFailure[ | |
| enabled] the following metrics are reported: | |
| * `action/auto_retry_count`: Number of automatic retries with tracing | |
| * `action/failures_on_auto_retry_count`: Number of failures on auto retry | |
| By comparing the values of these counters one can see how often the | |
| auto-retry succeeds. As explained link:#auto-retry-succeeded[above] if | |
| auto-retries succeed that's an issue with Gerrit that you may want to | |
| report. | |
| [[find-trace]] | |
| == Find log entries for a trace ID | |
| If tracing is enabled all log messages that correspond to the traced | |
| request have a `TRACE_ID` tag set, e.g.: | |
| ---- | |
| [2018-08-13 15:28:08,913] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : Received REST request: GET /a/accounts/self (parameters: [trace]) [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ] | |
| [2018-08-13 15:28:08,914] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : Calling user: admin [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ] | |
| [2018-08-13 15:28:08,942] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : REST call succeeded: 200 [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ] | |
| ---- | |
| By doing a grep with the trace ID over the error log the log entries | |
| that correspond to the request can be found. | |
| [TIP] | |
| Usually only server administrators have access to the logs. | |
| == Which information is captured in a trace? | |
| * request details | |
| ** REST API: request URL, request parameter names, calling user, | |
| response code, response body on errors | |
| ** SSH API: parameter names | |
| ** Git API: push options, magic branch parameter names | |
| * cache misses, cache evictions | |
| * reads from NoteDb, writes to NoteDb | |
| * reads of meta data files, writes of meta data files | |
| * index queries (with parameters and matches) | |
| * reindex events | |
| * permission checks (e.g. which rule is responsible for a deny) | |
| * timer metrics | |
| * all other logs | |
| GERRIT | |
| ------ | |
| Part of link:index.html[Gerrit Code Review] | |
| SEARCHBOX | |
| --------- |