| :linkattrs: | |
| = Request Cancellation and Deadlines | |
| [[motivation]] | |
| == Motivation | |
| Protect the Gerrit service by aborting requests that were cancelled or for which | |
| the deadline has exceeded. If these requests are not aborted, it can happen that | |
| too many of these requests are accumulated so that the server runs out of | |
| resources (e.g. threads). | |
| [[request-cancellation]] | |
| == Request Cancellation | |
| If a user cancels a request by disconnecting, ideally Gerrit should detect this | |
| and abort the request execution to avoid doing unnecessary work. If nobody is | |
| waiting for the response, Gerrit shouldn't spend resources to compute it. | |
| Detecting cancelled requests is not easily possible with all protocols that a | |
| client may use. At the moment Gerrit only detects request cancellations for git | |
| pushes, but not for other request types (in particular cancelled requests are | |
| not detected for REST calls over HTTP, SSH commands and git clone/fetch). | |
| [[server-side-deadlines]] | |
| == Server-side deadlines | |
| To limit the maximal execution time for requests, administrators can | |
| link:config-gerrit.html#deadline.id[configure server-side deadlines]. If a | |
| server-side deadline is exceeded by a matching request, the request is | |
| automatically aborted. In this case the client gets a proper error message | |
| informing the user about the exceeded deadline. | |
| Clients may override server-side deadlines by setting a | |
| link:#client-provided-deadlines[deadline] on the request. This means, if a | |
| request fails due to an exceeded server-side deadline, the client may repeat the | |
| request with a higher deadline or no deadline (deadline = 0) to get unblocked. | |
| Server-side deadlines are meant to protect the Gerrit service against resource | |
| exhaustion due to performence issues with a particular request. E.g. imagine a | |
| situation where requests for a certain REST endpoint are very slow. If more and | |
| more of such requests get stuck and are not being aborted, the Gerrit service | |
| may run out of threads, causing an outage for the entire Gerrit service. | |
| Server-side deadlines may prevent this because the slow requests get aborted | |
| after the deadline is exceeded, and hence the server resources are freed up. | |
| In some cases server-side deadlines may also lead to a better user experience, | |
| as it's better to tell the user that there is a performance issue, that prevents | |
| the execution of the request, than letting them wait indefinitely. | |
| Finally server-side deadlines can help ops engineers to detect performance | |
| issues more reliably and more quicky. For this alerts may be setup that are | |
| based on the link:metrics.html#cancellations[cancellation metrics]. | |
| [[receive-timeout]] | |
| === Receive Timeout | |
| For git pushes it is possible to configure a | |
| link:config-gerrit.html#receive.timeout[hard timeout]. In contrast to | |
| server-side deadlines, this timeout is not overridable by | |
| link:#client-provided-deadlines[client-provided deadlines]. | |
| [[client-provided-deadlines]] | |
| == Client-provided deadlines | |
| Clients can set a deadline on requests to limit the maximal execution time that | |
| they are willing to wait for a response. If the request doesn't finish within | |
| this deadline the request is aborted and the client receives an error, with a | |
| message telling them that the deadline has been exceeded. | |
| How to set a deadline on a request depends on the request type: | |
| [options="header",cols="1,6"] | |
| |======================= | |
| |Request Type |How to set a deadline? | |
| |REST over HTTP |Set the link:rest-api.html#deadline[X-Gerrit-Deadline header]. | |
| |SSH command |Set the link:cmd-index.html#deadline[deadline option]. | |
| |git push |Set the link:user-upload.html#deadline[deadline push option]. | |
| |git clone/fetch|Not supported. | |
| |======================= | |
| [[override-server-side-deadline]] | |
| === Override server-side deadline | |
| By setting a deadline on a request it is possible to override any | |
| link:#server-side-deadlines[server-side deadline], e.g. in order to increase it. | |
| Setting the deadline to `0` disables any server-side deadline. This allows | |
| clients to get unblocked if a request has previously failed due to an exceeded | |
| deadline. | |
| [NOTE] | |
| It is stronly discouraged for clients to permanently override | |
| link:#server-side-deadlines[server-side deadlines] with a higher deadline or to | |
| permanently disable them by always setting the deadline to `0`. If this becomes | |
| necessary the caller should get in touch with the Gerrit administrators to | |
| increase the server-side deadlines or resolve the performance issue in another | |
| way. | |
| [NOTE] | |
| It's not possible for clients to override the link:#receive-timeout[receive | |
| timeout] that is enforced on git push. | |
| [[faqs]] | |
| == FAQs | |
| [[deadline-exceeded-what-to-do]] | |
| === My request failed due to an execeeded deadline, what can I do? | |
| To get unblocked, you may repeat the request with deadlines disabled. To do this | |
| set the deadline to `0` on the request as explained | |
| link:#override-server-side-deadline[above]. | |
| If doing this becomes required frequently, please get in touch with the Gerrit | |
| administrators in order to investigate the performance issue and increase the | |
| server-side deadline if necessary. | |
| [NOTE] | |
| Setting deadlines for requests that are done from the Gerrit web UI is not | |
| possible. If exceeded deadlines occur frequently here, please get in touch with | |
| the Gerrit administrators in order to investigate the performance issue. | |
| [[push-fails-due-to-exceeded-deadline-but-cannot-be-overridden]] | |
| === My git push fails due to an exceeded deadline and I cannot override the deadline, what can I do? | |
| As explained link:#receive-timeout[above] a configured receive timeout cannot be | |
| overridden by clients. If pushes fail due to this timeout, get in touch with the | |
| Gerrit administrators in order to investigate the performance issue and increase | |
| the receive timeout if necessary. | |
| [[when-are-requests-aborted]] | |
| === How quickly does a request get aborted when it is cancelled or a deadline is exceeded? | |
| In order to know if a request should be aborted, Gerrit needs to explicitly | |
| check whether the request is cancelled or whether a deadline is exceeded. | |
| Gerrit does this check at the beginning and end of all performance critical | |
| steps and sub-steps. This means, the request is only aborted the next time such | |
| a step starts or finishes, which can also be never (e.g. if the request is stuck | |
| inside of a step). | |
| [NOTE] | |
| Technically the check whether a request should be aborted is done whenever the | |
| execution time of an operation or sub-step is captured, either by a timer | |
| metric or a `TraceTimer` ('TraceTimer` is the class that logs the execution time | |
| when the request is being link:user-request-tracing.html[traced]). | |
| [[how-are-requests-aborted]] | |
| === How does Gerrit abort requests? | |
| The exact response that is returned to the client depends on the request type | |
| and the cancellation reason: | |
| [options="header",cols="1,3,3"] | |
| |======================= | |
| |Request Type |Cancellation Reason|Response | |
| |REST over HTTP |Client Disconnected|The response is '499 Client Closed Request'. | |
| | |Server-side deadline exceeded|The response is '408 Server Deadline Exceeded'. | |
| | |Client-provided deadline exceeded|The response is '408 Client Provided Deadline Exceeded'. | |
| |SSH command |Client Disconnected|The error message is 'Client Closed Request'. | |
| | |Server-side deadline exceeded|The error message is 'Server Deadline Exceeded'. | |
| | |Client-provided deadline exceeded|The error message is 'Client Provided Deadline Exceeded'. | |
| |git push |Client Disconnected|The error status is 'Client Closed Request'. | |
| | |Server-side deadline exceeded|The error status is 'Server Deadline Exceeded'. | |
| | |Client-provided deadline exceeded|The error status is 'Client Provided Deadline Exceeded'. | |
| |git clone/fetch|Not supported. | |
| |======================= | |
| This means clients always get a proper error message telling the user why the | |
| request has been aborted. | |
| Errors due to aborted requests are usually not counted as internal server errors, | |
| but the link:metrics.html#cancellations[cancellation metrics] may be used to | |
| setup alerting for performance issues. | |
| [NOTE] | |
| During a request, cancellations can occur at any time. This means for non-atomic | |
| operations, it can happen that the operation is cancelled after some steps have | |
| already been successfully performed and before all steps have been executed, | |
| potentially leaving behind an inconsistent state (same as when a request fails | |
| due to an error). However for important steps, such a NoteDb updates that span | |
| multiple repositories, Gerrit ensures that they are not torn by cancellations. | |
| GERRIT | |
| ------ | |
| Part of link:index.html[Gerrit Code Review] | |
| SEARCHBOX | |
| --------- |