:linkattrs:
= Request Cancellation and Deadlines

[[motivation]]
== Motivation

Protect the Gerrit service by aborting requests that were cancelled or for which
the deadline has exceeded. If these requests are not aborted, it can happen that
too many of these requests are accumulated so that the server runs out of
resources (e.g. threads).

[[request-cancellation]]
== Request Cancellation

If a user cancels a request by disconnecting, ideally Gerrit should detect this
and abort the request execution to avoid doing unnecessary work. If nobody is
waiting for the response, Gerrit shouldn't spend resources to compute it.

Detecting cancelled requests is not easily possible with all protocols that a
client may use.

[[server-side-deadlines]]
== Server-side deadlines

To limit the maximal execution time for requests, administrators can
link:config-gerrit.html#deadline.id[configure server-side deadlines]. If a
server-side deadline is exceeded by a matching request, the request is
automatically aborted. In this case the client gets a proper error message
informing the user about the exceeded deadline.

[NOTE]
====
Deadlines are enforced cooperatively and may not interrupt execution immediately.
Cancellation is only evaluated at explicit check points; long-running blocking
operations may continue until the next check.
Admins should not assume a hard kill after exceeding deadline, and they should
not rely on deadlines as a precise resource cutoff.
====

Clients may override server-side deadlines by setting a
link:#client-provided-deadlines[deadline] on the request. This means, if a
request fails due to an exceeded server-side deadline, the client may repeat the
request with a higher deadline or no deadline (deadline = 0) to get unblocked.

Server-side deadlines are meant to protect the Gerrit service against resource
exhaustion due to performance issues with a particular request. E.g. imagine a
situation where requests for a certain REST endpoint are very slow. If more and
more of such requests get stuck and are not being aborted, the Gerrit service
may run out of threads, causing an outage for the entire Gerrit service.
Server-side deadlines may prevent this because the slow requests get aborted
after the deadline is exceeded, and hence the server resources are freed up.

In some cases server-side deadlines may also lead to a better user experience,
as it's better to tell the user that there is a performance issue, that prevents
the execution of the request, than letting them wait indefinitely.

Finally server-side deadlines can help ops engineers to detect performance
issues more reliably and more quickly. For this alerts may be setup that are
based on the link:metrics.html#cancellations[cancellation metrics].

[[receive-timeout]]
=== Receive Timeout

For git pushes it is possible to configure a
link:config-gerrit.html#receive.timeout[hard timeout]. In contrast to
server-side deadlines, this timeout is not overridable by
link:#client-provided-deadlines[client-provided deadlines].

[[client-provided-deadlines]]
== Client-provided deadlines

Clients can set a deadline on requests to limit the maximal execution time that
they are willing to wait for a response. If the request doesn't finish within
this deadline the request is aborted and the client receives an error, with a
message telling them that the deadline has been exceeded.

How to set a deadline on a request depends on the request type:

[options="header",cols="1,6"]
|=======================
|Request Type   |How to set a deadline?
|REST over HTTP |Set the link:rest-api.html#deadline[X-Gerrit-Deadline header].
|SSH command    |Set the link:cmd-index.html#deadline[deadline option].
|git push       |Set the link:user-upload.html#deadline[deadline push option].
|git clone/fetch|Not supported.
|=======================

[[override-server-side-deadline]]
=== Override server-side deadline

By setting a deadline on a request it is possible to override any
link:#server-side-deadlines[server-side deadline], e.g. in order to increase it.
Setting the deadline to `0` disables any server-side deadline. This allows
clients to get unblocked if a request has previously failed due to an exceeded
deadline.

[NOTE]
It is strongly discouraged for clients to permanently override
link:#server-side-deadlines[server-side deadlines] with a higher deadline or to
permanently disable them by always setting the deadline to `0`. If this becomes
necessary the caller should get in touch with the Gerrit administrators to
increase the server-side deadlines or resolve the performance issue in another
way.

[NOTE]
It's not possible for clients to override the link:#receive-timeout[receive
timeout] that is enforced on git push.

[[faqs]]
== FAQs

[[deadline-exceeded-what-to-do]]
=== My request failed due to an exceeded deadline, what can I do?

To get unblocked, you may repeat the request with deadlines disabled. To do this
set the deadline to `0` on the request as explained
link:#override-server-side-deadline[above].

If doing this becomes required frequently, please get in touch with the Gerrit
administrators in order to investigate the performance issue and increase the
server-side deadline if necessary.

[NOTE]
Setting deadlines for requests that are done from the Gerrit web UI is not
possible. If exceeded deadlines occur frequently here, please get in touch with
the Gerrit administrators in order to investigate the performance issue.

[[push-fails-due-to-exceeded-deadline-but-cannot-be-overridden]]
=== My git push fails due to an exceeded deadline and I cannot override the deadline, what can I do?

As explained link:#receive-timeout[above] a configured receive timeout cannot be
overridden by clients. If pushes fail due to this timeout, get in touch with the
Gerrit administrators in order to investigate the performance issue and increase
the receive timeout if necessary.

[[when-are-requests-aborted]]
=== How quickly does a request get aborted when it is cancelled or a deadline is exceeded?

In order to know if a request should be aborted, Gerrit needs to explicitly
check whether the request is cancelled or whether a deadline is exceeded.
Gerrit does this check at the beginning and end of all performance critical
steps and sub-steps. This means, the request is only aborted the next time such
a step starts or finishes, which can also be never (e.g. if the request is stuck
inside of a step).

[NOTE]
Technically the check whether a request should be aborted is done whenever the
execution time of an operation or sub-step is captured, either by a timer
metric or a `TraceTimer` ('TraceTimer` is the class that logs the execution time
when the request is being link:user-request-tracing.html[traced]).

[[how-are-requests-aborted]]
=== How does Gerrit abort requests?

The exact response that is returned to the client depends on the request type
and the cancellation reason:

[options="header",cols="1,3,3"]
|=======================
|Request Type   |Cancellation Reason|Response
|REST over HTTP |Client Disconnected|The response is '499 Client Closed Request'.
|               |Server-side deadline exceeded|The response is '500 Internal Server Error'.
|               |Client-provided deadline exceeded|The response is '408 Client Provided Deadline Exceeded'.
|SSH command    |Client Disconnected|The error message is 'Client Closed Request'.
|               |Server-side deadline exceeded|The error message is 'Server Deadline Exceeded'.
|               |Client-provided deadline exceeded|The error message is 'Client Provided Deadline Exceeded'.
|git push       |Client Disconnected|The error status is 'Client Closed Request'.
|               |Server-side deadline exceeded|The error status is 'Server Deadline Exceeded'.
|               |Client-provided deadline exceeded|The error status is 'Client Provided Deadline Exceeded'.
|git clone/fetch|Not supported.
|=======================

This means clients always get a proper error message telling the user why the
request has been aborted. A deadline response indicates an exceeded deadline,
not the actual execution outcome or that all work was rolled back.
Clients should treat deadline errors as indeterminate and not assume success or
failure.

Errors due to aborted requests are usually not counted as internal server errors,
but the link:metrics.html#cancellations[cancellation metrics] may be used to
setup alerting for performance issues.

[NOTE]
During a request, cancellations can occur at any time. This means for non-atomic
operations, it can happen that the operation is cancelled after some steps have
already been successfully performed and before all steps have been executed,
potentially leaving behind an inconsistent state (same as when a request fails
due to an error). However for important steps, such a NoteDb updates that span
multiple repositories, Gerrit ensures that they are not torn by cancellations.

GERRIT
------
Part of link:index.html[Gerrit Code Review]

SEARCHBOX
---------