blob: 7c39f1607a620a9e09ed37cfe97fdbd06db07b95 [file] [log] [blame]
Edwin Kempin40c29892021-09-01 15:58:05 +02001:linkattrs:
2= Request Cancellation and Deadlines
3
4[[motivation]]
5== Motivation
6
7Protect the Gerrit service by aborting requests that were cancelled or for which
8the deadline has exceeded. If these requests are not aborted, it can happen that
9too many of these requests are accumulated so that the server runs out of
10resources (e.g. threads).
11
12[[request-cancellation]]
13== Request Cancellation
14
15If a user cancels a request by disconnecting, ideally Gerrit should detect this
16and abort the request execution to avoid doing unnecessary work. If nobody is
17waiting for the response, Gerrit shouldn't spend resources to compute it.
18
19Detecting cancelled requests is not easily possible with all protocols that a
20client may use. At the moment Gerrit only detects request cancellations for git
21pushes, but not for other request types (in particular cancelled requests are
22not detected for REST calls over HTTP, SSH commands and git clone/fetch).
23
24[[server-side-deadlines]]
25== Server-side deadlines
26
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020027To limit the maximal execution time for requests, administrators can
28link:config-gerrit.html#deadline.id[configure server-side deadlines]. If a
29server-side deadline is exceeded by a matching request, the request is
30automatically aborted. In this case the client gets a proper error message
31informing the user about the exceeded deadline.
Edwin Kempin40c29892021-09-01 15:58:05 +020032
33Clients may override server-side deadlines by setting a
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020034link:#client-provided-deadlines[deadline] on the request. This means, if a
35request fails due to an exceeded server-side deadline, the client may repeat the
36request with a higher deadline or no deadline (deadline = 0) to get unblocked.
Edwin Kempin40c29892021-09-01 15:58:05 +020037
38Server-side deadlines are meant to protect the Gerrit service against resource
39exhaustion due to performence issues with a particular request. E.g. imagine a
40situation where requests for a certain REST endpoint are very slow. If more and
41more of such requests get stuck and are not being aborted, the Gerrit service
42may run out of threads, causing an outage for the entire Gerrit service.
43Server-side deadlines may prevent this because the slow requests get aborted
44after the deadline is exceeded, and hence the server resources are freed up.
45
46In some cases server-side deadlines may also lead to a better user experience,
47as it's better to tell the user that there is a performance issue, that prevents
48the execution of the request, than letting them wait indefinitely.
49
50Finally server-side deadlines can help ops engineers to detect performance
51issues more reliably and more quicky. For this alerts may be setup that are
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020052based on the link:metrics.html#cancellations[cancellation metrics].
Edwin Kempin40c29892021-09-01 15:58:05 +020053
54[[receive-timeout]]
55=== Receive Timeout
56
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020057For git pushes it is possible to configure a
58link:config-gerrit.html#receive.timeout[hard timeout]. In contrast to
59server-side deadlines, this timeout is not overridable by
60link:#client-provided-deadlines[client-provided deadlines].
Edwin Kempin40c29892021-09-01 15:58:05 +020061
62[[client-provided-deadlines]]
63== Client-provided deadlines
64
65Clients can set a deadline on requests to limit the maximal execution time that
66they are willing to wait for a response. If the request doesn't finish within
67this deadline the request is aborted and the client receives an error, with a
68message telling them that the deadline has been exceeded.
69
70How to set a deadline on a request depends on the request type:
71
72[options="header",cols="1,6"]
73|=======================
74|Request Type |How to set a deadline?
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020075|REST over HTTP |Set the link:rest-api.html#deadline[X-Gerrit-Deadline header].
76|SSH command |Set the link:cmd-index.html#deadline[deadline option].
77|git push |Set the link:user-upload.html#deadline[deadline push option].
Edwin Kempin40c29892021-09-01 15:58:05 +020078|git clone/fetch|Not supported.
79|=======================
80
81[[override-server-side-deadline]]
82=== Override server-side deadline
83
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020084By setting a deadline on a request it is possible to override any
85link:#server-side-deadlines[server-side deadline], e.g. in order to increase it.
86Setting the deadline to `0` disables any server-side deadline. This allows
87clients to get unblocked if a request has previously failed due to an exceeded
88deadline.
Edwin Kempin40c29892021-09-01 15:58:05 +020089
90[NOTE]
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020091It is stronly discouraged for clients to permanently override
92link:#server-side-deadlines[server-side deadlines] with a higher deadline or to
93permanently disable them by always setting the deadline to `0`. If this becomes
94necessary the caller should get in touch with the Gerrit administrators to
95increase the server-side deadlines or resolve the performance issue in another
96way.
Edwin Kempin40c29892021-09-01 15:58:05 +020097
98[NOTE]
Edwin Kempin6b3f55a2021-10-20 13:12:12 +020099It's not possible for clients to override the link:#receive-timeout[receive
100timeout] that is enforced on git push.
Edwin Kempin40c29892021-09-01 15:58:05 +0200101
Edwin Kempina78dd552021-09-02 09:18:46 +0200102[[faqs]]
103== FAQs
104
105[[deadline-exceeded-what-to-do]]
106=== My request failed due to an execeeded deadline, what can I do?
107
108To get unblocked, you may repeat the request with deadlines disabled. To do this
109set the deadline to `0` on the request as explained
Edwin Kempin6b3f55a2021-10-20 13:12:12 +0200110link:#override-server-side-deadline[above].
Edwin Kempina78dd552021-09-02 09:18:46 +0200111
112If doing this becomes required frequently, please get in touch with the Gerrit
113administrators in order to investigate the performance issue and increase the
114server-side deadline if necessary.
115
116[NOTE]
117Setting deadlines for requests that are done from the Gerrit web UI is not
118possible. If exceeded deadlines occur frequently here, please get in touch with
119the Gerrit administrators in order to investigate the performance issue.
120
121[[push-fails-due-to-exceeded-deadline-but-cannot-be-overridden]]
122=== My git push fails due to an exceeded deadline and I cannot override the deadline, what can I do?
123
Edwin Kempin6b3f55a2021-10-20 13:12:12 +0200124As explained link:#receive-timeout[above] a configured receive timeout cannot be
Edwin Kempina78dd552021-09-02 09:18:46 +0200125overridden by clients. If pushes fail due to this timeout, get in touch with the
126Gerrit administrators in order to investigate the performance issue and increase
127the receive timeout if necessary.
128
129[[when-are-requests-aborted]]
130=== How quickly does a request get aborted when it is cancelled or a deadline is exceeded?
131
132In order to know if a request should be aborted, Gerrit needs to explicitly
133check whether the request is cancelled or whether a deadline is exceeded.
134Gerrit does this check at the beginning and end of all performance critical
135steps and sub-steps. This means, the request is only aborted the next time such
136a step starts or finishes, which can also be never (e.g. if the request is stuck
137inside of a step).
138
139[NOTE]
140Technically the check whether a request should be aborted is done whenever the
141execution time of an operation or sub-step is captured, either by a timer
142metric or a `TraceTimer` ('TraceTimer` is the class that logs the execution time
Edwin Kempin6b3f55a2021-10-20 13:12:12 +0200143when the request is being link:user-request-tracing.html[traced]).
Edwin Kempina78dd552021-09-02 09:18:46 +0200144
145[[how-are-requests-aborted]]
146=== How does Gerrit abort requests?
147
148The exact response that is returned to the client depends on the request type
149and the cancellation reason:
150
151[options="header",cols="1,3,3"]
152|=======================
153|Request Type |Cancellation Reason|Response
154|REST over HTTP |Client Disconnected|The response is '499 Client Closed Request'.
155| |Server-side deadline exceeded|The response is '408 Server Deadline Exceeded'.
156| |Client-provided deadline exceeded|The response is '408 Client Provided Deadline Exceeded'.
157|SSH command |Client Disconnected|The error message is 'Client Closed Request'.
158| |Server-side deadline exceeded|The error message is 'Server Deadline Exceeded'.
159| |Client-provided deadline exceeded|The error message is 'Client Provided Deadline Exceeded'.
160|git push |Client Disconnected|The error status is 'Client Closed Request'.
161| |Server-side deadline exceeded|The error status is 'Server Deadline Exceeded'.
162| |Client-provided deadline exceeded|The error status is 'Client Provided Deadline Exceeded'.
163|git clone/fetch|Not supported.
164|=======================
165
166This means clients always get a proper error message telling the user why the
167request has been aborted.
168
169Errors due to aborted requests are usually not counted as internal server errors,
Edwin Kempin6b3f55a2021-10-20 13:12:12 +0200170but the link:metrics.html#cancellations[cancellation metrics] may be used to
171setup alerting for performance issues.
Edwin Kempina78dd552021-09-02 09:18:46 +0200172
Edwin Kempin0ef26862021-09-07 11:55:52 +0200173[NOTE]
174During a request, cancellations can occur at any time. This means for non-atomic
175operations, it can happen that the operation is cancelled after some steps have
176already been successfully performed and before all steps have been executed,
177potentially leaving behind an inconsistent state (same as when a request fails
178due to an error). However for important steps, such a NoteDb updates that span
179multiple repositories, Gerrit ensures that they are not torn by cancellations.
180
Edwin Kempin40c29892021-09-01 15:58:05 +0200181GERRIT
182------
183Part of link:index.html[Gerrit Code Review]
184
185SEARCHBOX
186---------