Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 1 | :linkattrs: |
| 2 | = Request Cancellation and Deadlines |
| 3 | |
| 4 | [[motivation]] |
| 5 | == Motivation |
| 6 | |
| 7 | Protect the Gerrit service by aborting requests that were cancelled or for which |
| 8 | the deadline has exceeded. If these requests are not aborted, it can happen that |
| 9 | too many of these requests are accumulated so that the server runs out of |
| 10 | resources (e.g. threads). |
| 11 | |
| 12 | [[request-cancellation]] |
| 13 | == Request Cancellation |
| 14 | |
| 15 | If a user cancels a request by disconnecting, ideally Gerrit should detect this |
| 16 | and abort the request execution to avoid doing unnecessary work. If nobody is |
| 17 | waiting for the response, Gerrit shouldn't spend resources to compute it. |
| 18 | |
| 19 | Detecting cancelled requests is not easily possible with all protocols that a |
| 20 | client may use. At the moment Gerrit only detects request cancellations for git |
| 21 | pushes, but not for other request types (in particular cancelled requests are |
| 22 | not detected for REST calls over HTTP, SSH commands and git clone/fetch). |
| 23 | |
| 24 | [[server-side-deadlines]] |
| 25 | == Server-side deadlines |
| 26 | |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 27 | To limit the maximal execution time for requests, administrators can |
| 28 | link:config-gerrit.html#deadline.id[configure server-side deadlines]. If a |
| 29 | server-side deadline is exceeded by a matching request, the request is |
| 30 | automatically aborted. In this case the client gets a proper error message |
| 31 | informing the user about the exceeded deadline. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 32 | |
| 33 | Clients may override server-side deadlines by setting a |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 34 | link:#client-provided-deadlines[deadline] on the request. This means, if a |
| 35 | request fails due to an exceeded server-side deadline, the client may repeat the |
| 36 | request with a higher deadline or no deadline (deadline = 0) to get unblocked. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 37 | |
| 38 | Server-side deadlines are meant to protect the Gerrit service against resource |
| 39 | exhaustion due to performence issues with a particular request. E.g. imagine a |
| 40 | situation where requests for a certain REST endpoint are very slow. If more and |
| 41 | more of such requests get stuck and are not being aborted, the Gerrit service |
| 42 | may run out of threads, causing an outage for the entire Gerrit service. |
| 43 | Server-side deadlines may prevent this because the slow requests get aborted |
| 44 | after the deadline is exceeded, and hence the server resources are freed up. |
| 45 | |
| 46 | In some cases server-side deadlines may also lead to a better user experience, |
| 47 | as it's better to tell the user that there is a performance issue, that prevents |
| 48 | the execution of the request, than letting them wait indefinitely. |
| 49 | |
| 50 | Finally server-side deadlines can help ops engineers to detect performance |
| 51 | issues more reliably and more quicky. For this alerts may be setup that are |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 52 | based on the link:metrics.html#cancellations[cancellation metrics]. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 53 | |
| 54 | [[receive-timeout]] |
| 55 | === Receive Timeout |
| 56 | |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 57 | For git pushes it is possible to configure a |
| 58 | link:config-gerrit.html#receive.timeout[hard timeout]. In contrast to |
| 59 | server-side deadlines, this timeout is not overridable by |
| 60 | link:#client-provided-deadlines[client-provided deadlines]. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 61 | |
| 62 | [[client-provided-deadlines]] |
| 63 | == Client-provided deadlines |
| 64 | |
| 65 | Clients can set a deadline on requests to limit the maximal execution time that |
| 66 | they are willing to wait for a response. If the request doesn't finish within |
| 67 | this deadline the request is aborted and the client receives an error, with a |
| 68 | message telling them that the deadline has been exceeded. |
| 69 | |
| 70 | How to set a deadline on a request depends on the request type: |
| 71 | |
| 72 | [options="header",cols="1,6"] |
| 73 | |======================= |
| 74 | |Request Type |How to set a deadline? |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 75 | |REST over HTTP |Set the link:rest-api.html#deadline[X-Gerrit-Deadline header]. |
| 76 | |SSH command |Set the link:cmd-index.html#deadline[deadline option]. |
| 77 | |git push |Set the link:user-upload.html#deadline[deadline push option]. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 78 | |git clone/fetch|Not supported. |
| 79 | |======================= |
| 80 | |
| 81 | [[override-server-side-deadline]] |
| 82 | === Override server-side deadline |
| 83 | |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 84 | By setting a deadline on a request it is possible to override any |
| 85 | link:#server-side-deadlines[server-side deadline], e.g. in order to increase it. |
| 86 | Setting the deadline to `0` disables any server-side deadline. This allows |
| 87 | clients to get unblocked if a request has previously failed due to an exceeded |
| 88 | deadline. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 89 | |
| 90 | [NOTE] |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 91 | It is stronly discouraged for clients to permanently override |
| 92 | link:#server-side-deadlines[server-side deadlines] with a higher deadline or to |
| 93 | permanently disable them by always setting the deadline to `0`. If this becomes |
| 94 | necessary the caller should get in touch with the Gerrit administrators to |
| 95 | increase the server-side deadlines or resolve the performance issue in another |
| 96 | way. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 97 | |
| 98 | [NOTE] |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 99 | It's not possible for clients to override the link:#receive-timeout[receive |
| 100 | timeout] that is enforced on git push. |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 101 | |
Edwin Kempin | a78dd55 | 2021-09-02 09:18:46 +0200 | [diff] [blame] | 102 | [[faqs]] |
| 103 | == FAQs |
| 104 | |
| 105 | [[deadline-exceeded-what-to-do]] |
| 106 | === My request failed due to an execeeded deadline, what can I do? |
| 107 | |
| 108 | To get unblocked, you may repeat the request with deadlines disabled. To do this |
| 109 | set the deadline to `0` on the request as explained |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 110 | link:#override-server-side-deadline[above]. |
Edwin Kempin | a78dd55 | 2021-09-02 09:18:46 +0200 | [diff] [blame] | 111 | |
| 112 | If doing this becomes required frequently, please get in touch with the Gerrit |
| 113 | administrators in order to investigate the performance issue and increase the |
| 114 | server-side deadline if necessary. |
| 115 | |
| 116 | [NOTE] |
| 117 | Setting deadlines for requests that are done from the Gerrit web UI is not |
| 118 | possible. If exceeded deadlines occur frequently here, please get in touch with |
| 119 | the Gerrit administrators in order to investigate the performance issue. |
| 120 | |
| 121 | [[push-fails-due-to-exceeded-deadline-but-cannot-be-overridden]] |
| 122 | === My git push fails due to an exceeded deadline and I cannot override the deadline, what can I do? |
| 123 | |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 124 | As explained link:#receive-timeout[above] a configured receive timeout cannot be |
Edwin Kempin | a78dd55 | 2021-09-02 09:18:46 +0200 | [diff] [blame] | 125 | overridden by clients. If pushes fail due to this timeout, get in touch with the |
| 126 | Gerrit administrators in order to investigate the performance issue and increase |
| 127 | the receive timeout if necessary. |
| 128 | |
| 129 | [[when-are-requests-aborted]] |
| 130 | === How quickly does a request get aborted when it is cancelled or a deadline is exceeded? |
| 131 | |
| 132 | In order to know if a request should be aborted, Gerrit needs to explicitly |
| 133 | check whether the request is cancelled or whether a deadline is exceeded. |
| 134 | Gerrit does this check at the beginning and end of all performance critical |
| 135 | steps and sub-steps. This means, the request is only aborted the next time such |
| 136 | a step starts or finishes, which can also be never (e.g. if the request is stuck |
| 137 | inside of a step). |
| 138 | |
| 139 | [NOTE] |
| 140 | Technically the check whether a request should be aborted is done whenever the |
| 141 | execution time of an operation or sub-step is captured, either by a timer |
| 142 | metric or a `TraceTimer` ('TraceTimer` is the class that logs the execution time |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 143 | when the request is being link:user-request-tracing.html[traced]). |
Edwin Kempin | a78dd55 | 2021-09-02 09:18:46 +0200 | [diff] [blame] | 144 | |
| 145 | [[how-are-requests-aborted]] |
| 146 | === How does Gerrit abort requests? |
| 147 | |
| 148 | The exact response that is returned to the client depends on the request type |
| 149 | and the cancellation reason: |
| 150 | |
| 151 | [options="header",cols="1,3,3"] |
| 152 | |======================= |
| 153 | |Request Type |Cancellation Reason|Response |
| 154 | |REST over HTTP |Client Disconnected|The response is '499 Client Closed Request'. |
| 155 | | |Server-side deadline exceeded|The response is '408 Server Deadline Exceeded'. |
| 156 | | |Client-provided deadline exceeded|The response is '408 Client Provided Deadline Exceeded'. |
| 157 | |SSH command |Client Disconnected|The error message is 'Client Closed Request'. |
| 158 | | |Server-side deadline exceeded|The error message is 'Server Deadline Exceeded'. |
| 159 | | |Client-provided deadline exceeded|The error message is 'Client Provided Deadline Exceeded'. |
| 160 | |git push |Client Disconnected|The error status is 'Client Closed Request'. |
| 161 | | |Server-side deadline exceeded|The error status is 'Server Deadline Exceeded'. |
| 162 | | |Client-provided deadline exceeded|The error status is 'Client Provided Deadline Exceeded'. |
| 163 | |git clone/fetch|Not supported. |
| 164 | |======================= |
| 165 | |
| 166 | This means clients always get a proper error message telling the user why the |
| 167 | request has been aborted. |
| 168 | |
| 169 | Errors due to aborted requests are usually not counted as internal server errors, |
Edwin Kempin | 6b3f55a | 2021-10-20 13:12:12 +0200 | [diff] [blame] | 170 | but the link:metrics.html#cancellations[cancellation metrics] may be used to |
| 171 | setup alerting for performance issues. |
Edwin Kempin | a78dd55 | 2021-09-02 09:18:46 +0200 | [diff] [blame] | 172 | |
Edwin Kempin | 0ef2686 | 2021-09-07 11:55:52 +0200 | [diff] [blame] | 173 | [NOTE] |
| 174 | During a request, cancellations can occur at any time. This means for non-atomic |
| 175 | operations, it can happen that the operation is cancelled after some steps have |
| 176 | already been successfully performed and before all steps have been executed, |
| 177 | potentially leaving behind an inconsistent state (same as when a request fails |
| 178 | due to an error). However for important steps, such a NoteDb updates that span |
| 179 | multiple repositories, Gerrit ensures that they are not torn by cancellations. |
| 180 | |
Edwin Kempin | 40c2989 | 2021-09-01 15:58:05 +0200 | [diff] [blame] | 181 | GERRIT |
| 182 | ------ |
| 183 | Part of link:index.html[Gerrit Code Review] |
| 184 | |
| 185 | SEARCHBOX |
| 186 | --------- |